feat(storage): full object checksum: parse finalize_time and server crc32c in async object stream#17261
feat(storage): full object checksum: parse finalize_time and server crc32c in async object stream#17261chandra-siri wants to merge 2 commits into
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces tracking for object finalization and full object CRC32C checksums in the asynchronous read stream. The feedback focuses on simplifying the production code by removing logic added solely to accommodate unit test mocks (such as checking for a .seconds attribute). Instead, it is recommended to mock finalize_time as a standard datetime.datetime or None in the unit tests, which allows the production code to rely on standard isinstance checks.
| # Since full object checksum validation is only required for finalized objects, | ||
| # check finalize_time (which is DatetimeWithNanoseconds/datetime in production, or mocked in tests). | ||
| finalize_time = getattr(response.metadata, "finalize_time", None) | ||
| if finalize_time: | ||
| is_finalized_val = False | ||
| if isinstance(finalize_time, datetime.datetime): | ||
| is_finalized_val = True | ||
| elif hasattr(finalize_time, "seconds") and finalize_time.seconds > 0: | ||
| is_finalized_val = True | ||
|
|
||
| if is_finalized_val: | ||
| self.is_finalized = True | ||
| if hasattr(response.metadata, "checksums") and response.metadata.checksums: | ||
| self.full_obj_server_crc32c = response.metadata.checksums.crc32c |
There was a problem hiding this comment.
Avoid adding production code logic (such as checking for a .seconds attribute) solely to satisfy unit test mocks. In production, finalize_time is a datetime.datetime (or DatetimeWithNanoseconds) object. By updating the unit tests to mock finalize_time as a datetime.datetime instance (or None when unfinalized), we can simplify this logic significantly and keep the production code clean.
| # Since full object checksum validation is only required for finalized objects, | |
| # check finalize_time (which is DatetimeWithNanoseconds/datetime in production, or mocked in tests). | |
| finalize_time = getattr(response.metadata, "finalize_time", None) | |
| if finalize_time: | |
| is_finalized_val = False | |
| if isinstance(finalize_time, datetime.datetime): | |
| is_finalized_val = True | |
| elif hasattr(finalize_time, "seconds") and finalize_time.seconds > 0: | |
| is_finalized_val = True | |
| if is_finalized_val: | |
| self.is_finalized = True | |
| if hasattr(response.metadata, "checksums") and response.metadata.checksums: | |
| self.full_obj_server_crc32c = response.metadata.checksums.crc32c | |
| # Since full object checksum validation is only required for finalized objects, | |
| # check finalize_time (which is DatetimeWithNanoseconds/datetime in production). | |
| finalize_time = getattr(response.metadata, "finalize_time", None) | |
| if isinstance(finalize_time, datetime.datetime): | |
| self.is_finalized = True | |
| checksums = getattr(response.metadata, "checksums", None) | |
| if checksums: | |
| self.full_obj_server_crc32c = getattr(checksums, "crc32c", None) |
| recv_response.metadata.finalize_time.seconds = 12345 | ||
| recv_response.metadata.checksums.crc32c = 98765 |
There was a problem hiding this comment.
Mock finalize_time as an actual datetime.datetime instance instead of a mock with a .seconds attribute. This allows the production code to use standard isinstance checks without mock-specific workarounds. (Note: Ensure datetime is imported in this test file).
| recv_response.metadata.finalize_time.seconds = 12345 | |
| recv_response.metadata.checksums.crc32c = 98765 | |
| recv_response.metadata.finalize_time = datetime.datetime(2024, 1, 1, tzinfo=datetime.timezone.utc) | |
| recv_response.metadata.checksums.crc32c = 98765 |
| recv_response.metadata = mock.MagicMock() | ||
| recv_response.metadata.generation = _TEST_GENERATION_NUMBER | ||
| recv_response.metadata.size = _TEST_OBJECT_SIZE | ||
| recv_response.metadata.finalize_time.seconds = 0 # NOT finalized! |
There was a problem hiding this comment.
PR 1 of 3: Extract finalize_time and full_obj_server_crc32c from the response metadata in _AsyncReadObjectStream.open().