Support checkpointing in Coco Reader #5162
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Category:
New feature (non-breaking change which adds functionality)
Description:
This PR adds checkpointing support to Coco Reader.
Additional information:
Affected modules and functionalities:
CocoLoader
inherits fromFileLabelLoaderBase
which has checkpointing support already implemented, so to enable checkpointing we just now inherit fromFileLabelLoaderBase<enable_checkpointing=true>
instead ofFileLabelLoaderBase<enable_checkpointing=false>
.CocoReader
was changed to inherit fromDataReader<..., supports_checkpointing=true>
.Key points relevant for the review:
CocoLoader
overridesPrepareMetadataImpl
, so a partFileLabelLoader
's implementation responsible for saving original sample order to a backup variable was copied from:DALI/dali/operators/reader/loader/file_label_loader.h
Lines 189 to 194 in a130ddb
to
CocoLoader
'sPrepareMetadataImpl
.Tests:
As
readers.coco
is a direct successor ofreaders.file
I decided not to duplicate all the param sets fromfile
and just create a set of few basic tests.Tests refactoring
As the reader checkpointing tests were getting repetitive and there are a few more to come, I decided to move the common part to
check_reader_checkpointing
. Some of file reader tests started failing after adding the assert checking if the test has enough data, so I lowered theiters_into_epoch
andbatch_size
in those.Checklist
Documentation
DALI team only
Requirements
REQ IDs: N/A
JIRA TASK: DALI-3690