Fix large mini-batch handling in parallel external source #3768
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…for storing capacity Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com>
JanuszL
approved these changes
Mar 29, 2022
mzient
reviewed
Mar 29, 2022
| if not proc.exitcode: | ||
| task_queue.close() | ||
| proc.join() | ||
| try: |
There was a problem hiding this comment.
Q: no enter/exit to use with proc?
There was a problem hiding this comment.
For plain proc not really, but good point that it looked like a good candidate for ctx manager.
mzient
approved these changes
Mar 29, 2022
… large batch test Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com>
Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com>
Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com>
|
CI MESSAGE: [4281330]: BUILD STARTED |
|
CI MESSAGE: [4281330]: BUILD PASSED |
|
!build |
|
CI MESSAGE: [4284870]: BUILD STARTED |
|
CI MESSAGE: [4284870]: BUILD PASSED |
cyyever
pushed a commit
to cyyever/DALI
that referenced
this pull request
May 13, 2022
* Add better error messaging on c-struct serialization * Use wider type for storing capacity * Add test for handling large sample in parallel external source Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com>
cyyever
pushed a commit
to cyyever/DALI
that referenced
this pull request
Jun 7, 2022
* Add better error messaging on c-struct serialization * Use wider type for storing capacity * Add test for handling large sample in parallel external source Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Category:
Bug fix (non-breaking change which fixes an issue)
Description:
Parallel external source passes data from workers to the main process through shared memory buffers.
If the capacity of a buffer exceeds
max_int, worker process fails to serialize the minibatch meta-data (which includes the capacity) when it is written into C-like structure (python'sstructpackage).This PR:
unsigned long long intinstead ofint.Additional information:
Affected modules and functionalities:
_multiprocmodule and tests.Key points relevant for the review:
unsigned long long intis too much and simply unsignedintthere will be good enough?Checklist
Tests
Documentation
DALI team only
Requirements
REQ IDs: N/A
JIRA TASK: DALI-2679