Ensure keeping py_pool alive until pipline is garbage collected #3245
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com>
|
!build |
|
CI MESSAGE: [2740575]: BUILD STARTED |
JanuszL
reviewed
Aug 11, 2021
dali/python/nvidia/dali/pipeline.py
Outdated
Comment on lines
555
to
558
| # Pipeline backend doesn't really do anything with the pool, sole point of this call | ||
| # is to ensure lifetime of the pool exceeds the lifetime of the pipeline's backend | ||
| # so that shared memory managed by the pool is not freed before pipline is garbage collected. | ||
| # Otherwise pipline may try to access freed memory which leads to crashes at the Python teardown |
There was a problem hiding this comment.
Suggested change
| # Pipeline backend doesn't really do anything with the pool, sole point of this call | |
| # is to ensure lifetime of the pool exceeds the lifetime of the pipeline's backend | |
| # so that shared memory managed by the pool is not freed before pipline is garbage collected. | |
| # Otherwise pipline may try to access freed memory which leads to crashes at the Python teardown | |
| # The sole point of this call is to ensure lifetime of the pool exceeds the lifetime of the pipeline's backend, | |
| # which runs external source operator that may access memory owned by the pool, | |
| # so that shared memory managed by the pool is not freed before pipline is garbage collected. | |
| # Otherwise pipline may try to access freed memory which leads to crashes at the Python teardown |
There was a problem hiding this comment.
Adjusted the sentence, please take a look as I've removed I think unnecessary "which" parts.
JanuszL
approved these changes
Aug 11, 2021
|
Do you think we can add a test for that? |
|
CI MESSAGE: [2740575]: BUILD PASSED |
Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com>
Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com>
JanuszL
reviewed
Aug 12, 2021
Comment on lines
29
to
42
| def create_py_file_reader(images_dir): | ||
| with open(os.path.join(images_dir, "image_list.txt"), 'r') as f: | ||
| file_label = [line.rstrip().split(' ') for line in f if line != ''] | ||
| files, labels = zip(*file_label) | ||
|
|
||
| def py_file_reader(sample_info): | ||
| sample_idx = sample_info.idx_in_epoch | ||
| jpeg_filename = files[sample_idx] | ||
| label = np.int32([labels[sample_idx]]) | ||
| with open(os.path.join(images_dir, jpeg_filename), 'rb') as f: | ||
| encoded_img = np.frombuffer(f.read(), dtype=np.uint8) | ||
| return encoded_img, label | ||
|
|
||
| return py_file_reader |
There was a problem hiding this comment.
Do we need that?
How about just:
jpeg_file = os.path.join(data_root, 'db', 'single', 'jpeg', '510', 'ship-1083562_640.jpg')
def create_py_file_reader(images_dir):
encoded_img = np.fromfile(jpeg_file, dtype=np.uint8)
label = 1
return encoded_img, label
Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com>
Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com>
Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com>
JanuszL
approved these changes
Aug 12, 2021
|
!build |
|
CI MESSAGE: [2750244]: BUILD STARTED |
klecki
approved these changes
Aug 12, 2021
|
CI MESSAGE: [2750244]: BUILD PASSED |
JanuszL
added
the
important-fix
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Signed-off-by: Kamil Tokarski ktokarski@nvidia.com
Description
What happened in this PR
Fixes an issue of random Python crashes at teardown if parallel external source is used.
That behavior was caused by garbage collection order issue: python pool objects, and in effect, shared memory wrappers
could be collected before pipeline backed, which led to pipeline accessing unmmaped memory regions.
Additional information
Affected modules and functionalities:
Pipeline/Pipeline backend
Key points relevant for the review:
Checklist
Tests
Documentation
DALI team only
Requirements
REQ IDs: N/A
JIRA TASK: DALI-2239