Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix sharding and shuffling in VideoLoaderDecoder #3411

Merged
merged 1 commit into from
Oct 8, 2021
Merged

Fix sharding and shuffling in VideoLoaderDecoder #3411

merged 1 commit into from
Oct 8, 2021

Conversation

JanuszL
Copy link
Contributor

@JanuszL JanuszL commented Oct 7, 2021

  • fixes calculation of sample start index for each shard when
    VideoLoaderDecoder is created
  • adds preshuffling when VideoLoaderDecoder is created to make
    sure that samples are globally mixed, not only inside the prefetch
    buffer
  • updates VideoReaderDecoderTest.CompareReaders to use sharding

Signed-off-by: Janusz Lisiecki jlisiecki@nvidia.com

Description

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Refactoring (Redesign of existing code that doesn't affect functionality)
  • Other (e.g. Documentation, Tests, Configuration)

What happened in this PR

  • fixes calculation of sample start index for each shard when
    VideoLoaderDecoder is created
  • adds preshuffling when VideoLoaderDecoder is created to make
    sure that samples are globally mixed, not only inside the prefetch
    buffer
  • updates VideoReaderDecoderTest.CompareReaders to use sharding

Additional information

  • Affected modules and functionalities:
    • video_loader_decoder.cc
    • video_reader_decoder_op_test
  • Key points relevant for the review:
    • NA

Checklist

Tests

  • Existing tests apply
  • New tests added
    • Python tests
    • GTests
    • Benchmark
    • Other
  • N/A

Documentation

  • Existing documentation applies
  • Documentation updated
    • Docstring
    • Doxygen
    • RST
    • Jupyter
    • Other
  • N/A

DALI team only

Requirements

  • Implements new requirements
  • Affects existing requirements
  • N/A

REQ IDs: N/A

JIRA TASK: N/A

@JanuszL
Fix sharding and shuffling in VideoLoaderDecoder
516ab39 
- fixes calculation of sample start index for each shard when
  VideoLoaderDecoder is created
- adds preshuffling when VideoLoaderDecoder is created to make
  sure that samples are globally mixed, not only inside the prefetch
  buffer
- updates VideoReaderDecoderTest.CompareReaders to use sharding

Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>
@JanuszL
Copy link
Contributor Author

JanuszL commented Oct 7, 2021

!build

@dali-automaton
Copy link
Collaborator

CI MESSAGE: [3140937]: BUILD STARTED

@awolant awolant self-assigned this Oct 7, 2021
@dali-automaton
Copy link
Collaborator

CI MESSAGE: [3140937]: BUILD PASSED

@JanuszL JanuszL merged commit 1dabb57 into NVIDIA:main Oct 8, 2021
@JanuszL JanuszL deleted the fix_VideoLoaderDecoder branch October 8, 2021 09:57
cyyever pushed a commit to cyyever/DALI that referenced this pull request Oct 17, 2021
@JanuszL @cyyever
Fix sharding and shuffling in VideoLoaderDecoder (NVIDIA#3411)
c158f6f 
- fixes calculation of sample start index for each shard when
  VideoLoaderDecoder is created
- adds preshuffling when VideoLoaderDecoder is created to make
  sure that samples are globally mixed, not only inside the prefetch
  buffer
- updates VideoReaderDecoderTest.CompareReaders to use sharding

Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>
cyyever pushed a commit to cyyever/DALI that referenced this pull request Jan 23, 2022
@JanuszL @cyyever
Fix sharding and shuffling in VideoLoaderDecoder (NVIDIA#3411)
3480f08 
- fixes calculation of sample start index for each shard when
  VideoLoaderDecoder is created
- adds preshuffling when VideoLoaderDecoder is created to make
  sure that samples are globally mixed, not only inside the prefetch
  buffer
- updates VideoReaderDecoderTest.CompareReaders to use sharding

Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>
cyyever pushed a commit to cyyever/DALI that referenced this pull request Jan 23, 2022
@JanuszL @cyyever
Fix sharding and shuffling in VideoLoaderDecoder (NVIDIA#3411)
cc85c13 
- fixes calculation of sample start index for each shard when
  VideoLoaderDecoder is created
- adds preshuffling when VideoLoaderDecoder is created to make
  sure that samples are globally mixed, not only inside the prefetch
  buffer
- updates VideoReaderDecoderTest.CompareReaders to use sharding

Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>
cyyever pushed a commit to cyyever/DALI that referenced this pull request Jan 23, 2022
@JanuszL @cyyever
Fix sharding and shuffling in VideoLoaderDecoder (NVIDIA#3411)
0f3c817 
- fixes calculation of sample start index for each shard when
  VideoLoaderDecoder is created
- adds preshuffling when VideoLoaderDecoder is created to make
  sure that samples are globally mixed, not only inside the prefetch
  buffer
- updates VideoReaderDecoderTest.CompareReaders to use sharding

Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>
cyyever pushed a commit to cyyever/DALI that referenced this pull request Jan 23, 2022
@JanuszL @cyyever
Fix sharding and shuffling in VideoLoaderDecoder (NVIDIA#3411)
0631178 
- fixes calculation of sample start index for each shard when
  VideoLoaderDecoder is created
- adds preshuffling when VideoLoaderDecoder is created to make
  sure that samples are globally mixed, not only inside the prefetch
  buffer
- updates VideoReaderDecoderTest.CompareReaders to use sharding

Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>
cyyever pushed a commit to cyyever/DALI that referenced this pull request Jan 23, 2022
@JanuszL @cyyever
Fix sharding and shuffling in VideoLoaderDecoder (NVIDIA#3411)
b8225ac 
- fixes calculation of sample start index for each shard when
  VideoLoaderDecoder is created
- adds preshuffling when VideoLoaderDecoder is created to make
  sure that samples are globally mixed, not only inside the prefetch
  buffer
- updates VideoReaderDecoderTest.CompareReaders to use sharding

Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>
cyyever pushed a commit to cyyever/DALI that referenced this pull request Jan 23, 2022
@JanuszL @cyyever
Fix sharding and shuffling in VideoLoaderDecoder (NVIDIA#3411)
8ddf1af 
- fixes calculation of sample start index for each shard when
  VideoLoaderDecoder is created
- adds preshuffling when VideoLoaderDecoder is created to make
  sure that samples are globally mixed, not only inside the prefetch
  buffer
- updates VideoReaderDecoderTest.CompareReaders to use sharding

Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>
cyyever pushed a commit to cyyever/DALI that referenced this pull request Jan 23, 2022
@JanuszL @cyyever
Fix sharding and shuffling in VideoLoaderDecoder (NVIDIA#3411)
c46f8b9 
- fixes calculation of sample start index for each shard when
  VideoLoaderDecoder is created
- adds preshuffling when VideoLoaderDecoder is created to make
  sure that samples are globally mixed, not only inside the prefetch
  buffer
- updates VideoReaderDecoderTest.CompareReaders to use sharding

Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>
cyyever pushed a commit to cyyever/DALI that referenced this pull request Jan 23, 2022
@JanuszL @cyyever
Fix sharding and shuffling in VideoLoaderDecoder (NVIDIA#3411)
c1bf6ab 
- fixes calculation of sample start index for each shard when
  VideoLoaderDecoder is created
- adds preshuffling when VideoLoaderDecoder is created to make
  sure that samples are globally mixed, not only inside the prefetch
  buffer
- updates VideoReaderDecoderTest.CompareReaders to use sharding

Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>
cyyever pushed a commit to cyyever/DALI that referenced this pull request Jan 23, 2022
@JanuszL @cyyever
Fix sharding and shuffling in VideoLoaderDecoder (NVIDIA#3411)
a042f12 
- fixes calculation of sample start index for each shard when
  VideoLoaderDecoder is created
- adds preshuffling when VideoLoaderDecoder is created to make
  sure that samples are globally mixed, not only inside the prefetch
  buffer
- updates VideoReaderDecoderTest.CompareReaders to use sharding

Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>
cyyever pushed a commit to cyyever/DALI that referenced this pull request Jan 23, 2022
@JanuszL @cyyever
Fix sharding and shuffling in VideoLoaderDecoder (NVIDIA#3411)
1438f0c 
- fixes calculation of sample start index for each shard when
  VideoLoaderDecoder is created
- adds preshuffling when VideoLoaderDecoder is created to make
  sure that samples are globally mixed, not only inside the prefetch
  buffer
- updates VideoReaderDecoderTest.CompareReaders to use sharding

Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>
cyyever pushed a commit to cyyever/DALI that referenced this pull request Jan 23, 2022
@JanuszL @cyyever
Fix sharding and shuffling in VideoLoaderDecoder (NVIDIA#3411)
c4f0c67 
- fixes calculation of sample start index for each shard when
  VideoLoaderDecoder is created
- adds preshuffling when VideoLoaderDecoder is created to make
  sure that samples are globally mixed, not only inside the prefetch
  buffer
- updates VideoReaderDecoderTest.CompareReaders to use sharding

Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>
cyyever pushed a commit to cyyever/DALI that referenced this pull request Jan 23, 2022
@JanuszL @cyyever
Fix sharding and shuffling in VideoLoaderDecoder (NVIDIA#3411)
644ef59 
- fixes calculation of sample start index for each shard when
  VideoLoaderDecoder is created
- adds preshuffling when VideoLoaderDecoder is created to make
  sure that samples are globally mixed, not only inside the prefetch
  buffer
- updates VideoReaderDecoderTest.CompareReaders to use sharding

Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>
cyyever pushed a commit to cyyever/DALI that referenced this pull request Jan 23, 2022
@JanuszL @cyyever
Fix sharding and shuffling in VideoLoaderDecoder (NVIDIA#3411)
d639e6f 
- fixes calculation of sample start index for each shard when
  VideoLoaderDecoder is created
- adds preshuffling when VideoLoaderDecoder is created to make
  sure that samples are globally mixed, not only inside the prefetch
  buffer
- updates VideoReaderDecoderTest.CompareReaders to use sharding

Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>
cyyever pushed a commit to cyyever/DALI that referenced this pull request Feb 21, 2022
@JanuszL @cyyever
Fix sharding and shuffling in VideoLoaderDecoder (NVIDIA#3411)
170b10e 
- fixes calculation of sample start index for each shard when
  VideoLoaderDecoder is created
- adds preshuffling when VideoLoaderDecoder is created to make
  sure that samples are globally mixed, not only inside the prefetch
  buffer
- updates VideoReaderDecoderTest.CompareReaders to use sharding

Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>
cyyever pushed a commit to cyyever/DALI that referenced this pull request May 13, 2022
@JanuszL @cyyever
Fix sharding and shuffling in VideoLoaderDecoder (NVIDIA#3411)
34ec07f 
- fixes calculation of sample start index for each shard when
  VideoLoaderDecoder is created
- adds preshuffling when VideoLoaderDecoder is created to make
  sure that samples are globally mixed, not only inside the prefetch
  buffer
- updates VideoReaderDecoderTest.CompareReaders to use sharding

Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>
cyyever pushed a commit to cyyever/DALI that referenced this pull request Jun 7, 2022
@JanuszL @cyyever
Fix sharding and shuffling in VideoLoaderDecoder (NVIDIA#3411)
c48ec3b 
- fixes calculation of sample start index for each shard when
  VideoLoaderDecoder is created
- adds preshuffling when VideoLoaderDecoder is created to make
  sure that samples are globally mixed, not only inside the prefetch
  buffer
- updates VideoReaderDecoderTest.CompareReaders to use sharding

Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@banasraf banasraf banasraf approved these changes

@awolant awolant awolant approved these changes

Assignees

@banasraf banasraf

@awolant awolant

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@JanuszL @dali-automaton @banasraf @awolant