Synchronize CUDA stream once in operator benchmark

[szkarpinski]

Description

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Refactoring (Redesign of existing code that doesn't affect functionality)
• Other (e.g. Documentation, Tests, Configuration)

What happened in this PR

CUDA stream was synchronized after each iteration in an operator benchmark, which introduced an overhead to the measurements, especially for small data and small batch sizes. In a real pipeline the synchronization would not happen after each operator but only at the end, so the introduced overhead was benchmark-specific and would not occur in real life.

This overhead was quite significant, when the operator execution was fast: the throughput of Copy measured on Titan V with copy_bench.cc for batch_size=1 and 3MiB images was decreased by 25% when synchronizing the stream after each iteration. The main source of the overhead was that the synchronization made it impossible to schedule copy with cudaMemcpyAsync while another copy was still in progress.

This PR moves the stream synchronization out of the loop, synchronizing the stream only once in a benchmark, so that the measured operator performance is closer to the actual one.

Additional information

Key points relevant for the review:

As synchronizing each loop introduces a benchmark-specific overhead, I consider it a bug that should be fixed. I am assuming that:

• Google Benchmark measures and reports the total execution time of all iterations, so spending most of the time in the last iteration does not impact the results
• There are no use cases in which synchronizing the stream each iterations is desired

If any of the above is false, making synchronization behaviour configurable with some optional parameter to RunGPU would be a better solution.

Checklist

Tests

• Existing tests apply
• New tests added
  • Python tests
  • GTests
  • Benchmark
  • Other
• N/A

Documentation

• Existing documentation applies
• Documentation updated
  • Docstring
  • Doxygen
  • RST
  • Jupyter
  • Other
• N/A

DALI team only

Requirements

• Implements new requirements
• Affects existing requirements
• N/A

REQ IDs: N/A

JIRA TASK: N/A

[jantonguirao]

!build

[dali-automaton]

CI MESSAGE: [3544555]: BUILD STARTED

[dali-automaton]

CI MESSAGE: [3544555]: BUILD PASSED