Add optional flags to disable SOA->legacy conversion and GPU->CPU transfer #132

makortel · 2018-08-14T08:46:18Z

This PR simplifies the raw2cluster and rechits CPU output so that they are always produced (since by default the products are always needed in CPU, resolves #67). Then it adds explicit configuration flags to switch off the conversion from CPU-SOA to CPU-legacy, and the GPU->CPU transfers altogether. It also adds two customization functions extending the "profiling workflow" to disable the data conversion, and, in addition, disable the transfers.

The customize functions are useful to study the technical performance of a "what if" case where we would replace the legacy dataformats with SOAs.

@felicepantaleo @fwyzard @VinInn

… to disable unnecessary GPU->CPU transfers

fwyzard · 2018-08-15T15:20:56Z

Validation summary

Reference release CMSSW_10_2_2_patch1 at 400b27e
Development branch CMSSW_10_2_X_Patatrack at 12f20c1
Testing PRs:

Add optional flags to disable SOA->legacy conversion and GPU->CPU transfer #132 at 60e931f

`makeTrackValidationPlots.py` plots

/RelValTTbar_13/CMSSW_10_2_1-PU25ns_102X_upgrade2018_realistic_v9_gcc7-v1/GEN-SIM-DIGI-RAW

tracking validation plots and summary for workflow 10824.5
tracking validation plots and summary for workflow 10824.8
tracking validation plots and summary for workflow 10824.7
tracking validation plots and summary for workflow 10824.9

/RelValZMM_13/CMSSW_10_2_1-102X_upgrade2018_realistic_v9_gcc7-v1/GEN-SIM-DIGI-RAW

tracking validation plots and summary for workflow 10824.5
tracking validation plots and summary for workflow 10824.8
tracking validation plots and summary for workflow 10824.7
tracking validation plots and summary for workflow 10824.9

DQM GUI plots

/RelValTTbar_13/CMSSW_10_2_1-PU25ns_102X_upgrade2018_realistic_v9_gcc7-v1/GEN-SIM-DIGI-RAW

reference DQM plots for reference release, workflow 10824.5
DQM plots for development release, workflow 10824.5
DQM plots for development release, workflow 10824.8
DQM plots for development release, workflow 10824.7
DQM plots for development release, workflow 10824.9
DQM plots for testing release, workflow 10824.5
DQM plots for testing release, workflow 10824.8
DQM plots for testing release, workflow 10824.7
DQM plots for testing release, workflow 10824.9
DQM comparison for reference workflow 10824.5
DQM comparison for workflow 10824.8
DQM comparison for workflow 10824.7
DQM comparison for workflow 10824.9

/RelValZMM_13/CMSSW_10_2_1-102X_upgrade2018_realistic_v9_gcc7-v1/GEN-SIM-DIGI-RAW

reference DQM plots for reference release, workflow 10824.5
DQM plots for development release, workflow 10824.5
DQM plots for development release, workflow 10824.8
DQM plots for development release, workflow 10824.7
DQM plots for development release, workflow 10824.9
DQM plots for testing release, workflow 10824.5
DQM plots for testing release, workflow 10824.8
DQM plots for testing release, workflow 10824.7
DQM plots for testing release, workflow 10824.9
DQM comparison for reference workflow 10824.5
DQM comparison for workflow 10824.8
DQM comparison for workflow 10824.7
DQM comparison for workflow 10824.9

logs and `nvprof`/`nvvp` profiles

/RelValTTbar_13/CMSSW_10_2_1-PU25ns_102X_upgrade2018_realistic_v9_gcc7-v1/GEN-SIM-DIGI-RAW

reference release, workflow 10824.5
- step3.py: log, visual profile and summary
development release, workflow 10824.5
- step3.py: log, visual profile and summary
- profile.py: log, visual profile and summary
development release, workflow 10824.8
- step3.py: log, visual profile and summary
- profile.py: log, visual profile and summary
- ❌ cuda-memcheck --tool initcheck --track-unused-memory no (report, log) found 16 errors
- ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
- ✔️ cuda-memcheck --tool synccheck (report, log) did not find any errors
development release, workflow 10824.7
- step3.py: log, visual profile and summary
development release, workflow 10824.9
- step3.py: log, visual profile and summary
- ✔️ cuda-memcheck --tool initcheck --track-unused-memory no (report, log) did not find any errors
- ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
- ✔️ cuda-memcheck --tool synccheck (report, log) did not find any errors
testing release, workflow 10824.5
- step3.py: log, visual profile and summary
- profile.py: log, visual profile and summary
testing release, workflow 10824.8
- step3.py: log, visual profile and summary
- profile.py: log, visual profile and summary
- ❌ cuda-memcheck --tool initcheck --track-unused-memory no (report, log) found 16 errors
- ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
- ✔️ cuda-memcheck --tool synccheck (report, log) did not find any errors
testing release, workflow 10824.7
- step3.py: log, visual profile and summary
testing release, workflow 10824.9
- step3.py: log, visual profile and summary
- ✔️ cuda-memcheck --tool initcheck --track-unused-memory no (report, log) did not find any errors
- ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
- ✔️ cuda-memcheck --tool synccheck (report, log) did not find any errors

/RelValZMM_13/CMSSW_10_2_1-102X_upgrade2018_realistic_v9_gcc7-v1/GEN-SIM-DIGI-RAW

reference release, workflow 10824.5
- step3.py: log, visual profile and summary
development release, workflow 10824.5
- step3.py: log, visual profile and summary
- profile.py: log, visual profile and summary
development release, workflow 10824.8
- step3.py: log, visual profile and summary
- profile.py: log, visual profile and summary
- ❌ cuda-memcheck --tool initcheck --track-unused-memory no (report, log) found 24 errors
- ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
- ✔️ cuda-memcheck --tool synccheck (report, log) did not find any errors
development release, workflow 10824.7
- step3.py: log, visual profile and summary
development release, workflow 10824.9
- step3.py: log, visual profile and summary
- ✔️ cuda-memcheck --tool initcheck --track-unused-memory no (report, log) did not find any errors
- ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
- ✔️ cuda-memcheck --tool synccheck (report, log) did not find any errors
testing release, workflow 10824.5
- step3.py: log, visual profile and summary
- profile.py: log, visual profile and summary
testing release, workflow 10824.8
- step3.py: log, visual profile and summary
- profile.py: log, visual profile and summary
- ❌ cuda-memcheck --tool initcheck --track-unused-memory no (report, log) found 24 errors
- ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
- ✔️ cuda-memcheck --tool synccheck (report, log) did not find any errors
testing release, workflow 10824.7
- step3.py: log, visual profile and summary
testing release, workflow 10824.9
- step3.py: log, visual profile and summary
- ✔️ cuda-memcheck --tool initcheck --track-unused-memory no (report, log) did not find any errors
- ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
- ✔️ cuda-memcheck --tool synccheck (report, log) did not find any errors

Logs

The full log is available at https://fwyzard.web.cern.ch/fwyzard/patatrack/pulls/3d906310ff6425d9a888f04aa92234c862a5fef7/log .

fwyzard · 2018-08-16T13:11:52Z

@makortel what is the use case of disabling the conversions but not the transfers ?

makortel · 2018-08-16T20:15:56Z

To separate the costs of "GPU-SOA -> CPU-SOA" and "CPU-SOA -> CPU-legacy", eventually showing that the latter is very costly.

(Ok, it's a bit cheating because if we want to persist the CPU-SOA on disk, very likely we can't use the cudaMallocHost for that, so we'd need a memcpy to another CPU-SOA, but I'd expect that to be relatively cheap. OTOH, HLT doesn't need to save clusters+hits to disk)

fwyzard · 2018-08-17T09:59:33Z

(Ok, it's a bit cheating because if we want to persist the CPU-SOA on disk, very likely we can't use the cudaMallocHost for that, so we'd need a memcpy to another CPU-SOA, but I'd expect that to be relatively cheap.

That's because the framework needs to own the memory ?

fwyzard · 2018-08-17T11:38:13Z

I hope I fixed the conflicts correctly...

fwyzard · 2018-08-17T14:12:17Z

Seems so.
However, I would rather leave 59fe1f5 out of this PR - but maybe not remove it from your own branch...

…nsfer (#132) Always produce the CPU cluster and rechit collections, since they are needed anyway. Add transfer and conversion flags to clusterizer, rechits and CA. Add customize functions to disable conversions to legacy formats, and to disable unnecessary GPU->CPU transfers.

makortel · 2018-08-17T19:20:43Z

(Ok, it's a bit cheating because if we want to persist the CPU-SOA on disk, very likely we can't use the cudaMallocHost for that, so we'd need a memcpy to another CPU-SOA, but I'd expect that to be relatively cheap.

That's because the framework needs to own the memory ?

It's more for serialization by ROOT. It may be possible to serialize cudaMallocHost-allocated objects such that they get deserialized with malloc, but it is not evident (to me) how to achieve that.

In addition, in principle another reason to copy soon from non-pageable memory to regular memory on CPU is that non-pageable memory is somewhat limited resource (ok, up to ~4 GB should be relatively safe still what I've read, so I'd expect it to take some time before we hit that limit in practice).

makortel · 2018-08-17T19:24:09Z

However, I would rather leave 59fe1f5 out of this PR - but maybe not remove it from your own branch...

That one is actually a crucial part of the more slimmed customizations. The output module (apparently) needs a non-transient product to consume to create the correct dependencies. I.e. consuming e.g. *_pixelTracksHitQuadruplets_*_* leads to nothing happen. So I took the liberty of adding a dummy producer in between, that would be made to produce the pixel track objects after we run the Riemann fit within CA producer.

…nsfer (#132) Always produce the CPU cluster and rechit collections, since they are needed anyway. Add transfer and conversion flags to clusterizer, rechits and CA. Add a skeleton for the future pixel track producer. Add customize functions to disable conversions to legacy formats, and to disable unnecessary GPU->CPU transfers.

fwyzard · 2018-08-17T20:44:05Z

That one is actually a crucial part of the more slimmed customizations.

Right, I should have grepped case insensitive... I've now restored it.

fwyzard · 2018-08-17T20:46:20Z

It's more for serialization by ROOT. It may be possible to serialize cudaMallocHost-allocated objects such that they get deserialized with malloc, but it is not evident (to me) how to achieve that.

OK, that's beyond my expertise - but why would it make a difference where the original memory comes from ?

Do you mean that if we use something like a vector with a CUDA-aware allocator, the allocator is part of the type, and ROOT needs to be aware of it ?

fwyzard · 2018-08-17T21:32:55Z

In addition, in principle another reason to copy soon from non-pageable memory to regular memory on CPU is that non-pageable memory is somewhat limited resource (ok, up to ~4 GB should be relatively safe still what I've read, so I'd expect it to take some time before we hit that limit in practice).

I think we can have as much non-pageable memory as the available physical memory.
For example, on vinavx2 I could cudaMallocHost (and then memset to make sure it was really allocated) up to 14 GB.

fwyzard · 2018-08-19T08:45:43Z

Going back to

It's more for serialization by ROOT. It may be possible to serialize cudaMallocHost-allocated objects such that they get deserialized with malloc, but it is not evident (to me) how to achieve that.

Reading the documentation for cudaHostRegister, it should be possible to create e.g. an std::vector without any special allocator, and register its memory buffer to CUDA / make it non-pageable (as long as it is not reallocated later on).

makortel · 2018-08-21T08:03:48Z

@fwyzard Replying collectively

OK, that's beyond my expertise - but why would it make a difference where the original memory comes from ?

Do you mean that if we use something like a vector with a CUDA-aware allocator, the allocator is part of the type, and ROOT needs to be aware of it ?

Something like that. If we store struct of pointers and ints, I'd guess ROOT can't serialize it without a custom streamer. With vector the allocator is indeed in the type. Maybe with some custom container class we could hide the allocator.

I think we can have as much non-pageable memory as the available physical memory.
For example, on vinavx2 I could cudaMallocHost (and then memset to make sure it was really allocated) up to 14 GB.

Ok, I'm happy to stand corrected. I just recalled people wondering in NVidia forums why they can't allocate more than ~4 GB non-pageable even if the system has more.

Reading the documentation for cudaHostRegister, it should be possible to create e.g. an std::vector without any special allocator, and register its memory buffer to CUDA / make it non-pageable (as long as it is not reallocated later on).

While cudaHostRegister looks nice it is claimed to be expensive. Of course we could measure which one is more expensive, cudaHostRegister or memcpy. Another concern I have with per-event-per-module-per-EDM-stream cudaMallocHost or cudaHostRegister is a synchronization with CUDA runtime/driver(?) as I saw in #125.

cmsbot · 2018-08-21T08:04:03Z

A new Pull Request was created by @makortel (Matti Kortelainen) for CMSSW_10_2_X_Patatrack.

It involves the following packages:

RecoLocalTracker/Configuration
RecoLocalTracker/SiPixelClusterizer
RecoLocalTracker/SiPixelRecHits
RecoPixelVertexing/Configuration
RecoPixelVertexing/PixelTrackFitting
RecoPixelVertexing/PixelTriplets
SimTracker/TrackerHitAssociation

@cmsbot, @fwyzard can you please review it and eventually sign? Thanks.

cms-bot commands are listed here

…nsfer (#132) Always produce the CPU cluster and rechit collections, since they are needed anyway. Add transfer and conversion flags to clusterizer, rechits and CA. Add a skeleton for the future pixel track producer. Add customize functions to disable conversions to legacy formats, and to disable unnecessary GPU->CPU transfers.

makortel added 7 commits August 14, 2018 10:02

Always produce the CPU cluster collection (since it needed anyway)

d0610bc

Always produce the CPU rechit collection (since it needed anyway)

e6e0adc

Add a skeleton for the future pixel track producer

59fe1f5

Add transfer and conversion flags to clusterizer

c92e4cc

Add transfer and conversion flags to rechits

2245a1c

Add transfer and conversion flags to CA

de9cc8b

Add customize functions to disable conversions to legacy formats, and…

60e931f

… to disable unnecessary GPU->CPU transfers

Merge branch 'CMSSW_10_2_X_Patatrack' into speedupWorkflow

bbd2f3e

fwyzard merged commit 89e6ad7 into cms-patatrack:CMSSW_10_2_X_Patatrack Aug 17, 2018

cmsbot added comparison-pending labels Aug 21, 2018

fwyzard removed the comparison-pending label Sep 2, 2018

fwyzard added this to the CMSSW_10_2_4_Patatrack milestone Sep 2, 2018

fwyzard removed alca-pending labels Sep 13, 2018

This was referenced Oct 8, 2020

Patatrack integration - Pixel local reconstruction (9/N) cms-sw/cmssw#31721

Merged

Patatrack integration - Pixel vertex reconstruction (11/N) cms-sw/cmssw#31723

Merged

fwyzard mentioned this pull request Nov 6, 2020

Patatrack integration - Pixel track reconstruction (10/N) cms-sw/cmssw#31722

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add optional flags to disable SOA->legacy conversion and GPU->CPU transfer #132

Add optional flags to disable SOA->legacy conversion and GPU->CPU transfer #132

makortel commented Aug 14, 2018

fwyzard commented Aug 15, 2018 •

edited

Loading

fwyzard commented Aug 16, 2018

makortel commented Aug 16, 2018

fwyzard commented Aug 17, 2018

fwyzard commented Aug 17, 2018

fwyzard commented Aug 17, 2018 •

edited

Loading

makortel commented Aug 17, 2018

makortel commented Aug 17, 2018

fwyzard commented Aug 17, 2018

fwyzard commented Aug 17, 2018

fwyzard commented Aug 17, 2018

fwyzard commented Aug 19, 2018

makortel commented Aug 21, 2018

cmsbot commented Aug 21, 2018

Add optional flags to disable SOA->legacy conversion and GPU->CPU transfer #132

Add optional flags to disable SOA->legacy conversion and GPU->CPU transfer #132

Conversation

makortel commented Aug 14, 2018

fwyzard commented Aug 15, 2018 • edited Loading

Validation summary

makeTrackValidationPlots.py plots

/RelValTTbar_13/CMSSW_10_2_1-PU25ns_102X_upgrade2018_realistic_v9_gcc7-v1/GEN-SIM-DIGI-RAW

/RelValZMM_13/CMSSW_10_2_1-102X_upgrade2018_realistic_v9_gcc7-v1/GEN-SIM-DIGI-RAW

DQM GUI plots

/RelValTTbar_13/CMSSW_10_2_1-PU25ns_102X_upgrade2018_realistic_v9_gcc7-v1/GEN-SIM-DIGI-RAW

/RelValZMM_13/CMSSW_10_2_1-102X_upgrade2018_realistic_v9_gcc7-v1/GEN-SIM-DIGI-RAW

logs and nvprof/nvvp profiles

/RelValTTbar_13/CMSSW_10_2_1-PU25ns_102X_upgrade2018_realistic_v9_gcc7-v1/GEN-SIM-DIGI-RAW

/RelValZMM_13/CMSSW_10_2_1-102X_upgrade2018_realistic_v9_gcc7-v1/GEN-SIM-DIGI-RAW

Logs

fwyzard commented Aug 16, 2018

makortel commented Aug 16, 2018

fwyzard commented Aug 17, 2018

fwyzard commented Aug 17, 2018

fwyzard commented Aug 17, 2018 • edited Loading

makortel commented Aug 17, 2018

makortel commented Aug 17, 2018

fwyzard commented Aug 17, 2018

fwyzard commented Aug 17, 2018

fwyzard commented Aug 17, 2018

fwyzard commented Aug 19, 2018

makortel commented Aug 21, 2018

cmsbot commented Aug 21, 2018

fwyzard commented Aug 15, 2018 •

edited

Loading

`makeTrackValidationPlots.py` plots

logs and `nvprof`/`nvvp` profiles

fwyzard commented Aug 17, 2018 •

edited

Loading