Redesign of the association maps, multivector manager, HGCAL Rechits and Validation with significant speedup of Phase-2 workflows #45865

felicepantaleo · 2024-09-03T13:42:22Z

PR description:

In the context of Next Generation Triggers and HGCAL TICL reconstruction, fast AssociatorMaps between Reco and Sim, Sim to Sim, and Reco to Reco are required in order to develop new reconstruction algorithms and study their performance. They have also been included in the HGCalValidator and in the TICLDumper, but can be extended to many other detectors (MTD for instance, as it is the new main offender).
The HGCAL Rechit producer was optimized by avoiding calling virtual functions for every rechit and to enable support for the MultiVectorManager.
The MultiVectorManager was optimized and together with the HGCAL RecHit map was used to optimize the E/Gamma reconstruction and the associators.
The HGCAL Validation was redesigned to make effective usage of the new associators. The framework was completely redesigned to generate automatically validation plots for any new collection of TICL tracksters against SimTracksters from Simclusters and CaloParticles.

Performance measurements:

Performed with 14_1_0_pre7 with TTbar PU200, 10 events, single thread, single stream

14_1_0_pre7:

Total time: 536s
Total allocated memory: 90GB

14_1_0_pre7+PR

Total time: 171s
Total allocated memory: 56GB

Performance comparison

Full performance comparison table here:
https://docs.google.com/spreadsheets/d/1vdmDDJJ07tf9ekQ8EC24sMqqvtCtiU2Mh6FawsIRu24/edit?usp=sharing

Notable numbers:

Overall Phase-2 TTbar PU200 Reconstruction+Validation workflows: speedup 313% , allocated memory -38%
Overall E/Gamma Phase-2 Reconstruction: 1.5x speedup
ElectronSeedProducer ecalDrivenElectronSeeds: 3x speedup
HGCalValidator: 30x speedup
New associators: between two orders of magnitude and infinite speedup depending if you want to run on the CaloParticles from pileup as well. In this case the associators in release will crash for the amount of resources needed.

@cms-sw/hgcal-dpg-l2 @cms-sw/egamma-pog-l2 @waredjeb @AuroraPerego @rovere

…aggregation by MultiVectorManager

…ociationMap

…icl::AssociationMap

…ator using ticl::AssociationMap

…ticl::AssociationMap

…icl::AssociationMap

…to use the new associators and automatically produce validation plots for new tracksters collections.

cmsbuild · 2024-09-03T13:43:00Z

cms-bot internal usage

cmsbuild · 2024-09-03T13:46:04Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-45865/41614

There are other open Pull requests which might conflict with changes you have proposed:
- File HLTrigger/Configuration/python/HLT_75e33/modules/hltTiclTracksterLinks_cfi.py modified in PR(s): Refactoring of TICL EnergyRegression and PID model #45821
- File HLTrigger/Configuration/python/HLT_75e33/modules/hltTiclTrackstersPassthrough_cfi.py modified in PR(s): Refactoring of TICL EnergyRegression and PID model #45821
- File RecoHGCal/TICL/plugins/PatternRecognitionbyPassthrough.cc modified in PR(s): Refactoring of TICL EnergyRegression and PID model #45821
- File RecoHGCal/TICL/plugins/PatternRecognitionbyPassthrough.h modified in PR(s): Refactoring of TICL EnergyRegression and PID model #45821
- File RecoHGCal/TICL/plugins/TrackstersProducer.cc modified in PR(s): Refactoring of TICL EnergyRegression and PID model #45821
- File RecoHGCal/TICL/python/iterativeTICL_cff.py modified in PR(s): Refactoring of TICL EnergyRegression and PID model #45821
- File SimDataFormats/Associations/src/classes_def.xml modified in PR(s): move dictionaries next to typedefs for TrackingParticle associations #37507

cmsbuild · 2024-09-05T13:46:55Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-45865/41641

There are other open Pull requests which might conflict with changes you have proposed:
- File HLTrigger/Configuration/python/HLT_75e33/modules/hltTiclTracksterLinks_cfi.py modified in PR(s): Refactoring of TICL EnergyRegression and PID model #45821
- File HLTrigger/Configuration/python/HLT_75e33/modules/hltTiclTrackstersPassthrough_cfi.py modified in PR(s): Refactoring of TICL EnergyRegression and PID model #45821
- File RecoHGCal/TICL/plugins/PatternRecognitionbyPassthrough.cc modified in PR(s): Refactoring of TICL EnergyRegression and PID model #45821
- File RecoHGCal/TICL/plugins/PatternRecognitionbyPassthrough.h modified in PR(s): Refactoring of TICL EnergyRegression and PID model #45821
- File RecoHGCal/TICL/plugins/TrackstersProducer.cc modified in PR(s): Refactoring of TICL EnergyRegression and PID model #45821
- File RecoHGCal/TICL/python/iterativeTICL_cff.py modified in PR(s): Refactoring of TICL EnergyRegression and PID model #45821
- File SimDataFormats/Associations/src/classes_def.xml modified in PR(s): move dictionaries next to typedefs for TrackingParticle associations #37507

cmsbuild · 2024-09-05T13:47:24Z

Pull request #45865 was updated. @Martin-Grunewald, @antoniovagnerini, @civanch, @jfernan2, @mandrenguyen, @mdhildreth, @mmusich, @nothingface0, @rvenditti, @srimanob, @subirsarkar, @syuvivida, @tjavaid can you please check and sign again.

cmsbuild · 2024-09-05T19:04:58Z

+1

Size: This PR adds an extra 52KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-232398/41316/summary.html
COMMIT: 28509ad
CMSSW: CMSSW_14_2_X_2024-09-05-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/45865/41316/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially added 1 lines to the logs
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 297 differences found in the comparisons
DQMHistoTests: Total files compared: 45
DQMHistoTests: Total histograms compared: 3422869
DQMHistoTests: Total failures: 17383
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3405466
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 2745.032 KiB( 44 files compared)
DQMHistoSizes: changed ( 24834.911,... ): 734.076 KiB HGCAL/HGCalValidator
DQMHistoSizes: changed ( 29896.203 ): -1659.424 KiB HGCAL/HGCalValidator
Checked 197 log files, 167 edm output root files, 45 DQM output files
TriggerResults: no differences found

cmsbuild · 2024-09-06T08:29:36Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-45865/41698

There are other open Pull requests which might conflict with changes you have proposed:
- File HLTrigger/Configuration/python/HLT_75e33/modules/hltTiclTracksterLinks_cfi.py modified in PR(s): Refactoring of TICL EnergyRegression and PID model #45821
- File HLTrigger/Configuration/python/HLT_75e33/modules/hltTiclTrackstersPassthrough_cfi.py modified in PR(s): Refactoring of TICL EnergyRegression and PID model #45821
- File RecoHGCal/TICL/plugins/PatternRecognitionbyPassthrough.cc modified in PR(s): Refactoring of TICL EnergyRegression and PID model #45821
- File RecoHGCal/TICL/plugins/PatternRecognitionbyPassthrough.h modified in PR(s): Refactoring of TICL EnergyRegression and PID model #45821
- File RecoHGCal/TICL/plugins/TrackstersProducer.cc modified in PR(s): Refactoring of TICL EnergyRegression and PID model #45821
- File RecoHGCal/TICL/python/iterativeTICL_cff.py modified in PR(s): Refactoring of TICL EnergyRegression and PID model #45821
- File SimDataFormats/Associations/src/classes_def.xml modified in PR(s): move dictionaries next to typedefs for TrackingParticle associations #37507

cmsbuild · 2024-09-06T08:30:01Z

Pull request #45865 was updated. @Martin-Grunewald, @antoniovagnerini, @civanch, @cmsbuild, @jfernan2, @mandrenguyen, @mdhildreth, @mmusich, @nothingface0, @rvenditti, @srimanob, @subirsarkar, @syuvivida, @tjavaid can you please check and sign again.

felicepantaleo · 2024-09-06T12:35:31Z

enable profiling

felicepantaleo · 2024-09-06T12:36:43Z

I'm enabling profiling even though the validation is not run in profiling workflows. Is it possible to specify another workflow for profiling?

felicepantaleo · 2024-09-06T12:36:50Z

@cmsbuild please test

jfernan2 · 2024-09-06T16:37:02Z

Profiling comparison tests run on wfs 29834.21 (D110 upgrade) and 12634.21 (Run3 2023) as set on cms-sw/cms-bot#2282

However, the timing comparison fails due to an unknown igprof (and VTune too) segmentation fault, this is a long standing problem which prevents to have timing studies in PRs unfortunately

#43166

cmsbuild · 2024-09-06T17:35:19Z

+1

Size: This PR adds an extra 52KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-232398/41378/summary.html
COMMIT: 8d8b7d4
CMSSW: CMSSW_14_2_X_2024-09-05-2300/el8_amd64_gcc12
Additional Tests: PROFILING
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/45865/41378/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 233 differences found in the comparisons
DQMHistoTests: Total files compared: 45
DQMHistoTests: Total histograms compared: 3422869
DQMHistoTests: Total failures: 14061
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3408788
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 2745.032 KiB( 44 files compared)
DQMHistoSizes: changed ( 24834.911,... ): 734.076 KiB HGCAL/HGCalValidator
DQMHistoSizes: changed ( 29896.203 ): -1659.424 KiB HGCAL/HGCalValidator
Checked 197 log files, 167 edm output root files, 45 DQM output files
TriggerResults: no differences found

felicepantaleo · 2024-09-07T11:24:22Z

Thanks @jfernan2 for clarifying. Unfortunately 29834.21 does not run any validation nor prevalidation during step3.

felicepantaleo and others added 17 commits August 29, 2024 12:33

Update MultiVectorManager to C++20 and improve it by avoiding copies.

1faf505

Introduce ticl::AssociationMap

ed85e4d

Improve RecHit data format and producers to be faster and to support …

f229809

…aggregation by MultiVectorManager

TICL Associators: Create Hit To Tracksters associator using ticl::Ass…

d64f2ab

…ociationMap

TICL Associators: Create LayerCluster to Trackster associator using t…

419ba45

…icl::AssociationMap

TICL Associators: Create hits to layerclusters and sim objects associ…

86e196d

…ator using ticl::AssociationMap

TICL Associators: Create SimCluster To Caloparticle associator using …

d46cbfc

…ticl::AssociationMap

TICL Associators: Create Trackster To SimTrackster associator using t…

704d51e

…icl::AssociationMap

ticlv5: rename Tracksters by Passthrough to Recovery

c7592bf

Double to float

d8dd2e4

Fix HLTCaloObjInRegionsProducer.cc to use the new HGCRecHit collection.

65bd5fe

Run new TICL associators

5361e0c

Adapt TICLDumper to use the new TICL Associators

0bcfc35

Validation is Automation: Redesign of the HGCAL Trackster Validation …

e660109

…to use the new associators and automatically produce validation plots for new tracksters collections.

Pass the new AssociationMaps to the TICLCandidateValidator

eed28af

fix hgcal event content for hlt and add new associators

b16adf4

Optimize egamma recostruction using MultiVectorManager and RecHitMap

acff6a7

cmsbuild added this to the CMSSW_14_2_X milestone Sep 3, 2024

cmsbuild added reconstruction-pending simulation-pending dqm-pending hlt-pending pending-signatures tests-pending orp-pending upgrade-pending code-checks-pending changes-dataformats labels Sep 3, 2024

cmsbuild added tests-started and removed tests-pending labels Sep 5, 2024

cmsbuild added code-checks-approved and removed code-checks-pending labels Sep 5, 2024

cmsbuild mentioned this pull request Sep 5, 2024

Refactoring of TICL EnergyRegression and PID model #45821

Open

cmsbuild added tests-approved and removed tests-started labels Sep 5, 2024

Remove duplicate dictionary from classes_def.xml

8d8b7d4

cmsbuild added tests-pending code-checks-pending and removed tests-approved code-checks-approved labels Sep 6, 2024

cmsbuild added code-checks-approved and removed code-checks-pending labels Sep 6, 2024

cmsbuild added tests-started and removed tests-pending labels Sep 6, 2024

cmsbuild added tests-approved and removed tests-started labels Sep 6, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Redesign of the association maps, multivector manager, HGCAL Rechits and Validation with significant speedup of Phase-2 workflows #45865

Redesign of the association maps, multivector manager, HGCAL Rechits and Validation with significant speedup of Phase-2 workflows #45865

felicepantaleo commented Sep 3, 2024 •

edited

Loading

cmsbuild commented Sep 3, 2024 •

edited

Loading

cmsbuild commented Sep 3, 2024

cmsbuild commented Sep 5, 2024

cmsbuild commented Sep 5, 2024

cmsbuild commented Sep 5, 2024

cmsbuild commented Sep 6, 2024

cmsbuild commented Sep 6, 2024

felicepantaleo commented Sep 6, 2024

felicepantaleo commented Sep 6, 2024

felicepantaleo commented Sep 6, 2024

jfernan2 commented Sep 6, 2024

cmsbuild commented Sep 6, 2024

felicepantaleo commented Sep 7, 2024

Redesign of the association maps, multivector manager, HGCAL Rechits and Validation with significant speedup of Phase-2 workflows #45865

Are you sure you want to change the base?

Redesign of the association maps, multivector manager, HGCAL Rechits and Validation with significant speedup of Phase-2 workflows #45865

Conversation

felicepantaleo commented Sep 3, 2024 • edited Loading

PR description:

Performance measurements:

Performance comparison

cmsbuild commented Sep 3, 2024 • edited Loading

cmsbuild commented Sep 3, 2024

cmsbuild commented Sep 5, 2024

cmsbuild commented Sep 5, 2024

cmsbuild commented Sep 5, 2024

Comparison Summary

cmsbuild commented Sep 6, 2024

cmsbuild commented Sep 6, 2024

felicepantaleo commented Sep 6, 2024

felicepantaleo commented Sep 6, 2024

felicepantaleo commented Sep 6, 2024

jfernan2 commented Sep 6, 2024

cmsbuild commented Sep 6, 2024

Comparison Summary

felicepantaleo commented Sep 7, 2024

felicepantaleo commented Sep 3, 2024 •

edited

Loading

cmsbuild commented Sep 3, 2024 •

edited

Loading