Add new dEdx calibration and estimator

[stahlleiton]

PR description:

This PR adds a new dEdx estimator meant for the 2024 PbPb UPC reco. It includes the following changes on top of "Run3_UPC" era:

• Add a uint8 variable in DeDxHitInfo to store the hit selections that would otherwise require access to hit cluster and track trajectory information to derive (not present in AOD). And adapted the DeDxHitInfoProducer to store the selections.
• Add a float variable in DeDxHit to store the error on the charge deposited used by the new LikelihoodFit dEdx estimator.
• New dEdx hit calibrator module that receives a collection of DeDxHitInfo, loops and correct hits using the DeDxCalibration conditions and store the output in a DeDxHit association vector.
• New dEdx estimator based on a likelihood fit using the path length, deposit charge and charge error of hits.
• Adapted the DeDxEstimatorProducer to read DeDxHit association vectors and use the LikelihoodFit dEdx estimator.

More details on the changes and validations done are documented in the slides

The code was adapted from Ferenc Sikler's dEdx framework and results. The details are also documented in the CADI HIN-12-016.

This PR requires #45024

@mandrenguyen @sikler

PR validation:

Tested workflows 180, 180.1, 181 and 181.1 with the dedxCalibration.db file using:

runTheMatrix.py -l 180,180.1,181,181.1 --command '--customise_commands "process.GlobalTag.toGet.extend([cms.PSet(record = cms.string("DeDxCalibrationRcd"), tag = cms.string("dedxCalibration"), connect = cms.string("sqlite_file:../dedxCalibration.db"))])"'

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

[cmsbuild]

cms-bot internal usage

[mandrenguyen]

"process.GlobalTag.toGet.extend([cms.PSet(record = cms.string("DeDxCalibrationRcd"), tag = cms.string("dedxCalibration"), connect = cms.string("sqlite_file:../dedxCalibration.db"))])"'

shouldn't we put this condition in the GT?

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-45016/40302

• This PR adds an extra 76KB to repository

[cmsbuild]

A new Pull Request was created by @stahlleiton for master.

It involves the following packages:

• CondCore/SiStripPlugins (db)
• CondFormats/DataRecord (alca, db)
• CondFormats/SiStripObjects (alca, db)
• DataFormats/TrackReco (reconstruction)
• RecoTracker/Configuration (reconstruction)
• RecoTracker/DeDx (reconstruction)

@cmsbuild, @francescobrivio, @saumyaphor4252, @perrotta, @mandrenguyen, @jfernan2, @consuegs can you please review it and eventually sign? Thanks.
@seemasharmafnal, @threus, @VinInn, @gpetruc, @VourMa, @erikbutz, @felicepantaleo, @PonIlya, @JanChyczynski, @tocheng, @JanFSchulte, @GiacomoSguazzoni, @rsreds, @mtosi, @yuanchao, @rovere, @mmusich, @missirol, @dgulhan this is something you requested to watch as well.
@antoniovilela, @sextonkennedy, @rappoccio you are the release manager for this.

cms-bot commands are listed here

[stahlleiton]

"process.GlobalTag.toGet.extend([cms.PSet(record = cms.string("DeDxCalibrationRcd"), tag = cms.string("dedxCalibration"), connect = cms.string("sqlite_file:../dedxCalibration.db"))])"'

shouldn't we put this condition in the GT?

Yes, but I thought in order to add to the GT the condition format had to exist in CMSSW.

Is it possible to add to the database before adding the cond format in CMSSW?

[mmusich]

Is it possible to add to the database before adding the cond format in CMSSW?

it is possible, see e.g.: https://cms-conddb.cern.ch/cmsDbBrowser/list/Prep/tags/DeDxCalibration_test_PR_45016 that I did for exercise.
What's not (normally) possible is to couple the newly created tag to the record DeDxCalibrationRcd in the GlobalTag, because said record will be unknown to the GlobalTag creation tools until when a pre-release with the new record is built and parsed (unless the tool is manually operated to feed the record in the known list before a pre-release is built).

[mmusich]

some unrequested comments by a non-reviewer (this PR caught my eye as a former domain maintainer of some of these packages).

[stahlleiton]

Is it possible to add to the database before adding the cond format in CMSSW?

it is possible, see e.g.: https://cms-conddb.cern.ch/cmsDbBrowser/list/Prep/tags/DeDxCalibration_test_PR_45016 that I did for exercise. What's not (normally) possible is to couple the newly created tag to the record DeDxCalibrationRcd in the GlobalTag, because said record will be unknown to the GlobalTag creation tools until when a pre-release with the new record is built and parsed (unless the tool is manually operated to feed the record in the known list before a pre-release is built).

ok, then maybe a better strategy would be to separate this PR into two, one with the cond format changes and another with the dEdx code changes, and get the cond format changes merged first.

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-45016/40309

• This PR adds an extra 88KB to repository

• Found files with invalid states:

  • RecoTracker/DeDx/python/dedxCalibrationDbCreation.py:
    • Added: b1d0770
    • Deleted: d503705

• There are other open Pull requests which might conflict with changes you have proposed:

  • File CondCore/Utilities/plugins/Module_2XML.cc modified in PR(s): Add new SimBeamSpotHLLHC CondFormat #43186
  • File CondCore/Utilities/src/CondDBFetch.cc modified in PR(s): Add new SimBeamSpotHLLHC CondFormat #43186
  • File CondCore/Utilities/src/CondDBImport.cc modified in PR(s): Add new SimBeamSpotHLLHC CondFormat #43186
  • File CondCore/Utilities/src/CondFormats.h modified in PR(s): Add new SimBeamSpotHLLHC CondFormat #43186

[cmsbuild]

Pull request #45016 was updated. @jfernan2, @saumyaphor4252, @consuegs, @mandrenguyen, @cmsbuild, @francescobrivio, @perrotta can you please check and sign again.

[mmusich]

ok, then maybe a better strategy would be to separate this PR into two, one with the cond format changes

That is indeed what's (generally) done in such cases. If this is urgent it can conceivably still enter in one go (upon agreement with alca/db, since it requires some manual hack).

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c3efdd/39950/summary.html
COMMIT: 2ed86ee
CMSSW: CMSSW_14_1_X_2024-06-18-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/45016/39950/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially removed 1 lines from the logs
• Reco comparison results: 6 differences found in the comparisons
• DQMHistoTests: Total files compared: 49
• DQMHistoTests: Total histograms compared: 3451538
• DQMHistoTests: Total failures: 6
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3451512
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
• Checked 206 log files, 170 edm output root files, 49 DQM output files
• TriggerResults: no differences found

[stahlleiton]

@perrotta /HIForward1/HIRun2023A-v1/RAW is not at T2_CH_CERN so I'm not 100% sure this will work. That's why I was asking.

Indeed I did not check... Then it would be better to request it, and being able to test this PR

Let's see what happens, they are on disk at KIT. I don't remember if the tests can anyway get the files through xrootd. If they fails I'll ask for a copy at CERN.

Seems the tests worked at the end

[mmusich]

Let's see what happens, they are on disk at KIT. I don't remember if the tests can anyway get the files through xrootd. If they fails I'll ask for a copy at CERN.

As far as I understand this is expected to work by construction, because the input files are reused from an existing Relval test

cmssw/Configuration/PyReleaseValidation/python/relval_steps.py

Line 620 in 9030bf6

steps['RunUPC2023']={'INPUT':InputInfo(dataSet='/HIForward1/HIRun2023A-v1/RAW',label='upc2023',events=10000,location='STD',ls=Run2023UPC)}

therefore they have already been cached in the cmsbuild user area (see logs of wfs 142.901,142.902)

[mandrenguyen]

+1

[AdrianoDee]

+pdmv

[stahlleiton]

@cms-sw/upgrade-l2 : please review/sign this PR , thanks

[srimanob]

+Upgrade

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @rappoccio, @sextonkennedy, @antoniovilela (and backports should be raised in the release meeting by the corresponding L2)

[antoniovilela]

+1

[mmusich]

starting from CMSSW_14_1_X_2024-06-21-2300 (when this PR was merged), there is a bunch of relvals that are consistently failing in every IB:

• 136.901, 136.902, 136.903, 136.904.

All of them have, see e.g. this log:

----- Begin Fatal Exception 22-Jun-2024 14:15:47 CEST-----------------------
An exception of category 'ProductNotFound' occurred while
   [0] Processing  Event run: 276092 lumi: 133 event: 133239690 stream: 0
   [1] Running path 'MINIAODSIMoutput_step'
   [2] Prefetching for module PoolOutputModule/'MINIAODSIMoutput'
   [3] Prefetching for module PATIsolatedTrackProducer/'isolatedTracks'
   [4] Calling method for module DeDxHitInfoProducer/'dedxHitInfo'
Exception Message:
Principal::getByToken: Found zero products matching all criteria
Looking for type: SiPixelClusterShapeCache
Looking for module label: siPixelClusterShapeCache
Looking for productInstanceName: 

   Additional Info:
      [a] If you wish to continue processing events after a ProductNotFound exception,
add "TryToContinue = cms.untracked.vstring('ProductNotFound')" to the "options" PSet in the configuration.

----- End Fatal Exception -------------------------------------------------

this PR is likely the culprit.

[stahlleiton]

starting from CMSSW_14_1_X_2024-06-21-2300 (when this PR was merged), there is a bunch of relvals that are consistently failing in every IB:

* 136.901, 136.902, 136.903, 136.904.

All of them have, see e.g. this log:

----- Begin Fatal Exception 22-Jun-2024 14:15:47 CEST-----------------------
An exception of category 'ProductNotFound' occurred while
   [0] Processing  Event run: 276092 lumi: 133 event: 133239690 stream: 0
   [1] Running path 'MINIAODSIMoutput_step'
   [2] Prefetching for module PoolOutputModule/'MINIAODSIMoutput'
   [3] Prefetching for module PATIsolatedTrackProducer/'isolatedTracks'
   [4] Calling method for module DeDxHitInfoProducer/'dedxHitInfo'
Exception Message:
Principal::getByToken: Found zero products matching all criteria
Looking for type: SiPixelClusterShapeCache
Looking for module label: siPixelClusterShapeCache
Looking for productInstanceName: 

   Additional Info:
      [a] If you wish to continue processing events after a ProductNotFound exception,
add "TryToContinue = cms.untracked.vstring('ProductNotFound')" to the "options" PSet in the configuration.

----- End Fatal Exception -------------------------------------------------

this PR is likely the culprit.

Thanks for reporting it. I will work on addressing this issue asap, but I wonder why this was not found by the PR tests.

[mmusich]

but I wonder why this was not found by the PR tests.

The failing workflows are not run as part of PR tests.

[stahlleiton]

but I wonder why this was not found by the PR tests.

The failing workflows are not run as part of PR tests.

ok, thanks for the clarification. Apart from 136.901, 136.902, 136.903, 136.904, are there any other relvals that are also failing (to test against all that are failing)

[mmusich]

Apart from 136.901, 136.902, 136.903, 136.904, are there any other relvals that are also failing (to test against all that are failing)

In the last IBs there are many other relval failures (https://cmssdt.cern.ch/SDT/html/cmssdt-ib/#/ib/CMSSW_14_1_X), but none of them seems to be related to this PR.
There are unit tests in TauAnalysis/MCEmbeddingTools that show the same error, and are probably related.

[stahlleiton]

• 136.901, 136.902, 136.903, 136.904.

The PR #45288 address this issue