DQM perLS in one shot #41196
DQM perLS in one shot #41196
Conversation
|
-code-checks Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-41196/34892
Code check has found code style and quality issues which could be resolved by applying following patch(s)
|
|
please test |
|
+code-checks Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-41196/34894
|
|
A new Pull Request was created by @jfernan2 for master. It involves the following packages:
@emanueleusai, @cmsbuild, @syuvivida, @rvenditti, @micsucmed, @pmandrik can you please review it and eventually sign? Thanks. cms-bot commands are listed here |
|
@cmsbuild please test |
|
@cms-sw/dqm-l2 could you please trigger the tests at your earliest convenience. I don't have privileges as in the past :-) |
|
@cmsbuild please test |
|
+1 Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-30f993/31623/summary.html Comparison SummarySummary:
|
|
+code-checks Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-41196/35134
|
|
Pull request #41196 was updated. @emanueleusai, @cmsbuild, @syuvivida, @rvenditti, @micsucmed, @pmandrik can you please check and sign again. |
|
please test |
|
+1 Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-30f993/31926/summary.html Comparison SummarySummary:
|
|
+1 |
cmsbuild
added
dqm-approved
fully-signed
and removed
dqm-pending
pending-signatures
labels
|
This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @rappoccio (and backports should be raised in the release meeting by the corresponding L2) |
|
+1 |
PR description:
DQMIO data tier is saved perRun basis up to now, while perLS mode was activated in the so called nanoDQMIO format which is described here:
https://twiki.cern.ch/twiki/bin/view/CMS/PerLsDQMIO
This implied running RECO-DQM steps twice, one perRun and one perLS. This PR attempts to create the output in one shot to reduce computing resource needs, by enabling perLS mode for the list of (needed) MEs requested by DPG/POGs in [1].
These changes result in an (expected) increase of the DQMIO data tier (usual perRun MEs + additional dedicated perLS MEs) but maintaining at the same time (plain root) DQM format (to be uploaded to DQM GUI) after HARVESTING identical (up to rounding errors).
With this single shot mode, there will only be one output file, with both per-run and per-lumi histograms in it (depending on the ME). For the per-lumi MEs, no per-run results will be present, but that is fine, the harvesting job will make per-run histograms from the per-lumi ones. The Harvesting Job will see per-lumi histograms (which only matters if there is endLumi code, which is rare outside online), and save per-Job histograms in the end (which will be full runs for the usual Harvesting jobs). The plain ROOT file should be the same, up to rounding errors, which the DQM BinByBin tool may detect in this PR.
Needless to say that, future updates of the list of MEs in file [1], will chnage the content of per LS MEs to be saved in the future.
PR validation:
Uisng file: /store/data/Run2022G/Muon/RAW/v1/000/362/596/00000/6e355fc9-e4e9-478a-9ce4-d3b6c1d6135e.root from /Muon/Run2022G which contains 8 LS from run 362596, I performed standard RECO and DQM with command in [2]:
Checked MEs from [1] are created in a per LS basis (identified one typo in [1] and 3 MEs from Tracking POG which apparently do not exist in this run[3], so I flagged them in [1] file)
Checked DQMIO output does not explode (average increase is 72kB per LS with the current version of [1]). Indeed, standard DQM workflow already contains 1366 MEs in perLS mode by default (used by DPGs and POGs to created plots vs LS in DQM GUI), out of which 14 are again requested in [1].
Hence, we are adding only 148 new MEs in per LS to the standard DQMIO data tier, which acocunt for 578545 extra bytes / 8 LSs = 72318 bytes = 72.3 kB per LS. An increase totally acceptable bearing in mind that DQMIO datasets are small (order of of dozens of GB) compared to the rest of CMS datasets in DAS[4]
The total numbers of MEs in default DQMIO for a given run and in the new version after this PR iare the following:
Default (before):
Run 362596, 72672 MEs (per Run MEs)
Run 362596, Lumi 31-38, 1514 MEs (per LS MEs)
After this PR:
Run 362596, 72820 MEs (per Run MEs)
Run 362596, Lumi 31-38, 1366 MEs (per LS MEs)
and the type composition is detailed in [5]
Checked that DQM Harvesting works out of the box taking as input the new DQMIO with mixed per Run and per LS MEs:
https://tinyurl.com/27pmw76f (differences in the bin by bin tool ONLY for process time in DQM modules)
runTheMatrix.py -l limited -t 8 works
[1] DQMServices/Core/python/nanoDQMIO_perLSoutput_cff.py
[2] cmsDriver.py step2 --conditions 124X_dataRun3_v10 --data --datatier DQMIO --era Run3 --eventcontent DQM --filein file:/eos/cms/store/data/Run2022B/SingleMuon/RAW/v1/000/355/094/00000/1ed76311-a4df-4699-885e-965087521f03.root --fileout file:step2.root --nStreams 2 --nThreads 8 --no_exec --number 10 --process RECO --python_filename step_2_nano_cfg.py --scenario pp --step RAW2DIGI,L1Reco,RECO,DQM
[3] https://tinyurl.com/2oy3k7vc does not contain the following MEs:
Tracking/TrackParameters/GeneralProperties/Chi2oNDF_lumiFlag_GenTk
Tracking/TrackParameters/GeneralProperties/NumberOfRecHitsPerTrack_lumiFlag_GenTk
Tracking/TrackParameters/GeneralProperties/NumberOfTracks_lumiFlag_GenTk
requested in [1]
[4] https://cmsweb.cern.ch/das/request?view=list&limit=50&instance=prod%2Fglobal&input=size+dataset%3D%2FMuon%2FRun2022G-PromptReco-v1%2FDQMIO
[5] Before (default DQM) and now (after this PR) perLS ME composition:
+--------+------+------------+-----------+------------------+----------+
| Run | Lumi | FirstIndex | LastIndex | Type | ME Count |
+--------+------+------------+-----------+------------------+----------+
| 362596 | 31 | 0 | 984 | 3 (TH1Fs) | 985 |
| 362596 | 31 | 0 | 19 | 5 (TH1Ds) | 20 |
| 362596 | 31 | 0 | 38 | 6 (TH2Fs) | 39 |
| 362596 | 31 | 0 | 265 | 10 (TProfiles) | 266 |
| 362596 | 31 | 0 | 55 | 11 (TProfile2Ds) | 56 |
+--------+------+------------+-----------+------------------+----------+
| Run | Lumi | FirstIndex | LastIndex | Type | ME Count |
+--------+------+------------+-----------+------------------+----------+
| 362596 | 31 | 0 | 0 | 0 (Ints) | 1 |
| 362596 | 31 | 0 | 1109 | 3 (TH1Fs) | 1110 |
| 362596 | 31 | 0 | 19 | 5 (TH1Ds) | 20 |
| 362596 | 31 | 0 | 60 | 6 (TH2Fs) | 61 |
| 362596 | 31 | 0 | 265 | 10 (TProfiles) | 266 |
| 362596 | 31 | 0 | 55 | 11 (TProfile2Ds) | 56 |
Used dqmiodumpmetadata.py , dqmiodumpindices.py and dqmiolistmes.py scripts for these checks