DeepTauId failures in RelVals (Incompatible shapes)

[AdrianoDee]

Running RelVals we are observing some failures due to a tensorflow exception coming from DeepTauId module. Some examples listed here.

1) 2023 Data reHLT + reRECO

In HLTDR3_2023 step in path HLT_VBF_DoubleMediumDeepTauPFTauHPS20_eta2p1_v7 in 14_0_0_pre3 RelVals

Fatal Exception (Exit code: 8001)
An exception of category 'InvalidRun' occurred while
[0] Processing Event run: 367131 lumi: 11 event: 22076365 stream: 0
[1] Running path 'HLT_VBF_DoubleMediumDeepTauPFTauHPS20_eta2p1_v7'
[2] Calling method for module DeepTauId/'hltHpsPFTauDeepTauProducerForVBFIsoTau'
Exception Message:
error while running session: INVALID_ARGUMENT: Incompatible shapes: [0,1,1,64] vs. [154]
[[{{node inner_muon_norm_1/FusedBatchNorm_1/Mul}}]]

with the config here, that is what we get from wf 141.035 running L1REPACK:Full,HLT:@relval2024 (HLT pointing at GRun here). The error here. The wf on Stats2.

Also in the same step in 13_3_0_pre5 RunDisplacedJet2023C in a different path (HLT_DoubleMediumDeepTauPFTauHPS30_L2NN_eta2p1_PFJet60_v6 ) run in HLT:@relval2023. The error here. The wf on Stats2.

2) 2022 Data reHLT + reRECO

Much rarer in AODNANORUN3_reHLT_2022 step in deepTau2017v2p1ForMini in RunJetMET2022D with 14_0_0 The error here. The wf on Stats2.

Fatal Exception (Exit code: 8001)
An exception of category 'InvalidRun' occurred while
[0] Processing Event run: 357735 lumi: 20 event: 32782226 stream: 0
[1] Running path 'NANOEDMAODoutput_step'
[2] Prefetching for module PoolOutputModule/'NANOEDMAODoutput'
[3] Prefetching for module SimpleCandidateFlatTableProducer/'boostedTauTable'
[4] Prefetching for module PATObjectCrossLinker/'linkedObjects'
[5] Prefetching for module PATJetRefSelector/'finalJetsPuppi'
[6] Prefetching for module PATJetUserDataEmbedder/'updatedJetsPuppiWithUserData'
[7] Prefetching for module PATJetUpdater/'updatedJetsPuppi'
[8] Prefetching for module PATJetSelector/'slimmedJetsPuppi'
[9] Prefetching for module PATJetUpdater/'updatedPatJetsTransientCorrectedSlimmedPuppiWithDeepTags'
[10] Prefetching for module BoostedJetONNXJetTagsProducer/'pfParticleNetFromMiniAODAK4PuppiCentralJetTagsSlimmedPuppiWithDeepTags'
[11] Prefetching for module ParticleNetFeatureEvaluator/'pfParticleNetFromMiniAODAK4PuppiCentralTagInfosSlimmedPuppiWithDeepTags'
[12] Prefetching for module PATTauIDEmbedder/'slimmedTaus'
[13] Calling method for module DeepTauId/'deepTau2017v2p1ForMini'
Exception Message:
error while running session: INVALID_ARGUMENT: Incompatible shapes: [0,1,1,64] vs. [154]
[[{{node inner_muon_norm_1/FusedBatchNorm_1/Mul}}]]

3) MC 2023

In DigiPU_2023PU step in hltHpsPFTauDeepTauProducer in RelValTenTau_15_500 with 13_3_0_pre1 (at the moment the first occurrence I found). The error here. The wf on Stats2.

Fatal Exception (Exit code: 8001)
An exception of category 'InvalidRun' occurred while
[0] Processing Event run: 1 lumi: 18 event: 1707 stream: 1
[1] Running path 'HLT_DoubleMediumDeepTauPFTauHPS30_L2NN_eta2p1_OneProng_M5to80_v2'
[2] Calling method for module DeepTauId/'hltHpsPFTauDeepTauProducer'
Exception Message:
error while running session: INVALID_ARGUMENT: Incompatible shapes: [0,1,1,38] vs. [92]
[[{{node inner_hadrons_norm_1/FusedBatchNorm_1/Mul}}]]

CPU

At the moment it appears that in all cases the jobs were running on Intel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz (or on a Gold one), Cascade Lake (see #44333 (comment)).

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @AdrianoDee.

@Dr15Jones, @antoniovilela, @smuzaffar, @makortel, @sextonkennedy, @rappoccio can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[AdrianoDee]

assign hlt

[AdrianoDee]

assign pdmv

[cmsbuild]

New categories assigned: hlt,pdmv

@Martin-Grunewald,@mmusich,@AdrianoDee,@sunilUIET,@miquork you have been requested to review this Pull request/Issue and eventually sign? Thanks

[mmusich]

@cms-sw/tau-pog-l2 FYI

[mmusich]

type tau

[mmusich]

just as an observation this path is not new (first included in the GRun menu in 2022, https://its.cern.ch/jira/browse/CMSHLT-2289)

EDIT but was touched recently in https://its.cern.ch/jira/browse/CMSHLT-3052

[mmusich]

@cms-sw/pdmv-l2

In data reHLT+reRECO RelVals we are observing some failures at HLTDR_2023 step in path HLT_VBF_DoubleMediumDeepTauPFTauHPS20_eta2p1_v7

Please help filling in some information:

• In which release is this happening?
• Is it reproducibile?
• Does it affect all jobs of the relvals?
• Is there a pattern w.r.t. the CPU microarchitecture of the node on which the job lands?

[Martin-Grunewald]

I can't find it in the Dashboard. Since it is labelled HLTDR_2023, and the path in question is not in the Fake* menus, it must be in some 13_X release running the actual 2023 HLT with the 2023 version of that path.

[AdrianoDee]

Quick answers:

• this happened both in 14_0_0_pre3 and 14_0_0 but I'm tracking it back to older releases (coming back as soon as I find the first occurrence);
• it just happens on a fraction of the jobs and the fraction itself is quite random (fluctuates in the order of few percentages of the events failing).

For the reproducibility and the CPU pattern I'll need a moment to check those.

[Martin-Grunewald]

Hmm well, in 14_X, HLTDR_2023 should (now) run the Fake* menus, while the real HLT menus should be within HLTDR_2024.

[mmusich]

in 14_X, HLTDR_2023 should (now) run the Fake* menus, while the real HLT menus should be within HLTDR_2024

Indeed the configuration linked above has
L1REPACK:Full,HLT:@relval2024, but in absence of real 2024 data we're running the 2024 menu on 2023 data.

[AdrianoDee]

I see the same (similar) error

Fatal Exception (Exit code: 8001)
An exception of category 'InvalidRun' occurred while
[0] Processing Event run: 367131 lumi: 122 event: 206577729 stream: 1
[1] Running path 'HLT_DoubleMediumDeepTauPFTauHPS30_L2NN_eta2p1_PFJet60_v6'
[2] Calling method for module DeepTauId/'hltHpsPFTauDeepTauProducer'
Exception Message:
error while running session: INVALID_ARGUMENT: Incompatible shapes: [0,1,1,38] vs. [92]
[[{{node inner_hadrons_norm_1/FusedBatchNorm_1/Mul}}]]

in 13_3_0_pre5 RunDisplacedJet2023C running L1REPACK:Full,HLT:@relval2023.

[mmusich]

HLT_DoubleMediumDeepTauPFTauHPS30_L2NN_eta2p1_PFJet60_v6

This is a different path, so it points to a general problem with DeepTauId (path-aspecific)

[Dr15Jones]

For context, it appears the exception comes from here:

cmssw/PhysicsTools/TensorFlow/src/TensorFlow.cc

Lines 272 to 275 in ff51428

	Status status = session->Run(runOptions, inputs, outputNames, {}, outputs, nullptr, threadPoolOptions);
	if (!status.ok()) {
	throw cms::Exception("InvalidRun") << "error while running session: " << status.ToString();
	}

[makortel]

assign ml

[makortel]

assign reconstruction

[cmsbuild]

New categories assigned: ml,reconstruction

@jfernan2,@mandrenguyen,@valsdav,@wpmccormack you have been requested to review this Pull request/Issue and eventually sign? Thanks

[makortel]

This failure was now seen in Tier0 PromptReco https://cms-talk.web.cern.ch/t/update-t0-skim-config-for-2024-pp-collision/36794/5 .

[mmusich]

urgent

This failure was now seen in Tier0 PromptReco https://cms-talk.web.cern.ch/t/update-t0-skim-config-for-2024-pp-collision/36794/5

I can prepare a PR with guards to avoid the execution of the model with empty inputs, and in parallel investigate more deeply this TF behaviour.

@valsdav, we have established that this issue can affect Prompt Reconstruction and (potentially, when the new nodes for the HLT farm arrive) also online trigger operations. Please prepare PRs with guards to avoid the execution of the model with empty inputs.
Thank you.

Marco (as ORM)

[mmusich]

for record, the proposed fixes are:

• DeepTau - Do not call TF inference with empty grid #44455 (master)
• [backport] DeepTau - Do not call TF inference with empty grid #44456 (14.0.X)

[jfernan2]

+1
solved by #44455

[valsdav]

+ml

Basic guards to solve the empty input problem in DeepTauId are in place, but the reason of the empty grid needs to be investigated with Tau experts.

A more general guard for empty inputs will be added (see #44481)

[AdrianoDee]

+pdmv
(really only the reporter)

[mmusich]

... hlt will sign once the 14.0.X PR is merged and tested in IBs.

[mmusich]

but the reason of the empty grid needs to be investigated with Tau experts.

@cms-sw/reconstruction-l2 this looks like needs a separate issue. Can you open one?

[mmusich]

+hlt

• no issues observed after the 14.0.X PR got merged and tested in IBs.

[cmsbuild]

This issue is fully signed and ready to be closed.

[makortel]

@cmsbuild, please close