HLT crash in run 383631 - 'VertexException' and HcalDigisProducerGPU:hltHcalDigisGPU error #45555
Comments
|
cms-bot internal usage |
|
A new Issue was created by @jalimena. @Dr15Jones, @antoniovilela, @makortel, @mandrenguyen, @rappoccio, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks. cms-bot commands are listed here |
|
For record, also using repacked data with the following script: #!/bin/bash -ex
# CMSSW_14_0_11_MULTIARCHS
hltGetConfiguration run:383631 \
--globaltag 140X_dataRun3_HLT_v3 \
--data \
--no-prescale \
--no-output \
--max-events -1 \
--input '/store/group/tsg/FOG/error_stream_root/run383631/run383631_ls0444_index000423_fu-c2b14-43-01_pid675389.root,/store/group/tsg/FOG/error_stream_root/run383631/run383631_ls0445_index000025_fu-c2b14
-43-01_pid675389.root,/store/group/tsg/FOG/error_stream_root/run383631/run383631_ls0666_index000047_fu-c2b14-43-01_pid675067.root,/store/group/tsg/FOG/error_stream_root/run383631/run383631_ls0445_index00
0002_fu-c2b14-43-01_pid675389.root,/store/group/tsg/FOG/error_stream_root/run383631/run383631_ls0666_index000027_fu-c2b14-43-01_pid675067.root,/store/group/tsg/FOG/error_stream_root/run383631/run383631_l
s0666_index000065_fu-c2b14-43-01_pid675067.root' > hlt.py
cat <<@EOF >> hlt.py
process.options.wantSummary = True
process.options.numberOfThreads = 1
process.options.numberOfStreams = 0
@EOF
cmsRun hlt.py &> hlt.logdoesn't reproduce. |
|
@jalimena where can we find the full stack trace for all threads? |
sorry i have uploaded the complete stack trace from the DAQ shifter in the issue description. Let me know if it looks incomplete. |
|
As the "regular" reproducing script (see #45555 (comment)) doesn't lead to a crash I have explored the option of running filling with junk the memory on the host and device allocators (cf #44923 (comment)). when doing the same in Not sure if that's expected. [1] Click me#!/bin/bash -ex
# CMSSW_14_0_11_MULTIARCHS
hltGetConfiguration run:383631 \
--globaltag 140X_dataRun3_HLT_v3 \
--data \
--no-prescale \
--no-output \
--max-events -1 \
--input '/store/group/tsg/FOG/error_stream_root/run383631/run383631_ls0444_index000423_fu-c2b14-43-01_pid675389.root,/store/group/tsg/FOG/error_stream_root/run383631/run383631_ls0445_index000025_fu-c2b14
-43-01_pid675389.root,/store/group/tsg/FOG/error_stream_root/run383631/run383631_ls0666_index000047_fu-c2b14-43-01_pid675067.root,/store/group/tsg/FOG/error_stream_root/run383631/run383631_ls0445_index00
0002_fu-c2b14-43-01_pid675389.root,/store/group/tsg/FOG/error_stream_root/run383631/run383631_ls0666_index000027_fu-c2b14-43-01_pid675067.root,/store/group/tsg/FOG/error_stream_root/run383631/run383631_l
s0666_index000065_fu-c2b14-43-01_pid675067.root' > hlt.py
cat <<@EOF >> hlt.py
del process.MessageLogger
process.load('FWCore.MessageService.MessageLogger_cfi')
process.MessageLogger.CUDAService = {}
process.MessageLogger.AlpakaService = {}
process.load('HeterogeneousCore.CUDAServices.CUDAService_cfi')
from HeterogeneousCore.AlpakaServices.AlpakaServiceCudaAsync_cfi import AlpakaServiceCudaAsync as _AlpakaServiceCudaAsync
process.AlpakaServiceCudaAsync = _AlpakaServiceCudaAsync.clone(
verbose = True,
hostAllocator = dict(
binGrowth = 2,
minBin = 8, # 256 bytes
maxBin = 30, # 1 GB
maxCachedBytes = 64*1024*1024*1024, # 64 GB
maxCachedFraction = 0.8, # or 80%, whatever is less
fillAllocations = True,
fillAllocationValue = 0xA5,
fillReallocations = True,
fillReallocationValue = 0x69,
fillDeallocations = True,
fillDeallocationValue = 0x5A,
fillCaches = True,
fillCacheValue = 0x96
),
deviceAllocator = dict(
binGrowth = 2,
minBin = 8, # 256 bytes
maxBin = 30, # 1 GB
maxCachedBytes = 8*1024*1024*1024, # 8 GB
maxCachedFraction = 0.8, # or 80%, whatever is less
fillAllocations = True,
fillAllocationValue = 0xA5,
fillReallocations = True,
fillReallocationValue = 0x69,
fillDeallocations = True,
fillDeallocationValue = 0x5A,
fillCaches = True,
fillCacheValue = 0x96
)
)
process.options.wantSummary = True
process.options.numberOfThreads = 1
process.options.numberOfStreams = 0
@EOF
cmsRun hlt.py &> hlt.log |
Thanks. There's a possibly relevant error from the PrimaryVertexProducer, and I guess a question for physics is whether having the BasicSingleVertexState throw on invalid errors is the desired behavior: The detailed stack trace shows that stack trace interpretation is getting complicated by the async GPU calls--the |
|
I didn't notice at first glance: So the plot thickens and looks to go in the direction of the already observed #41914 in which we suspected beamspot issues. |
|
assign hlt, reconstruction, alca |
|
type tracking |
|
@cms-sw/tracking-pog-l2 FYI |
|
New categories assigned: hlt,reconstruction,alca @Martin-Grunewald,@mmusich,@jfernan2,@mandrenguyen,@saumyaphor4252,@perrotta,@consuegs you have been requested to review this Pull request/Issue and eventually sign? Thanks |
cmsbuild
added
reconstruction-pending
hlt-pending
alca-pending
pending-signatures
tracking
and removed
pending-assignment
labels
|
spoon-feeding a null-covariance matrix fake BeamSpot to the HLT with: diff --git a/RecoVertex/BeamSpotProducer/plugins/BeamSpotOnlineProducer.cc b/RecoVertex/BeamSpotProducer/plugins/BeamSpotOnlineProducer.cc
index 83aa832cfa5..56f0c8948b6 100644
--- a/RecoVertex/BeamSpotProducer/plugins/BeamSpotOnlineProducer.cc
+++ b/RecoVertex/BeamSpotProducer/plugins/BeamSpotOnlineProducer.cc
@@ -108,7 +108,7 @@ void BeamSpotOnlineProducer::produce(Event& iEvent, const EventSetup& iSetup) {
edm::LogWarning("BeamSpotFromDB")
<< "Online Beam Spot producer falls back to DB value because the ESProducer returned a fake beamspot ";
}
- fallBackToDB = true;
+ //fallBackToDB = true;
} else {
// translate from BeamSpotObjects to reco::BeamSpot
// in case we need to switch to LHC reference frame
diff --git a/RecoVertex/BeamSpotProducer/plugins/OnlineBeamSpotESProducer.cc b/RecoVertex/BeamSpotProducer/plugins/OnlineBeamSpotESProducer.cc
index 0b8c4233c7c..2c42e2b6d63 100644
--- a/RecoVertex/BeamSpotProducer/plugins/OnlineBeamSpotESProducer.cc
+++ b/RecoVertex/BeamSpotProducer/plugins/OnlineBeamSpotESProducer.cc
@@ -55,13 +55,21 @@ OnlineBeamSpotESProducer::OnlineBeamSpotESProducer(const edm::ParameterSet& p)
fakeBS_.setPosition(0.0001, 0.0001, 0.0001);
fakeBS_.setType(-1);
// Set diagonal covariance, i.e. errors on the parameters
- fakeBS_.setCovariance(0, 0, 5e-10);
- fakeBS_.setCovariance(1, 1, 5e-10);
- fakeBS_.setCovariance(2, 2, 0.002);
- fakeBS_.setCovariance(3, 3, 0.002);
- fakeBS_.setCovariance(4, 4, 5e-11);
- fakeBS_.setCovariance(5, 5, 5e-11);
- fakeBS_.setCovariance(6, 6, 1e-09);
+ fakeBS_.setCovariance(0, 0, 0.);
+ fakeBS_.setCovariance(1, 1, 0.);
+ fakeBS_.setCovariance(2, 2, 0.);
+ fakeBS_.setCovariance(3, 3, 0.);
+ fakeBS_.setCovariance(4, 4, 0.);
+ fakeBS_.setCovariance(5, 5, 0.);
+ fakeBS_.setCovariance(6, 6, 0.);
bsHLTToken_ = cc.consumesFrom<BeamSpotOnlineObjects, BeamSpotOnlineHLTObjectsRcd>();
bsLegacyToken_ = cc.consumesFrom<BeamSpotOnlineObjects, BeamSpotOnlineLegacyObjectsRcd>();
@@ -179,7 +187,7 @@ std::shared_ptr<const BeamSpotObjects> OnlineBeamSpotESProducer::produce(const B
return std::shared_ptr<const BeamSpotObjects>(best, edm::do_nothing_deleter());
}
edm::LogWarning("OnlineBeamSpotESProducer")
- << "None of the Online BeamSpots in the ES is suitable, \n returning a fake one(fallback to PCL).";
+ << "None of the Online BeamSpots in the ES is suitable, \n returning a fake one (fallback to PCL).";
return std::shared_ptr<const BeamSpotObjects>(&fakeBS_, edm::do_nothing_deleter());
}I sort of reproduce the crash: the questions are:
|
there's no such payload in the DB for run 383631:
|
does it look like there was a glitch in the DB access or did the zeroes show up from another part of the logic in the BS producer? |
Possible, but it's not clear from where it would read it then (I would have expected a framework exception).
By looking at the code it's not clear to me from where it could come. |
It could indeed make sense to demote the exception to |
After are more careful look I convinced myself this case has to stay as an exception. Generally we want to stop processing as soon as possible when this situation happens (e.g. in this case to prevent calling Modules in different streams throwing exceptions should be fine, and two independent modules processed by one stream throwing exceptions should also be fine. From the (core) framework perspective the I would add tests for the "Alpaka framework" that an Alpaka module throwing an exception in |
|
Written that, I wonder what the other exception which I'd bet is the cause for std::terminate() here 🤦 . I'll fix that.
|
Just to add some information, indeed the hltd log doesn't seem to mention any fallback to PCL, but the online DQM monitoring under But in any case, as Marco mentioned above, the payload consumed from PCL would have been OK. |
It's not clear why e.g. at least this: cmssw/RecoVertex/BeamSpotProducer/plugins/OnlineBeamSpotESProducer.cc Lines 181 to 182 in 4c86b77 is not in the logs, but for sure this: cmssw/RecoVertex/BeamSpotProducer/plugins/BeamSpotOnlineProducer.cc Lines 108 to 109 in 4c86b77 is manually suppressed in the HLT menu itself via the following snippet: process.MessageLogger = cms.Service( "MessageLogger",
suppressWarning = cms.untracked.vstring( 'hltL3MuonsIOHit',
'hltL3MuonsOIHit',
'hltL3MuonsOIState',
'hltLightPFTracks',
'hltOnlineBeamSpot',
'hltPixelTracks',
'hltPixelTracksForHighMult',
'hltSiPixelClusters',
'hltSiPixelDigis' ),
suppressFwkInfo = cms.untracked.vstring( ),
suppressInfo = cms.untracked.vstring( ),
suppressDebug = cms.untracked.vstring( ),
debugModules = cms.untracked.vstring( ),
cerr = cms.untracked.PSet(
INFO = cms.untracked.PSet( limit = cms.untracked.int32( 0 ) ),
noTimeStamps = cms.untracked.bool( False ),
FwkReport = cms.untracked.PSet(
reportEvery = cms.untracked.int32( 1 ),
limit = cms.untracked.int32( 0 )
),
default = cms.untracked.PSet( limit = cms.untracked.int32( 10000000 ) ),
Root_NoDictionary = cms.untracked.PSet( limit = cms.untracked.int32( 0 ) ),
FwkJob = cms.untracked.PSet( limit = cms.untracked.int32( 0 ) ),
FwkSummary = cms.untracked.PSet(
reportEvery = cms.untracked.int32( 1 ),
limit = cms.untracked.int32( 10000000 )
),
threshold = cms.untracked.string( "INFO" ),
enableStatistics = cms.untracked.bool( False )
),
suppressError = cms.untracked.vstring( 'hltL3TkTracksFromL2IOHit',
'hltL3TkTracksFromL2OIHit',
'hltL3TkTracksFromL2OIState',
'hltOnlineBeamSpot' )
)I propose to lift the suppression at https://its.cern.ch/jira/browse/CMSHLT-3298 |
|
@mmusich one possibility for a missing message is if the MessageLogger gets too many request for messages at the same time it will drop some of them in order to keep up with the requests. |
that's a possibility, but I am not very convinced as there are other warnings from the same event that crashed (just before the crash). |
@cms-sw/db-l2 @PonIlya is it possible to have an audit of the P5 frontier squid at the time of the crash ( 25-Jul-2024 08:49:19 )? |
|
|
@PonIlya , thank you for the reply.
to be perfectly honest I am not sure, but there are indications that somehow the connection to the DB "glitched" during this period of time. As a DB expert you are perhaps in a better position to judge if there was any anomaly during that time. |
|
@mmusich Unfortunately, the logs are no longer available for audit. Let me know if a similar issue arises again so that I can respond more quickly. The logs are deleted once they reach a certain size. For example, as of today, only logs from August 7th are available. |
In run 383631 (pp collisions, release
CMSSW_14_0_11_MULTIARCHS), we got this error:Complete stack trace should be here:
old_hlt_run383631_pid675389.log
I tried to reproduce it, but no success:
Possibly related to #41914
@cms-sw/hlt-l2 FYI
@cms-sw/heterogeneous-l2 FYI
Juliette, for FOG
The text was updated successfully, but these errors were encountered: