Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ARM] Crashes in Cling #43802

Open
makortel opened this issue Jan 26, 2024 · 22 comments
Open

[ARM] Crashes in Cling #43802

makortel opened this issue Jan 26, 2024 · 22 comments
Labels
core-pending pending-signatures root

Comments

@makortel
Copy link
Contributor

This issue is to continue to track the crashes in Cling we're seeing on ARM that were first reported in #43577 (comment). Repeating the stack traces:

On el8_aarch64_gcc12 CMSSW_14_0_X_2023-12-13-2300 two workflows crashed in a way that looks possibly related

11024.0 step 3

Other TBB threads are in tbb::detail::r1::futex_wait()

Thread 1 (Thread 0x4000269bd080 (LWP 63750) "cmsRun"):
#3  0x000040002bf5ce8c in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00004001acff001c in ?? ()
#6  0x0000400039d3c590 in cling::Interpreter::RunFunction(clang::FunctionDecl const*, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCling.so
#7  0x0000400039d3c590 in cling::Interpreter::RunFunction(clang::FunctionDecl const*, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCling.so
#8  0x0000400039d3cc3c in cling::Interpreter::EvaluateInternal(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, cling::CompilationOptions, cling::Value*, cling::Transaction**, unsigned long) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCling.so
#9  0x0000400039e18844 in cling::MetaSema::actOnxCommand(llvm::StringRef, llvm::StringRef, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCling.so
#10 0x0000400039e25134 in cling::MetaParser::isXCommand(cling::MetaSema::ActionResult&, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCling.so
#11 0x0000400039e26ff4 in cling::MetaParser::isCommand(cling::MetaSema::ActionResult&, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCling.so
#12 0x0000400039e122ec in cling::MetaProcessor::process(llvm::StringRef, cling::Interpreter::CompilationResult&, cling::Value*, bool) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCling.so
#13 0x0000400039c602dc in HandleInterpreterException(cling::MetaProcessor*, char const*, cling::Interpreter::CompilationResult&, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCling.so
#14 0x0000400039c72d40 in TCling::ProcessLine(char const*, TInterpreter::EErrorCode*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCling.so
#15 0x0000400039c732b4 in TCling::ProcessLineSynch(char const*, TInterpreter::EErrorCode*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCling.so
#16 0x0000400025f398a8 in TApplication::ExecuteFile(char const*, int*, bool) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCore.so
#17 0x0000400039c71dd8 in TCling::ExecuteMacro(char const*, TInterpreter::EErrorCode*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCling.so
#18 0x0000400025f8ec5c in TROOT::Macro(char const*, int*, bool) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCore.so
#19 0x0000400025f72a90 in TPluginManager::LoadHandlerMacros(char const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCore.so
#20 0x0000400025f730ec in TPluginManager::LoadHandlersFromPluginDirs(char const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCore.so
#21 0x0000400025f74164 in TPluginManager::FindHandler(char const*, char const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCore.so
#22 0x000040002579899c in ROOT::Math::Factory::CreateMinimizer(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libMathCore.so
#23 0x000040002579a270 in ROOT::Fit::FitConfig::CreateMinimizer() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libMathCore.so
#24 0x00004000257b1dc0 in ROOT::Fit::Fitter::DoInitMinimizer() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libMathCore.so
#25 0x00004000257b827c in bool ROOT::Fit::Fitter::DoMinimization<ROOT::Fit::PoissonLikelihoodFCN<ROOT::Math::IBaseFunctionMultiDimTempl<double>, ROOT::Math::IParametricFunctionMultiDimTempl<double> > >(std::unique_ptr<ROOT::Fit::PoissonLikelihoodFCN<ROOT::Math::IBaseFunctionMultiDimTempl<double>, ROOT::Math::IParametricFunctionMultiDimTempl<double> >, std::default_delete<ROOT::Fit::PoissonLikelihoodFCN<ROOT::Math::IBaseFunctionMultiDimTempl<double>, ROOT::Math::IParametricFunctionMultiDimTempl<double> > > >, ROOT::Math::IBaseFunctionMultiDimTempl<double> const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libMathCore.so
#26 0x00004000257b5590 in ROOT::Fit::Fitter::DoBinnedLikelihoodFit(bool, ROOT::EExecutionPolicy const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libMathCore.so
#27 0x000040002c60a3a4 in TFitResultPtr HFit::Fit<TH1>(TH1*, TF1*, Foption_t&, ROOT::Math::MinimizerOptions const&, char const*, ROOT::Fit::DataRange&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libHist.so
#28 0x000040002c5fff14 in ROOT::Fit::FitObject(TH1*, TF1*, Foption_t&, ROOT::Math::MinimizerOptions const&, char const*, ROOT::Fit::DataRange&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libHist.so
#29 0x000040002c6c97e0 in TH1::Fit(TF1*, char const*, char const*, double, double) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libHist.so
#30 0x000040006b49055c in PVFitter::runFitter() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/lib/el8_aarch64_gcc12/libRecoVertexBeamSpotProducer.so
#31 0x000040006b4871d8 in BeamFitter::runPVandTrkFitter() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/lib/el8_aarch64_gcc12/libRecoVertexBeamSpotProducer.so
#32 0x00004000c5aa0e94 in AlcaBeamMonitor::globalEndLuminosityBlock(edm::LuminosityBlock const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/lib/el8_aarch64_gcc12/pluginAlcaBeamMonitor.so
#33 0x00004000c5aad9fc in virtual thunk to edm::one::impl::LuminosityBlockCacheHolder<edm::one::EDProducerBase, alcabeammonitor::BeamSpotInfo>::doEndLuminosityBlock_(edm::LuminosityBlock const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/lib/el8_aarch64_gcc12/pluginAlcaBeamMonitor.so
#34 0x0000400024dadcc4 in edm::one::EDProducerBase::doEndLuminosityBlock(edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so

Current Modules:
Module: AlcaBeamMonitor:AlcaBeamMonitor (crashed)
Module: none
Module: none
Module: none

https://cmssdt.cern.ch/SDT/cgi-bin/logreader/el8_aarch64_gcc12/CMSSW_14_0_X_2023-12-13-2300/pyRelValMatrixLogs/run/11024.0_TTbar_13+2018PU/step3_TTbar_13+2018PU.log#/

11025.0 step 3

Other TBB threads are in tbb::detail::r1::futex_wait

Thread 3 (Thread 0x400053389250 (LWP 2727347) "cmsRun"):
#3  0x000040000fdece8c in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00004001aafe001c in ?? ()
#6  0x000040001dcac590 in cling::Interpreter::RunFunction(clang::FunctionDecl const*, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCling.so
#7  0x000040001dcac590 in cling::Interpreter::RunFunction(clang::FunctionDecl const*, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCling.so
#8  0x000040001dcacc3c in cling::Interpreter::EvaluateInternal(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, cling::CompilationOptions, cling::Value*, cling::Transaction**, unsigned long) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCling.so
#9  0x000040001dd88844 in cling::MetaSema::actOnxCommand(llvm::StringRef, llvm::StringRef, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCling.so
#10 0x000040001dd95134 in cling::MetaParser::isXCommand(cling::MetaSema::ActionResult&, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCling.so
#11 0x000040001dd96ff4 in cling::MetaParser::isCommand(cling::MetaSema::ActionResult&, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCling.so
#12 0x000040001dd822ec in cling::MetaProcessor::process(llvm::StringRef, cling::Interpreter::CompilationResult&, cling::Value*, bool) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCling.so
#13 0x000040001dbd02dc in HandleInterpreterException(cling::MetaProcessor*, char const*, cling::Interpreter::CompilationResult&, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCling.so
#14 0x000040001dbe2d40 in TCling::ProcessLine(char const*, TInterpreter::EErrorCode*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCling.so
#15 0x000040001dbe32b4 in TCling::ProcessLineSynch(char const*, TInterpreter::EErrorCode*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCling.so
#16 0x0000400009e898a8 in TApplication::ExecuteFile(char const*, int*, bool) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCore.so
#17 0x000040001dbe1dd8 in TCling::ExecuteMacro(char const*, TInterpreter::EErrorCode*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCling.so
#18 0x0000400009edec5c in TROOT::Macro(char const*, int*, bool) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCore.so
#19 0x0000400009ec2a90 in TPluginManager::LoadHandlerMacros(char const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCore.so
#20 0x0000400009ec30ec in TPluginManager::LoadHandlersFromPluginDirs(char const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCore.so
#21 0x0000400009ec4164 in TPluginManager::FindHandler(char const*, char const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCore.so
#22 0x00004000096e899c in ROOT::Math::Factory::CreateMinimizer(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libMathCore.so
#23 0x00004000096ea270 in ROOT::Fit::FitConfig::CreateMinimizer() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libMathCore.so
#24 0x0000400009701dc0 in ROOT::Fit::Fitter::DoInitMinimizer() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libMathCore.so
#25 0x000040000970827c in bool ROOT::Fit::Fitter::DoMinimization<ROOT::Fit::PoissonLikelihoodFCN<ROOT::Math::IBaseFunctionMultiDimTempl<double>, ROOT::Math::IParametricFunctionMultiDimTempl<double> > >(std::unique_ptr<ROOT::Fit::PoissonLikelihoodFCN<ROOT::Math::IBaseFunctionMultiDimTempl<double>, ROOT::Math::IParametricFunctionMultiDimTempl<double> >, std::default_delete<ROOT::Fit::PoissonLikelihoodFCN<ROOT::Math::IBaseFunctionMultiDimTempl<double>, ROOT::Math::IParametricFunctionMultiDimTempl<double> > > >, ROOT::Math::IBaseFunctionMultiDimTempl<double> const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libMathCore.so
#26 0x0000400009705590 in ROOT::Fit::Fitter::DoBinnedLikelihoodFit(bool, ROOT::EExecutionPolicy const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libMathCore.so
#27 0x000040001054a3a4 in TFitResultPtr HFit::Fit<TH1>(TH1*, TF1*, Foption_t&, ROOT::Math::MinimizerOptions const&, char const*, ROOT::Fit::DataRange&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libHist.so
#28 0x000040001053ff14 in ROOT::Fit::FitObject(TH1*, TF1*, Foption_t&, ROOT::Math::MinimizerOptions const&, char const*, ROOT::Fit::DataRange&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libHist.so
#29 0x00004000106097e0 in TH1::Fit(TF1*, char const*, char const*, double, double) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libHist.so
#30 0x000040004f3c055c in PVFitter::runFitter() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/lib/el8_aarch64_gcc12/libRecoVertexBeamSpotProducer.so
#31 0x000040004f3b71d8 in BeamFitter::runPVandTrkFitter() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/lib/el8_aarch64_gcc12/libRecoVertexBeamSpotProducer.so
#32 0x00004000ab9f0e94 in AlcaBeamMonitor::globalEndLuminosityBlock(edm::LuminosityBlock const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/lib/el8_aarch64_gcc12/pluginAlcaBeamMonitor.so
#33 0x00004000ab9fd9fc in virtual thunk to edm::one::impl::LuminosityBlockCacheHolder<edm::one::EDProducerBase, alcabeammonitor::BeamSpotInfo>::doEndLuminosityBlock_(edm::LuminosityBlock const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/lib/el8_aarch64_gcc12/pluginAlcaBeamMonitor.so
#34 0x0000400008cfdcc4 in edm::one::EDProducerBase::doEndLuminosityBlock(edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so

Module: AlcaBeamMonitor:AlcaBeamMonitor (crashed)
Module: none
Module: none
Module: none

https://cmssdt.cern.ch/SDT/cgi-bin/logreader/el8_aarch64_gcc12/CMSSW_14_0_X_2023-12-13-2300/pyRelValMatrixLogs/run/11025.0_ZEE_13+2018PU/step3_ZEE_13+2018PU.log#/
@makortel makortel mentioned this issue Jan 26, 2024
Closed
@cmsbuild
Copy link
Contributor

cmsbuild commented Jan 26, 2024 •
edited
Loading

cms-bot internal usage

@cmsbuild
Copy link
Contributor

A new Issue was created by @makortel Matti Kortelainen.

@rappoccio, @sextonkennedy, @Dr15Jones, @smuzaffar, @antoniovilela, @makortel can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

@makortel
Copy link
Contributor Author

Some more from CMSSW_14_0_X_2024-01-25-2300 on el9_aarch64_gcc12

140.58 step 3

Thread 1 (Thread 0x40000fd3b360 (LWP 1572278) "cmsRun"):
#3  0x000040001543cf3c in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x000040018cff001c in ?? ()
#6  0x000040002052ad94 in cling::IncrementalExecutor::executeWrapper(llvm::StringRef, cling::Value*) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCling.so
#7  0x000040002052ad94 in cling::IncrementalExecutor::executeWrapper(llvm::StringRef, cling::Value*) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCling.so
#8  0x0000000000000000 in ?? ()

Current Modules:
Module: AlcaBeamMonitor:AlcaBeamMonitor (crashed)
Module: none
Module: none
Module: none

https://cmssdt.cern.ch/SDT/cgi-bin/logreader/el9_aarch64_gcc12/CMSSW_14_0_X_2024-01-25-2300/pyRelValMatrixLogs/run/140.58_RunHI2018/step3_RunHI2018.log#/

11024.0 step 3

Thread 5 (Thread 0x400064e19180 (LWP 1567392) "cmsRun"):
#3  0x000040001f24cf3c in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00004001b2fe001c in ?? ()
#6  0x000040002d09bea0 in cling::Interpreter::RunFunction(clang::FunctionDecl const*, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCling.so
#7  0x000040002d09bea0 in cling::Interpreter::RunFunction(clang::FunctionDecl const*, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCling.so
#8  0x000040002d09c54c in cling::Interpreter::EvaluateInternal(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, cling::CompilationOptions, cling::Value*, cling::Transaction**, unsigned long) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCling.so
#9  0x000040002d16cb44 in cling::MetaSema::actOnxCommand(llvm::StringRef, llvm::StringRef, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCling.so
#10 0x000040002d179434 in cling::MetaParser::isXCommand(cling::MetaSema::ActionResult&, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCling.so
#11 0x000040002d17b2f4 in cling::MetaParser::isCommand(cling::MetaSema::ActionResult&, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCling.so
#12 0x000040002d1665ec in cling::MetaProcessor::process(llvm::StringRef, cling::Interpreter::CompilationResult&, cling::Value*, bool) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCling.so
#13 0x000040002cfbfcfc in HandleInterpreterException(cling::MetaProcessor*, char const*, cling::Interpreter::CompilationResult&, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCling.so
#14 0x000040002cfd2790 in TCling::ProcessLine(char const*, TInterpreter::EErrorCode*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCling.so
#15 0x000040002cfd2d04 in TCling::ProcessLineSynch(char const*, TInterpreter::EErrorCode*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCling.so
#16 0x00004000191e9888 in TApplication::ExecuteFile(char const*, int*, bool) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCore.so
#17 0x000040002cfd1828 in TCling::ExecuteMacro(char const*, TInterpreter::EErrorCode*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCling.so
#18 0x000040001923eb9c in TROOT::Macro(char const*, int*, bool) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCore.so
#19 0x0000400019222980 in TPluginManager::LoadHandlerMacros(char const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCore.so
#20 0x0000400019222fdc in TPluginManager::LoadHandlersFromPluginDirs(char const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCore.so
#21 0x0000400019224064 in TPluginManager::FindHandler(char const*, char const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCore.so
#22 0x0000400018a4835c in ROOT::Math::Factory::CreateMinimizer(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libMathCore.so
#23 0x0000400018a49c30 in ROOT::Fit::FitConfig::CreateMinimizer() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libMathCore.so
#24 0x0000400018a617a0 in ROOT::Fit::Fitter::DoInitMinimizer() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libMathCore.so
#25 0x0000400018a67840 in bool ROOT::Fit::Fitter::DoMinimization<ROOT::Fit::PoissonLikelihoodFCN<ROOT::Math::IBaseFunctionMultiDimTempl<double>, ROOT::Math::IParametricFunctionMultiDimTempl<double> > >(std::unique_ptr<ROOT::Fit::PoissonLikelihoodFCN<ROOT::Math::IBaseFunctionMultiDimTempl<double>, ROOT::Math::IParametricFunctionMultiDimTempl<double> >, std::default_delete<ROOT::Fit::PoissonLikelihoodFCN<ROOT::Math::IBaseFunctionMultiDimTempl<double>, ROOT::Math::IParametricFunctionMultiDimTempl<double> > > >, ROOT::Math::IBaseFunctionMultiDimTempl<double> const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libMathCore.so
#26 0x0000400018a64ddc in ROOT::Fit::Fitter::DoBinnedLikelihoodFit(bool, ROOT::EExecutionPolicy const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libMathCore.so
#27 0x000040001f8fa26c in TFitResultPtr HFit::Fit<TH1>(TH1*, TF1*, Foption_t&, ROOT::Math::MinimizerOptions const&, char const*, ROOT::Fit::DataRange&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libHist.so
#28 0x000040001f8f00e4 in ROOT::Fit::FitObject(TH1*, TF1*, Foption_t&, ROOT::Math::MinimizerOptions const&, char const*, ROOT::Fit::DataRange&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libHist.so
#29 0x000040001f9b9630 in TH1::Fit(TF1*, char const*, char const*, double, double) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libHist.so
#30 0x000040005e95058c in PVFitter::runFitter() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/libRecoVertexBeamSpotProducer.so
#31 0x000040005e9471ac in BeamFitter::runPVandTrkFitter() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/libRecoVertexBeamSpotProducer.so
#32 0x00004000bb0a0ee4 in AlcaBeamMonitor::globalEndLuminosityBlock(edm::LuminosityBlock const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/pluginAlcaBeamMonitor.so
#33 0x00004000bb0ad9cc in virtual thunk to edm::one::impl::LuminosityBlockCacheHolder<edm::one::EDProducerBase, alcabeammonitor::BeamSpotInfo>::doEndLuminosityBlock_(edm::LuminosityBlock const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/pluginAlcaBeamMonitor.so
#34 0x000040001805e6b4 in edm::one::EDProducerBase::doEndLuminosityBlock(edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#35 0x00004000180456f0 in edm::WorkerT<edm::one::EDProducerBase>::implDoEnd(edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/libFWCoreFramework.so

Current Modules:
Module: AlcaBeamMonitor:AlcaBeamMonitor (crashed)
Module: none
Module: none
Module: none

https://cmssdt.cern.ch/SDT/cgi-bin/logreader/el9_aarch64_gcc12/CMSSW_14_0_X_2024-01-25-2300/pyRelValMatrixLogs/run/11024.0_TTbar_13+2018PU/step3_TTbar_13+2018PU.log#/

25034.114 step 3

Thread 5 (Thread 0x400059a09180 (LWP 489105) "cmsRun"):
#3  0x0000400012adcf3c in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x000040028bfe001c in ?? ()
#6  0x000040002096bea0 in cling::Interpreter::RunFunction(clang::FunctionDecl const*, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCling.so
#7  0x000040002096bea0 in cling::Interpreter::RunFunction(clang::FunctionDecl const*, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCling.so
#8  0x000040002096c54c in cling::Interpreter::EvaluateInternal(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, cling::CompilationOptions, cling::Value*, cling::Transaction**, unsigned long) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCling.so
#9  0x0000400020a3cb44 in cling::MetaSema::actOnxCommand(llvm::StringRef, llvm::StringRef, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCling.so
#10 0x0000400020a49434 in cling::MetaParser::isXCommand(cling::MetaSema::ActionResult&, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCling.so
#11 0x0000400020a4b2f4 in cling::MetaParser::isCommand(cling::MetaSema::ActionResult&, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCling.so
#12 0x0000400020a365ec in cling::MetaProcessor::process(llvm::StringRef, cling::Interpreter::CompilationResult&, cling::Value*, bool) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCling.so
#13 0x000040002088fcfc in HandleInterpreterException(cling::MetaProcessor*, char const*, cling::Interpreter::CompilationResult&, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCling.so
#14 0x00004000208a2790 in TCling::ProcessLine(char const*, TInterpreter::EErrorCode*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCling.so
#15 0x00004000208a2d04 in TCling::ProcessLineSynch(char const*, TInterpreter::EErrorCode*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCling.so
#16 0x000040000ca39888 in TApplication::ExecuteFile(char const*, int*, bool) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCore.so
#17 0x00004000208a1828 in TCling::ExecuteMacro(char const*, TInterpreter::EErrorCode*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCling.so
#18 0x000040000ca8eb9c in TROOT::Macro(char const*, int*, bool) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCore.so
#19 0x000040000ca72980 in TPluginManager::LoadHandlerMacros(char const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCore.so
#20 0x000040000ca72fdc in TPluginManager::LoadHandlersFromPluginDirs(char const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCore.so
#21 0x000040000ca74064 in TPluginManager::FindHandler(char const*, char const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCore.so
#22 0x000040000c29835c in ROOT::Math::Factory::CreateMinimizer(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libMathCore.so
#23 0x000040000c299c30 in ROOT::Fit::FitConfig::CreateMinimizer() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libMathCore.so
#24 0x000040000c2b17a0 in ROOT::Fit::Fitter::DoInitMinimizer() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libMathCore.so
#25 0x000040000c2b7490 in bool ROOT::Fit::Fitter::DoMinimization<ROOT::Fit::Chi2FCN<ROOT::Math::IBaseFunctionMultiDimTempl<double>, ROOT::Math::IParametricFunctionMultiDimTempl<double> > >(std::unique_ptr<ROOT::Fit::Chi2FCN<ROOT::Math::IBaseFunctionMultiDimTempl<double>, ROOT::Math::IParametricFunctionMultiDimTempl<double> >, std::default_delete<ROOT::Fit::Chi2FCN<ROOT::Math::IBaseFunctionMultiDimTempl<double>, ROOT::Math::IParametricFunctionMultiDimTempl<double> > > >, ROOT::Math::IBaseFunctionMultiDimTempl<double> const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libMathCore.so
#26 0x000040000c2b448c in ROOT::Fit::Fitter::DoLeastSquareFit(ROOT::EExecutionPolicy const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libMathCore.so
#27 0x00004000131c9138 in TFitResultPtr HFit::Fit<TGraph>(TGraph*, TF1*, Foption_t&, ROOT::Math::MinimizerOptions const&, char const*, ROOT::Fit::DataRange&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libHist.so
#28 0x00004000131c006c in ROOT::Fit::FitObject(TGraph*, TF1*, Foption_t&, ROOT::Math::MinimizerOptions const&, char const*, ROOT::Fit::DataRange&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libHist.so
#29 0x000040001326d0d4 in TGraph::Fit(TF1*, char const*, char const*, double, double) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libHist.so
#30 0x0000400090e150dc in CSCSimHitMatcher::fitHitsInChamber(unsigned int, float&, float&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/libValidationMuonHits.so
#31 0x00004000959796e8 in CSCStubResolutionValidation::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/libValidationMuonCSCDigis.so
#32 0x000040000b8b760c in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#33 0x000040000b89c310 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#34 0x000040000b82ac14 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#35 0x000040000b82b434 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#36 0x000040000bdc50cc in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/libFWCoreConcurrency.so

Thread 4 (Thread 0x400056da9180 (LWP 489104) "cmsRun"):
#3  0x0000400012ad924c in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x0000400039a02100 in std::_Sp_counted_ptr_inplace<SiPixelRecHit, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/pluginRecoTrackerMeasurementDetPlugins.so
#6  0x00004000bdd01358 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/pluginRecoTrackerCkfPatternPlugins.so
#7  0x00004000bdd013a0 in TrajectoryMeasurement::~TrajectoryMeasurement() [clone .lto_priv.0] () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/pluginRecoTrackerCkfPatternPlugins.so
#8  0x00004000bdd014b0 in cmsutils::bqueue<TrajectoryMeasurement>::~bqueue() [clone .lto_priv.0] () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/pluginRecoTrackerCkfPatternPlugins.so
#9  0x00004000bdd0172c in std::vector<TempTrajectory, std::allocator<TempTrajectory> >::_M_erase(__gnu_cxx::__normal_iterator<TempTrajectory*, std::vector<TempTrajectory, std::allocator<TempTrajectory> > >, __gnu_cxx::__normal_iterator<TempTrajectory*, std::vector<TempTrajectory, std::allocator<TempTrajectory> > >) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/pluginRecoTrackerCkfPatternPlugins.so
#10 0x00004000bdd039e0 in GroupedCkfTrajectoryBuilder::groupedLimitedCandidates(TrajectorySeed const&, TempTrajectory const&, TrajectoryFilter const*, Propagator const*, bool, std::vector<TempTrajectory, std::allocator<TempTrajectory> >&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/pluginRecoTrackerCkfPatternPlugins.so
#11 0x00004000bdd0573c in GroupedCkfTrajectoryBuilder::rebuildTrajectories(TempTrajectory const&, TrajectorySeed const&, std::vector<Trajectory, std::allocator<Trajectory> >&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/pluginRecoTrackerCkfPatternPlugins.so
#12 0x00004000bc7b9d18 in cms::CkfTrackCandidateMakerBase::produceBase(edm::Event&, edm::EventSetup const&)::{lambda(unsigned long)#1}::operator()(unsigned long) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/libRecoTrackerCkfPattern.so
#13 0x00004000bc7ba9fc in cms::CkfTrackCandidateMakerBase::produceBase(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/libRecoTrackerCkfPattern.so
#14 0x000040000b8b760c in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#15 0x000040000b89c310 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#16 0x000040000b82ac14 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#17 0x000040000b82b434 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#18 0x000040000bdc50cc in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/libFWCoreConcurrency.so

Thread 3 (Thread 0x400056399180 (LWP 489103) "cmsRun"):
#3  0x0000400012ad924c in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00004000e153a20c in std::_Hashtable<std::pair<unsigned int, EncodedEventId>, std::pair<std::pair<unsigned int, EncodedEventId> const, edm::Ref<std::vector<TrackingParticle, std::allocator<TrackingParticle> >, TrackingParticle, edm::refhelper::FindUsingAdvance<std::vector<TrackingParticle, std::allocator<TrackingParticle> >, TrackingParticle> > >, std::allocator<std::pair<std::pair<unsigned int, EncodedEventId> const, edm::Ref<std::vector<TrackingParticle, std::allocator<TrackingParticle> >, TrackingParticle, edm::refhelper::FindUsingAdvance<std::vector<TrackingParticle, std::allocator<TrackingParticle> >, TrackingParticle> > > >, std::__detail::_Select1st, std::equal_to<std::pair<unsigned int, EncodedEventId> >, UniqueSimTrackIdHash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_find_before_node(unsigned long, std::pair<unsigned int, EncodedEventId> const&, unsigned long) const [clone .constprop.0] () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/pluginSimTrackerTrackerHitAssociationPlugins.so
#6  0x00004000e153a310 in std::_Hashtable<std::pair<unsigned int, EncodedEventId>, std::pair<std::pair<unsigned int, EncodedEventId> const, edm::Ref<std::vector<TrackingParticle, std::allocator<TrackingParticle> >, TrackingParticle, edm::refhelper::FindUsingAdvance<std::vector<TrackingParticle, std::allocator<TrackingParticle> >, TrackingParticle> > >, std::allocator<std::pair<std::pair<unsigned int, EncodedEventId> const, edm::Ref<std::vector<TrackingParticle, std::allocator<TrackingParticle> >, TrackingParticle, edm::refhelper::FindUsingAdvance<std::vector<TrackingParticle, std::allocator<TrackingParticle> >, TrackingParticle> > > >, std::__detail::_Select1st, std::equal_to<std::pair<unsigned int, EncodedEventId> >, UniqueSimTrackIdHash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::find(std::pair<unsigned int, EncodedEventId> const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/pluginSimTrackerTrackerHitAssociationPlugins.so
#7  0x00004000e1540310 in ClusterTPAssociationProducer::produce(edm::StreamID, edm::Event&, edm::EventSetup const&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/pluginSimTrackerTrackerHitAssociationPlugins.so
#8  0x000040000b89e638 in edm::global::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#9  0x000040000b896540 in edm::WorkerT<edm::global::EDProducerBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#10 0x000040000b82ac14 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#11 0x000040000b82b434 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#12 0x000040000bdc50cc in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/libFWCoreConcurrency.so

Thread 1 (Thread 0x40000d4cb360 (LWP 485190) "cmsRun"):
#3  0x0000400012ad924c in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00004000e1a1b44c in std::__detail::_Map_base<DetId, std::pair<DetId const, std::vector<std::pair<int, float>, std::allocator<std::pair<int, float> > > >, std::allocator<std::pair<DetId const, std::vector<std::pair<int, float>, std::allocator<std::pair<int, float> > > > >, std::__detail::_Select1st, std::equal_to<DetId>, std::hash<DetId>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true>, true>::operator[](DetId const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/pluginSimCalorimetryHGCalAssociatorProducersAuto.so
#6  0x00004000e1a1c89c in TSToSimTSHitLCAssociatorByEnergyScoreImpl::makeConnections(edm::Handle<std::vector<ticl::Trackster, std::allocator<ticl::Trackster> > > const&, edm::Handle<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> > > const&, edm::Handle<std::vector<SimCluster, std::allocator<SimCluster> > > const&, edm::Handle<std::vector<CaloParticle, std::allocator<CaloParticle> > > const&, edm::Handle<std::vector<ticl::Trackster, std::allocator<ticl::Trackster> > > const&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/pluginSimCalorimetryHGCalAssociatorProducersAuto.so
#7  0x00004000e1a20b18 in TSToSimTSHitLCAssociatorByEnergyScoreImpl::associateSimToReco(edm::Handle<std::vector<ticl::Trackster, std::allocator<ticl::Trackster> > > const&, edm::Handle<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> > > const&, edm::Handle<std::vector<SimCluster, std::allocator<SimCluster> > > const&, edm::Handle<std::vector<CaloParticle, std::allocator<CaloParticle> > > const&, edm::Handle<std::vector<ticl::Trackster, std::allocator<ticl::Trackster> > > const&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/pluginSimCalorimetryHGCalAssociatorProducersAuto.so
#8  0x00004000e1a2046c in TSToSimTSHitLCAssociatorEDProducer::produce(edm::StreamID, edm::Event&, edm::EventSetup const&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/pluginSimCalorimetryHGCalAssociatorProducersAuto.so
#9  0x000040000b89e638 in edm::global::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#10 0x000040000b896540 in edm::WorkerT<edm::global::EDProducerBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#11 0x000040000b82ac14 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#12 0x000040000b82b434 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/libFWCoreFramework.so

Current Modules:
Module: CSCDigiValidation:cscDigiValidation (crashed)
Module: ClusterTPAssociationProducer:tpClusterProducer
Module: TSToSimTSHitLCAssociatorEDProducer:tracksterSimTracksterAssociationPRbyCLUE3D
Module: CkfTrackCandidateMaker:initialStepTrackCandidates

https://cmssdt.cern.ch/SDT/cgi-bin/logreader/el9_aarch64_gcc12/CMSSW_14_0_X_2024-01-25-2300/pyRelValMatrixLogs/run/25034.114_TTbar_14TeV+2026D98PU_OTInefficiency10PC/step3_TTbar_14TeV+2026D98PU_OTInefficiency10PC.log#/

@makortel
Copy link
Contributor Author

assign core

@cmsbuild
Copy link
Contributor

New categories assigned: core

@Dr15Jones,@makortel,@smuzaffar you have been requested to review this Pull request/Issue and eventually sign? Thanks

@makortel
Copy link
Contributor Author

type root

@cmsbuild cmsbuild added the root label Jan 26, 2024
@makortel
Copy link
Contributor Author

@pcanal @vgvassilev

@makortel
Copy link
Contributor Author

Responding to @vgvassilev's #43577 (comment)

Can we run with valgrind to make sure there are no obvious memory errors?

I ran valgrind for step3 of workflow 11025.0 in CMSSW_14_0_X_2024-01-11-2300. Below is a summary of different kinds of warnings

==25280== Conditional jump or move depends on uninitialised value(s)
==25280==    at 0x46AF40E1: std::_Function_handler<void (long, long), tensorflow::functor::OneHot<Eigen::ThreadPoolDevice, float, int>::Compute(Eigen::ThreadPoolDevice const&, Eigen::TensorMap<Eigen::Tensor<int const, 2, 1, long>, 16, Eigen::MakePointer> const&, Eigen::TensorMap<Eigen::TensorFixedSize<float const, Eigen::Sizes<>, 1, long>, 16, Eigen::MakePointer> const&, Eigen::TensorMap<Eigen::TensorFixedSize<float const, Eigen::Sizes<>, 1, long>, 16, Eigen::MakePointer> const&, Eigen::TensorMap<Eigen::Tensor<float, 3, 1, long>, 16, Eigen::MakePointer>*)::{lambda(long, long)#1}>::_M_invoke(std::_Any_data const&, long&&, long&&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/tensorflow/2.12.0-150b7aa8b33fac0cef686f0aee2109fb/lib/libtensorflow_cc.so.2.12.0)
==25280==    by 0x46AF7033: Eigen::ThreadPoolDevice::parallelFor(long, Eigen::TensorOpCost const&, std::function<long (long)>, std::function<void (long, long)>) const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/tensorflow/2.12.0-150b7aa8b33fac0cef686f0aee2109fb/lib/libtensorflow_cc.so.2.12.0)
==25280==    by 0x46B1963E: tensorflow::OneHotOp<Eigen::ThreadPoolDevice, float, int>::Compute(tensorflow::OpKernelContext*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/tensorflow/2.12.0-150b7aa8b33fac0cef686f0aee2109fb/lib/libtensorflow_cc.so.2.12.0)
==25280==    by 0x5657CF6A: tensorflow::ThreadPoolDevice::Compute(tensorflow::OpKernel*, tensorflow::OpKernelContext*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/tensorflow/2.12.0-150b7aa8b33fac0cef686f0aee2109fb/lib/libtensorflow_framework.so.2.12.0)
==25280==    by 0x5666AEF0: tensorflow::(anonymous namespace)::ExecutorState<tensorflow::SimplePropagatorState>::ProcessInline(tensorflow::SimplePropagatorState::TaggedNodeReadyQueue*, long) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/tensorflow/2.12.0-150b7aa8b33fac0cef686f0aee2109fb/lib/libtensorflow_framework.so.2.12.0)
==25280==    by 0x5666BFD5: tensorflow::(anonymous namespace)::ExecutorState<tensorflow::SimplePropagatorState>::Process(tensorflow::SimplePropagatorState::TaggedNode, long) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/tensorflow/2.12.0-150b7aa8b33fac0cef686f0aee2109fb/lib/libtensorflow_framework.so.2.12.0)
==25280==    by 0x4FDCC99B: tsl::thread::ThreadPool::Schedule(std::function<void ()>) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/tensorflow/2.12.0-150b7aa8b33fac0cef686f0aee2109fb/lib/libtensorflow_cc.so.2.12.0)
==25280==    by 0x4FF0AD77: std::_Function_handler<void (std::function<void ()>), tensorflow::DirectSession::RunInternal(long, tensorflow::RunOptions const&, tensorflow::CallFrameInterface*, tensorflow::DirectSession::ExecutorsAndKeys*, tensorflow::RunMetadata*, tsl::thread::ThreadPoolOptions const&)::{lambda(std::function<void ()>)#6}>::_M_invoke(std::_Any_data const&, std::function<void ()>&&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/tensorflow/2.12.0-150b7aa8b33fac0cef686f0aee2109fb/lib/libtensorflow_cc.so.2.12.0)
==25280==    by 0x56650923: void tensorflow::(anonymous namespace)::ExecutorState<tensorflow::SimplePropagatorState>::RunTask<std::_Bind<void (tensorflow::(anonymous namespace)::ExecutorState<tensorflow::SimplePropagatorState>::*(tensorflow::(anonymous namespace)::ExecutorState<tensorflow::SimplePropagatorState>*, tensorflow::SimplePropagatorState::TaggedNode, long))(tensorflow::SimplePropagatorState::TaggedNode, long)> >(std::_Bind<void (tensorflow::(anonymous namespace)::ExecutorState<tensorflow::SimplePropagatorState>::*(tensorflow::(anonymous namespace)::ExecutorState<tensorflow::SimplePropagatorState>*, tensorflow::SimplePropagatorState::TaggedNode, long))(tensorflow::SimplePropagatorState::TaggedNode, long)>&&, int) [clone .constprop.0] (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/tensorflow/2.12.0-150b7aa8b33fac0cef686f0aee2109fb/lib/libtensorflow_framework.so.2.12.0)
==25280==    by 0x5665636D: tensorflow::(anonymous namespace)::ExecutorState<tensorflow::SimplePropagatorState>::ScheduleReady(absl::lts_20220623::InlinedVector<tensorflow::SimplePropagatorState::TaggedNode, 8ul, std::allocator<tensorflow::SimplePropagatorState::TaggedNode> >*, tensorflow::SimplePropagatorState::TaggedNodeReadyQueue*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/tensorflow/2.12.0-150b7aa8b33fac0cef686f0aee2109fb/lib/libtensorflow_framework.so.2.12.0)
==25280==    by 0x5665FFA4: tensorflow::(anonymous namespace)::ExecutorState<tensorflow::SimplePropagatorState>::NodeDone(tsl::Status const&, absl::lts_20220623::InlinedVector<tensorflow::SimplePropagatorState::TaggedNode, 8ul, std::allocator<tensorflow::SimplePropagatorState::TaggedNode> >*, tensorflow::NodeExecStatsInterface*, tensorflow::SimplePropagatorState::TaggedNodeReadyQueue*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/tensorflow/2.12.0-150b7aa8b33fac0cef686f0aee2109fb/lib/libtensorflow_framework.so.2.12.0)
==25280==    by 0x5666A20B: tensorflow::(anonymous namespace)::ExecutorState<tensorflow::SimplePropagatorState>::ProcessInline(tensorflow::SimplePropagatorState::TaggedNodeReadyQueue*, long) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/tensorflow/2.12.0-150b7aa8b33fac0cef686f0aee2109fb/lib/libtensorflow_framework.so.2.12.0)
==25280==    by 0x5666BFD5: tensorflow::(anonymous namespace)::ExecutorState<tensorflow::SimplePropagatorState>::Process(tensorflow::SimplePropagatorState::TaggedNode, long) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/tensorflow/2.12.0-150b7aa8b33fac0cef686f0aee2109fb/lib/libtensorflow_framework.so.2.12.0)
==25280==    by 0x4FDCC99B: tsl::thread::ThreadPool::Schedule(std::function<void ()>) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/tensorflow/2.12.0-150b7aa8b33fac0cef686f0aee2109fb/lib/libtensorflow_cc.so.2.12.0)
==25280==    by 0x4FF0AD77: std::_Function_handler<void (std::function<void ()>), tensorflow::DirectSession::RunInternal(long, tensorflow::RunOptions const&, tensorflow::CallFrameInterface*, tensorflow::DirectSession::ExecutorsAndKeys*, tensorflow::RunMetadata*, tsl::thread::ThreadPoolOptions const&)::{lambda(std::function<void ()>)#6}>::_M_invoke(std::_Any_data const&, std::function<void ()>&&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/tensorflow/2.12.0-150b7aa8b33fac0cef686f0aee2109fb/lib/libtensorflow_cc.so.2.12.0)
==25280==    by 0x56656270: tensorflow::(anonymous namespace)::ExecutorState<tensorflow::SimplePropagatorState>::ScheduleReady(absl::lts_20220623::InlinedVector<tensorflow::SimplePropagatorState::TaggedNode, 8ul, std::allocator<tensorflow::SimplePropagatorState::TaggedNode> >*, tensorflow::SimplePropagatorState::TaggedNodeReadyQueue*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/tensorflow/2.12.0-150b7aa8b33fac0cef686f0aee2109fb/lib/libtensorflow_framework.so.2.12.0)
==25280==    by 0x5665AF07: tensorflow::(anonymous namespace)::ExecutorImpl::RunAsync(tensorflow::Executor::Args const&, std::function<void (tsl::Status const&)>) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/tensorflow/2.12.0-150b7aa8b33fac0cef686f0aee2109fb/lib/libtensorflow_framework.so.2.12.0)
==25280==    by 0x4FF1D619: tensorflow::DirectSession::RunInternal(long, tensorflow::RunOptions const&, tensorflow::CallFrameInterface*, tensorflow::DirectSession::ExecutorsAndKeys*, tensorflow::RunMetadata*, tsl::thread::ThreadPoolOptions const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/tensorflow/2.12.0-150b7aa8b33fac0cef686f0aee2109fb/lib/libtensorflow_cc.so.2.12.0)
==25280==    by 0x4FF2034D: tensorflow::DirectSession::Run(tensorflow::RunOptions const&, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor> > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<tensorflow::Tensor, std::allocator<tensorflow::Tensor> >*, tensorflow::RunMetadata*, tsl::thread::ThreadPoolOptions const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/tensorflow/2.12.0-150b7aa8b33fac0cef686f0aee2109fb/lib/libtensorflow_cc.so.2.12.0)
==25280==    by 0x4342D4D4: tensorflow::run(tensorflow::Session*, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor> > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<tensorflow::Tensor, std::allocator<tensorflow::Tensor> >*, tsl::thread::ThreadPoolOptions const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libPhysicsToolsTensorFlow.so)
==25280==  Uninitialised value was created by a heap allocation
==25280==    at 0x4036601: posix_memalign (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/valgrind/3.22.0-e989c977f8949fc48f1bc1206f1f87c8/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==25280==    by 0x4FDF4371: tsl::port::AlignedMalloc(unsigned long, int) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/tensorflow/2.12.0-150b7aa8b33fac0cef686f0aee2109fb/lib/libtensorflow_cc.so.2.12.0)
==25280==    by 0x56413E89: tsl::(anonymous namespace)::CPUAllocator::AllocateRaw(unsigned long, unsigned long) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/tensorflow/2.12.0-150b7aa8b33fac0cef686f0aee2109fb/lib/libtensorflow_framework.so.2.12.0)
==25280==    by 0x564BABB4: tensorflow::Tensor::Tensor(tsl::Allocator*, tensorflow::DataType, tensorflow::TensorShape const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/tensorflow/2.12.0-150b7aa8b33fac0cef686f0aee2109fb/lib/libtensorflow_framework.so.2.12.0)
==25280==    by 0x5F0BC1E5: (anonymous namespace)::TfDnn::operator()(std::vector<reco::Track, std::allocator<reco::Track> > const&, reco::BeamSpot const&, std::vector<reco::Vertex, std::allocator<reco::Vertex> > const&) const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/pluginRecoTrackerFinalTrackSelectorsPlugins.so)
==25280==    by 0x5F0BE105: TrackMVAClassifier<(anonymous namespace)::TfDnn, void>::computeMVA(std::vector<reco::Track, std::allocator<reco::Track> > const&, reco::BeamSpot const&, std::vector<reco::Vertex, std::allocator<reco::Vertex> > const&, std::vector<std::pair<float, bool>, std::allocator<std::pair<float, bool> > >&) const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/pluginRecoTrackerFinalTrackSelectorsPlugins.so)
==25280==    by 0x5F13CAF7: TrackMVAClassifierBase::produce(edm::Event&, edm::EventSetup const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libRecoTrackerFinalTrackSelectors.so)
==25280==    by 0x4C6EBA0: edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4C5347D: edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4BDE3E8: std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4BDE983: edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x420BF27: tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreConcurrency.so)
==25280==    by 0x6401290: UnknownInlinedFun (task_dispatcher.h:322)
==25280==    by 0x6401290: UnknownInlinedFun (task_dispatcher.h:458)
==25280==    by 0x6401290: tbb::detail::r1::task_dispatcher::execute_and_wait(tbb::detail::d1::task*, tbb::detail::d1::wait_context&, tbb::detail::d1::task_group_context&) (task_dispatcher.cpp:168)
==25280==    by 0x4B6178A: edm::FinalWaitingTask::wait() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4B715F9: edm::EventProcessor::processRuns() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4B6B690: edm::EventProcessor::runToCompletion() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4074EE: tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/bin/slc7_amd64_gcc12/cmsRun)
==25280==    by 0x63ED9BC: tbb::detail::r1::task_arena_impl::execute(tbb::detail::d1::task_arena_base&, tbb::detail::d1::delegate_base&) (arena.cpp:688)
==25280==    by 0x408ED1: main::{lambda()#1}::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/bin/slc7_amd64_gcc12/cmsRun)
==25280==    by 0x40518B: main (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/bin/slc7_amd64_gcc12/cmsRun)

Similar report repeats for Use of uninitialised value of size 8

==25280== Conditional jump or move depends on uninitialised value(s)
==25280==    at 0x138E3C3A: TrackerTopology::print[abi:cxx11](DetId) const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libDataFormatsTrackerCommon.so)
==25280==    by 0x7D3348D2: MuonAssociatorByHitsHelper::getMatchedIds(std::vector<std::unique_ptr<std::pair<unsigned int, std::vector<std::pair<unsigned int, EncodedEventId>, std::allocator<std::pair<unsigned int, EncodedEventId> > > >, std::default_delete<std::pair<unsigned int, std::vector<std::pair<unsigned int, EncodedEventId>, std::allocator<std::pair<unsigned int, EncodedEventId> > > > > >, std::allocator<std::unique_ptr<std::pair<unsigned int, std::vector<std::pair<unsigned int, EncodedEventId>, std::allocator<std::pair<unsigned int, EncodedEventId> > > >, std::default_delete<std::pair<unsigned int, std::vector<std::pair<unsigned int, EncodedEventId>, std::allocator<std::pair<unsigned int, EncodedEventId> > > > > > > >&, std::vector<std::unique_ptr<std::pair<unsigned int, std::vector<std::pair<unsigned int, EncodedEventId>, std::allocator<std::pair<unsigned int, EncodedEventId> > > >, std::default_delete<std::pair<unsigned int, std::vector<std::pair<unsigned int, EncodedEventId>, std::allocator<std::pair<unsigned int, EncodedEventId> > > > > >, std::allocator<std::unique_ptr<std::pair<unsigned int, std::vector<std::pair<unsigned int, EncodedEventId>, std::allocator<std::pair<unsigned int, EncodedEventId> > > >, std::default_delete<std::pair<unsigned int, std::vector<std::pair<unsigned int, EncodedEventId>, std::allocator<std::pair<unsigned int, EncodedEventId> > > > > > > >&, std::vector<std::unique_ptr<std::pair<unsigned int, std::vector<std::pair<unsigned int, EncodedEventId>, std::allocator<std::pair<unsigned int, EncodedEventId> > > >, std::default_delete<std::pair<unsigned int, std::vector<std::pair<unsigned int, EncodedEventId>, std::allocator<std::pair<unsigned int, EncodedEventId> > > > > >, std::allocator<std::unique_ptr<std::pair<unsigned int, std::vector<std::pair<unsigned int, EncodedEventId>, std::allocator<std::pair<unsigned int, EncodedEventId> > > >, std::default_delete<std::pair<unsigned int, std::vector<std::pair<unsigned int, EncodedEventId>, std::allocator<std::pair<unsigned int, EncodedEventId> > > > > > > >&, std::vector<std::unique_ptr<std::pair<unsigned int, std::vector<std::pair<unsigned int, EncodedEventId>, std::allocator<std::pair<unsigned int, EncodedEventId> > > >, std::default_delete<std::pair<unsigned int, std::vector<std::pair<unsigned int, EncodedEventId>, std::allocator<std::pair<unsigned int, EncodedEventId> > > > > >, std::allocator<std::unique_ptr<std::pair<unsigned int, std::vector<std::pair<unsigned int, EncodedEventId>, std::allocator<std::pair<unsigned int, EncodedEventId> > > >, std::default_delete<std::pair<unsigned int, std::vector<std::pair<unsigned int, EncodedEventId>, std::allocator<std::pair<unsigned int, EncodedEventId> > > > > > > >&, int&, int&, int&, int&, int&, int&, int&, int&, int&, int&, int&, int&, int&, int&, int&, int&, int&, int&, int&, int&, __gnu_cxx::__normal_iterator<TrackingRecHit* const*, std::vector<TrackingRecHit*, std::allocator<TrackingRecHit*> > >, __gnu_cxx::__normal_iterator<TrackingRecHit* const*, std::vector<TrackingRecHit*, std::allocator<TrackingRecHit*> > >, TrackerHitAssociator const*, DTHitAssociator const&, CSCHitAssociator const&, RPCHitAssociator const&, GEMHitAssociator const&, bool, TrackerTopology const*) const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libSimMuonMCTruth.so)
==25280==    by 0x7D33A4A7: MuonAssociatorByHitsHelper::associateRecoToSimIndices(std::vector<std::pair<__gnu_cxx::__normal_iterator<TrackingRecHit* const*, std::vector<TrackingRecHit*, std::allocator<TrackingRecHit*> > >, __gnu_cxx::__normal_iterator<TrackingRecHit* const*, std::vector<TrackingRecHit*, std::allocator<TrackingRecHit*> > > >, std::allocator<std::pair<__gnu_cxx::__normal_iterator<TrackingRecHit* const*, std::vector<TrackingRecHit*, std::allocator<TrackingRecHit*> > >, __gnu_cxx::__normal_iterator<TrackingRecHit* const*, std::vector<TrackingRecHit*, std::allocator<TrackingRecHit*> > > > > > const&, edm::RefVector<std::vector<TrackingParticle, std::allocator<TrackingParticle> >, TrackingParticle, edm::refhelper::FindUsingAdvance<std::vector<TrackingParticle, std::allocator<TrackingParticle> >, TrackingParticle> > const&, MuonAssociatorByHitsHelper::Resources const&) const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libSimMuonMCTruth.so)
==25280==    by 0x7D32FB9B: MuonAssociatorByHits::associateRecoToSim(edm::RefToBaseVector<reco::Track> const&, edm::RefVector<std::vector<TrackingParticle, std::allocator<TrackingParticle> >, TrackingParticle, edm::refhelper::FindUsingAdvance<std::vector<TrackingParticle, std::allocator<TrackingParticle> >, TrackingParticle> > const&, edm::Event const*, edm::EventSetup const*) const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libSimMuonMCTruth.so)
==25280==    by 0x7D4874C8: MuonAssociatorEDProducer::produce(edm::Event&, edm::EventSetup const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/pluginSimMuonMCTruthPlugins.so)
==25280==    by 0x4C6EBA0: edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4C5347D: edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4BDE3E8: std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4BDE983: edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x420BF27: tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreConcurrency.so)
==25280==    by 0x6401290: UnknownInlinedFun (task_dispatcher.h:322)
==25280==    by 0x6401290: UnknownInlinedFun (task_dispatcher.h:458)
==25280==    by 0x6401290: tbb::detail::r1::task_dispatcher::execute_and_wait(tbb::detail::d1::task*, tbb::detail::d1::wait_context&, tbb::detail::d1::task_group_context&) (task_dispatcher.cpp:168)
==25280==    by 0x4B6178A: edm::FinalWaitingTask::wait() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4B715F9: edm::EventProcessor::processRuns() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4B6B690: edm::EventProcessor::runToCompletion() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4074EE: tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/bin/slc7_amd64_gcc12/cmsRun)
==25280==    by 0x63ED9BC: tbb::detail::r1::task_arena_impl::execute(tbb::detail::d1::task_arena_base&, tbb::detail::d1::delegate_base&) (arena.cpp:688)
==25280==    by 0x408ED1: main::{lambda()#1}::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/bin/slc7_amd64_gcc12/cmsRun)
==25280==    by 0x40518B: main (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/bin/slc7_amd64_gcc12/cmsRun)
==25280==  Uninitialised value was created by a stack allocation
==25280==    at 0x5F1DAC06: TrackerTopologyEP::produce(TrackerTopologyRcd const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/pluginGeometryTrackerNumberingBuilderPlugins.so)

==25280== Conditional jump or move depends on uninitialised value(s)
==25280==    at 0x643878B: ??? (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/xz/5.2.5-6f3f49b07db84e10c9be594a1176c114/lib/liblzma.so.5.2.5)
==25280==    by 0x6437922: ??? (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/xz/5.2.5-6f3f49b07db84e10c9be594a1176c114/lib/liblzma.so.5.2.5)
==25280==    by 0x643BE40: ??? (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/xz/5.2.5-6f3f49b07db84e10c9be594a1176c114/lib/liblzma.so.5.2.5)
==25280==    by 0x64396D4: ??? (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/xz/5.2.5-6f3f49b07db84e10c9be594a1176c114/lib/liblzma.so.5.2.5)
==25280==    by 0x6441B80: ??? (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/xz/5.2.5-6f3f49b07db84e10c9be594a1176c114/lib/liblzma.so.5.2.5)
==25280==    by 0x6436963: ??? (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/xz/5.2.5-6f3f49b07db84e10c9be594a1176c114/lib/liblzma.so.5.2.5)
==25280==    by 0x642E14B: ??? (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/xz/5.2.5-6f3f49b07db84e10c9be594a1176c114/lib/liblzma.so.5.2.5)
==25280==    by 0x642FB46: ??? (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/xz/5.2.5-6f3f49b07db84e10c9be594a1176c114/lib/liblzma.so.5.2.5)
==25280==    by 0x642AB55: lzma_code (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/xz/5.2.5-6f3f49b07db84e10c9be594a1176c114/lib/liblzma.so.5.2.5)
==25280==    by 0x5BA1763: R__zipLZMA (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libCore.so)
==25280==    by 0x4F795FC: TBasket::WriteBuffer() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x4F89237: std::_Function_handler<void (), ROOT::Internal::TBranchIMTHelper::Run<TBranch::WriteBasketImpl(TBasket*, int, ROOT::Internal::TBranchIMTHelper*)::{lambda()#1}>(TBranch::WriteBasketImpl(TBasket*, int, ROOT::Internal::TBranchIMTHelper*)::{lambda()#1} const&)::{lambda()#1}>::_M_invoke(std::_Any_data const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x763660D: tbb::detail::d1::function_task<std::function<void ()> >::execute(tbb::detail::d1::execution_data&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libImt.so)
==25280==    by 0x6401290: UnknownInlinedFun (task_dispatcher.h:322)
==25280==    by 0x6401290: UnknownInlinedFun (task_dispatcher.h:458)
==25280==    by 0x6401290: tbb::detail::r1::task_dispatcher::execute_and_wait(tbb::detail::d1::task*, tbb::detail::d1::wait_context&, tbb::detail::d1::task_group_context&) (task_dispatcher.cpp:168)
==25280==    by 0x7636398: ROOT::Experimental::TTaskGroup::Wait() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libImt.so)
==25280==    by 0x5002D35: TTree::Fill() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x8D5712C7: tbb::detail::d1::task_arena_function<edm::RootOutputTree::fillTree()::{lambda()#1}, void>::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libIOPoolOutput.so)
==25280==    by 0x63EE29D: UnknownInlinedFun (arena.cpp:757)
==25280==    by 0x63EE29D: UnknownInlinedFun (_template_helpers.h:230)
==25280==    by 0x63EE29D: tbb::detail::r1::isolate_within_arena(tbb::detail::d1::delegate_base&, long) (arena.cpp:758)
==25280==    by 0x8D571DFF: edm::RootOutputTree::fillTree() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libIOPoolOutput.so)
==25280==    by 0x8D56FC58: edm::RootOutputFile::fillBranches(edm::BranchType const&, edm::OccurrenceForOutput const&, unsigned int, std::vector<edm::StoredProductProvenance, std::allocator<edm::StoredProductProvenance> >*, edm::ProductProvenanceRetriever const*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libIOPoolOutput.so)
==25280==  Uninitialised value was created by a stack allocation
==25280==    at 0x988789B0: pat::PATElectronProducer::produce(edm::Event&, edm::EventSetup const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/pluginPhysicsToolsPatAlgos_plugins.so)

Repeats for Use of uninitialised value of size 8

==25280== Conditional jump or move depends on uninitialised value(s)
==25280==    at 0x65C74B52: .LTHUNK17.lto_priv.0 (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/pluginDPGAnalysisSkims.so)
==25280==    by 0x65C751A1: virtual thunk to edm::global::impl::LuminosityBlockCacheHolder<edm::global::EDFilterBase, leef::LumiErrors>::doEndLuminosityBlock_(edm::LuminosityBlock const&, edm::EventSetup const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/pluginDPGAnalysisSkims.so)
==25280==    by 0x4C55DBA: edm::global::EDFilterBase::doEndLuminosityBlock(edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4C4D46F: edm::WorkerT<edm::global::EDFilterBase>::implDoEnd(edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4B9F12E: std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3>::Context const*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4B8D9DA: edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3> >::execute() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4B5188D: tbb::detail::d1::function_task<edm::WaitingTaskHolder::doneWaiting(std::__exception_ptr::exception_ptr)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x6401290: UnknownInlinedFun (task_dispatcher.h:322)
==25280==    by 0x6401290: UnknownInlinedFun (task_dispatcher.h:458)
==25280==    by 0x6401290: tbb::detail::r1::task_dispatcher::execute_and_wait(tbb::detail::d1::task*, tbb::detail::d1::wait_context&, tbb::detail::d1::task_group_context&) (task_dispatcher.cpp:168)
==25280==    by 0x4B6178A: edm::FinalWaitingTask::wait() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4B715F9: edm::EventProcessor::processRuns() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4B6B690: edm::EventProcessor::runToCompletion() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4074EE: tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/bin/slc7_amd64_gcc12/cmsRun)
==25280==    by 0x63ED9BC: tbb::detail::r1::task_arena_impl::execute(tbb::detail::d1::task_arena_base&, tbb::detail::d1::delegate_base&) (arena.cpp:688)
==25280==    by 0x408ED1: main::{lambda()#1}::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/bin/slc7_amd64_gcc12/cmsRun)
==25280==    by 0x40518B: main (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/bin/slc7_amd64_gcc12/cmsRun)
==25280==  Uninitialised value was created by a heap allocation
==25280==    at 0x402EEE1: operator new(unsigned long) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/valgrind/3.22.0-e989c977f8949fc48f1bc1206f1f87c8/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==25280==    by 0x65C79BD0: edm::WorkerMaker<LogErrorEventFilter>::makeModule(edm::ParameterSet const&) const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/pluginDPGAnalysisSkims.so)
==25280==    by 0x4C49B67: edm::Maker::makeModule(edm::MakeModuleParams const&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4BC08C7: edm::Factory::makeModule(edm::MakeModuleParams const&, edm::ModuleTypeResolverMaker const*, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4BCF58F: edm::ModuleRegistry::getModule(edm::MakeModuleParams const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4C4A33F: edm::WorkerRegistry::getWorker(edm::WorkerParams const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4C4A701: edm::WorkerManager::getWorker(edm::ParameterSet&, edm::ProductRegistry&, edm::PreallocationConfiguration const*, std::shared_ptr<edm::ProcessConfiguration const>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4C257FD: edm::(anonymous namespace)::getWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, edm::ParameterSet&, edm::WorkerManager&, edm::ProductRegistry&, edm::PreallocationConfiguration const*, std::shared_ptr<edm::ProcessConfiguration const>) [clone .lto_priv.0] (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4C2FA11: edm::StreamSchedule::fillWorkers(edm::ParameterSet&, edm::ProductRegistry&, edm::PreallocationConfiguration const*, std::shared_ptr<edm::ProcessConfiguration const>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, std::vector<edm::WorkerInPath, std::allocator<edm::WorkerInPath> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, edm::ConditionalTaskHelper const&, std::unordered_set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4C322A3: edm::StreamSchedule::fillTrigPath(edm::ParameterSet&, edm::ProductRegistry&, edm::PreallocationConfiguration const*, std::shared_ptr<edm::ProcessConfiguration const>, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<edm::HLTGlobalStatus>, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, edm::ConditionalTaskHelper const&, std::unordered_set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4C285A6: edm::StreamSchedule::StreamSchedule(std::shared_ptr<edm::TriggerResultInserter>, std::vector<edm::propagate_const<std::shared_ptr<edm::PathStatusInserter> >, std::allocator<edm::propagate_const<std::shared_ptr<edm::PathStatusInserter> > > >&, std::vector<edm::propagate_const<std::shared_ptr<edm::EndPathStatusInserter> >, std::allocator<edm::propagate_const<std::shared_ptr<edm::EndPathStatusInserter> > > >&, std::shared_ptr<edm::ModuleRegistry>, edm::ParameterSet&, edm::service::TriggerNamesService const&, edm::PreallocationConfiguration const&, edm::ProductRegistry&, edm::ExceptionToActionTable const&, std::shared_ptr<edm::ActivityRegistry>, std::shared_ptr<edm::ProcessConfiguration const>, edm::StreamID, edm::ProcessContext const*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4C0EAAF: edm::Schedule::Schedule(edm::ParameterSet&, edm::service::TriggerNamesService const&, edm::ProductRegistry&, edm::ExceptionToActionTable const&, std::shared_ptr<edm::ActivityRegistry>, std::shared_ptr<edm::ProcessConfiguration const>, edm::PreallocationConfiguration const&, edm::ProcessContext const*, edm::ModuleTypeResolverMaker const*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4C22D2C: edm::ScheduleItems::initModules(edm::ParameterSet&, edm::service::TriggerNamesService const&, edm::PreallocationConfiguration const&, edm::ProcessContext const*, edm::ModuleTypeResolverMaker const*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4BA0B95: tbb::detail::d1::function_task<edm::EventProcessor::init(std::shared_ptr<edm::ProcessDesc>&, edm::ServiceToken const&, edm::serviceregistry::ServiceLegacy)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) [clone .lto_priv.0] (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x6401290: UnknownInlinedFun (task_dispatcher.h:322)
==25280==    by 0x6401290: UnknownInlinedFun (task_dispatcher.h:458)
==25280==    by 0x6401290: tbb::detail::r1::task_dispatcher::execute_and_wait(tbb::detail::d1::task*, tbb::detail::d1::wait_context&, tbb::detail::d1::task_group_context&) (task_dispatcher.cpp:168)
==25280==    by 0x4B682B1: edm::EventProcessor::init(std::shared_ptr<edm::ProcessDesc>&, edm::ServiceToken const&, edm::serviceregistry::ServiceLegacy) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4B6AB8B: edm::EventProcessor::EventProcessor(std::shared_ptr<edm::ProcessDesc>, edm::ServiceToken const&, edm::serviceregistry::ServiceLegacy) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4073AE: tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/bin/slc7_amd64_gcc12/cmsRun)
==25280==    by 0x63ED9BC: tbb::detail::r1::task_arena_impl::execute(tbb::detail::d1::task_arena_base&, tbb::detail::d1::delegate_base&) (arena.cpp:688)
==25280==    by 0x408ED1: main::{lambda()#1}::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/bin/slc7_amd64_gcc12/cmsRun)

@makortel
Copy link
Contributor Author

More from the same valgrind run

==25280== Conditional jump or move depends on uninitialised value(s)
==25280==    at 0x7D9DDCE: ZSTD_HcFindBestMatch_noDict_4 (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/zstd/1.5.2-0530c990dd9ca27a8d966ab201ed2dfe/lib/libzstd.so.1.5.2)
==25280==    by 0x7DBAE18: ZSTD_compressBlock_lazy2 (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/zstd/1.5.2-0530c990dd9ca27a8d966ab201ed2dfe/lib/libzstd.so.1.5.2)
==25280==    by 0x7D754A2: ZSTD_compressBlock_internal (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/zstd/1.5.2-0530c990dd9ca27a8d966ab201ed2dfe/lib/libzstd.so.1.5.2)
==25280==    by 0x7D7705E: ZSTD_compress_frameChunk (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/zstd/1.5.2-0530c990dd9ca27a8d966ab201ed2dfe/lib/libzstd.so.1.5.2)
==25280==    by 0x7D79F65: ZSTD_compressEnd (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/zstd/1.5.2-0530c990dd9ca27a8d966ab201ed2dfe/lib/libzstd.so.1.5.2)
==25280==    by 0x7D7DE3F: ZSTD_compressCCtx (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/zstd/1.5.2-0530c990dd9ca27a8d966ab201ed2dfe/lib/libzstd.so.1.5.2)
==25280==    by 0x5BA1C14: R__zipZSTD (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libCore.so)
==25280==    by 0x4F795FC: TBasket::WriteBuffer() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x4F8800E: TBranch::WriteBasketImpl(TBasket*, int, ROOT::Internal::TBranchIMTHelper*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x4F88C15: TBranch::FlushBaskets() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x4F88C5B: TBranch::FlushBaskets() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x4FF76F1: TTree::FlushBasketsImpl() const::{lambda()#1}::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x7639D53: tbb::detail::d1::start_for<tbb::detail::d1::blocked_range<unsigned int>, tbb::detail::d1::parallel_for_body_wrapper<std::function<void (unsigned int)>, unsigned int>, tbb::detail::d1::auto_partitioner const>::execute(tbb::detail::d1::execution_data&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libImt.so)
==25280==    by 0x6401290: UnknownInlinedFun (task_dispatcher.h:322)
==25280==    by 0x6401290: UnknownInlinedFun (task_dispatcher.h:458)
==25280==    by 0x6401290: tbb::detail::r1::task_dispatcher::execute_and_wait(tbb::detail::d1::task*, tbb::detail::d1::wait_context&, tbb::detail::d1::task_group_context&) (task_dispatcher.cpp:168)
==25280==    by 0x7638BB2: tbb::detail::d1::task_arena_function<ROOT::TThreadExecutor::ParallelFor(unsigned int, unsigned int, unsigned int, std::function<void (unsigned int)> const&)::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libImt.so)
==25280==    by 0x63EE29D: UnknownInlinedFun (arena.cpp:757)
==25280==    by 0x63EE29D: UnknownInlinedFun (_template_helpers.h:230)
==25280==    by 0x63EE29D: tbb::detail::r1::isolate_within_arena(tbb::detail::d1::delegate_base&, long) (arena.cpp:758)
==25280==    by 0x7637492: tbb::detail::d1::task_arena_function<ROOT::TThreadExecutor::ParallelFor(unsigned int, unsigned int, unsigned int, std::function<void (unsigned int)> const&)::{lambda()#1}, void>::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libImt.so)
==25280==    by 0x63ED9BC: tbb::detail::r1::task_arena_impl::execute(tbb::detail::d1::task_arena_base&, tbb::detail::d1::delegate_base&) (arena.cpp:688)
==25280==    by 0x7639198: ROOT::TThreadExecutor::ParallelFor(unsigned int, unsigned int, unsigned int, std::function<void (unsigned int)> const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libImt.so)
==25280==    by 0x4FF8A8E: TTree::FlushBasketsImpl() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==  Uninitialised value was created by a stack allocation
==25280==    at 0x7D48F8B3: MuonSimClassifier::convertAndPush(TrackingParticle const&, std::vector<reco::GenParticle, std::allocator<reco::GenParticle> >&, edm::Ref<std::vector<TrackingParticle, std::allocator<TrackingParticle> >, TrackingParticle, edm::refhelper::FindUsingAdvance<std::vector<TrackingParticle, std::allocator<TrackingParticle> >, TrackingParticle> > const&, edm::Handle<std::vector<reco::GenParticle, std::allocator<reco::GenParticle> > > const&) const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/pluginSimMuonMCTruthPlugins.so)

==25280== Conditional jump or move depends on uninitialised value(s)
==25280==    at 0x554C093: TFree::GetBestFree(TList*, int) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libRIO.so)
==25280==    by 0x558B4E3: TKey::Create(int, TFile*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libRIO.so)
==25280==    by 0x4F7997D: TBasket::WriteBuffer() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x4F8800E: TBranch::WriteBasketImpl(TBasket*, int, ROOT::Internal::TBranchIMTHelper*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x4F88C15: TBranch::FlushBaskets() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x4F88C5B: TBranch::FlushBaskets() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x4FF76F1: TTree::FlushBasketsImpl() const::{lambda()#1}::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x7639D53: tbb::detail::d1::start_for<tbb::detail::d1::blocked_range<unsigned int>, tbb::detail::d1::parallel_for_body_wrapper<std::function<void (unsigned int)>, unsigned int>, tbb::detail::d1::auto_partitioner const>::execute(tbb::detail::d1::execution_data&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libImt.so)
==25280==    by 0x6401290: UnknownInlinedFun (task_dispatcher.h:322)
==25280==    by 0x6401290: UnknownInlinedFun (task_dispatcher.h:458)
==25280==    by 0x6401290: tbb::detail::r1::task_dispatcher::execute_and_wait(tbb::detail::d1::task*, tbb::detail::d1::wait_context&, tbb::detail::d1::task_group_context&) (task_dispatcher.cpp:168)
==25280==    by 0x7638BB2: tbb::detail::d1::task_arena_function<ROOT::TThreadExecutor::ParallelFor(unsigned int, unsigned int, unsigned int, std::function<void (unsigned int)> const&)::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libImt.so)
==25280==    by 0x63EE29D: UnknownInlinedFun (arena.cpp:757)
==25280==    by 0x63EE29D: UnknownInlinedFun (_template_helpers.h:230)
==25280==    by 0x63EE29D: tbb::detail::r1::isolate_within_arena(tbb::detail::d1::delegate_base&, long) (arena.cpp:758)
==25280==    by 0x7637492: tbb::detail::d1::task_arena_function<ROOT::TThreadExecutor::ParallelFor(unsigned int, unsigned int, unsigned int, std::function<void (unsigned int)> const&)::{lambda()#1}, void>::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libImt.so)
==25280==    by 0x63ED9BC: tbb::detail::r1::task_arena_impl::execute(tbb::detail::d1::task_arena_base&, tbb::detail::d1::delegate_base&) (arena.cpp:688)
==25280==    by 0x7639198: ROOT::TThreadExecutor::ParallelFor(unsigned int, unsigned int, unsigned int, std::function<void (unsigned int)> const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libImt.so)
==25280==    by 0x4FF8A8E: TTree::FlushBasketsImpl() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x4FF98EA: TTree::AutoSave(char const*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x8D56F2C8: edm::RootOutputFile::finishEndFile() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libIOPoolOutput.so)
==25280==    by 0x8D55AA64: edm::PoolOutputModule::finishEndFile() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libIOPoolOutput.so)
==25280==    by 0x8D55ABB2: edm::PoolOutputModule::reallyCloseFile() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libIOPoolOutput.so)
==25280==    by 0x4C105C4: edm::Schedule::closeOutputFiles() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==  Uninitialised value was created by a heap allocation
==25280==    at 0x402EEE1: operator new(unsigned long) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/valgrind/3.22.0-e989c977f8949fc48f1bc1206f1f87c8/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==25280==    by 0x4F88113: TBranch::WriteBasketImpl(TBasket*, int, ROOT::Internal::TBranchIMTHelper*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x4F889C9: TBranch::FillImpl(ROOT::Internal::TBranchIMTHelper*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x4F92C5C: TBranchElement::FillImpl(ROOT::Internal::TBranchIMTHelper*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x50030D6: TTree::Fill() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x8D569F70: edm::RootOutputFile::writeParentageRegistry() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libIOPoolOutput.so)
==25280==    by 0x8D55AB7E: edm::PoolOutputModule::reallyCloseFile() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libIOPoolOutput.so)
==25280==    by 0x4C105C4: edm::Schedule::closeOutputFiles() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4B6658C: edm::EventProcessor::closeOutputFiles() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4B6B779: edm::EventProcessor::runToCompletion() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4074EE: tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/bin/slc7_amd64_gcc12/cmsRun)
==25280==    by 0x63ED9BC: tbb::detail::r1::task_arena_impl::execute(tbb::detail::d1::task_arena_base&, tbb::detail::d1::delegate_base&) (arena.cpp:688)
==25280==    by 0x408ED1: main::{lambda()#1}::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/bin/slc7_amd64_gcc12/cmsRun)
==25280==    by 0x40518B: main (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/bin/slc7_amd64_gcc12/cmsRun)

==25280== Syscall param lseek(offset) contains uninitialised byte(s)
==25280==    at 0x704FDDD: __lseek64 (in /usr/lib64/libpthread-2.17.so)
==25280==    by 0xE9A456E: edm::storage::File::position(long, edm::storage::Storage::Relative) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libUtilitiesStorageFactory.so)
==25280==    by 0xE9A30F1: edm::storage::StorageAccountProxy::position(long, edm::storage::Storage::Relative) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libUtilitiesStorageFactory.so)
==25280==    by 0xFCC7332: TStorageFactoryFile::SysSeek(int, long long, int) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libIOPoolTFileAdaptor.so)
==25280==    by 0x554F841: TFile::Seek(long long, TFile::ERelativeTo) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libRIO.so)
==25280==    by 0x558B262: TKey::WriteFileKeepBuffer(TFile*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libRIO.so)
==25280==    by 0x4F7949F: TBasket::WriteBuffer() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x4F8800E: TBranch::WriteBasketImpl(TBasket*, int, ROOT::Internal::TBranchIMTHelper*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x4F88C15: TBranch::FlushBaskets() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x4F88C5B: TBranch::FlushBaskets() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x4FF76F1: TTree::FlushBasketsImpl() const::{lambda()#1}::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x7639D53: tbb::detail::d1::start_for<tbb::detail::d1::blocked_range<unsigned int>, tbb::detail::d1::parallel_for_body_wrapper<std::function<void (unsigned int)>, unsigned int>, tbb::detail::d1::auto_partitioner const>::execute(tbb::detail::d1::execution_data&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libImt.so)
==25280==    by 0x6401290: UnknownInlinedFun (task_dispatcher.h:322)
==25280==    by 0x6401290: UnknownInlinedFun (task_dispatcher.h:458)
==25280==    by 0x6401290: tbb::detail::r1::task_dispatcher::execute_and_wait(tbb::detail::d1::task*, tbb::detail::d1::wait_context&, tbb::detail::d1::task_group_context&) (task_dispatcher.cpp:168)
==25280==    by 0x7638BB2: tbb::detail::d1::task_arena_function<ROOT::TThreadExecutor::ParallelFor(unsigned int, unsigned int, unsigned int, std::function<void (unsigned int)> const&)::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libImt.so)
==25280==    by 0x63EE29D: UnknownInlinedFun (arena.cpp:757)
==25280==    by 0x63EE29D: UnknownInlinedFun (_template_helpers.h:230)
==25280==    by 0x63EE29D: tbb::detail::r1::isolate_within_arena(tbb::detail::d1::delegate_base&, long) (arena.cpp:758)
==25280==    by 0x7637492: tbb::detail::d1::task_arena_function<ROOT::TThreadExecutor::ParallelFor(unsigned int, unsigned int, unsigned int, std::function<void (unsigned int)> const&)::{lambda()#1}, void>::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libImt.so)
==25280==    by 0x63ED9BC: tbb::detail::r1::task_arena_impl::execute(tbb::detail::d1::task_arena_base&, tbb::detail::d1::delegate_base&) (arena.cpp:688)
==25280==    by 0x7639198: ROOT::TThreadExecutor::ParallelFor(unsigned int, unsigned int, unsigned int, std::function<void (unsigned int)> const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libImt.so)
==25280==    by 0x4FF8A8E: TTree::FlushBasketsImpl() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x4FF98EA: TTree::AutoSave(char const*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==  Uninitialised value was created by a heap allocation
==25280==    at 0x402EEE1: operator new(unsigned long) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/valgrind/3.22.0-e989c977f8949fc48f1bc1206f1f87c8/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==25280==    by 0x4F88113: TBranch::WriteBasketImpl(TBasket*, int, ROOT::Internal::TBranchIMTHelper*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x4F889C9: TBranch::FillImpl(ROOT::Internal::TBranchIMTHelper*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x4F92C5C: TBranchElement::FillImpl(ROOT::Internal::TBranchIMTHelper*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x50030D6: TTree::Fill() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x8D569F70: edm::RootOutputFile::writeParentageRegistry() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libIOPoolOutput.so)
==25280==    by 0x8D55AB7E: edm::PoolOutputModule::reallyCloseFile() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libIOPoolOutput.so)
==25280==    by 0x4C105C4: edm::Schedule::closeOutputFiles() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4B6658C: edm::EventProcessor::closeOutputFiles() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4B6B779: edm::EventProcessor::runToCompletion() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4074EE: tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/bin/slc7_amd64_gcc12/cmsRun)
==25280==    by 0x63ED9BC: tbb::detail::r1::task_arena_impl::execute(tbb::detail::d1::task_arena_base&, tbb::detail::d1::delegate_base&) (arena.cpp:688)
==25280==    by 0x408ED1: main::{lambda()#1}::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/bin/slc7_amd64_gcc12/cmsRun)
==25280==    by 0x40518B: main (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/bin/slc7_amd64_gcc12/cmsRun)

==25280== Conditional jump or move depends on uninitialised value(s)
==25280==    at 0x7D9DD06: ZSTD_HcFindBestMatch_noDict_4 (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/zstd/1.5.2-0530c990dd9ca27a8d966ab201ed2dfe/lib/libzstd.so.1.5.2)
==25280==    by 0x7DBAB16: ZSTD_compressBlock_lazy2 (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/zstd/1.5.2-0530c990dd9ca27a8d966ab201ed2dfe/lib/libzstd.so.1.5.2)
==25280==    by 0x7D754A2: ZSTD_compressBlock_internal (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/zstd/1.5.2-0530c990dd9ca27a8d966ab201ed2dfe/lib/libzstd.so.1.5.2)
==25280==    by 0x7D7705E: ZSTD_compress_frameChunk (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/zstd/1.5.2-0530c990dd9ca27a8d966ab201ed2dfe/lib/libzstd.so.1.5.2)
==25280==    by 0x7D79F65: ZSTD_compressEnd (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/zstd/1.5.2-0530c990dd9ca27a8d966ab201ed2dfe/lib/libzstd.so.1.5.2)
==25280==    by 0x7D7DE3F: ZSTD_compressCCtx (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/zstd/1.5.2-0530c990dd9ca27a8d966ab201ed2dfe/lib/libzstd.so.1.5.2)
==25280==    by 0x5BA1C14: R__zipZSTD (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libCore.so)
==25280==    by 0x4F795FC: TBasket::WriteBuffer() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x4F8800E: TBranch::WriteBasketImpl(TBasket*, int, ROOT::Internal::TBranchIMTHelper*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x4F88C15: TBranch::FlushBaskets() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x4F88C5B: TBranch::FlushBaskets() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x4FF76F1: TTree::FlushBasketsImpl() const::{lambda()#1}::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x7639D53: tbb::detail::d1::start_for<tbb::detail::d1::blocked_range<unsigned int>, tbb::detail::d1::parallel_for_body_wrapper<std::function<void (unsigned int)>, unsigned int>, tbb::detail::d1::auto_partitioner const>::execute(tbb::detail::d1::execution_data&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libImt.so)
==25280==    by 0x6401290: UnknownInlinedFun (task_dispatcher.h:322)
==25280==    by 0x6401290: UnknownInlinedFun (task_dispatcher.h:458)
==25280==    by 0x6401290: tbb::detail::r1::task_dispatcher::execute_and_wait(tbb::detail::d1::task*, tbb::detail::d1::wait_context&, tbb::detail::d1::task_group_context&) (task_dispatcher.cpp:168)
==25280==    by 0x7638BB2: tbb::detail::d1::task_arena_function<ROOT::TThreadExecutor::ParallelFor(unsigned int, unsigned int, unsigned int, std::function<void (unsigned int)> const&)::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libImt.so)
==25280==    by 0x63EE29D: UnknownInlinedFun (arena.cpp:757)
==25280==    by 0x63EE29D: UnknownInlinedFun (_template_helpers.h:230)
==25280==    by 0x63EE29D: tbb::detail::r1::isolate_within_arena(tbb::detail::d1::delegate_base&, long) (arena.cpp:758)
==25280==    by 0x7637492: tbb::detail::d1::task_arena_function<ROOT::TThreadExecutor::ParallelFor(unsigned int, unsigned int, unsigned int, std::function<void (unsigned int)> const&)::{lambda()#1}, void>::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libImt.so)
==25280==    by 0x63ED9BC: tbb::detail::r1::task_arena_impl::execute(tbb::detail::d1::task_arena_base&, tbb::detail::d1::delegate_base&) (arena.cpp:688)
==25280==    by 0x7639198: ROOT::TThreadExecutor::ParallelFor(unsigned int, unsigned int, unsigned int, std::function<void (unsigned int)> const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libImt.so)
==25280==    by 0x4FF8A8E: TTree::FlushBasketsImpl() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==  Uninitialised value was created by a stack allocation
==25280==    at 0xADA39931: TcdsRawToDigi::produce(edm::Event&, edm::EventSetup const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/pluginEventFilterOnlineMetaDataRawToDigiAuto.so)

@makortel
Copy link
Contributor Author

So I didn't see anything obvious related to Cling, but we might want to follow up

  • Tensorflow (not sure how though)
  • TrackerTopologyEP::produce()
  • pat::PATElectronProducer::produce()
  • edm::WorkerMaker<LogErrorEventFilter>::makeModule()
  • MuonSimClassifier::convertAndPush()
  • TcdsRawToDigi::produce()

@pcanal
Copy link
Contributor

pcanal commented Jan 26, 2024

The following errors usually indicates uninitialised memory in the user data.

==25280== Conditional jump or move depends on uninitialised value(s)
==25280==    at 0x7D9DDCE: ZSTD_HcFindBestMatch_noDict_4 (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/zstd/1.5.2-0530c990dd9ca27a8d966ab201ed2dfe/lib/libzstd.so.1.5.2)
==25280==    by 0x7DBAE18: ZSTD_compressBlock_lazy2 (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/zstd/1.5.2-0530c990dd9ca27a8d966ab201ed2dfe/lib/libzstd.so.1.5.2)
==25280==    by 0x7D754A2: ZSTD_compressBlock_internal (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/zstd/1.5.2-0530c990dd9ca27a8d966ab201ed2dfe/lib/libzstd.so.1.5.2)
==25280==    by 0x7D7705E: ZSTD_compress_frameChunk (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/zstd/1.5.2-0530c990dd9ca27a8d966ab201ed2dfe/lib/libzstd.so.1.5.2)
==25280==    by 0x7D79F65: ZSTD_compressEnd (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/zstd/1.5.2-0530c990dd9ca27a8d966ab201ed2dfe/lib/libzstd.so.1.5.2)
==25280==    by 0x7D7DE3F: ZSTD_compressCCtx (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/zstd/1.5.2-0530c990dd9ca27a8d966ab201ed2dfe/lib/libzstd.so.1.5.2)
==25280==    by 0x5BA1C14: R__zipZSTD (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libCore.so)

and, at least in the first case, valgrind points to:

==25280==  Uninitialised value was created by a stack allocation
==25280==    at 0x7D48F8B3: MuonSimClassifier::convertAndPush(TrackingParticle const&, std::vector<reco::GenParticle, std::allocator<reco::GenParticle> >&, edm::Ref<std::vector<TrackingParticle, std::allocator<TrackingParticle> >, TrackingParticle, edm::refhelper::FindUsingAdvance<std::vector<TrackingParticle, std::allocator<TrackingParticle> >, TrackingParticle> > const&, edm::Handle<std::vector<reco::GenParticle, std::allocator<reco::GenParticle> > > const&) const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/pluginSimMuonMCTruthPlugins.so)

@pcanal
Copy link
Contributor

pcanal commented Jan 26, 2024 •
edited
Loading 

==25280== Conditional jump or move depends on uninitialised value(s)
==25280==    at 0x554C093: TFree::GetBestFree(TList*, int) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libRIO.so)
==25280==    by 0x558B4E3: TKey::Create(int, TFile*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libRIO.so)

is a bit worrisome and might be an issue in ROOT (or valgrind)

@makortel
Copy link
Contributor Author

The following errors usually indicates uninitialised memory in the user data.
...
and, at least in the first case, valgrind points to:

Thanks for confirming, this was my interpretation as well.

@pcanal
Copy link
Contributor

pcanal commented Jan 26, 2024

The following is the most unexpected of all and may indicate either that valgrind is not ready for the platform or that there is a serious memory corruption somewhere. Since the symbol mentions LTO, did you also try without LTO turned on.

==25280== Conditional jump or move depends on uninitialised value(s)
==25280==    at 0x65C74B52: .LTHUNK17.lto_priv.0 (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/pluginDPGAnalysisSkims.so)
==25280==    by 0x65C751A1: virtual thunk to edm::global::impl::LuminosityBlockCacheHolder<edm::global::EDFilterBase, leef::LumiErrors>::doEndLuminosityBlock_(edm::LuminosityBlock const&, edm::EventSetup const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/pluginDPGAnalysisSkims.so)

@makortel
Copy link
Contributor Author 

==25280== Conditional jump or move depends on uninitialised value(s)
==25280==    at 0x554C093: TFree::GetBestFree(TList*, int) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libRIO.so)
==25280==    by 0x558B4E3: TKey::Create(int, TFile*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libRIO.so)

is a bit worrisome and might be an issue in ROOT (or valgrind)

Ok, there were more of those. I attached a file that should contain all of them (in case that would be helpful).
valgrind_root.txt

@makortel
Copy link
Contributor Author

Since the symbol mentions LTO, did you also try without LTO turned on.

I did not (and at this stage I'd have to move to newer IB). @smuzaffar Is it possible to turn off LTO for select package(s) in a local build, or would that require using the NONLTO build?

@smuzaffar
Copy link
Contributor

smuzaffar commented Jan 26, 2024 •
edited
Loading

@makortel , currently you can not disable LTO for individual packages. But what you can do is to

  • checkout cmssw packages you want to build without LTO
  • update config/Self.xml file in your devel area and set ENABLE_LTO=0
  • run scram setup self
  • run scram build -j $(nproc)

this should allow you to build the checked out packages without LTO

@aandvalenzuela
Copy link
Contributor

aandvalenzuela commented Feb 27, 2024 •
edited
Loading

Hello,

I am not 100% sure it is fully related, but workflow 25034.114 is consistently failing for el8_aarch64_gcc12 in CSCDigiValidation:cscDigiValidation as reported above but with a different stacktrace:

Thread 6 (Thread 0x4000a6b89250 (LWP 92865) "cmsRun"):
#0  0x00004000372d41e8 in epoll_pwait () from /lib64/libc.so.6
#1  0x000040003bedbdec in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#2  0x000040003bed7efc in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#3  0x000040003bee10e0 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#4  0x00004000371c78b8 in start_thread () from /lib64/libpthread.so.0
#5  0x0000400037223afc in thread_start () from /lib64/libc.so.6
Thread 5 (Thread 0x40007df09250 (LWP 92716) "cmsRun"):
#0  0x00004000372a4b3c in nanosleep () from /lib64/libc.so.6
#1  0x00004000372a49e0 in sleep () from /lib64/libc.so.6
#2  0x000040003c3a927c in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000040010553cc70 in void (anonymous namespace)::getSimTrackId<PixelDigiSimLink>(std::vector<std::pair<unsigned int, EncodedEventId>, std::allocator<std::pair<unsigned int, EncodedEventId> > >&, edm::Handle<edm::DetSetVector<PixelDigiSimLink> > const&, DetId const&, unsigned int) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/pluginSimTrackerTrackerHitAssociationPlugins.so
#5  0x000040010553fb30 in ClusterTPAssociationProducer::produce(edm::StreamID, edm::Event&, edm::EventSetup const&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/pluginSimTrackerTrackerHitAssociationPlugins.so
#6  0x00004000351c64e8 in edm::global::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#7  0x00004000351b8d00 in edm::WorkerT<edm::global::EDProducerBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#8  0x000040003514c2c4 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#9  0x000040003514cae4 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#10 0x00004000356e50dc in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/libFWCoreConcurrency.so
#11 0x0000400036c32dfc in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x40016abcb900, waiter=<synthetic pointer>..., this=0x400037b4b480) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#12 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (waiter=<synthetic pointer>..., t=0x0, this=0x400037b4b480) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#13 tbb::detail::r1::arena::process (tls=..., this=0x400037b4ad00) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/arena.cpp:137
#14 tbb::detail::r1::market::process (this=0x400037b4b580, j=...) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/market.cpp:599
#15 0x0000400036c3af2c in tbb::detail::r1::rml::private_worker::run (this=0x40003b618000) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#16 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x40003b618000) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#17 0x00004000371c78b8 in start_thread () from /lib64/libpthread.so.0
#18 0x0000400037223afc in thread_start () from /lib64/libc.so.6
Thread 4 (Thread 0x40007d4f9250 (LWP 92715) "cmsRun"):
#0  0x00004000372ca91c in poll () from /lib64/libc.so.6
#1  0x000040003c3f333c in full_read.constprop () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x000040003c3ac5e4 in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  0x000040003c3acf5c in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x0000400213ff001c in ?? ()
#6  0x000040004a27d300 in cling::Interpreter::RunFunction(clang::FunctionDecl const*, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/external/el8_aarch64_gcc12/lib/libCling.so
#7  0x000040004a27d300 in cling::Interpreter::RunFunction(clang::FunctionDecl const*, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/external/el8_aarch64_gcc12/lib/libCling.so
#8  0x000040004a27d9ac in cling::Interpreter::EvaluateInternal(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, cling::CompilationOptions, cling::Value*, cling::Transaction**, unsigned long) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/external/el8_aarch64_gcc12/lib/libCling.so
#9  0x000040004a34f744 in cling::MetaSema::actOnxCommand(llvm::StringRef, llvm::StringRef, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/external/el8_aarch64_gcc12/lib/libCling.so
#10 0x000040004a35c034 in cling::MetaParser::isXCommand(cling::MetaSema::ActionResult&, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/external/el8_aarch64_gcc12/lib/libCling.so
#11 0x000040004a35def4 in cling::MetaParser::isCommand(cling::MetaSema::ActionResult&, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/external/el8_aarch64_gcc12/lib/libCling.so
#12 0x000040004a3491ec in cling::MetaProcessor::process(llvm::StringRef, cling::Interpreter::CompilationResult&, cling::Value*, bool) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/external/el8_aarch64_gcc12/lib/libCling.so
#13 0x000040004a1a08ac in HandleInterpreterException(cling::MetaProcessor*, char const*, cling::Interpreter::CompilationResult&, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/external/el8_aarch64_gcc12/lib/libCling.so
#14 0x000040004a1b3340 in TCling::ProcessLine(char const*, TInterpreter::EErrorCode*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/external/el8_aarch64_gcc12/lib/libCling.so
#15 0x000040004a1b38b4 in TCling::ProcessLineSynch(char const*, TInterpreter::EErrorCode*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/external/el8_aarch64_gcc12/lib/libCling.so
#16 0x0000400036357258 in TApplication::ExecuteFile(char const*, int*, bool) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/external/el8_aarch64_gcc12/lib/libCore.so
#17 0x000040004a1b23d8 in TCling::ExecuteMacro(char const*, TInterpreter::EErrorCode*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/external/el8_aarch64_gcc12/lib/libCling.so
#18 0x00004000363ac11c in TROOT::Macro(char const*, int*, bool) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/external/el8_aarch64_gcc12/lib/libCore.so
#19 0x00004000363900a0 in TPluginManager::LoadHandlerMacros(char const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/external/el8_aarch64_gcc12/lib/libCore.so
#20 0x00004000363906fc in TPluginManager::LoadHandlersFromPluginDirs(char const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/external/el8_aarch64_gcc12/lib/libCore.so
#21 0x0000400036391774 in TPluginManager::FindHandler(char const*, char const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/external/el8_aarch64_gcc12/lib/libCore.so
#22 0x0000400035bb7cbc in ROOT::Math::Factory::CreateMinimizer(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/external/el8_aarch64_gcc12/lib/libMathCore.so
#23 0x0000400035bb9680 in ROOT::Fit::FitConfig::CreateMinimizer() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/external/el8_aarch64_gcc12/lib/libMathCore.so
#24 0x0000400035bd11d0 in ROOT::Fit::Fitter::DoInitMinimizer() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/external/el8_aarch64_gcc12/lib/libMathCore.so
#25 0x0000400035bd80fc in bool ROOT::Fit::Fitter::DoMinimization<ROOT::Fit::Chi2FCN<ROOT::Math::IBaseFunctionMultiDimTempl<double>, ROOT::Math::IParametricFunctionMultiDimTempl<double> > >(std::unique_ptr<ROOT::Fit::Chi2FCN<ROOT::Math::IBaseFunctionMultiDimTempl<double>, ROOT::Math::IParametricFunctionMultiDimTempl<double> >, std::default_delete<ROOT::Fit::Chi2FCN<ROOT::Math::IBaseFunctionMultiDimTempl<double>, ROOT::Math::IParametricFunctionMultiDimTempl<double> > > >, ROOT::Math::IBaseFunctionMultiDimTempl<double> const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/external/el8_aarch64_gcc12/lib/libMathCore.so
#26 0x0000400035bd3f08 in ROOT::Fit::Fitter::DoLeastSquareFit(ROOT::EExecutionPolicy const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/external/el8_aarch64_gcc12/lib/libMathCore.so
#27 0x000040003ca9ae60 in TFitResultPtr HFit::Fit<TGraph>(TGraph*, TF1*, Foption_t&, ROOT::Math::MinimizerOptions const&, char const*, ROOT::Fit::DataRange&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/external/el8_aarch64_gcc12/lib/libHist.so
#28 0x000040003ca8f340 in ROOT::Fit::FitObject(TGraph*, TF1*, Foption_t&, ROOT::Math::MinimizerOptions const&, char const*, ROOT::Fit::DataRange&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/external/el8_aarch64_gcc12/lib/libHist.so
#29 0x000040003cb3c734 in TGraph::Fit(TF1*, char const*, char const*, double, double) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/external/el8_aarch64_gcc12/lib/libHist.so
#30 0x00004000c21a50cc in CSCSimHitMatcher::fitHitsInChamber(unsigned int, float&, float&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/libValidationMuonHits.so
#31 0x00004000c44f9700 in CSCStubResolutionValidation::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/libValidationMuonCSCDigis.so
#32 0x00004000351da61c in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#33 0x00004000351bf1a0 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#34 0x000040003514c2c4 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#35 0x000040003514cae4 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#36 0x00004000356e50dc in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/libFWCoreConcurrency.so
#37 0x0000400036c32dfc in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x40016ac69300, waiter=<synthetic pointer>..., this=0x400037b4b380) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#38 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (waiter=<synthetic pointer>..., t=0x0, this=0x400037b4b380) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#39 tbb::detail::r1::arena::process (tls=..., this=0x400037b4ad00) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/arena.cpp:137
#40 tbb::detail::r1::market::process (this=0x400037b4b580, j=...) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/market.cpp:599
#41 0x0000400036c3af2c in tbb::detail::r1::rml::private_worker::run (this=0x40003b618100) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#42 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x40003b618100) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#43 0x00004000371c78b8 in start_thread () from /lib64/libpthread.so.0
#44 0x0000400037223afc in thread_start () from /lib64/libc.so.6
Thread 3 (Thread 0x40007cae9250 (LWP 92714) "cmsRun"):
#0  0x00004000372a4b3c in nanosleep () from /lib64/libc.so.6
#1  0x00004000372a49e0 in sleep () from /lib64/libc.so.6
#2  0x000040003c3a927c in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00004000df56c868 in std::__detail::_Map_base<DetId, std::pair<DetId const, std::vector<std::pair<int, float>, std::allocator<std::pair<int, float> > > >, std::allocator<std::pair<DetId const, std::vector<std::pair<int, float>, std::allocator<std::pair<int, float> > > > >, std::__detail::_Select1st, std::equal_to<DetId>, std::hash<DetId>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true>, true>::operator[](DetId const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/pluginSimCalorimetryHGCalAssociatorProducersAuto.so
#5  0x00004000df56ff34 in TSToSimTSHitLCAssociatorByEnergyScoreImpl::makeConnections(edm::Handle<std::vector<ticl::Trackster, std::allocator<ticl::Trackster> > > const&, edm::Handle<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> > > const&, edm::Handle<std::vector<SimCluster, std::allocator<SimCluster> > > const&, edm::Handle<std::vector<CaloParticle, std::allocator<CaloParticle> > > const&, edm::Handle<std::vector<ticl::Trackster, std::allocator<ticl::Trackster> > > const&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/pluginSimCalorimetryHGCalAssociatorProducersAuto.so
#6  0x00004000df5729a8 in TSToSimTSHitLCAssociatorByEnergyScoreImpl::associateRecoToSim(edm::Handle<std::vector<ticl::Trackster, std::allocator<ticl::Trackster> > > const&, edm::Handle<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> > > const&, edm::Handle<std::vector<SimCluster, std::allocator<SimCluster> > > const&, edm::Handle<std::vector<CaloParticle, std::allocator<CaloParticle> > > const&, edm::Handle<std::vector<ticl::Trackster, std::allocator<ticl::Trackster> > > const&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/pluginSimCalorimetryHGCalAssociatorProducersAuto.so
#7  0x00004000df56e9ec in TSToSimTSHitLCAssociatorEDProducer::produce(edm::StreamID, edm::Event&, edm::EventSetup const&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/pluginSimCalorimetryHGCalAssociatorProducersAuto.so
#8  0x00004000351c64e8 in edm::global::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#9  0x00004000351b8d00 in edm::WorkerT<edm::global::EDProducerBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#10 0x000040003514c2c4 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#11 0x000040003514cae4 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#12 0x00004000356e50dc in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/libFWCoreConcurrency.so
#13 0x0000400036c32dfc in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x40013b781e00, waiter=<synthetic pointer>..., this=0x400037b4b400) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#14 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (waiter=<synthetic pointer>..., t=0x0, this=0x400037b4b400) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#15 tbb::detail::r1::arena::process (tls=..., this=0x400037b4ad00) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/arena.cpp:137
#16 tbb::detail::r1::market::process (this=0x400037b4b580, j=...) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/market.cpp:599
#17 0x0000400036c3af2c in tbb::detail::r1::rml::private_worker::run (this=0x40003b618080) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#18 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x40003b618080) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#19 0x00004000371c78b8 in start_thread () from /lib64/libpthread.so.0
#20 0x0000400037223afc in thread_start () from /lib64/libc.so.6
Thread 2 (Thread 0x400040879250 (LWP 92602) "cmsRun"):
#0  0x00004000371d2578 in waitpid () from /lib64/libpthread.so.0
#1  0x000040003c3a93f0 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x000040003c3ac550 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  0x0000400036f5eb8c in std::execute_native_thread_routine (__p=0x4000390ac5d0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82
#4  0x00004000371c78b8 in start_thread () from /lib64/libpthread.so.0
#5  0x0000400037223afc in thread_start () from /lib64/libc.so.6
Thread 1 (Thread 0x400036ddd080 (LWP 92020) "cmsRun"):
#0  0x00004000372a4b3c in nanosleep () from /lib64/libc.so.6
#1  0x00004000372a49e0 in sleep () from /lib64/libc.so.6
#2  0x000040003c3a927c in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00004000cd40bf80 in std::_Hashtable<unsigned int, std::pair<unsigned int const, MTDHit>, std::allocator<std::pair<unsigned int const, MTDHit> >, std::__detail::_Select1st, std::equal_to<unsigned int>, std::hash<unsigned int>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::find(unsigned int const&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/pluginValidationMtdValidationAuto.so
#5  0x00004000cd41f624 in EtlLocalRecoValidation::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/pluginValidationMtdValidationAuto.so
#6  0x00004000351da61c in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#7  0x00004000351bf1a0 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#8  0x000040003514c2c4 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#9  0x000040003514cae4 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#10 0x00004000356e50dc in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/libFWCoreConcurrency.so
#11 0x0000400036c38774 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=0x400118c4c400, this=0x400037b4b300) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#12 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x400037b4b300) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#13 tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:168
#14 0x00004000350c7044 in edm::FinalWaitingTask::wait() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#15 0x00004000350d0668 in edm::EventProcessor::processRuns() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#16 0x00004000350d0bf8 in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#17 0x0000000000406fe4 in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#18 0x0000400036c310d8 in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/arena.cpp:688
#19 0x0000000000408854 in main::{lambda()#1}::operator()() const ()
#20 0x0000000000404954 in main ()

Current Modules:

Module: CSCDigiValidation:cscDigiValidation (crashed)
Module: EtlLocalRecoValidation:etlLocalRecoValid
Module: TSToSimTSHitLCAssociatorEDProducer:tracksterSimTracksterAssociationLinkingbyCLUE3D
Module: ClusterTPAssociationProducer:hltTPClusterProducer

A fatal system signal has occurred: segmentation violation

Log from today's IBs at CMSSW_14_1_X_2024-02-26-2300.

RelVal 140.56 is also failing (as 140.58 before) showing:

#5  0x000040003fef3afc in thread_start () from /lib64/libc.so.6
Thread 1 (Thread 0x40003faad080 (LWP 317044) "cmsRun"):
#0  0x000040003ff74b3c in nanosleep () from /lib64/libc.so.6
#1  0x000040003ff749e0 in sleep () from /lib64/libc.so.6
#2  0x00004000451f927c in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000040003fef37e4 in syscall () from /lib64/libc.so.6
#5  0x000040003f908f68 in tbb::detail::r1::futex_wait (comparand=2, futex=0xffffc4b36758) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0xffffc4b36758) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::sleep_node<tbb::detail::r1::market_context>::wait (this=0xffffc4b36728) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/concurrent_monitor.h:170
#8  tbb::detail::r1::concurrent_monitor_base<tbb::detail::r1::market_context>::commit_wait (this=<optimized out>, node=...) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/concurrent_monitor.h:232
#9  tbb::detail::r1::concurrent_monitor_base<tbb::detail::r1::market_context>::commit_wait (node=..., this=0x40004081ab18) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/concurrent_monitor.h:228
#10 tbb::detail::r1::concurrent_monitor_base<tbb::detail::r1::market_context>::wait<tbb::detail::r1::sleep_node<tbb::detail::r1::market_context>, tbb::detail::r1::external_waiter::pause(tbb::detail::r1::arena_slot&)::{lambda()#1}&>(tbb::detail::r1::external_waiter::pause(tbb::detail::r1::arena_slot&)::{lambda()#1}&, tbb::detail::r1::sleep_node<tbb::detail::r1::market_context>&&) (node=..., pred=<synthetic pointer>..., this=0x40004081ab18) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/concurrent_monitor.h:262
#11 tbb::detail::r1::sleep_waiter::sleep<tbb::detail::r1::external_waiter::pause(tbb::detail::r1::arena_slot&)::{lambda()#1}>(unsigned long, tbb::detail::r1::external_waiter::pause(tbb::detail::r1::arena_slot&)::{lambda()#1}) (this=0xffffc4b366d0, wakeup_condition=..., uniq_tag=70369926930688) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/waiters.h:118
#12 tbb::detail::r1::external_waiter::pause (this=0xffffc4b366d0) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/waiters.h:144
#13 tbb::detail::r1::external_waiter::pause (this=<optimized out>) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/waiters.h:137
#14 tbb::detail::r1::task_dispatcher::receive_or_steal_task<false, tbb::detail::r1::external_waiter> (critical_allowed=true, fifo_allowed=true, isolation=0, waiter=..., ed=..., tls=..., this=0x4000407f3d80) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:231
#15 tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=0x0, this=0x4000407f3d80) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:350
#16 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x4000407f3d80) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#17 tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:168
#18 0x000040003dd97044 in edm::FinalWaitingTask::wait() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#19 0x000040003dda0668 in edm::EventProcessor::processRuns() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#20 0x000040003dda0bf8 in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#21 0x0000000000406fe4 in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#22 0x000040003f9010d8 in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/arena.cpp:688
#23 0x0000000000408854 in main::{lambda()#1}::operator()() const ()
#24 0x0000000000404954 in main ()

Current Modules:

Module: AlcaBeamMonitor:AlcaBeamMonitor (crashed)
Module: none
Module: none
Module: none

A fatal system signal has occurred: segmentation violation

@makortel makortel mentioned this issue Mar 5, 2024
Closed
@iarspider
Copy link
Contributor

Two new crashes in CMSSW_14_1_ROOT6_X_2024-03-06-2300:

140.58

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Thu Mar  7 05:58:42 CET 2024
Thread 8 (Thread 0x4000c3729250 (LWP 513138) "cmsRun"):
#0  0x000040003beedc58 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
#1  0x000040006da1d02c in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libtensorflow_cc.so.2
#2  0x000040006da1d720 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libtensorflow_cc.so.2
#3  0x000040006da1abdc in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libtensorflow_cc.so.2
#4  0x0000400077d7112c in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libtensorflow_framework.so.2
#5  0x000040003bee78b8 in start_thread () from /lib64/libpthread.so.0
#6  0x000040003bf43afc in thread_start () from /lib64/libc.so.6
Thread 7 (Thread 0x4000c3519250 (LWP 513137) "cmsRun"):
#0  0x000040003beedc58 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
#1  0x000040006da1d02c in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libtensorflow_cc.so.2
#2  0x000040006da1d720 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libtensorflow_cc.so.2
#3  0x000040006da1abdc in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libtensorflow_cc.so.2
#4  0x0000400077d7112c in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libtensorflow_framework.so.2
#5  0x000040003bee78b8 in start_thread () from /lib64/libpthread.so.0
#6  0x000040003bf43afc in thread_start () from /lib64/libc.so.6
Thread 6 (Thread 0x4000c2309250 (LWP 513136) "cmsRun"):
#0  0x000040003beedc58 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
#1  0x000040006da1d02c in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libtensorflow_cc.so.2
#2  0x000040006da1d720 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libtensorflow_cc.so.2
#3  0x000040006da1abdc in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libtensorflow_cc.so.2
#4  0x0000400077d7112c in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libtensorflow_framework.so.2
#5  0x000040003bee78b8 in start_thread () from /lib64/libpthread.so.0
#6  0x000040003bf43afc in thread_start () from /lib64/libc.so.6
Thread 5 (Thread 0x400086a09250 (LWP 512865) "cmsRun"):
#0  0x000040003bfc4b3c in nanosleep () from /lib64/libc.so.6
#1  0x000040003bfc49e0 in sleep () from /lib64/libc.so.6
#2  0x00004000411a927c in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000040003bf437e4 in syscall () from /lib64/libc.so.6
#5  0x000040003b949818 in tbb::detail::r1::futex_wait (comparand=2, futex=0x400040408024) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x400040408024) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  0x000040003b95b09c in tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x400040408020) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x400040408000) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x400040408000) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x000040003bee78b8 in start_thread () from /lib64/libpthread.so.0
#11 0x000040003bf43afc in thread_start () from /lib64/libc.so.6
Thread 4 (Thread 0x4000839d9250 (LWP 512864) "cmsRun"):
#0  0x000040003bfc4b3c in nanosleep () from /lib64/libc.so.6
#1  0x000040003bfc49e0 in sleep () from /lib64/libc.so.6
#2  0x00004000411a927c in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000040003bf437e4 in syscall () from /lib64/libc.so.6
#5  0x000040003b949818 in tbb::detail::r1::futex_wait (comparand=2, futex=0x400040408124) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x400040408124) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  0x000040003b95b09c in tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x400040408120) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x400040408100) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x400040408100) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x000040003bee78b8 in start_thread () from /lib64/libpthread.so.0
#11 0x000040003bf43afc in thread_start () from /lib64/libc.so.6
Thread 3 (Thread 0x400082fc9250 (LWP 512863) "cmsRun"):
#0  0x000040003bfea91c in poll () from /lib64/libc.so.6
#1  0x00004000411f333c in full_read.constprop () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x00004000411ac5e4 in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  0x00004000411acf5c in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x000040003b0b16f0 in TPluginManager::RemoveHandler(char const*, char const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCore.so
#6  0x000040003b0b1818 in TPluginManager::AddHandler(char const*, char const*, char const*, char const*, char const*, char const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCore.so
#7  0x000040004cce89c4 in cling::IncrementalExecutor::executeWrapper(llvm::StringRef, cling::Value*) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCling.so
#8  0x000040004cc75b60 in cling::Interpreter::RunFunction(clang::FunctionDecl const*, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCling.so
#9  0x000040004cc7620c in cling::Interpreter::EvaluateInternal(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, cling::CompilationOptions, cling::Value*, cling::Transaction**, unsigned long) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCling.so
#10 0x000040004cd54ad0 in cling::MetaSema::actOnxCommand(llvm::StringRef, llvm::StringRef, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCling.so
#11 0x000040004cd61624 in cling::MetaParser::isXCommand(cling::MetaSema::ActionResult&, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCling.so
#12 0x000040004cd634d4 in cling::MetaParser::isCommand(cling::MetaSema::ActionResult&, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCling.so
#13 0x000040004cd4e55c in cling::MetaProcessor::process(llvm::StringRef, cling::Interpreter::CompilationResult&, cling::Value*, bool) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCling.so
#14 0x000040004cb91d9c in HandleInterpreterException(cling::MetaProcessor*, char const*, cling::Interpreter::CompilationResult&, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCling.so
#15 0x000040004cba48d0 in TCling::ProcessLine(char const*, TInterpreter::EErrorCode*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCling.so
#16 0x000040004cba4e44 in TCling::ProcessLineSynch(char const*, TInterpreter::EErrorCode*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCling.so
#17 0x000040003b079048 in TApplication::ExecuteFile(char const*, int*, bool) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCore.so
#18 0x000040004cba3968 in TCling::ExecuteMacro(char const*, TInterpreter::EErrorCode*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCling.so
#19 0x000040003b0ce8ec in TROOT::Macro(char const*, int*, bool) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCore.so
#20 0x000040003b0b27d0 in TPluginManager::LoadHandlerMacros(char const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCore.so
#21 0x000040003b0b2e2c in TPluginManager::LoadHandlersFromPluginDirs(char const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCore.so
#22 0x000040003b0b3ea4 in TPluginManager::FindHandler(char const*, char const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCore.so
#23 0x000040003a8e7b9c in ROOT::Math::Factory::CreateMinimizer(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libMathCore.so
#24 0x000040003a8e9560 in ROOT::Fit::FitConfig::CreateMinimizer() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libMathCore.so
#25 0x000040003a900f30 in ROOT::Fit::Fitter::DoInitMinimizer() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libMathCore.so
#26 0x000040003a9071dc in bool ROOT::Fit::Fitter::DoMinimization<ROOT::Fit::PoissonLikelihoodFCN<ROOT::Math::IBaseFunctionMultiDimTempl<double>, ROOT::Math::IParametricFunctionMultiDimTempl<double> > >(std::unique_ptr<ROOT::Fit::PoissonLikelihoodFCN<ROOT::Math::IBaseFunctionMultiDimTempl<double>, ROOT::Math::IParametricFunctionMultiDimTempl<double> >, std::default_delete<ROOT::Fit::PoissonLikelihoodFCN<ROOT::Math::IBaseFunctionMultiDimTempl<double>, ROOT::Math::IParametricFunctionMultiDimTempl<double> > > >, ROOT::Math::IBaseFunctionMultiDimTempl<double> const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libMathCore.so
#27 0x000040003a9046c0 in ROOT::Fit::Fitter::DoBinnedLikelihoodFit(bool, ROOT::EExecutionPolicy const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libMathCore.so
#28 0x0000400043af8a4c in TFitResultPtr HFit::Fit<TH1>(TH1*, TF1*, Foption_t&, ROOT::Math::MinimizerOptions const&, char const*, ROOT::Fit::DataRange&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libHist.so
#29 0x0000400043aee194 in ROOT::Fit::FitObject(TH1*, TF1*, Foption_t&, ROOT::Math::MinimizerOptions const&, char const*, ROOT::Fit::DataRange&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libHist.so
#30 0x0000400043bb9be0 in TH1::Fit(TF1*, char const*, char const*, double, double) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libHist.so
#31 0x000040007e96c910 in BSFitter::Fit_d0phi() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/libRecoVertexBeamSpotProducer.so
#32 0x000040007e96d454 in BSFitter::Fit_ited0phi() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/libRecoVertexBeamSpotProducer.so
#33 0x000040007e96d940 in BSFitter::Fit(double*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/libRecoVertexBeamSpotProducer.so
#34 0x000040007e96e098 in BSFitter::Fit() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/libRecoVertexBeamSpotProducer.so
#35 0x000040007e976e20 in BeamFitter::runFitterNoTxt() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/libRecoVertexBeamSpotProducer.so
#36 0x000040007e977294 in BeamFitter::runPVandTrkFitter() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/libRecoVertexBeamSpotProducer.so
#37 0x00004000bc3e0ee4 in AlcaBeamMonitor::globalEndLuminosityBlock(edm::LuminosityBlock const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/pluginAlcaBeamMonitor.so
#38 0x00004000bc3ed9dc in virtual thunk to edm::one::impl::LuminosityBlockCacheHolder<edm::one::EDProducerBase, alcabeammonitor::BeamSpotInfo>::doEndLuminosityBlock_(edm::LuminosityBlock const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/pluginAlcaBeamMonitor.so
#39 0x0000400039eef144 in edm::one::EDProducerBase::doEndLuminosityBlock(edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#40 0x0000400039ed7d00 in edm::WorkerT<edm::one::EDProducerBase>::implDoEnd(edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#41 0x0000400039e2a6c0 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#42 0x0000400039e17acc in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3> >::execute()::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#43 0x0000400039e2e430 in edm::SerialTaskQueue::QueuedTask<edm::SerialTaskQueueChain::push<edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3> >::execute()::{lambda()#1}&>(tbb::detail::d1::task_group&, edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3> >::execute()::{lambda()#1}&)::{lambda()#1}>::execute() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#44 0x000040003a4065b0 in tbb::detail::d1::function_task<edm::SerialTaskQueue::spawn(edm::SerialTaskQueue::TaskBase&)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/libFWCoreConcurrency.so
#45 0x000040003b952dfc in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x40003c718900, waiter=<synthetic pointer>..., this=0x40003c86b400) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#46 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (waiter=<synthetic pointer>..., t=0x0, this=0x40003c86b400) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#47 tbb::detail::r1::arena::process (tls=..., this=0x40003c86ad00) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/arena.cpp:137
#48 tbb::detail::r1::market::process (this=0x40003c86b580, j=...) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/market.cpp:599
#49 0x000040003b95af2c in tbb::detail::r1::rml::private_worker::run (this=0x400040408080) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#50 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x400040408080) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#51 0x000040003bee78b8 in start_thread () from /lib64/libpthread.so.0
#52 0x000040003bf43afc in thread_start () from /lib64/libc.so.6
Thread 2 (Thread 0x4000450d9250 (LWP 512780) "cmsRun"):
#0  0x000040003bef2578 in waitpid () from /lib64/libpthread.so.0
#1  0x00004000411a93f0 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x00004000411ac550 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  0x000040003bc7eb8c in std::execute_native_thread_routine (__p=0x40003e0ab230) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82
#4  0x000040003bee78b8 in start_thread () from /lib64/libpthread.so.0
#5  0x000040003bf43afc in thread_start () from /lib64/libc.so.6
Thread 1 (Thread 0x40003bafd0f0 (LWP 512617) "cmsRun"):
#0  0x000040003bfc4b3c in nanosleep () from /lib64/libc.so.6
#1  0x000040003bfc49e0 in sleep () from /lib64/libc.so.6
#2  0x00004000411a927c in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000040003bf437e4 in syscall () from /lib64/libc.so.6
#5  0x000040003b958f68 in tbb::detail::r1::futex_wait (comparand=2, futex=0xffffd9801418) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0xffffd9801418) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::sleep_node<tbb::detail::r1::market_context>::wait (this=0xffffd98013e8) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/concurrent_monitor.h:170
#8  tbb::detail::r1::concurrent_monitor_base<tbb::detail::r1::market_context>::commit_wait (this=<optimized out>, node=...) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/concurrent_monitor.h:232
#9  tbb::detail::r1::concurrent_monitor_base<tbb::detail::r1::market_context>::commit_wait (node=..., this=0x40003c86b598) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/concurrent_monitor.h:228
#10 tbb::detail::r1::concurrent_monitor_base<tbb::detail::r1::market_context>::wait<tbb::detail::r1::sleep_node<tbb::detail::r1::market_context>, tbb::detail::r1::external_waiter::pause(tbb::detail::r1::arena_slot&)::{lambda()#1}&>(tbb::detail::r1::external_waiter::pause(tbb::detail::r1::arena_slot&)::{lambda()#1}&, tbb::detail::r1::sleep_node<tbb::detail::r1::market_context>&&) (node=..., pred=<synthetic pointer>..., this=0x40003c86b598) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/concurrent_monitor.h:262
#11 tbb::detail::r1::sleep_waiter::sleep<tbb::detail::r1::external_waiter::pause(tbb::detail::r1::arena_slot&)::{lambda()#1}>(unsigned long, tbb::detail::r1::external_waiter::pause(tbb::detail::r1::arena_slot&)::{lambda()#1}) (this=0xffffd9801390, wakeup_condition=..., uniq_tag=70369862305792) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/waiters.h:118
#12 tbb::detail::r1::external_waiter::pause (this=0xffffd9801390) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/waiters.h:144
#13 tbb::detail::r1::external_waiter::pause (this=<optimized out>) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/waiters.h:137
#14 tbb::detail::r1::task_dispatcher::receive_or_steal_task<false, tbb::detail::r1::external_waiter> (critical_allowed=true, fifo_allowed=true, isolation=0, waiter=..., ed=..., tls=..., this=0x40003c86b300) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:231
#15 tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=0x0, this=0x40003c86b300) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:350
#16 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x40003c86b300) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#17 tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:168
#18 0x0000400039de6fc4 in edm::FinalWaitingTask::wait() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#19 0x0000400039df05e8 in edm::EventProcessor::processRuns() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#20 0x0000400039df0b78 in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#21 0x0000000000406fe4 in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#22 0x000040003b9510d8 in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/arena.cpp:688
#23 0x0000000000408854 in main::{lambda()#1}::operator()() const ()
#24 0x0000000000404954 in main ()

Current Modules:

Module: AlcaBeamMonitor:AlcaBeamMonitor (crashed)
Module: none
Module: none
Module: none

and 11025.0

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Thu Mar  7 06:14:29 CET 2024
Thread 22 (Thread 0x4000d7209250 (LWP 532101) "cmsRun"):
#0  0x0000400028d3dc58 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
#1  0x000040005ce3d02c in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libtensorflow_cc.so.2
#2  0x000040005ce3d720 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libtensorflow_cc.so.2
#3  0x000040005ce3abdc in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libtensorflow_cc.so.2
#4  0x000040006719112c in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libtensorflow_framework.so.2
#5  0x0000400028d378b8 in start_thread () from /lib64/libpthread.so.0
#6  0x0000400028d93afc in thread_start () from /lib64/libc.so.6
Thread 21 (Thread 0x4000d2e99250 (LWP 532100) "cmsRun"):
#0  0x0000400028d3dc58 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
#1  0x000040005ce3d02c in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libtensorflow_cc.so.2
#2  0x000040005ce3d720 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libtensorflow_cc.so.2
#3  0x000040005ce3abdc in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libtensorflow_cc.so.2
#4  0x000040006719112c in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libtensorflow_framework.so.2
#5  0x0000400028d378b8 in start_thread () from /lib64/libpthread.so.0
#6  0x0000400028d93afc in thread_start () from /lib64/libc.so.6
Thread 20 (Thread 0x4000d2c89250 (LWP 532099) "cmsRun"):
#0  0x0000400028d3dc58 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
#1  0x000040005ce3d02c in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libtensorflow_cc.so.2
#2  0x000040005ce3d720 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libtensorflow_cc.so.2
#3  0x000040005ce3abdc in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libtensorflow_cc.so.2
#4  0x000040006719112c in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libtensorflow_framework.so.2
#5  0x0000400028d378b8 in start_thread () from /lib64/libpthread.so.0
#6  0x0000400028d93afc in thread_start () from /lib64/libc.so.6
Thread 19 (Thread 0x400083439250 (LWP 531609) "cmsRun"):
#0  0x0000400028d406d8 in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x0000400028d407f4 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x000040002dc88a70 in XrdCl::JobManager::RunJobs() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdCl.so.3
#3  0x000040002dc88b4c in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdCl.so.3
#4  0x0000400028d378b8 in start_thread () from /lib64/libpthread.so.0
#5  0x0000400028d93afc in thread_start () from /lib64/libc.so.6
Thread 18 (Thread 0x400083229250 (LWP 531608) "cmsRun"):
#0  0x0000400028d406d8 in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x0000400028d407f4 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x000040002dc88a70 in XrdCl::JobManager::RunJobs() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdCl.so.3
#3  0x000040002dc88b4c in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdCl.so.3
#4  0x0000400028d378b8 in start_thread () from /lib64/libpthread.so.0
#5  0x0000400028d93afc in thread_start () from /lib64/libc.so.6
Thread 17 (Thread 0x400083019250 (LWP 531607) "cmsRun"):
#0  0x0000400028d406d8 in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x0000400028d407f4 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x000040002dc88a70 in XrdCl::JobManager::RunJobs() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdCl.so.3
#3  0x000040002dc88b4c in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdCl.so.3
#4  0x0000400028d378b8 in start_thread () from /lib64/libpthread.so.0
#5  0x0000400028d93afc in thread_start () from /lib64/libc.so.6
Thread 16 (Thread 0x400082e09250 (LWP 531606) "cmsRun"):
#0  0x0000400028d41fb0 in nanosleep () from /lib64/libpthread.so.0
#1  0x000040002da51b24 in XrdSysTimer::Wait(int) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#2  0x000040002dc131ac in XrdCl::TaskManager::RunTasks() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdCl.so.3
#3  0x000040002dc1331c in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdCl.so.3
#4  0x0000400028d378b8 in start_thread () from /lib64/libpthread.so.0
#5  0x0000400028d93afc in thread_start () from /lib64/libc.so.6
Thread 15 (Thread 0x400082bf9250 (LWP 531605) "cmsRun"):
#0  0x0000400028e441e8 in epoll_pwait () from /lib64/libc.so.6
#1  0x000040002da4bdec in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#2  0x000040002da47efc in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#3  0x000040002da510e0 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#4  0x0000400028d378b8 in start_thread () from /lib64/libpthread.so.0
#5  0x0000400028d93afc in thread_start () from /lib64/libc.so.6
Thread 14 (Thread 0x4000829e9250 (LWP 531604) "cmsRun"):
#0  0x0000400028e441e8 in epoll_pwait () from /lib64/libc.so.6
#1  0x000040002da4bdec in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#2  0x000040002da47efc in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#3  0x000040002da510e0 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#4  0x0000400028d378b8 in start_thread () from /lib64/libpthread.so.0
#5  0x0000400028d93afc in thread_start () from /lib64/libc.so.6
Thread 13 (Thread 0x4000827d9250 (LWP 531603) "cmsRun"):
#0  0x0000400028e441e8 in epoll_pwait () from /lib64/libc.so.6
#1  0x000040002da4bdec in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#2  0x000040002da47efc in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#3  0x000040002da510e0 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#4  0x0000400028d378b8 in start_thread () from /lib64/libpthread.so.0
#5  0x0000400028d93afc in thread_start () from /lib64/libc.so.6
Thread 12 (Thread 0x4000825c9250 (LWP 531602) "cmsRun"):
#0  0x0000400028e441e8 in epoll_pwait () from /lib64/libc.so.6
#1  0x000040002da4bdec in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#2  0x000040002da47efc in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#3  0x000040002da510e0 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#4  0x0000400028d378b8 in start_thread () from /lib64/libpthread.so.0
#5  0x0000400028d93afc in thread_start () from /lib64/libc.so.6
Thread 11 (Thread 0x4000823b9250 (LWP 531601) "cmsRun"):
#0  0x0000400028e441e8 in epoll_pwait () from /lib64/libc.so.6
#1  0x000040002da4bdec in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#2  0x000040002da47efc in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#3  0x000040002da510e0 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#4  0x0000400028d378b8 in start_thread () from /lib64/libpthread.so.0
#5  0x0000400028d93afc in thread_start () from /lib64/libc.so.6
Thread 10 (Thread 0x4000821a9250 (LWP 531600) "cmsRun"):
#0  0x0000400028e441e8 in epoll_pwait () from /lib64/libc.so.6
#1  0x000040002da4bdec in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#2  0x000040002da47efc in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#3  0x000040002da510e0 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#4  0x0000400028d378b8 in start_thread () from /lib64/libpthread.so.0
#5  0x0000400028d93afc in thread_start () from /lib64/libc.so.6
Thread 9 (Thread 0x400081f99250 (LWP 531599) "cmsRun"):
#0  0x0000400028e441e8 in epoll_pwait () from /lib64/libc.so.6
#1  0x000040002da4bdec in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#2  0x000040002da47efc in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#3  0x000040002da510e0 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#4  0x0000400028d378b8 in start_thread () from /lib64/libpthread.so.0
#5  0x0000400028d93afc in thread_start () from /lib64/libc.so.6
Thread 8 (Thread 0x400081d89250 (LWP 531598) "cmsRun"):
#0  0x0000400028e441e8 in epoll_pwait () from /lib64/libc.so.6
#1  0x000040002da4bdec in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#2  0x000040002da47efc in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#3  0x000040002da510e0 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#4  0x0000400028d378b8 in start_thread () from /lib64/libpthread.so.0
#5  0x0000400028d93afc in thread_start () from /lib64/libc.so.6
Thread 7 (Thread 0x400081b79250 (LWP 531597) "cmsRun"):
#0  0x0000400028e441e8 in epoll_pwait () from /lib64/libc.so.6
#1  0x000040002da4bdec in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#2  0x000040002da47efc in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#3  0x000040002da510e0 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#4  0x0000400028d378b8 in start_thread () from /lib64/libpthread.so.0
#5  0x0000400028d93afc in thread_start () from /lib64/libc.so.6
Thread 6 (Thread 0x400081969250 (LWP 531596) "cmsRun"):
#0  0x0000400028e441e8 in epoll_pwait () from /lib64/libc.so.6
#1  0x000040002da4bdec in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#2  0x000040002da47efc in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#3  0x000040002da510e0 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#4  0x0000400028d378b8 in start_thread () from /lib64/libpthread.so.0
#5  0x0000400028d93afc in thread_start () from /lib64/libc.so.6
Thread 5 (Thread 0x400072a09250 (LWP 531583) "cmsRun"):
#0  0x0000400028e14b3c in nanosleep () from /lib64/libc.so.6
#1  0x0000400028e149e0 in sleep () from /lib64/libc.so.6
#2  0x000040002df1927c in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000400028d937e4 in syscall () from /lib64/libc.so.6
#5  0x0000400028799818 in tbb::detail::r1::futex_wait (comparand=2, futex=0x400029da8024) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x400029da8024) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  0x00004000287ab09c in tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x400029da8020) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x400029da8000) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x400029da8000) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x0000400028d378b8 in start_thread () from /lib64/libpthread.so.0
#11 0x0000400028d93afc in thread_start () from /lib64/libc.so.6
Thread 4 (Thread 0x40006f869250 (LWP 531582) "cmsRun"):
#0  0x0000400028e14b3c in nanosleep () from /lib64/libc.so.6
#1  0x0000400028e149e0 in sleep () from /lib64/libc.so.6
#2  0x000040002df1927c in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000400028d937e4 in syscall () from /lib64/libc.so.6
#5  0x0000400028799818 in tbb::detail::r1::futex_wait (comparand=2, futex=0x400029da8124) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x400029da8124) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  0x00004000287ab09c in tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x400029da8120) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x400029da8100) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x400029da8100) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x0000400028d378b8 in start_thread () from /lib64/libpthread.so.0
#11 0x0000400028d93afc in thread_start () from /lib64/libc.so.6
Thread 3 (Thread 0x40006ee59250 (LWP 531581) "cmsRun"):
#0  0x0000400028e14b3c in nanosleep () from /lib64/libc.so.6
#1  0x0000400028e149e0 in sleep () from /lib64/libc.so.6
#2  0x000040002df1927c in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000400028d937e4 in syscall () from /lib64/libc.so.6
#5  0x0000400028799818 in tbb::detail::r1::futex_wait (comparand=2, futex=0x400029da80a4) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x400029da80a4) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  0x00004000287ab09c in tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x400029da80a0) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x400029da8080) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x400029da8080) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x0000400028d378b8 in start_thread () from /lib64/libpthread.so.0
#11 0x0000400028d93afc in thread_start () from /lib64/libc.so.6
Thread 2 (Thread 0x4000323c9250 (LWP 531548) "cmsRun"):
#0  0x0000400028d42578 in waitpid () from /lib64/libpthread.so.0
#1  0x000040002df193f0 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x000040002df1c550 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  0x0000400028aceb8c in std::execute_native_thread_routine (__p=0x40002b0ab230) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82
#4  0x0000400028d378b8 in start_thread () from /lib64/libpthread.so.0
#5  0x0000400028d93afc in thread_start () from /lib64/libc.so.6
Thread 1 (Thread 0x40002894d0f0 (LWP 531083) "cmsRun"):
#0  0x0000400028e3a91c in poll () from /lib64/libc.so.6
#1  0x000040002df6333c in full_read.constprop () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x000040002df1c5e4 in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  0x000040002df1cf5c in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00004001b1fe0024 in ?? ()
#6  0x000040003bd45b60 in cling::Interpreter::RunFunction(clang::FunctionDecl const*, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCling.so
#7  0x000040003bd45b60 in cling::Interpreter::RunFunction(clang::FunctionDecl const*, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCling.so
#8  0x000040003bd4620c in cling::Interpreter::EvaluateInternal(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, cling::CompilationOptions, cling::Value*, cling::Transaction**, unsigned long) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCling.so
#9  0x000040003be24ad0 in cling::MetaSema::actOnxCommand(llvm::StringRef, llvm::StringRef, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCling.so
#10 0x000040003be31624 in cling::MetaParser::isXCommand(cling::MetaSema::ActionResult&, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCling.so
#11 0x000040003be334d4 in cling::MetaParser::isCommand(cling::MetaSema::ActionResult&, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCling.so
#12 0x000040003be1e55c in cling::MetaProcessor::process(llvm::StringRef, cling::Interpreter::CompilationResult&, cling::Value*, bool) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCling.so
#13 0x000040003bc61d9c in HandleInterpreterException(cling::MetaProcessor*, char const*, cling::Interpreter::CompilationResult&, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCling.so
#14 0x000040003bc748d0 in TCling::ProcessLine(char const*, TInterpreter::EErrorCode*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCling.so
#15 0x000040003bc74e44 in TCling::ProcessLineSynch(char const*, TInterpreter::EErrorCode*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCling.so
#16 0x0000400027ec9048 in TApplication::ExecuteFile(char const*, int*, bool) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCore.so
#17 0x000040003bc73968 in TCling::ExecuteMacro(char const*, TInterpreter::EErrorCode*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCling.so
#18 0x0000400027f1e8ec in TROOT::Macro(char const*, int*, bool) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCore.so
#19 0x0000400027f027d0 in TPluginManager::LoadHandlerMacros(char const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCore.so
#20 0x0000400027f02e2c in TPluginManager::LoadHandlersFromPluginDirs(char const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCore.so
#21 0x0000400027f03ea4 in TPluginManager::FindHandler(char const*, char const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCore.so
#22 0x0000400027737b9c in ROOT::Math::Factory::CreateMinimizer(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libMathCore.so
#23 0x0000400027739560 in ROOT::Fit::FitConfig::CreateMinimizer() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libMathCore.so
#24 0x0000400027750f30 in ROOT::Fit::Fitter::DoInitMinimizer() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libMathCore.so
#25 0x00004000277571dc in bool ROOT::Fit::Fitter::DoMinimization<ROOT::Fit::PoissonLikelihoodFCN<ROOT::Math::IBaseFunctionMultiDimTempl<double>, ROOT::Math::IParametricFunctionMultiDimTempl<double> > >(std::unique_ptr<ROOT::Fit::PoissonLikelihoodFCN<ROOT::Math::IBaseFunctionMultiDimTempl<double>, ROOT::Math::IParametricFunctionMultiDimTempl<double> >, std::default_delete<ROOT::Fit::PoissonLikelihoodFCN<ROOT::Math::IBaseFunctionMultiDimTempl<double>, ROOT::Math::IParametricFunctionMultiDimTempl<double> > > >, ROOT::Math::IBaseFunctionMultiDimTempl<double> const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libMathCore.so
#26 0x00004000277546c0 in ROOT::Fit::Fitter::DoBinnedLikelihoodFit(bool, ROOT::EExecutionPolicy const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libMathCore.so
#27 0x000040002e5c8a4c in TFitResultPtr HFit::Fit<TH1>(TH1*, TF1*, Foption_t&, ROOT::Math::MinimizerOptions const&, char const*, ROOT::Fit::DataRange&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libHist.so
#28 0x000040002e5be194 in ROOT::Fit::FitObject(TH1*, TF1*, Foption_t&, ROOT::Math::MinimizerOptions const&, char const*, ROOT::Fit::DataRange&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libHist.so
#29 0x000040002e689be0 in TH1::Fit(TF1*, char const*, char const*, double, double) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libHist.so
#30 0x000040006ada05fc in PVFitter::runFitter() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/libRecoVertexBeamSpotProducer.so
#31 0x000040006ad97278 in BeamFitter::runPVandTrkFitter() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/libRecoVertexBeamSpotProducer.so
#32 0x00004000c6660ee4 in AlcaBeamMonitor::globalEndLuminosityBlock(edm::LuminosityBlock const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/pluginAlcaBeamMonitor.so
#33 0x00004000c666d9dc in virtual thunk to edm::one::impl::LuminosityBlockCacheHolder<edm::one::EDProducerBase, alcabeammonitor::BeamSpotInfo>::doEndLuminosityBlock_(edm::LuminosityBlock const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/pluginAlcaBeamMonitor.so
#34 0x0000400026d3f144 in edm::one::EDProducerBase::doEndLuminosityBlock(edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#35 0x0000400026d27d00 in edm::WorkerT<edm::one::EDProducerBase>::implDoEnd(edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#36 0x0000400026c7a6c0 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#37 0x0000400026c67acc in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3> >::execute()::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#38 0x0000400026c7e430 in edm::SerialTaskQueue::QueuedTask<edm::SerialTaskQueueChain::push<edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3> >::execute()::{lambda()#1}&>(tbb::detail::d1::task_group&, edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3> >::execute()::{lambda()#1}&)::{lambda()#1}>::execute() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#39 0x00004000272565b0 in tbb::detail::d1::function_task<edm::SerialTaskQueue::spawn(edm::SerialTaskQueue::TaskBase&)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/libFWCoreConcurrency.so
#40 0x00004000287a8774 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=0x400108faff00, this=0x4000296bb300) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#41 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x4000296bb300) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#42 tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:168
#43 0x0000400026c36fc4 in edm::FinalWaitingTask::wait() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#44 0x0000400026c405e8 in edm::EventProcessor::processRuns() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#45 0x0000400026c40b78 in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#46 0x0000000000406fe4 in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#47 0x00004000287a10d8 in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/arena.cpp:688
#48 0x0000000000408854 in main::{lambda()#1}::operator()() const ()
#49 0x0000000000404954 in main ()

Current Modules:

Module: AlcaBeamMonitor:AlcaBeamMonitor (crashed)
Module: none
Module: none
Module: none

@vgvassilev
Copy link
Contributor

Any chance to cook up a ROOT-only reproducer?

@makortel makortel mentioned this issue Mar 17, 2024
Open
@iarspider
Copy link
Contributor

Crash in CMSSW_14_1_X_2024-05-28-1100

Thread 5 (Thread 0x40005eb29220 (LWP 2760826) "cmsRun"):
#0  0x000040001402a91c in poll () from /lib64/libc.so.6
#1  0x000040001943f700 in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02839/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-05-27-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x000040001943f934 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02839/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-05-27-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000400273fe001c in ?? ()
#5  0x0000400027420464 in cling::IncrementalExecutor::executeWrapper(llvm::StringRef, cling::Value*) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02839/el8_aarch64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-05-28-1100/external/el8_aarch64_gcc12/lib/libCling.so
#6  0x0000400027420464 in cling::IncrementalExecutor::executeWrapper(llvm::StringRef, cling::Value*) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02839/el8_aarch64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-05-28-1100/external/el8_aarch64_gcc12/lib/libCling.so
#7  0x0000000000000000 in ?? ()

@aandvalenzuela
Copy link
Contributor

aandvalenzuela commented Jul 30, 2024 •
edited
Loading

Hello,

We see multiple of those crashed in cling reporting two types of stack trace:

  • For el8_aarch64_gcc12:
    • RelVal 140.102, 141.003, 141.042, 141.045 and 141.111 reporting the following stack trace:
    Thread 3 (Thread 0x400069a35220 (LWP 675231) "cmsRun"):
#0  0x00004000213d6954 in poll () from /lib64/libc.so.6
#1  0x00004000278a4c90 in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02848/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-29-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x00004000278a4ec4 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02848/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-29-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000040010bfff01c in ?? ()
#5  0x0000400034ccef10 in cling::Interpreter::RunFunction(clang::FunctionDecl const*, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02848/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-29-2300/external/el8_aarch64_gcc12/lib/libCling.so
#6  0x0000400034ccef10 in cling::Interpreter::RunFunction(clang::FunctionDecl const*, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02848/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-29-2300/external/el8_aarch64_gcc12/lib/libCling.so
#7  0x0000400034cd03ac in cling::Interpreter::EvaluateInternal(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, cling::CompilationOptions, cling::Value*, cling::Transaction**, unsigned long) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02848/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-29-2300/external/el8_aarch64_gcc12/lib/libCling.so
#8  0x0000400034da722c in cling::MetaSema::actOnxCommand(llvm::StringRef, llvm::StringRef, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02848/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-29-2300/external/el8_aarch64_gcc12/lib/libCling.so
#9  0x0000400034db678c in cling::MetaParser::isXCommand(cling::MetaSema::ActionResult&, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02848/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-29-2300/external/el8_aarch64_gcc12/lib/libCling.so
#10 0x0000400034db6d90 in cling::MetaParser::isCommand(cling::MetaSema::ActionResult&, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02848/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-29-2300/external/el8_aarch64_gcc12/lib/libCling.so
#11 0x0000400034da073c in cling::MetaProcessor::process(llvm::StringRef, cling::Interpreter::CompilationResult&, cling::Value*, bool) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02848/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-29-2300/external/el8_aarch64_gcc12/lib/libCling.so
#12 0x0000400034bef6fc in HandleInterpreterException(cling::MetaProcessor*, char const*, cling::Interpreter::CompilationResult&, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02848/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-29-2300/external/el8_aarch64_gcc12/lib/libCling.so
#13 0x0000400034c054dc in TCling::ProcessLine(char const*, TInterpreter::EErrorCode*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02848/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-29-2300/external/el8_aarch64_gcc12/lib/libCling.so
#14 0x0000400034c059f4 in TCling::ProcessLineSynch(char const*, TInterpreter::EErrorCode*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02848/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-29-2300/external/el8_aarch64_gcc12/lib/libCling.so
#15 0x00004000207f6d28 in TApplication::ExecuteFile(char const*, int*, bool) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02848/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-29-2300/external/el8_aarch64_gcc12/lib/libCore.so
#16 0x0000400034bf916c in TCling::ExecuteMacro(char const*, TInterpreter::EErrorCode*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02848/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-29-2300/external/el8_aarch64_gcc12/lib/libCling.so
#17 0x000040002084db5c in TROOT::Macro(char const*, int*, bool) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02848/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-29-2300/external/el8_aarch64_gcc12/lib/libCore.so
#18 0x0000400020831420 in TPluginManager::LoadHandlerMacros(char const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02848/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-29-2300/external/el8_aarch64_gcc12/lib/libCore.so
#19 0x0000400020831a7c in TPluginManager::LoadHandlersFromPluginDirs(char const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02848/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-29-2300/external/el8_aarch64_gcc12/lib/libCore.so
#20 0x0000400020832af4 in TPluginManager::FindHandler(char const*, char const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02848/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-29-2300/external/el8_aarch64_gcc12/lib/libCore.so
#21 0x000040001fed7d40 in ROOT::Math::Factory::CreateMinimizer(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02848/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-29-2300/external/el8_aarch64_gcc12/lib/libMathCore.so
#22 0x000040001fed9948 in ROOT::Fit::FitConfig::CreateMinimizer() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02848/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-29-2300/external/el8_aarch64_gcc12/lib/libMathCore.so
#23 0x000040001fef484c in ROOT::Fit::Fitter::DoInitMinimizer() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02848/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-29-2300/external/el8_aarch64_gcc12/lib/libMathCore.so
#24 0x000040001fefa45c in bool ROOT::Fit::Fitter::DoMinimization<ROOT::Fit::PoissonLikelihoodFCN<ROOT::Math::IBaseFunctionMultiDimTempl<double>, ROOT::Math::IParametricFunctionMultiDimTempl<double> > >(std::unique_ptr<ROOT::Fit::PoissonLikelihoodFCN<ROOT::Math::IBaseFunctionMultiDimTempl<double>, ROOT::Math::IParametricFunctionMultiDimTempl<double> >, std::default_delete<ROOT::Fit::PoissonLikelihoodFCN<ROOT::Math::IBaseFunctionMultiDimTempl<double>, ROOT::Math::IParametricFunctionMultiDimTempl<double> > > >, ROOT::Math::IBaseFunctionMultiDimTempl<double> const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02848/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-29-2300/external/el8_aarch64_gcc12/lib/libMathCore.so
#25 0x000040001fef7770 in ROOT::Fit::Fitter::DoBinnedLikelihoodFit(bool, ROOT::EExecutionPolicy const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02848/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-29-2300/external/el8_aarch64_gcc12/lib/libMathCore.so
#26 0x0000400027bcc044 in TFitResultPtr HFit::Fit<TH1>(TH1*, TF1*, Foption_t&, ROOT::Math::MinimizerOptions const&, char const*, ROOT::Fit::DataRange&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02848/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-29-2300/external/el8_aarch64_gcc12/lib/libHist.so
#27 0x0000400027bc31d4 in ROOT::Fit::FitObject(TH1*, TF1*, Foption_t&, ROOT::Math::MinimizerOptions const&, char const*, ROOT::Fit::DataRange&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02848/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-29-2300/external/el8_aarch64_gcc12/lib/libHist.so
#28 0x0000400027c90ee0 in TH1::Fit(TF1*, char const*, char const*, double, double) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02848/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-29-2300/external/el8_aarch64_gcc12/lib/libHist.so
#29 0x0000400065703930 in PVFitter::runFitter() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02848/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-29-2300/lib/el8_aarch64_gcc12/libRecoVertexBeamSpotProducer.so
#30 0x00004000656fb098 in BeamFitter::runPVandTrkFitter() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02848/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-29-2300/lib/el8_aarch64_gcc12/libRecoVertexBeamSpotProducer.so
#31 0x000040009e4b7450 in AlcaBeamMonitor::globalEndLuminosityBlock(edm::LuminosityBlock const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02848/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-29-2300/lib/el8_aarch64_gcc12/pluginAlcaBeamMonitor.so
#32 0x000040009e4c0f4c in virtual thunk to edm::one::impl::LuminosityBlockCacheHolder<edm::one::EDProducerBase, alcabeammonitor::BeamSpotInfo>::doEndLuminosityBlock_(edm::LuminosityBlock const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02848/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-29-2300/lib/el8_aarch64_gcc12/pluginAlcaBeamMonitor.so
#33 0x000040001f6296e4 in edm::one::EDProducerBase::doEndLuminosityBlock(edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02848/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-29-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#34 0x000040001f616c00 in edm::WorkerT<edm::one::EDProducerBase>::implDoEnd(edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02848/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-29-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#35 0x000040001f558348 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02848/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-29-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#36 0x000040001f55ba30 in edm::SerialTaskQueue::QueuedTask<edm::SerialTaskQueueChain::push<edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3> >::execute()::{lambda()#1}&>(tbb::detail::d1::task_group&, edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3> >::execute()::{lambda()#1}&)::{lambda()#1}>::execute() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02848/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-29-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#37 0x000040001f899728 in tbb::detail::d1::function_task<edm::SerialTaskQueue::spawn(edm::SerialTaskQueue::TaskBase&)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02848/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-29-2300/lib/el8_aarch64_gcc12/libFWCoreConcurrency.so
#38 0x00004000201ac79c in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x4000e7133700, waiter=<synthetic pointer>..., this=0x400021bab400) at /data/cmsbld/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-cd51b72bd1dc6777edf83e38784c1702/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#39 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (waiter=<synthetic pointer>..., t=0x0, this=0x400021bab400) at /data/cmsbld/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-cd51b72bd1dc6777edf83e38784c1702/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#40 tbb::detail::r1::arena::process (tls=..., this=0x400021baad00) at /data/cmsbld/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-cd51b72bd1dc6777edf83e38784c1702/tbb-v2021.9.0/src/tbb/arena.cpp:137
#41 tbb::detail::r1::market::process (this=0x400021ba7080, j=...) at /data/cmsbld/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-cd51b72bd1dc6777edf83e38784c1702/tbb-v2021.9.0/src/tbb/market.cpp:599
#42 0x00004000201b4ffc in tbb::detail::r1::rml::private_worker::run (this=0x400026efc080) at /data/cmsbld/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-cd51b72bd1dc6777edf83e38784c1702/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#43 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x400026efc080) at /data/cmsbld/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-cd51b72bd1dc6777edf83e38784c1702/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#44 0x00004000212de8b8 in start_thread () from /lib64/libpthread.so.0
#45 0x000040002132fafc in thread_start () from /lib64/libc.so.6
Thread 2 (Thread 0x4000295ac220 (LWP 674946) "cmsRun"):
#0  0x00004000212e9580 in waitpid () from /lib64/libpthread.so.0
#1  0x00004000278a1e24 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02848/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-29-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x00004000278a4a78 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02848/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-29-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  0x00004000210ceb8c in std::execute_native_thread_routine (__p=0x4000230ac790) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82
#4  0x00004000212de8b8 in start_thread () from /lib64/libpthread.so.0
#5  0x000040002132fafc in thread_start () from /lib64/libc.so.6
Thread 1 (Thread 0x40001f3f26b0 (LWP 671459) "cmsRun"):
#0  0x00004000213b0b3c in nanosleep () from /lib64/libc.so.6
#1  0x00004000213b09e0 in sleep () from /lib64/libc.so.6
#2  0x00004000278a1cb8 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02848/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-29-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000040002132f7e4 in syscall () from /lib64/libc.so.6
#5  0x00004000201b2ff8 in tbb::detail::r1::futex_wait (comparand=2, futex=0xfffff21e1318) at /data/cmsbld/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-cd51b72bd1dc6777edf83e38784c1702/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0xfffff21e1318) at /data/cmsbld/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-cd51b72bd1dc6777edf83e38784c1702/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::sleep_node<tbb::detail::r1::market_context>::wait (this=0xfffff21e12e8) at /data/cmsbld/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-cd51b72bd1dc6777edf83e38784c1702/tbb-v2021.9.0/src/tbb/concurrent_monitor.h:170
#8  tbb::detail::r1::concurrent_monitor_base<tbb::detail::r1::market_context>::commit_wait (this=<optimized out>, node=...) at /data/cmsbld/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-cd51b72bd1dc6777edf83e38784c1702/tbb-v2021.9.0/src/tbb/concurrent_monitor.h:232
#9  tbb::detail::r1::concurrent_monitor_base<tbb::detail::r1::market_context>::commit_wait (node=..., this=0x400021ba7098) at /data/cmsbld/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-cd51b72bd1dc6777edf83e38784c1702/tbb-v2021.9.0/src/tbb/concurrent_monitor.h:228
#10 tbb::detail::r1::concurrent_monitor_base<tbb::detail::r1::market_context>::wait<tbb::detail::r1::sleep_node<tbb::detail::r1::market_context>, tbb::detail::r1::external_waiter::pause(tbb::detail::r1::arena_slot&)::{lambda()#1}&>(tbb::detail::r1::external_waiter::pause(tbb::detail::r1::arena_slot&)::{lambda()#1}&, tbb::detail::r1::sleep_node<tbb::detail::r1::market_context>&&) (node=..., pred=..., this=<optimized out>) at /data/cmsbld/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-cd51b72bd1dc6777edf83e38784c1702/tbb-v2021.9.0/src/tbb/concurrent_monitor.h:262
#11 tbb::detail::r1::sleep_waiter::sleep<tbb::detail::r1::external_waiter::pause(tbb::detail::r1::arena_slot&)::{lambda()#1}>(unsigned long, tbb::detail::r1::external_waiter::pause(tbb::detail::r1::arena_slot&)::{lambda()#1}) (this=0xfffff21e1290, wakeup_condition=..., uniq_tag=70369489793536) at /data/cmsbld/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-cd51b72bd1dc6777edf83e38784c1702/tbb-v2021.9.0/src/tbb/waiters.h:118
#12 tbb::detail::r1::external_waiter::pause (this=0xfffff21e1290) at /data/cmsbld/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-cd51b72bd1dc6777edf83e38784c1702/tbb-v2021.9.0/src/tbb/waiters.h:144
#13 tbb::detail::r1::external_waiter::pause (this=<optimized out>) at /data/cmsbld/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-cd51b72bd1dc6777edf83e38784c1702/tbb-v2021.9.0/src/tbb/waiters.h:137
#14 tbb::detail::r1::task_dispatcher::receive_or_steal_task<false, tbb::detail::r1::external_waiter> (critical_allowed=true, fifo_allowed=true, isolation=0, waiter=..., ed=..., tls=..., this=0x400021bab300) at /data/cmsbld/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-cd51b72bd1dc6777edf83e38784c1702/tbb-v2021.9.0/src/tbb/task_dispatcher.h:231
#15 tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=0x0, this=0x400021bab300) at /data/cmsbld/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-cd51b72bd1dc6777edf83e38784c1702/tbb-v2021.9.0/src/tbb/task_dispatcher.h:350
#16 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x400021bab300) at /data/cmsbld/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-cd51b72bd1dc6777edf83e38784c1702/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#17 tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-cd51b72bd1dc6777edf83e38784c1702/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:168
#18 0x000040001f517544 in edm::FinalWaitingTask::wait() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02848/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-29-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#19 0x000040001f5241bc in edm::EventProcessor::processRuns() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02848/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-29-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#20 0x000040001f52465c in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02848/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-29-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#21 0x0000000000407afc in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#22 0x00004000201aaa68 in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-cd51b72bd1dc6777edf83e38784c1702/tbb-v2021.9.0/src/tbb/arena.cpp:688
#23 0x0000000000409398 in main::{lambda()#1}::operator()() const ()
#24 0x0000000000404524 in main ()

Current Modules:

Module: AlcaBeamMonitor:AlcaBeamMonitor (crashed)
Module: none
Module: none
Module: none
    • RelVals 140.104, 141.008, 141.034 show cling failures on cling::IncrementalExecutor::executeWrapper only:
    Thread 4 (Thread 0x400052d69220 (LWP 481699) "cmsRun"):
#0  0x0000400009da1954 in poll () from /lib64/libc.so.6
#1  0x0000400010272c90 in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02848/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-29-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x0000400010272ec4 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02848/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-29-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000400102fff01c in ?? ()
#5  0x000040001d744854 in cling::IncrementalExecutor::executeWrapper(llvm::StringRef, cling::Value*) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02848/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-29-2300/external/el8_aarch64_gcc12/lib/libCling.so
#6  0x000040001d744854 in cling::IncrementalExecutor::executeWrapper(llvm::StringRef, cling::Value*) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02848/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-29-2300/external/el8_aarch64_gcc12/lib/libCling.so
#7  0x0000000000000000 in ?? ()
Thread 3 (Thread 0x400052368220 (LWP 481698) "cmsRun"):
#0  0x0000400009d7bb3c in nanosleep () from /lib64/libc.so.6
#1  0x0000400009d7b9e0 in sleep () from /lib64/libc.so.6
#2  0x000040001026fcb8 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02848/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-29-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000400009cfa7e4 in syscall () from /lib64/libc.so.6
#5  0x0000400008b82858 in tbb::detail::r1::futex_wait (comparand=2, futex=0x40000acc20a4) at /data/cmsbld/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-cd51b72bd1dc6777edf83e38784c1702/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x40000acc20a4) at /data/cmsbld/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-cd51b72bd1dc6777edf83e38784c1702/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  0x0000400008b9416c in tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x40000acc20a0) at /data/cmsbld/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-cd51b72bd1dc6777edf83e38784c1702/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x40000acc2080) at /data/cmsbld/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-cd51b72bd1dc6777edf83e38784c1702/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x40000acc2080) at /data/cmsbld/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-cd51b72bd1dc6777edf83e38784c1702/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x00004000095c18b8 in start_thread () from /lib64/libpthread.so.0
#11 0x0000400009cfaafc in thread_start () from /lib64/libc.so.6
Thread 2 (Thread 0x40001284f220 (LWP 481283) "cmsRun"):
#0  0x00004000095cc580 in waitpid () from /lib64/libpthread.so.0
#1  0x000040001026fe24 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02848/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-29-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x0000400010272a78 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02848/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-29-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  0x0000400009aceb8c in std::execute_native_thread_routine (__p=0x40000c0ac790) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82
#4  0x00004000095c18b8 in start_thread () from /lib64/libpthread.so.0
#5  0x0000400009cfaafc in thread_start () from /lib64/libc.so.6
Thread 1 (Thread 0x400008fe8010 (LWP 477370) "cmsRun"):
#0  0x0000400009d7bb3c in nanosleep () from /lib64/libc.so.6
#1  0x0000400009d7b9e0 in sleep () from /lib64/libc.so.6
#2  0x000040001026fcb8 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02848/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-29-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000400009cfa7e4 in syscall () from /lib64/libc.so.6
#5  0x0000400008b91ff8 in tbb::detail::r1::futex_wait (comparand=2, futex=0xffffc3a92d28) at /data/cmsbld/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-cd51b72bd1dc6777edf83e38784c1702/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0xffffc3a92d28) at /data/cmsbld/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-cd51b72bd1dc6777edf83e38784c1702/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::sleep_node<tbb::detail::r1::market_context>::wait (this=0xffffc3a92cf8) at /data/cmsbld/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-cd51b72bd1dc6777edf83e38784c1702/tbb-v2021.9.0/src/tbb/concurrent_monitor.h:170
#8  tbb::detail::r1::concurrent_monitor_base<tbb::detail::r1::market_context>::commit_wait (this=<optimized out>, node=...) at /data/cmsbld/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-cd51b72bd1dc6777edf83e38784c1702/tbb-v2021.9.0/src/tbb/concurrent_monitor.h:232
#9  tbb::detail::r1::concurrent_monitor_base<tbb::detail::r1::market_context>::commit_wait (node=..., this=0x40000a54f098) at /data/cmsbld/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-cd51b72bd1dc6777edf83e38784c1702/tbb-v2021.9.0/src/tbb/concurrent_monitor.h:228
#10 tbb::detail::r1::concurrent_monitor_base<tbb::detail::r1::market_context>::wait<tbb::detail::r1::sleep_node<tbb::detail::r1::market_context>, tbb::detail::r1::external_waiter::pause(tbb::detail::r1::arena_slot&)::{lambda()#1}&>(tbb::detail::r1::external_waiter::pause(tbb::detail::r1::arena_slot&)::{lambda()#1}&, tbb::detail::r1::sleep_node<tbb::detail::r1::market_context>&&) (node=..., pred=..., this=<optimized out>) at /data/cmsbld/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-cd51b72bd1dc6777edf83e38784c1702/tbb-v2021.9.0/src/tbb/concurrent_monitor.h:262
#11 tbb::detail::r1::sleep_waiter::sleep<tbb::detail::r1::external_waiter::pause(tbb::detail::r1::arena_slot&)::{lambda()#1}>(unsigned long, tbb::detail::r1::external_waiter::pause(tbb::detail::r1::arena_slot&)::{lambda()#1}) (this=0xffffc3a92ca0, wakeup_condition=..., uniq_tag=70369095619072) at /data/cmsbld/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-cd51b72bd1dc6777edf83e38784c1702/tbb-v2021.9.0/src/tbb/waiters.h:118
#12 tbb::detail::r1::external_waiter::pause (this=0xffffc3a92ca0) at /data/cmsbld/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-cd51b72bd1dc6777edf83e38784c1702/tbb-v2021.9.0/src/tbb/waiters.h:144
#13 tbb::detail::r1::external_waiter::pause (this=<optimized out>) at /data/cmsbld/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-cd51b72bd1dc6777edf83e38784c1702/tbb-v2021.9.0/src/tbb/waiters.h:137
#14 tbb::detail::r1::task_dispatcher::receive_or_steal_task<false, tbb::detail::r1::external_waiter> (critical_allowed=true, fifo_allowed=true, isolation=0, waiter=..., ed=..., tls=..., this=0x40000a553300) at /data/cmsbld/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-cd51b72bd1dc6777edf83e38784c1702/tbb-v2021.9.0/src/tbb/task_dispatcher.h:231
#15 tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=0x0, this=0x40000a553300) at /data/cmsbld/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-cd51b72bd1dc6777edf83e38784c1702/tbb-v2021.9.0/src/tbb/task_dispatcher.h:350
#16 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x40000a553300) at /data/cmsbld/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-cd51b72bd1dc6777edf83e38784c1702/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#17 tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-cd51b72bd1dc6777edf83e38784c1702/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:168
#18 0x0000400007d17544 in edm::FinalWaitingTask::wait() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02848/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-29-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#19 0x0000400007d241bc in edm::EventProcessor::processRuns() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02848/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-29-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#20 0x0000400007d2465c in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02848/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-29-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#21 0x0000000000407afc in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#22 0x0000400008b89a68 in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-cd51b72bd1dc6777edf83e38784c1702/tbb-v2021.9.0/src/tbb/arena.cpp:688
#23 0x0000000000409398 in main::{lambda()#1}::operator()() const ()
#24 0x0000000000404524 in main ()

Current Modules:

Module: AlcaBeamMonitor:AlcaBeamMonitor (crashed)
Module: none
Module: none
Module: none
  • For el9_aarch64_gcc12 we also see these two types of stack traces:
    • RelVal 29834.99 fails with full stack trace. See complete stack trace.
    • RelVal 11846.0 fails in cling::IncrementalExecutor::executeWrapper only. See complete stack trace.

@dan131riley dan131riley mentioned this issue Aug 9, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
core-pending pending-signatures root
Projects
ROOT prioritization
Status: Needs debugging in ROOT
Milestone
No milestone
Development

No branches or pull requests
7 participants
@makortel @iarspider @smuzaffar @cmsbuild @pcanal @vgvassilev @aandvalenzuela