Segmentation violation in workflows 33034.0 and 33434.0 (geometry D72)

[silviodonato]

In CMSSW_11_2_X_2020-10-21-2300, two worklows give seg violation

33034.0
TTbar_14TeV+2026D72+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+DigiTrigger+RecoGlobal+HARVESTGlobal
33434.0
TTbar_14TeV+2026D73+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+DigiTrigger+RecoGlobal+HARVESTGlobal

Module: LXXXCorrectorProducer:ak4CaloL3AbsoluteCorrector (crashed)

cmsDriver.py step3 --conditions auto:phase2_realistic_T21 -s RAW2DIGI,L1Reco,RECO,RECOSIM,PAT,VALIDATION:@phase2Validation+@miniAODValidation,DQM:@phase2+@miniAODDQM --datatier GEN-SIM-RECO,MINIAODSIM,DQMIO -n 10 --geometry Extended2026D72 --era Phase2C11 --eventcontent FEVTDEBUGHLT,MINIAODSIM,DQM --customise Validation/Performance/TimeMemorySummary.customiseWithTimeMemorySummary --prefix '/data/cmsbld/jenkins/workspace/ib-run-relvals/cms-bot/monitor_workflow.py timeout --signal SIGTERM 9000 ' --filein file:step2.root --fileout file:step3.root --suffix "-j JobReport3.xml " --nThreads 4 > step3_TTbar_14TeV+2026D72+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+DigiTrigger+RecoGlobal+HARVESTGlobal.log 2>&1

https://cmssdt.cern.ch/SDT/cgi-bin/logreader/slc7_amd64_gcc820/CMSSW_11_2_X_2020-10-21-2300/pyRelValMatrixLogs/run/33034.0_TTbar_14TeV+2026D72+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+DigiTrigger+RecoGlobal+HARVESTGlobal/step3_TTbar_14TeV+2026D72+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+DigiTrigger+RecoGlobal+HARVESTGlobal.log#/64-64

Running single thread I got

Thu Oct 22 10:52:43 CEST 2020
Thread 5 (Thread 0x7f2e667ff700 (LWP 24312)):
#0  0x00007f2ed3f95a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2ed457862c in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_11_1_0_pre7-slc7_amd64_gcc820/build/CMSSW_11_1_0_pre7-build/BUILD/slc7_amd64_gcc820/external/gcc/8.2.0-bcolbf/gcc-8.2.0/obj/x86_64-unknown-linux-gnu/libstdc++-v3/include/x86_64-unknown-linux-gnu/bits/gthr-default.h:864
#2  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:53
#3  0x00007f2e71bfec1c in Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc820/cms/cmssw-patch/CMSSW_11_2_X_2020-10-21-2300/external/slc7_amd64_gcc820/lib/libtensorflow_framework.so.2
#4  0x00007f2e71bfbb43 in std::_Function_handler<void (), tensorflow::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc820/cms/cmssw-patch/CMSSW_11_2_X_2020-10-21-2300/external/slc7_amd64_gcc820/lib/libtensorflow_framework.so.2
#5  0x00007f2ed457dd2f in std::execute_native_thread_routine (__p=0x7f2e7f868180) at ../../../../../libstdc++-v3/src/c++11/thread.cc:80
#6  0x00007f2ed3f91ea5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f2ed3cba8dd in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f2e675ff700 (LWP 24311)):
#0  0x00007f2ed3f95a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2ed457862c in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_11_1_0_pre7-slc7_amd64_gcc820/build/CMSSW_11_1_0_pre7-build/BUILD/slc7_amd64_gcc820/external/gcc/8.2.0-bcolbf/gcc-8.2.0/obj/x86_64-unknown-linux-gnu/libstdc++-v3/include/x86_64-unknown-linux-gnu/bits/gthr-default.h:864
#2  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:53
#3  0x00007f2e71bfec1c in Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc820/cms/cmssw-patch/CMSSW_11_2_X_2020-10-21-2300/external/slc7_amd64_gcc820/lib/libtensorflow_framework.so.2
#4  0x00007f2e71bfbb43 in std::_Function_handler<void (), tensorflow::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc820/cms/cmssw-patch/CMSSW_11_2_X_2020-10-21-2300/external/slc7_amd64_gcc820/lib/libtensorflow_framework.so.2
#5  0x00007f2ed457dd2f in std::execute_native_thread_routine (__p=0x7f2e68d9d6b0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:80
#6  0x00007f2ed3f91ea5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f2ed3cba8dd in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f2e68209700 (LWP 24310)):
#0  0x00007f2ed3f95a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2ed457862c in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_11_1_0_pre7-slc7_amd64_gcc820/build/CMSSW_11_1_0_pre7-build/BUILD/slc7_amd64_gcc820/external/gcc/8.2.0-bcolbf/gcc-8.2.0/obj/x86_64-unknown-linux-gnu/libstdc++-v3/include/x86_64-unknown-linux-gnu/bits/gthr-default.h:864
#2  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:53
#3  0x00007f2e71bfec1c in Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc820/cms/cmssw-patch/CMSSW_11_2_X_2020-10-21-2300/external/slc7_amd64_gcc820/lib/libtensorflow_framework.so.2
#4  0x00007f2e71bfbb43 in std::_Function_handler<void (), tensorflow::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc820/cms/cmssw-patch/CMSSW_11_2_X_2020-10-21-2300/external/slc7_amd64_gcc820/lib/libtensorflow_framework.so.2
#5  0x00007f2ed457dd2f in std::execute_native_thread_routine (__p=0x7f2e68d9d590) at ../../../../../libstdc++-v3/src/c++11/thread.cc:80
#6  0x00007f2ed3f91ea5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f2ed3cba8dd in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f2eaa29b700 (LWP 23679)):
#0  0x00007f2ed3f991d9 in waitpid () from /lib64/libpthread.so.0
#1  0x00007f2ec776ffd7 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc820/cms/cmssw-patch/CMSSW_11_2_X_2020-10-21-2300/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#2  0x00007f2ec7770a9a in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc820/cms/cmssw-patch/CMSSW_11_2_X_2020-10-21-2300/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  0x00007f2ed457dd2f in std::execute_native_thread_routine (__p=0x7f2ec951d1f0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:80
#4  0x00007f2ed3f91ea5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f2ed3cba8dd in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f2ed22a3540 (LWP 22477)):
#0  0x00007f2ed3cafc3d in poll () from /lib64/libc.so.6
#1  0x00007f2ec777043f in full_read.constprop () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc820/cms/cmssw-patch/CMSSW_11_2_X_2020-10-21-2300/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#2  0x00007f2ec7770b7c in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc820/cms/cmssw-patch/CMSSW_11_2_X_2020-10-21-2300/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  0x00007f2ec7771a59 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc820/cms/cmssw-patch/CMSSW_11_2_X_2020-10-21-2300/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007f2ea012364a in MTDRingForwardLayer::MTDRingForwardLayer(std::vector<ForwardDetRing const*, std::allocator<ForwardDetRing const*> > const&) () from /cvmfs/cms-ib.cern.ch/nweek-02651/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_X_2020-10-21-1100/lib/slc7_amd64_gcc820/libRecoMTDDetLayers.so
#6  0x00007f2ea0120f1d in MTDRingForwardDoubleLayer::MTDRingForwardDoubleLayer(std::vector<ForwardDetRing const*, std::allocator<ForwardDetRing const*> > const&, std::vector<ForwardDetRing const*, std::allocator<ForwardDetRing const*> > const&) () from /cvmfs/cms-ib.cern.ch/nweek-02651/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_X_2020-10-21-1100/lib/slc7_amd64_gcc820/libRecoMTDDetLayers.so
#7  0x00007f2e9dd17eda in ETLDetLayerGeometryBuilder::buildLayer(int, int, std::vector<unsigned int, std::allocator<unsigned int> >&, MTDGeometry const&) () from /cvmfs/cms-ib.cern.ch/nweek-02651/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_X_2020-10-21-1100/lib/slc7_amd64_gcc820/pluginRecoMTDDetLayersPlugins.so
#8  0x00007f2e9dd18080 in ETLDetLayerGeometryBuilder::buildLayers(MTDGeometry const&) () from /cvmfs/cms-ib.cern.ch/nweek-02651/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_X_2020-10-21-1100/lib/slc7_amd64_gcc820/pluginRecoMTDDetLayersPlugins.so
#9  0x00007f2e9dd19127 in MTDDetLayerGeometryESProducer::produce(MTDRecoGeometryRecord const&) () from /cvmfs/cms-ib.cern.ch/nweek-02651/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_X_2020-10-21-1100/lib/slc7_amd64_gcc820/pluginRecoMTDDetLayersPlugins.so
#10 0x00007f2e9dd1d76d in decltype ({parm#1}()) edm::convertException::wrap<edm::eventsetup::Callback<MTDDetLayerGeometryESProducer, std::unique_ptr<MTDDetLayerGeometry, std::default_delete<MTDDetLayerGeometry> >, MTDRecoGeometryRecord, edm::eventsetup::CallbackSimpleDecorator<MTDRecoGeometryRecord> >::runProducerAsync(std::__exception_ptr::exception_ptr const*, edm::eventsetup::EventSetupRecordImpl const*, edm::EventSetupImpl const*, edm::ServiceToken const&)::{lambda()#1}::operator()() const::{lambda()#1}>(edm::eventsetup::Callback<MTDDetLayerGeometryESProducer, std::unique_ptr<MTDDetLayerGeometry, std::default_delete<MTDDetLayerGeometry> >, MTDRecoGeometryRecord, edm::eventsetup::CallbackSimpleDecorator<MTDRecoGeometryRecord> >::runProducerAsync(std::__exception_ptr::exception_ptr const*, edm::eventsetup::EventSetupRecordImpl const*, edm::EventSetupImpl const*, edm::ServiceToken const&)::{lambda()#1}::operator()() const::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02651/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_X_2020-10-21-1100/lib/slc7_amd64_gcc820/pluginRecoMTDDetLayersPlugins.so
#11 0x00007f2e9dd1d9c1 in edm::eventsetup::Callback<MTDDetLayerGeometryESProducer, std::unique_ptr<MTDDetLayerGeometry, std::default_delete<MTDDetLayerGeometry> >, MTDRecoGeometryRecord, edm::eventsetup::CallbackSimpleDecorator<MTDRecoGeometryRecord> >::runProducerAsync(std::__exception_ptr::exception_ptr const*, edm::eventsetup::EventSetupRecordImpl const*, edm::EventSetupImpl const*, edm::ServiceToken const&)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/nweek-02651/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_X_2020-10-21-1100/lib/slc7_amd64_gcc820/pluginRecoMTDDetLayersPlugins.so
#12 0x00007f2e9dd1e9ff in void edm::SerialTaskQueueChain::actionToRun<edm::eventsetup::Callback<MTDDetLayerGeometryESProducer, std::unique_ptr<MTDDetLayerGeometry, std::default_delete<MTDDetLayerGeometry> >, MTDRecoGeometryRecord, edm::eventsetup::CallbackSimpleDecorator<MTDRecoGeometryRecord> >::runProducerAsync(std::__exception_ptr::exception_ptr const*, edm::eventsetup::EventSetupRecordImpl const*, edm::EventSetupImpl const*, edm::ServiceToken const&)::{lambda()#1}&>(edm::eventsetup::Callback<MTDDetLayerGeometryESProducer, std::unique_ptr<MTDDetLayerGeometry, std::default_delete<MTDDetLayerGeometry> >, MTDRecoGeometryRecord, edm::eventsetup::CallbackSimpleDecorator<MTDRecoGeometryRecord> >::runProducerAsync(std::__exception_ptr::exception_ptr const*, edm::eventsetup::EventSetupRecordImpl const*, edm::EventSetupImpl const*, edm::ServiceToken const&)::{lambda()#1}&) () from /cvmfs/cms-ib.cern.ch/nweek-02651/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_X_2020-10-21-1100/lib/slc7_amd64_gcc820/pluginRecoMTDDetLayersPlugins.so
#13 0x00007f2e9dd1ea71 in edm::SerialTaskQueue::QueuedTask<edm::SerialTaskQueueChain::push<edm::eventsetup::Callback<MTDDetLayerGeometryESProducer, std::unique_ptr<MTDDetLayerGeometry, std::default_delete<MTDDetLayerGeometry> >, MTDRecoGeometryRecord, edm::eventsetup::CallbackSimpleDecorator<MTDRecoGeometryRecord> >::runProducerAsync(std::__exception_ptr::exception_ptr const*, edm::eventsetup::EventSetupRecordImpl const*, edm::EventSetupImpl const*, edm::ServiceToken const&)::{lambda()#1}>(edm::eventsetup::Callback<MTDDetLayerGeometryESProducer, std::unique_ptr<MTDDetLayerGeometry, std::default_delete<MTDDetLayerGeometry> >, MTDRecoGeometryRecord, edm::eventsetup::CallbackSimpleDecorator<MTDRecoGeometryRecord> >::runProducerAsync(std::__exception_ptr::exception_ptr const*, edm::eventsetup::EventSetupRecordImpl const*, edm::EventSetupImpl const*, edm::ServiceToken const&)::{lambda()#1}&&)::{lambda()#1}>::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02651/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_X_2020-10-21-1100/lib/slc7_amd64_gcc820/pluginRecoMTDDetLayersPlugins.so
#14 0x00007f2ed4f86bfd in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::process_bypass_loop (this=this@entry=0x7f2ed0d5b200, context_guard=..., t=t@entry=0x7f2d893c1e40, isolation=isolation@entry=0) at ../../src/tbb/custom_scheduler.h:393
#15 0x00007f2ed4f86ef5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f2ed0d5b200, parent=..., child=<optimized out>) at ../../include/tbb/task.h:1003
#16 0x00007f2ed6818555 in tbb::interface7::internal::delegated_function<edm::Worker::esPrefetchAsync(edm::WaitingTask*, edm::EventSetupImpl const&, edm::Transition, edm::ServiceToken const&)::{lambda()#1}::operator()()::{lambda()#1} const, void>::operator()() const () from /cvmfs/cms-ib.cern.ch/nweek-02651/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_X_2020-10-21-1100/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#17 0x00007f2ed4f81bc1 in tbb::interface7::internal::task_arena_base::internal_execute (this=0x7f2ed6913fe0 <edm::esTaskArena()::s_arena>, d=...) at ../../src/tbb/arena.cpp:1105
#18 0x00007f2ed6817d72 in edm::FunctorTask<edm::Worker::esPrefetchAsync(edm::WaitingTask*, edm::EventSetupImpl const&, edm::Transition, edm::ServiceToken const&)::{lambda()#1}>::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02651/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_X_2020-10-21-1100/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#19 0x00007f2ed4f86bfd in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::process_bypass_loop (this=this@entry=0x7f2ed0d5b200, context_guard=..., t=t@entry=0x7f2d893d5140, isolation=isolation@entry=0) at ../../src/tbb/custom_scheduler.h:393
#20 0x00007f2ed4f86ef5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f2ed0d5b200, parent=..., child=<optimized out>) at ../../include/tbb/task.h:1003
#21 0x00007f2ed670f355 in edm::EventProcessor::processLumis(std::shared_ptr<void> const&) () from /cvmfs/cms-ib.cern.ch/nweek-02651/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_X_2020-10-21-1100/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#22 0x00007f2ed671749e in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/nweek-02651/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_X_2020-10-21-1100/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#23 0x000000000040f59d in tbb::interface7::internal::delegated_function<main::{lambda()#1}::operator()() const::{lambda()#1} const, void>::operator()() const ()
#24 0x00007f2ed4f81bc1 in tbb::interface7::internal::task_arena_base::internal_execute (this=0x7ffdbf961f30, d=...) at ../../src/tbb/arena.cpp:1105
#25 0x0000000000410561 in main::{lambda()#1}::operator()() const ()
#26 0x000000000040eff5 in main ()

Current Modules:

Module: none (crashed)

A fatal system signal has occurred: segmentation violation

[cmsbuild]

A new Issue was created by @silviodonato Silvio Donato.

@Dr15Jones, @dpiparo, @silviodonato, @smuzaffar, @makortel, @qliphy can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[silviodonato]

assign reconstruction
(actually I'm not sure about the origin of the problem)

PRs merged
#31314 from JanFSchulte: Integrating VectorHits reconstruction for the Phase2 OT - reprise
#31700 from guitargeek: Adapt EgammaHadTower and ElectronHcalHelper classes to use ESGetToken
#31859 from Dr15Jones: Removed unused fwkJobReports from MessageLogger in HLTrigger Subsystem
#31668 from mandrenguyen: Bad PF candidate filter for 2018 PbPb data
#31836 from makortel: Complete EventSetup-consumes migration for {Ecal,Hcal}HaloDataProducer
#31842 from bsunanda: Phase2-gex25 Add a new scenario 2026D74 with V14 HGCal and GE0
#31788 from mmusich: move from MagneticField_AutoFromDBCurrent_cff to MagneticField in all Tracker-related configurations
#31832 from bsunanda: Run3-gex24 Try to sort out the issue of SimHit for muons using dd4hep
#31808 from namapane: Update DT vdrift calibration code to allow using "new" DB format
#31865 from Dr15Jones: Updated MessageLogger tests to remove obsolete usages
#31876 from bsunanda: Run4-hgx265 Update tests for ESGetToken and also for checking content in geometry
#31885 from ianna: [DD4hep] Re-introduce namespace in Ecal barrel
#31881 from jeongeun: drop type specs in RecoHGCal
#31883 from bsunanda: Run3-gex25 Add a few sens.xml with expanded list of sensitive volumes
#31882 from bsunanda: Run3-gex26 Correct for preshower numbering scheme

[cmsbuild]

New categories assigned: reconstruction

@slava77,@perrotta,@jpata you have been requested to review this Pull request/Issue and eventually sign? Thanks

[silviodonato]

assign upgrade, geometry

[cmsbuild]

New categories assigned: geometry,upgrade

@Dr15Jones,@cvuosalo,@mdhildreth,@makortel,@ianna,@kpedro88,@civanch you have been requested to review this Pull request/Issue and eventually sign? Thanks

[silviodonato]

cc @fabiocos

[silviodonato]

the workflow was added with #31842
@bsunanda

[silviodonato]

unassign reconstruction

[fabiocos]

@silviodonato @kpedro88 unfortunately #31842 did not just conflict with mine, but activated workflows that are bound to crash without #31765 . So It is not simply a matter of rebasing mine, those need to be switched off for the time being. I prepare a PR to fix this for the time being.

[fabiocos]

the problem should be fixed by #31899