Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SIGSEGV in HGVHistoProducerAlgo #43942

Closed
iarspider opened this issue Feb 12, 2024 · 35 comments
Closed

SIGSEGV in HGVHistoProducerAlgo #43942

iarspider opened this issue Feb 12, 2024 · 35 comments
Labels
dqm-pending pending-signatures reconstruction-pending upgrade-pending

Comments

@iarspider
Copy link
Contributor 

Thread 3 (Thread 0x14dd8ae4e700 (LWP 624626) "cmsRun"):
#0  0x000014dddbb8e301 in poll () from /lib64/libc.so.6
#1  0x000014ddd73e6efc in (anonymous namespace)::full_read(int, char*, unsigned long, int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x000014ddd73e6984 in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  0x000014ddd73e628f in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x000014dd5120b3e3 in HGVHistoProducerAlgo::layerClusters_to_CaloParticles(HGVHistoProducerAlgoHistograms const&, edm::Handle<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> > >, std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> > const&, edm::Handle<std::vector<CaloParticle, std::allocator<CaloParticle> > >, std::vector<CaloParticle, std::allocator<CaloParticle> > const&, std::vector<unsigned long, std::allocator<unsigned long> > const&, std::vector<unsigned long, std::allocator<unsigned long> > const&, std::unordered_map<DetId, HGCRecHit const*, std::hash<DetId>, std::equal_to<DetId>, std::allocator<std::pair<DetId const, HGCRecHit const*> > > const&, unsigned int, edm::AssociationMap<edm::OneToManyWithQualityGeneric<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> >, std::vector<CaloParticle, std::allocator<CaloParticle> >, float, unsigned int, edm::RefProd<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> > >, edm::RefProd<std::vector<CaloParticle, std::allocator<CaloParticle> > >, edm::Ref<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> >, reco::CaloCluster, edm::refhelper::FindUsingAdvance<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> >, reco::CaloCluster> >, edm::Ref<std::vector<CaloParticle, std::allocator<CaloParticle> >, CaloParticle, edm::refhelper::FindUsingAdvance<std::vector<CaloParticle, std::allocator<CaloParticle> >, CaloParticle> > > > const&, edm::AssociationMap<edm::OneToManyWithQualityGeneric<std::vector<CaloParticle, std::allocator<CaloParticle> >, std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> >, std::pair<float, float>, unsigned int, edm::RefProd<std::vector<CaloParticle, std::allocator<CaloParticle> > >, edm::RefProd<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> > >, edm::Ref<std::vector<CaloParticle, std::allocator<CaloParticle> >, CaloParticle, edm::refhelper::FindUsingAdvance<std::vector<CaloParticle, std::allocator<CaloParticle> >, CaloParticle> >, edm::Ref<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> >, reco::CaloCluster, edm::refhelper::FindUsingAdvance<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> >, reco::CaloCluster> > > > const&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/libValidationHGCalValidation.so
#6  0x000014dd51210716 in HGVHistoProducerAlgo::fill_generic_cluster_histos(HGVHistoProducerAlgoHistograms const&, int, edm::Handle<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> > >, std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> > const&, edm::Handle<std::vector<CaloParticle, std::allocator<CaloParticle> > >, std::vector<CaloParticle, std::allocator<CaloParticle> > const&, std::vector<unsigned long, std::allocator<unsigned long> > const&, std::vector<unsigned long, std::allocator<unsigned long> > const&, std::unordered_map<DetId, HGCRecHit const*, std::hash<DetId>, std::equal_to<DetId>, std::allocator<std::pair<DetId const, HGCRecHit const*> > > const&, std::map<double, double, std::less<double>, std::allocator<std::pair<double const, double> > >, unsigned int, std::vector<int, std::allocator<int> >, edm::AssociationMap<edm::OneToManyWithQualityGeneric<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> >, std::vector<CaloParticle, std::allocator<CaloParticle> >, float, unsigned int, edm::RefProd<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> > >, edm::RefProd<std::vector<CaloParticle, std::allocator<CaloParticle> > >, edm::Ref<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> >, reco::CaloCluster, edm::refhelper::FindUsingAdvance<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> >, reco::CaloCluster> >, edm::Ref<std::vector<CaloParticle, std::allocator<CaloParticle> >, CaloParticle, edm::refhelper::FindUsingAdvance<std::vector<CaloParticle, std::allocator<CaloParticle> >, CaloParticle> > > > const&, edm::AssociationMap<edm::OneToManyWithQualityGeneric<std::vector<CaloParticle, std::allocator<CaloParticle> >, std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> >, std::pair<float, float>, unsigned int, edm::RefProd<std::vector<CaloParticle, std::allocator<CaloParticle> > >, edm::RefProd<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> > >, edm::Ref<std::vector<CaloParticle, std::allocator<CaloParticle> >, CaloParticle, edm::refhelper::FindUsingAdvance<std::vector<CaloParticle, std::allocator<CaloParticle> >, CaloParticle> >, edm::Ref<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> >, reco::CaloCluster, edm::refhelper::FindUsingAdvance<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> >, reco::CaloCluster> > > > const&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/libValidationHGCalValidation.so
#7  0x000014dd5132040c in HGCalValidator::dqmAnalyze(edm::Event const&, edm::EventSetup const&, HGCalValidatorHistograms const&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/pluginValidationHGCalValidationAuto.so
#8  0x000014ddde6cd0ff in edm::global::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#9  0x000014ddde6c12cb in edm::WorkerT<edm::global::EDProducerBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#10 0x000014ddde614fab in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#11 0x000014ddde614dd5 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#12 0x000014ddde614afd in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#13 0x000014ddde61617d in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#14 0x000014ddde2a7f2b in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::$_0>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/libFWCoreConcurrency.so
#15 0x000014dddccf895b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x14dcf0436b00, waiter=..., this=0x14ddda3c9480) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#16 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x14ddda3c9480) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#17 tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/arena.cpp:137
#18 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/market.cpp:599
#19 0x000014dddccfab0e in tbb::detail::r1::rml::private_worker::run (this=0x14ddd83b4080) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#20 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14ddd83b4080) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#21 0x000014dddbe371ca in start_thread () from /lib64/libpthread.so.0
#22 0x000014dddbaa3e73 in clone () from /lib64/libc.so.6

Full log

Full stacktrace A fatal system signal has occurred: segmentation violation The following is the call stack containing the origin of the signal.

Mon Feb 12 07:24:39 CET 2024
Thread 22 (Thread 0x14dd3afff700 (LWP 629352) "cmsRun"):
#0 0x000014dddbe3d45c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000014dda0fb984e in Eigen::ThreadPoolTempltsl::thread::EigenEnvironment::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#2 0x000014dda0fb9e03 in Eigen::ThreadPoolTempltsl::thread::EigenEnvironment::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#3 0x000014dda0fb7578 in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#4 0x000014dd93d29a02 in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/external/el8_amd64_gcc12/lib/libtensorflow_framework.so.2
#5 0x000014dddbe371ca in start_thread () from /lib64/libpthread.so.0
#6 0x000014dddbaa3e73 in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x14dd3b9ff700 (LWP 629348) "cmsRun"):
#0 0x000014dddbe3d45c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000014dda0fb984e in Eigen::ThreadPoolTempltsl::thread::EigenEnvironment::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#2 0x000014dda0fb9e03 in Eigen::ThreadPoolTempltsl::thread::EigenEnvironment::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#3 0x000014dda0fb7578 in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#4 0x000014dd93d29a02 in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/external/el8_amd64_gcc12/lib/libtensorflow_framework.so.2
#5 0x000014dddbe371ca in start_thread () from /lib64/libpthread.so.0
#6 0x000014dddbaa3e73 in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x14dd3be40700 (LWP 629347) "cmsRun"):
#0 0x000014dddbe3d45c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000014dda0fb984e in Eigen::ThreadPoolTempltsl::thread::EigenEnvironment::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#2 0x000014dda0fb9e03 in Eigen::ThreadPoolTempltsl::thread::EigenEnvironment::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#3 0x000014dda0fb7578 in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#4 0x000014dd93d29a02 in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/external/el8_amd64_gcc12/lib/libtensorflow_framework.so.2
#5 0x000014dddbe371ca in start_thread () from /lib64/libpthread.so.0
#6 0x000014dddbaa3e73 in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x14dd699ba700 (LWP 626121) "cmsRun"):
#0 0x000014dddbe3fda6 in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1 0x000014dddbe3fe98 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2 0x000014ddd79a8706 in XrdCl::JobManager::RunJobs() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#3 0x000014ddd79a87b9 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#4 0x000014dddbe371ca in start_thread () from /lib64/libpthread.so.0
#5 0x000014dddbaa3e73 in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x14dd69bbb700 (LWP 626120) "cmsRun"):
#0 0x000014dddbe3fda6 in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1 0x000014dddbe3fe98 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2 0x000014ddd79a8706 in XrdCl::JobManager::RunJobs() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#3 0x000014ddd79a87b9 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#4 0x000014dddbe371ca in start_thread () from /lib64/libpthread.so.0
#5 0x000014dddbaa3e73 in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x14dd69dbc700 (LWP 626119) "cmsRun"):
#0 0x000014dddbe3fda6 in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1 0x000014dddbe3fe98 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2 0x000014ddd79a8706 in XrdCl::JobManager::RunJobs() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#3 0x000014ddd79a87b9 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#4 0x000014dddbe371ca in start_thread () from /lib64/libpthread.so.0
#5 0x000014dddbaa3e73 in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x14dd69fbd700 (LWP 626118) "cmsRun"):
#0 0x000014dddbe41180 in nanosleep () from /lib64/libpthread.so.0
#1 0x000014ddd7a97f08 in XrdSysTimer::Wait(int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2 0x000014ddd791d03c in XrdCl::TaskManager::RunTasks() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#3 0x000014ddd791d179 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#4 0x000014dddbe371ca in start_thread () from /lib64/libpthread.so.0
#5 0x000014dddbaa3e73 in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x14dd6a1be700 (LWP 626117) "cmsRun"):
#0 0x000014dddbb99247 in epoll_wait () from /lib64/libc.so.6
#1 0x000014ddd7a92282 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2 0x000014ddd7a8e1ed in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3 0x000014ddd7a97617 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4 0x000014dddbe371ca in start_thread () from /lib64/libpthread.so.0
#5 0x000014dddbaa3e73 in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x14dd6a3bf700 (LWP 626116) "cmsRun"):
#0 0x000014dddbb99247 in epoll_wait () from /lib64/libc.so.6
#1 0x000014ddd7a92282 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2 0x000014ddd7a8e1ed in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3 0x000014ddd7a97617 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4 0x000014dddbe371ca in start_thread () from /lib64/libpthread.so.0
#5 0x000014dddbaa3e73 in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x14dd6a5c0700 (LWP 626115) "cmsRun"):
#0 0x000014dddbb99247 in epoll_wait () from /lib64/libc.so.6
#1 0x000014ddd7a92282 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2 0x000014ddd7a8e1ed in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3 0x000014ddd7a97617 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4 0x000014dddbe371ca in start_thread () from /lib64/libpthread.so.0
#5 0x000014dddbaa3e73 in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x14dd6a7c1700 (LWP 626114) "cmsRun"):
#0 0x000014dddbb99247 in epoll_wait () from /lib64/libc.so.6
#1 0x000014ddd7a92282 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2 0x000014ddd7a8e1ed in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3 0x000014ddd7a97617 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4 0x000014dddbe371ca in start_thread () from /lib64/libpthread.so.0
#5 0x000014dddbaa3e73 in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x14dd6a9c2700 (LWP 626113) "cmsRun"):
#0 0x000014dddbb99247 in epoll_wait () from /lib64/libc.so.6
#1 0x000014ddd7a92282 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2 0x000014ddd7a8e1ed in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3 0x000014ddd7a97617 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4 0x000014dddbe371ca in start_thread () from /lib64/libpthread.so.0
#5 0x000014dddbaa3e73 in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x14dd6abc3700 (LWP 626112) "cmsRun"):
#0 0x000014dddbb99247 in epoll_wait () from /lib64/libc.so.6
#1 0x000014ddd7a92282 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2 0x000014ddd7a8e1ed in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3 0x000014ddd7a97617 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4 0x000014dddbe371ca in start_thread () from /lib64/libpthread.so.0
#5 0x000014dddbaa3e73 in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x14dd6adc4700 (LWP 626111) "cmsRun"):
#0 0x000014dddbb99247 in epoll_wait () from /lib64/libc.so.6
#1 0x000014ddd7a92282 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2 0x000014ddd7a8e1ed in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3 0x000014ddd7a97617 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4 0x000014dddbe371ca in start_thread () from /lib64/libpthread.so.0
#5 0x000014dddbaa3e73 in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x14dd6afc5700 (LWP 626110) "cmsRun"):
#0 0x000014dddbb99247 in epoll_wait () from /lib64/libc.so.6
#1 0x000014ddd7a92282 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2 0x000014ddd7a8e1ed in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3 0x000014ddd7a97617 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4 0x000014dddbe371ca in start_thread () from /lib64/libpthread.so.0
#5 0x000014dddbaa3e73 in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x14dd6b1c6700 (LWP 626109) "cmsRun"):
#0 0x000014dddbb99247 in epoll_wait () from /lib64/libc.so.6
#1 0x000014ddd7a92282 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2 0x000014ddd7a8e1ed in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3 0x000014ddd7a97617 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4 0x000014dddbe371ca in start_thread () from /lib64/libpthread.so.0
#5 0x000014dddbaa3e73 in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x14dd6b3c7700 (LWP 626108) "cmsRun"):
#0 0x000014dddbb99247 in epoll_wait () from /lib64/libc.so.6
#1 0x000014ddd7a92282 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2 0x000014ddd7a8e1ed in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3 0x000014ddd7a97617 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4 0x000014dddbe371ca in start_thread () from /lib64/libpthread.so.0
#5 0x000014dddbaa3e73 in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x14dd893ff700 (LWP 624628) "cmsRun"):
#0 0x000014dddbb63918 in nanosleep () from /lib64/libc.so.6
#1 0x000014dddbb6381e in sleep () from /lib64/libc.so.6
#2 0x000014ddd73e5f14 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3
#4 0x000014dddc0eafe8 in __ieee754_asin_fma () from /lib64/libm.so.6
#5 0x000014ddb27ed25c in ClosestApproachInRPhi::zCoord(Vector3DBase<float, GlobalTag> const&, Point3DBase<float, GlobalTag> const&, double, double, double, double, double) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/libTrackingToolsPatternTools.so
#6 0x000014ddb27ecbb7 in ClosestApproachInRPhi::compute(int const&, Vector3DBase<float, GlobalTag> const&, Point3DBase<float, GlobalTag> const&, int const&, Vector3DBase<float, GlobalTag> const&, Point3DBase<float, GlobalTag> const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/libTrackingToolsPatternTools.so
#7 0x000014ddb27ecde1 in ClosestApproachInRPhi::calculate(FreeTrajectoryState const&, FreeTrajectoryState const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/libTrackingToolsPatternTools.so
#8 0x000014dd1ddb1dcd in V0Fitter::fitAll(edm::Event const&, edm::EventSetup const&, std::vector<reco::VertexCompositeCandidate, std::allocatorreco::VertexCompositeCandidate >&, std::vector<reco::VertexCompositeCandidate, std::allocatorreco::VertexCompositeCandidate >&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/pluginRecoVertexV0Producer.so
#9 0x000014dd1ddb9682 in V0Producer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/pluginRecoVertexV0Producer.so
#10 0x000014ddde6e06fb in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#11 0x000014ddde6c3edb in edm::WorkerTedm::stream::EDProducerAdaptorBase::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#12 0x000014ddde614fab in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#13 0x000014ddde614dd5 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#14 0x000014ddde614afd in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#15 0x000014ddde61617d in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#16 0x000014ddde2a7f2b in tbb::detail::d1::function_taskedm::WaitingTaskList::announce()::$_0::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/libFWCoreConcurrency.so
#17 0x000014dddccf895b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x14dcf042db00, waiter=..., this=0x14ddda3c9500) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#18 tbb::detail::r1::task_dispatcher::local_wait_for_alltbb::detail::r1::outermost_worker_waiter (t=0x0, waiter=..., this=0x14ddda3c9500) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#19 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/arena.cpp:137
#20 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/market.cpp:599
#21 0x000014dddccfab0e in tbb::detail::r1::rml::private_worker::run (this=0x14ddd83b4000) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#22 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14ddd83b4000) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#23 0x000014dddbe371ca in start_thread () from /lib64/libpthread.so.0
#24 0x000014dddbaa3e73 in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x14dd8a44d700 (LWP 624627) "cmsRun"):
#0 0x000014dddbb63918 in nanosleep () from /lib64/libc.so.6
#1 0x000014dddbb6381e in sleep () from /lib64/libc.so.6
#2 0x000014ddd73e5f14 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3
#4 0x000014dd5d7aabec in DAClusterizerInZ_vect::update(double, DAClusterizerInZ_vect::track_t&, DAClusterizerInZ_vect::vertex_t&, double, bool) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/libRecoVertexPrimaryVertexProducer.so
#5 0x000014dd5d7abab0 in DAClusterizerInZ_vect::thermalize(double, DAClusterizerInZ_vect::track_t&, DAClusterizerInZ_vect::vertex_t&, double, double) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/libRecoVertexPrimaryVertexProducer.so
#6 0x000014dd5d7add31 in DAClusterizerInZ_vect::vertices_no_blocks(std::vector<reco::TransientTrack, std::allocatorreco::TransientTrack > const&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/libRecoVertexPrimaryVertexProducer.so
#7 0x000014dd5d7b20e8 in DAClusterizerInZ_vect::vertices(std::vector<reco::TransientTrack, std::allocatorreco::TransientTrack > const&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/libRecoVertexPrimaryVertexProducer.so
#8 0x000014dd1df04890 in PrimaryVertexProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/pluginRecoVertexPrimaryVertexProducerPlugins.so
#9 0x000014ddde6e06fb in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#10 0x000014ddde6c3edb in edm::WorkerTedm::stream::EDProducerAdaptorBase::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#11 0x000014ddde614fab in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#12 0x000014ddde614dd5 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#13 0x000014ddde614afd in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#14 0x000014ddde61617d in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#15 0x000014ddde2a7f2b in tbb::detail::d1::function_taskedm::WaitingTaskList::announce()::$_0::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/libFWCoreConcurrency.so
#16 0x000014dddccf895b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x14dcbcb52500, waiter=..., this=0x14ddda3c9400) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#17 tbb::detail::r1::task_dispatcher::local_wait_for_alltbb::detail::r1::outermost_worker_waiter (t=0x0, waiter=..., this=0x14ddda3c9400) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#18 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/arena.cpp:137
#19 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/market.cpp:599
#20 0x000014dddccfab0e in tbb::detail::r1::rml::private_worker::run (this=0x14ddd83b4100) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#21 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14ddd83b4100) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#22 0x000014dddbe371ca in start_thread () from /lib64/libpthread.so.0
#23 0x000014dddbaa3e73 in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x14dd8ae4e700 (LWP 624626) "cmsRun"):
#0 0x000014dddbb8e301 in poll () from /lib64/libc.so.6
#1 0x000014ddd73e6efc in (anonymous namespace)::full_read(int, char*, unsigned long, int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2 0x000014ddd73e6984 in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3 0x000014ddd73e628f in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#4
#5 0x000014dd5120b3e3 in HGVHistoProducerAlgo::layerClusters_to_CaloParticles(HGVHistoProducerAlgoHistograms const&, edm::Handle<std::vector<reco::CaloCluster, std::allocatorreco::CaloCluster > >, std::vector<reco::CaloCluster, std::allocatorreco::CaloCluster > const&, edm::Handle<std::vector<CaloParticle, std::allocator > >, std::vector<CaloParticle, std::allocator > const&, std::vector<unsigned long, std::allocator > const&, std::vector<unsigned long, std::allocator > const&, std::unordered_map<DetId, HGCRecHit const*, std::hash, std::equal_to, std::allocator<std::pair<DetId const, HGCRecHit const*> > > const&, unsigned int, edm::AssociationMap<edm::OneToManyWithQualityGeneric<std::vector<reco::CaloCluster, std::allocatorreco::CaloCluster >, std::vector<CaloParticle, std::allocator >, float, unsigned int, edm::RefProd<std::vector<reco::CaloCluster, std::allocatorreco::CaloCluster > >, edm::RefProd<std::vector<CaloParticle, std::allocator > >, edm::Ref<std::vector<reco::CaloCluster, std::allocatorreco::CaloCluster >, reco::CaloCluster, edm::refhelper::FindUsingAdvance<std::vector<reco::CaloCluster, std::allocatorreco::CaloCluster >, reco::CaloCluster> >, edm::Ref<std::vector<CaloParticle, std::allocator >, CaloParticle, edm::refhelper::FindUsingAdvance<std::vector<CaloParticle, std::allocator >, CaloParticle> > > > const&, edm::AssociationMap<edm::OneToManyWithQualityGeneric<std::vector<CaloParticle, std::allocator >, std::vector<reco::CaloCluster, std::allocatorreco::CaloCluster >, std::pair<float, float>, unsigned int, edm::RefProd<std::vector<CaloParticle, std::allocator > >, edm::RefProd<std::vector<reco::CaloCluster, std::allocatorreco::CaloCluster > >, edm::Ref<std::vector<CaloParticle, std::allocator >, CaloParticle, edm::refhelper::FindUsingAdvance<std::vector<CaloParticle, std::allocator >, CaloParticle> >, edm::Ref<std::vector<reco::CaloCluster, std::allocatorreco::CaloCluster >, reco::CaloCluster, edm::refhelper::FindUsingAdvance<std::vector<reco::CaloCluster, std::allocatorreco::CaloCluster >, reco::CaloCluster> > > > const&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/libValidationHGCalValidation.so
#6 0x000014dd51210716 in HGVHistoProducerAlgo::fill_generic_cluster_histos(HGVHistoProducerAlgoHistograms const&, int, edm::Handle<std::vector<reco::CaloCluster, std::allocatorreco::CaloCluster > >, std::vector<reco::CaloCluster, std::allocatorreco::CaloCluster > const&, edm::Handle<std::vector<CaloParticle, std::allocator > >, std::vector<CaloParticle, std::allocator > const&, std::vector<unsigned long, std::allocator > const&, std::vector<unsigned long, std::allocator > const&, std::unordered_map<DetId, HGCRecHit const*, std::hash, std::equal_to, std::allocator<std::pair<DetId const, HGCRecHit const*> > > const&, std::map<double, double, std::less, std::allocator<std::pair<double const, double> > >, unsigned int, std::vector<int, std::allocator >, edm::AssociationMap<edm::OneToManyWithQualityGeneric<std::vector<reco::CaloCluster, std::allocatorreco::CaloCluster >, std::vector<CaloParticle, std::allocator >, float, unsigned int, edm::RefProd<std::vector<reco::CaloCluster, std::allocatorreco::CaloCluster > >, edm::RefProd<std::vector<CaloParticle, std::allocator > >, edm::Ref<std::vector<reco::CaloCluster, std::allocatorreco::CaloCluster >, reco::CaloCluster, edm::refhelper::FindUsingAdvance<std::vector<reco::CaloCluster, std::allocatorreco::CaloCluster >, reco::CaloCluster> >, edm::Ref<std::vector<CaloParticle, std::allocator >, CaloParticle, edm::refhelper::FindUsingAdvance<std::vector<CaloParticle, std::allocator >, CaloParticle> > > > const&, edm::AssociationMap<edm::OneToManyWithQualityGeneric<std::vector<CaloParticle, std::allocator >, std::vector<reco::CaloCluster, std::allocatorreco::CaloCluster >, std::pair<float, float>, unsigned int, edm::RefProd<std::vector<CaloParticle, std::allocator > >, edm::RefProd<std::vector<reco::CaloCluster, std::allocatorreco::CaloCluster > >, edm::Ref<std::vector<CaloParticle, std::allocator >, CaloParticle, edm::refhelper::FindUsingAdvance<std::vector<CaloParticle, std::allocator >, CaloParticle> >, edm::Ref<std::vector<reco::CaloCluster, std::allocatorreco::CaloCluster >, reco::CaloCluster, edm::refhelper::FindUsingAdvance<std::vector<reco::CaloCluster, std::allocatorreco::CaloCluster >, reco::CaloCluster> > > > const&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/libValidationHGCalValidation.so
#7 0x000014dd5132040c in HGCalValidator::dqmAnalyze(edm::Event const&, edm::EventSetup const&, HGCalValidatorHistograms const&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/pluginValidationHGCalValidationAuto.so
#8 0x000014ddde6cd0ff in edm::global::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#9 0x000014ddde6c12cb in edm::WorkerTedm::global::EDProducerBase::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#10 0x000014ddde614fab in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#11 0x000014ddde614dd5 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#12 0x000014ddde614afd in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#13 0x000014ddde61617d in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#14 0x000014ddde2a7f2b in tbb::detail::d1::function_taskedm::WaitingTaskList::announce()::$_0::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/libFWCoreConcurrency.so
#15 0x000014dddccf895b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x14dcf0436b00, waiter=..., this=0x14ddda3c9480) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#16 tbb::detail::r1::task_dispatcher::local_wait_for_alltbb::detail::r1::outermost_worker_waiter (t=0x0, waiter=..., this=0x14ddda3c9480) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#17 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/arena.cpp:137
#18 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/market.cpp:599
#19 0x000014dddccfab0e in tbb::detail::r1::rml::private_worker::run (this=0x14ddd83b4080) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#20 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14ddd83b4080) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#21 0x000014dddbe371ca in start_thread () from /lib64/libpthread.so.0
#22 0x000014dddbaa3e73 in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x14ddb2b14700 (LWP 623518) "cmsRun"):
#0 0x000014dddbe41672 in waitpid () from /lib64/libpthread.so.0
#1 0x000014ddd73e6c21 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2 0x000014dddc4cba73 in std::execute_native_thread_routine (__p=0x14ddcf33c960) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82
#3 0x000014dddbe371ca in start_thread () from /lib64/libpthread.so.0
#4 0x000014dddbaa3e73 in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x14dddafc1640 (LWP 615529) "cmsRun"):
#0 0x000014dddbb63918 in nanosleep () from /lib64/libc.so.6
#1 0x000014dddbb6381e in sleep () from /lib64/libc.so.6
#2 0x000014ddd73e5f14 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3
#4 0x000014dd43472a12 in TSToSimTSHitLCAssociatorByEnergyScoreImpl::makeConnections(edm::Handle<std::vector<ticl::Trackster, std::allocatorticl::Trackster > > const&, edm::Handle<std::vector<reco::CaloCluster, std::allocatorreco::CaloCluster > > const&, edm::Handle<std::vector<SimCluster, std::allocator > > const&, edm::Handle<std::vector<CaloParticle, std::allocator > > const&, edm::Handle<std::vector<ticl::Trackster, std::allocatorticl::Trackster > > const&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/pluginSimCalorimetryHGCalAssociatorProducersAuto.so
#5 0x000014dd43474923 in TSToSimTSHitLCAssociatorByEnergyScoreImpl::associateSimToReco(edm::Handle<std::vector<ticl::Trackster, std::allocatorticl::Trackster > > const&, edm::Handle<std::vector<reco::CaloCluster, std::allocatorreco::CaloCluster > > const&, edm::Handle<std::vector<SimCluster, std::allocator > > const&, edm::Handle<std::vector<CaloParticle, std::allocator > > const&, edm::Handle<std::vector<ticl::Trackster, std::allocatorticl::Trackster > > const&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/pluginSimCalorimetryHGCalAssociatorProducersAuto.so
#6 0x000014dd4347803f in TSToSimTSHitLCAssociatorEDProducer::produce(edm::StreamID, edm::Event&, edm::EventSetup const&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/pluginSimCalorimetryHGCalAssociatorProducersAuto.so
#7 0x000014ddde6cd0ff in edm::global::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#8 0x000014ddde6c12cb in edm::WorkerTedm::global::EDProducerBase::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#9 0x000014ddde614fab in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#10 0x000014ddde614dd5 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#11 0x000014ddde614afd in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#12 0x000014ddde61617d in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#13 0x000014ddde2a7f2b in tbb::detail::d1::function_taskedm::WaitingTaskList::announce()::$_0::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/libFWCoreConcurrency.so
#14 0x000014dddcd01281 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=, this=0x14ddda3c9380) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#15 tbb::detail::r1::task_dispatcher::local_wait_for_alltbb::detail::r1::external_waiter (waiter=..., t=, this=0x14ddda3c9380) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#16 tbb::detail::r1::task_dispatcher::execute_and_wait (t=, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:168
#17 0x000014ddde58d308 in void tbb::detail::d0::try_call_proxytbb::detail::d1::task_group_base::wait()::{lambda()#1}::on_completiontbb::detail::d1::task_group_base::wait()::{lambda()#2}(tbb::detail::d1::task_group_base::wait()::{lambda()#2}) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#18 0x000014ddde58bf75 in edm::FinalWaitingTask::wait() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#19 0x000014ddde5633e5 in edm::EventProcessor::processRuns() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#20 0x000014ddde560b10 in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-02-11-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#21 0x00005628150e8965 in tbb::detail::d1::task_arena_function<main::$_0::operator()() const::{lambda()#1}, void>::operator()() const ()
#22 0x000014dddcced9ad in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/arena.cpp:688
#23 0x00005628150e7d9d in main::$_0::operator()() const ()
#24 0x00005628150e582f in main ()

Current Modules:

Module: HGCalValidator:hgcalValidator (crashed)
Module: V0Producer:generalV0Candidates
Module: TSToSimTSHitLCAssociatorEDProducer:tracksterSimTracksterAssociationLinkingPU
Module: PrimaryVertexProducer:unsortedOfflinePrimaryVertices

A fatal system signal has occurred: segmentation violation
timeout: the monitored command dumped core

Out of 5 local tries, it reproduced only once, one failure was in a different part of code, other runs were fine. Also happened in ROOT626_X IB: link
@iarspider iarspider changed the title SIGSEGV in different methods of HGVHistoProducerAlgo SIGSEGV in HGVHistoProducerAlgo Feb 12, 2024
@cmsbuild
Copy link
Contributor

cmsbuild commented Feb 12, 2024 •
edited
Loading

cms-bot internal usage

@cmsbuild
Copy link
Contributor

A new Issue was created by @iarspider .

@Dr15Jones, @antoniovilela, @rappoccio, @makortel, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

@iarspider
Copy link
Contributor Author

assign Validation/HGCalValidation

@cmsbuild
Copy link
Contributor

New categories assigned: dqm

@rvenditti,@syuvivida,@tjavaid,@nothingface0,@antoniovagnerini you have been requested to review this Pull request/Issue and eventually sign? Thanks

@makortel
Copy link
Contributor

assign upgade

@makortel
Copy link
Contributor

@cms-sw/hgcal-dpg-l2

@dan131riley
Copy link

We're seen a variety of crashes that look like memory corruption in 25034.{21,99,114,999,9921}. I suspect these are all related. A lot of them have TSToSimTSHitLCAssociatorEDProducer running on another thread at the time of crash. ASAN is seeing buffer overflows in MTDThresholdClusterizer (stack trace below). I'm seeing those in valgrind too, but don't have line numbers yet.

==30200== Invalid write of size 4
==30200==    at 0x9F3F7325: MTDThresholdClusterizer::copy_to_buffer(__gnu_cxx::__normal_iterator<FTLRecHit const*, std::vector<FTLRecHit, std::allocator<FTLRecHit> > >, MTDGeometry const*, MTDTopology const*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02824/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-11-0000/lib/slc7_amd64_gcc12/libRecoLocalFastTimeFTLClusterizer.so)
==30200==    by 0x9F3F7DB9: MTDThresholdClusterizer::clusterize(edm::SortedCollection<FTLRecHit, edm::StrictWeakOrdering<FTLRecHit> > const&, MTDGeometry const*, MTDTopology const*, edmNew::DetSetVector<FTLCluster>&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02824/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-11-0000/lib/slc7_amd64_gcc12/libRecoLocalFastTimeFTLClusterizer.so)
==30200==    by 0x9F3D7262: MTDClusterProducer::produce(edm::Event&, edm::EventSetup const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02824/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-11-0000/lib/slc7_amd64_gcc12/pluginRecoLocalFastTimeFTLClusterizerPlugins.so)

==30200==  Address 0x71532f844 is 16 bytes after a block of size 4 free'd
==30200==    at 0x4032779: operator delete(void*, unsigned long) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02824/slc7_amd64_gcc12/external/valgrind/3.22.0-e989c977f8949fc48f1bc1206f1f87c8/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==30200==    by 0x9F3F7C07: MTDThresholdClusterizer::clusterize(edm::SortedCollection<FTLRecHit, edm::StrictWeakOrdering<FTLRecHit> > const&, MTDGeometry const*, MTDTopology const*, edmNew::DetSetVector<FTLCluster>&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02824/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-11-0000/lib/slc7_amd64_gcc12/libRecoLocalFastTimeFTLClusterizer.so)
==30200==    by 0x9F3D7262: MTDClusterProducer::produce(edm::Event&, edm::EventSetup const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02824/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-11-0000/lib/slc7_amd64_gcc12/pluginRecoLocalFastTimeFTLClusterizerPlugins.so)

==30200==  Block was alloc'd at
==30200==    at 0x402EEE1: operator new(unsigned long) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02824/slc7_amd64_gcc12/external/valgrind/3.22.0-e989c977f8949fc48f1bc1206f1f87c8/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==30200==    by 0x9F3F5D8F: MTDThresholdClusterizer::make_cluster(FTLCluster::FTLHitPos const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02824/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-11-0000/lib/slc7_amd64_gcc12/libRecoLocalFastTimeFTLClusterizer.so)
==30200==    by 0x9F3F7AF2: MTDThresholdClusterizer::clusterize(edm::SortedCollection<FTLRecHit, edm::StrictWeakOrdering<FTLRecHit> > const&, MTDGeometry const*, MTDTopology const*, edmNew::DetSetVector<FTLCluster>&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02824/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-11-0000/lib/slc7_amd64_gcc12/libRecoLocalFastTimeFTLClusterizer.so)
==30200==    by 0x9F3D7262: MTDClusterProducer::produce(edm::Event&, edm::EventSetup const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02824/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-11-0000/lib/slc7_amd64_gcc12/pluginRecoLocalFastTimeFTLClusterizerPlugins.so)

==3131784==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x61d004aab8b0 at pc 0x1489a85fc260 bp 0x14899e220090 sp 0x14899e220088
    #0 0x1489a85fc25f in MTDThresholdClusterizer::copy_to_buffer(__gnu_cxx::__normal_iterator<FTLRecHit const*, std::vector<FTLRecHit, std::allocator<FTLRecHit> > >, MTDGeometry const*, MTDTopology const*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02823/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASAN_X_2024-02-09-2300/lib/el8_amd64_gcc12/libRecoLocalFastTimeFTLClusterizer.so+0x1725f)
    #1 0x1489a85fd61a in MTDThresholdClusterizer::clusterize(edm::SortedCollection<FTLRecHit, edm::StrictWeakOrdering<FTLRecHit> > const&, MTDGeometry const*, MTDTopology const*, edmNew::DetSetVector<FTLCluster>&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02823/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASAN_X_2024-02-09-2300/lib/el8_amd64_gcc12/libRecoLocalFastTimeFTLClusterizer.so+0x1861a)
    #2 0x1489a8670bf3 in void MTDClusterProducer::run<edm::SortedCollection<FTLRecHit, edm::StrictWeakOrdering<FTLRecHit> > >(edm::SortedCollection<FTLRecHit, edm::StrictWeakOrdering<FTLRecHit> > const&, edmNew::DetSetVector<FTLCluster>&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02823/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASAN_X_2024-02-09-2300/lib/el8_amd64_gcc12/pluginRecoLocalFastTimeFTLClusterizerPlugins.so+0x59bf3)
    #3 0x1489a8666ace in MTDClusterProducer::produce(edm::Event&, edm::EventSetup const&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02823/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASAN_X_2024-02-09-2300/lib/el8_amd64_gcc12/pluginRecoLocalFastTimeFTLClusterizerPlugins.so+0x4face)
    #4 0x1489fd943c90 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02823/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASAN_X_2024-02-09-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so+0x999c90)
    #5 0x1489fd8be4c8 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02823/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASAN_X_2024-02-09-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so+0x9144c8)
    #6 0x1489fd585827 in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02823/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASAN_X_2024-02-09-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so+0x5db827)
    #7 0x1489fd585ed2 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02823/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASAN_X_2024-02-09-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so+0x5dbed2)
    #8 0x1489fd591655 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02823/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASAN_X_2024-02-09-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so+0x5e7655)
    #9 0x1489fc7363be in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02823/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASAN_X_2024-02-09-2300/lib/el8_amd64_gcc12/libFWCoreConcurrency.so+0x123be)
    #10 0x1489fb06695a in tbb::detail::d1::task* tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter>(tbb::detail::d1::task*, tbb::detail::r1::outermost_worker_waiter&) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
    #11 0x1489fb06695a in tbb::detail::d1::task* tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter>(tbb::detail::d1::task*, tbb::detail::r1::outermost_worker_waiter&) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
    #12 0x1489fb06695a in tbb::detail::r1::arena::process(tbb::detail::r1::thread_data&) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/arena.cpp:137
    #13 0x1489fb06695a in tbb::detail::r1::market::process(rml::job&) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/market.cpp:599
    #14 0x1489fb068b0d in tbb::detail::r1::rml::private_worker::run() /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/private_server.cpp:271
    #15 0x1489fb068b0d in tbb::detail::r1::rml::private_worker::thread_routine(void*) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/private_server.cpp:221
    #16 0x1489fa1a51c9 in start_thread (/lib64/libpthread.so.0+0x81c9)
    #17 0x1489f9e11e72 in __clone (/lib64/libc.so.6+0x39e72)

0x61d004aab8b0 is located 48 bytes to the right of 2048-byte region [0x61d004aab080,0x61d004aab880)
allocated by thread T4 here:
    #0 0x1489fdbaf6d8 in operator new(unsigned long) ../../../../libsanitizer/asan/asan_new_delete.cpp:95
    #1 0x1489a8603e05 in std::vector<GeomDetEnumerators::Location, std::allocator<GeomDetEnumerators::Location> >::_M_fill_insert(__gnu_cxx::__normal_iterator<GeomDetEnumerators::Location*, std::vector<GeomDetEnumerators::Location, std::allocator<GeomDetEnumerators::Location> > >, unsigned long, GeomDetEnumerators::Location const&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02823/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASAN_X_2024-02-09-2300/lib/el8_amd64_gcc12/libRecoLocalFastTimeFTLClusterizer.so+0x1ee05)
    #2 0x1489a86064bf in MTDArrayBuffer::setSize(unsigned int, unsigned int) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02823/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASAN_X_2024-02-09-2300/lib/el8_amd64_gcc12/libRecoLocalFastTimeFTLClusterizer.so+0x214bf)
    #3 0x1489a85fa808 in MTDThresholdClusterizer::setup(MTDGeometry const*, MTDTopology const*, DetId const&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02823/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASAN_X_2024-02-09-2300/lib/el8_amd64_gcc12/libRecoLocalFastTimeFTLClusterizer.so+0x15808)
    #4 0x1489a85fd4a0 in MTDThresholdClusterizer::clusterize(edm::SortedCollection<FTLRecHit, edm::StrictWeakOrdering<FTLRecHit> > const&, MTDGeometry const*, MTDTopology const*, edmNew::DetSetVector<FTLCluster>&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02823/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASAN_X_2024-02-09-2300/lib/el8_amd64_gcc12/libRecoLocalFastTimeFTLClusterizer.so+0x184a0)
    #5 0x1489a8670bf3 in void MTDClusterProducer::run<edm::SortedCollection<FTLRecHit, edm::StrictWeakOrdering<FTLRecHit> > >(edm::SortedCollection<FTLRecHit, edm::StrictWeakOrdering<FTLRecHit> > const&, edmNew::DetSetVector<FTLCluster>&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02823/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASAN_X_2024-02-09-2300/lib/el8_amd64_gcc12/pluginRecoLocalFastTimeFTLClusterizerPlugins.so+0x59bf3)
    #6 0x1489a8666ace in MTDClusterProducer::produce(edm::Event&, edm::EventSetup const&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02823/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASAN_X_2024-02-09-2300/lib/el8_amd64_gcc12/pluginRecoLocalFastTimeFTLClusterizerPlugins.so+0x4face)
    #7 0x1489fd943c90 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02823/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASAN_X_2024-02-09-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so+0x999c90)
    #8 0x1489fd8be4c8 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02823/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASAN_X_2024-02-09-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so+0x9144c8)
    #9 0x1489fd585827 in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02823/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASAN_X_2024-02-09-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so+0x5db827)
    #10 0x1489fd585ed2 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02823/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASAN_X_2024-02-09-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so+0x5dbed2)
    #11 0x1489fd591655 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02823/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASAN_X_2024-02-09-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so+0x5e7655)
    #12 0x1489fc7363be in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02823/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASAN_X_2024-02-09-2300/lib/el8_amd64_gcc12/libFWCoreConcurrency.so+0x123be)
    #13 0x1489fb06695a in tbb::detail::d1::task* tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter>(tbb::detail::d1::task*, tbb::detail::r1::outermost_worker_waiter&) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
    #14 0x1489fb06695a in tbb::detail::d1::task* tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter>(tbb::detail::d1::task*, tbb::detail::r1::outermost_worker_waiter&) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
    #15 0x1489fb06695a in tbb::detail::r1::arena::process(tbb::detail::r1::thread_data&) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/arena.cpp:137
    #16 0x1489fb06695a in tbb::detail::r1::market::process(rml::job&) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/market.cpp:599
    #17 0x1489fb068b0d in tbb::detail::r1::rml::private_worker::run() /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/private_server.cpp:271
    #18 0x1489fb068b0d in tbb::detail::r1::rml::private_worker::thread_routine(void*) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/private_server.cpp:221

Thread T4 created by T2 here:
    #0 0x1489fdb40136 in __interceptor_pthread_create ../../../../libsanitizer/asan/asan_interceptors.cpp:207
    #1 0x1489fb06816f in tbb::detail::r1::rml::internal::thread_monitor::launch(void* (*)(void*), void*, unsigned long) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:208
    #2 0x1489fb06816f in tbb::detail::r1::rml::private_worker::wake_or_launch() /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/private_server.cpp:305
    #3 0x1489fb06816f in tbb::detail::r1::rml::private_server::wake_some(int) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/private_server.cpp:412

Thread T2 created by T0 here:
    #0 0x1489fdb40136 in __interceptor_pthread_create ../../../../libsanitizer/asan/asan_interceptors.cpp:207
    #1 0x1489fb06816f in tbb::detail::r1::rml::internal::thread_monitor::launch(void* (*)(void*), void*, unsigned long) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:208
    #2 0x1489fb06816f in tbb::detail::r1::rml::private_worker::wake_or_launch() /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/private_server.cpp:305
    #3 0x1489fb06816f in tbb::detail::r1::rml::private_server::wake_some(int) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/private_server.cpp:412

SUMMARY: AddressSanitizer: heap-buffer-overflow (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02823/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASAN_X_2024-02-09-2300/lib/el8_amd64_gcc12/libRecoLocalFastTimeFTLClusterizer.so+0x1725f) in MTDThresholdClusterizer::copy_to_buffer(__gnu_cxx::__normal_iterator<FTLRecHit const*, std::vector<FTLRecHit, std::allocator<FTLRecHit> > >, MTDGeometry const*, MTDTopology const*)
Shadow bytes around the buggy address:
  0x0c3a8094d6c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c3a8094d6d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c3a8094d6e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c3a8094d6f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c3a8094d700: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c3a8094d710: fa fa fa fa fa fa[fa]fa fa fa fa fa fa fa fa fa
  0x0c3a8094d720: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c3a8094d730: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c3a8094d740: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c3a8094d750: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c3a8094d760: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==3131784==ABORTING

@makortel
Copy link
Contributor

ASAN is seeing buffer overflows in MTDThresholdClusterizer (stack trace below).

FYI @cms-sw/mtd-dpg-l2

@makortel
Copy link
Contributor

Seems like the problems started in CMSSW_14_1_X_2024-02-09-2300

@dan131riley
Copy link

There looks to be an indexing problem in MTDThresholdClusterizer. If add a check

diff --git a/RecoLocalFastTime/FTLClusterizer/interface/MTDArrayBuffer.h b/RecoLocalFastTime/FTLClusterizer/interface/MTDArrayBuffer.h
index 0a67b7cdffc..f8619d2e13c 100644
--- a/RecoLocalFastTime/FTLClusterizer/interface/MTDArrayBuffer.h
+++ b/RecoLocalFastTime/FTLClusterizer/interface/MTDArrayBuffer.h
@@ -105,7 +105,10 @@ public:
   uint size() const { return hitEnergy_vec.size(); }

   /// Definition of indexing within the buffer.
-  uint index(uint row, uint col) const { return col * nrows + row; }
+  uint index(uint row, uint col) const {
+    assert(inside(row,col));
+    return col * nrows + row;
+  }
   uint index(const FTLCluster::FTLHitPos& pix) const { return index(pix.row(), pix.col()); }

 private:

and run WF 25034.114 (possibly rerunning step2 a few times) the assertion fires with this (truncated) stack trace from gdb:

#4  0x00007f53f83c8634 in MTDArrayBuffer::index (this=0x7f5384791e98, row=7, col=16) at /mnt/data1/dsr/tmp/CMSSW_14_1_X_2024-02-11-0000/src/RecoLocalFastTime/FTLClusterizer/interface/MTDArrayBuffer.h:109
#5  0x00007f53f83c8dd8 in MTDArrayBuffer::set (this=0x7f5384791e98, row=7, col=16, subDet=GeomDetEnumerators::endcap, energy=0.0234375, time=10.7347965, time_error=0.0390000008, local_error=..., local_point=..., xpos=0) at /mnt/data1/dsr/tmp/CMSSW_14_1_X_2024-02-11-0000/src/RecoLocalFastTime/FTLClusterizer/interface/MTDArrayBuffer.h:183
#6  0x00007f53f83cb914 in MTDThresholdClusterizer::copy_to_buffer (this=0x7f5384791e40, itr={
  id_ = {
    static kDetMask = 15,
    static kSubdetMask = 7,
    static kDetOffset = 28,
    static kSubdetOffset = 25,
    id_ = 1665274953
  },
  energy_ = 0.0234375,
  time_ = 10.7347965,
  timeError_ = 0.0390000008,
  position_ = -1,
  positionError_ = -1,
  row_ = 7 '\a',
  column_ = 16 '\020',
  flagBits_ = 1 '\001'
}, geom=0x7f535822f780, topo=0x7f53560d9600) at src/RecoLocalFastTime/FTLClusterizer/src/MTDThresholdClusterizer.cc:239
#7  0x00007f53f83cb044 in MTDThresholdClusterizer::clusterize (this=0x7f5384791e40, input=..., geom=0x7f535822f780, topo=0x7f53560d9600, output=...) at src/RecoLocalFastTime/FTLClusterizer/src/MTDThresholdClusterizer.cc:150
#8  0x00007f53f83fe076 in MTDClusterProducer::run<edm::SortedCollection<FTLRecHit, edm::StrictWeakOrdering<FTLRecHit> > > (this=0x7f5384ce1800, input=..., output=...) at src/RecoLocalFastTime/FTLClusterizer/plugins/MTDClusterProducer.cc:152
#9  0x00007f53f83fc997 in MTDClusterProducer::produce (this=0x7f5384ce1800, e=..., es=...) at src/RecoLocalFastTime/FTLClusterizer/plugins/MTDClusterProducer.cc:135
(gdb) p nrows
$1 = 32
(gdb) p ncols
$2 = 16

valgrind report that led me to this:

==85350== Invalid write of size 4
==85350==    at 0x9F3BE2DB: UnknownInlinedFun (RecoLocalFastTime/FTLClusterizer/interface/MTDArrayBuffer.h:179)
==85350==    by 0x9F3BE2DB: MTDThresholdClusterizer::copy_to_buffer(__gnu_cxx::__normal_iterator<FTLRecHit const*, std::vector<FTLRecHit, std::allocator<FTLRecHit> > >, MTDGeometry const*, MTDTopology const*) (RecoLocalFastTime/FTLClusterizer/src/MTDThresholdClusterizer.cc:239)
==85350==    by 0x9F3BEDB9: MTDThresholdClusterizer::clusterize(edm::SortedCollection<FTLRecHit, edm::StrictWeakOrdering<FTLRecHit> > const&, MTDGeometry const*, MTDTopology const*, edmNew::DetSetVector<FTLCluster>&) (RecoLocalFastTime/FTLClusterizer/src/MTDThresholdClusterizer.cc:150)
==85350==    by 0x9F39E262: MTDClusterProducer::produce(edm::Event&, edm::EventSetup const&) (RecoLocalFastTime/FTLClusterizer/plugins/MTDClusterProducer.cc:135)

==85350==  Address 0x1fe3c6554 is 4 bytes after a block of size 2,048 alloc'd
==85350==    at 0x402EEE1: operator new(unsigned long) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02824/slc7_amd64_gcc12/external/valgrind/3.22.0-e989c977f8949fc48f1bc1206f1f87c8/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==85350==    by 0x9F3BDE9B: UnknownInlinedFun (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02824/slc7_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/bits/new_allocator.h:137)
==85350==    by 0x9F3BDE9B: UnknownInlinedFun (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02824/slc7_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/bits/alloc_traits.h:464)
==85350==    by 0x9F3BDE9B: UnknownInlinedFun (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02824/slc7_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/bits/stl_vector.h:378)
==85350==    by 0x9F3BDE9B: UnknownInlinedFun (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02824/slc7_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/bits/vector.tcc:581)
==85350==    by 0x9F3BDE9B: UnknownInlinedFun (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02824/slc7_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/bits/stl_vector.h:1032)
==85350==    by 0x9F3BDE9B: UnknownInlinedFun (RecoLocalFastTime/FTLClusterizer/interface/MTDArrayBuffer.h:135)
==85350==    by 0x9F3BDE9B: MTDThresholdClusterizer::setup(MTDGeometry const*, MTDTopology const*, DetId const&) (RecoLocalFastTime/FTLClusterizer/src/MTDThresholdClusterizer.cc:77)
==85350==    by 0x9F3BE8D7: MTDThresholdClusterizer::clusterize(edm::SortedCollection<FTLRecHit, edm::StrictWeakOrdering<FTLRecHit> > const&, MTDGeometry const*, MTDTopology const*, edmNew::DetSetVector<FTLCluster>&) (RecoLocalFastTime/FTLClusterizer/src/MTDThresholdClusterizer.cc:139)
==85350==    by 0x9F39E262: MTDClusterProducer::produce(edm::Event&, edm::EventSetup const&) (RecoLocalFastTime/FTLClusterizer/plugins/MTDClusterProducer.cc:135)

==85350== Invalid write of size 8
==85350==    at 0x9F3BE317: UnknownInlinedFun (RecoLocalFastTime/FTLClusterizer/interface/MTDArrayBuffer.h:183)
==85350==    by 0x9F3BE317: MTDThresholdClusterizer::copy_to_buffer(__gnu_cxx::__normal_iterator<FTLRecHit const*, std::vector<FTLRecHit, std::allocator<FTLRecHit> > >, MTDGeometry const*, MTDTopology const*) (RecoLocalFastTime/FTLClusterizer/src/MTDThresholdClusterizer.cc:239)
==85350==    by 0x9F3BEDB9: MTDThresholdClusterizer::clusterize(edm::SortedCollection<FTLRecHit, edm::StrictWeakOrdering<FTLRecHit> > const&, MTDGeometry const*, MTDTopology const*, edmNew::DetSetVector<FTLCluster>&) (RecoLocalFastTime/FTLClusterizer/src/MTDThresholdClusterizer.cc:150)
==85350==    by 0x9F39E262: MTDClusterProducer::produce(edm::Event&, edm::EventSetup const&) (RecoLocalFastTime/FTLClusterizer/plugins/MTDClusterProducer.cc:135)

==85350==  Address 0x1fe3c8de0 is 16 bytes after a block of size 8,192 alloc'd
==85350==    at 0x402EEE1: operator new(unsigned long) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02824/slc7_amd64_gcc12/external/valgrind/3.22.0-e989c977f8949fc48f1bc1206f1f87c8/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==85350==    by 0x9F3BDFCF: UnknownInlinedFun (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02824/slc7_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/bits/new_allocator.h:137)
==85350==    by 0x9F3BDFCF: UnknownInlinedFun (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02824/slc7_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/bits/alloc_traits.h:464)
==85350==    by 0x9F3BDFCF: UnknownInlinedFun (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02824/slc7_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/bits/stl_vector.h:378)
==85350==    by 0x9F3BDFCF: UnknownInlinedFun (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02824/slc7_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/bits/vector.tcc:657)
==85350==    by 0x9F3BDFCF: UnknownInlinedFun (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02824/slc7_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/bits/stl_vector.h:1011)
==85350==    by 0x9F3BDFCF: UnknownInlinedFun (RecoLocalFastTime/FTLClusterizer/interface/MTDArrayBuffer.h:139)
==85350==    by 0x9F3BDFCF: MTDThresholdClusterizer::setup(MTDGeometry const*, MTDTopology const*, DetId const&) (RecoLocalFastTime/FTLClusterizer/src/MTDThresholdClusterizer.cc:77)
==85350==    by 0x9F3BE8D7: MTDThresholdClusterizer::clusterize(edm::SortedCollection<FTLRecHit, edm::StrictWeakOrdering<FTLRecHit> > const&, MTDGeometry const*, MTDTopology const*, edmNew::DetSetVector<FTLCluster>&) (RecoLocalFastTime/FTLClusterizer/src/MTDThresholdClusterizer.cc:139)
==85350==    by 0x9F39E262: MTDClusterProducer::produce(edm::Event&, edm::EventSetup const&) (RecoLocalFastTime/FTLClusterizer/plugins/MTDClusterProducer.cc:135)

@makortel
Copy link
Contributor

assign reconstruction

@makortel
Copy link
Contributor

assign upgrade

@cmsbuild
Copy link
Contributor

New categories assigned: reconstruction,upgrade

@jfernan2,@mandrenguyen,@srimanob,@subirsarkar you have been requested to review this Pull request/Issue and eventually sign? Thanks

@makortel
Copy link
Contributor

Running 25034.999 on one thread in CMSSW_14_1_ASAN_X_2024-02-12-2300 succeeded. This might hint towards some kind of threading (or ordering) problem (or my single test was lucky).

@makortel
Copy link
Contributor

By reverting #43762 I got the workflow 25034.999 to run in CMSSW_14_1_ASAN_X_2024-02-12-2300 with 4 threads.

@makortel makortel mentioned this issue Feb 13, 2024
Merged
@dan131riley
Copy link

Running 25034.999 on one thread in CMSSW_14_1_ASAN_X_2024-02-12-2300 succeeded. This might hint towards some kind of threading (or ordering) problem (or my single test was lucky).

The problem seems to be only 30-50% reproducible, it took me a couple of step2 runs to produce a step2.root that causes the indexing error assertion failure. Once I had a file that hit the assertion multi-threaded, it also failed single threaded.

@fabiocos
Copy link
Contributor

@parbol @martinamalberti FYI

The new ETL digitization is not directly touching the clustering code, that is stable since my last update a couple of years ago. And the crashing job refers to geometry scenario D98, in production since a while. But there was an update in local reconstruction in the new PR...

@fabiocos
Copy link
Contributor

@parbol I notice that in the debugging by @dan131riley the problem appears in the pixel (16,32), i.e. the maximum allowed value for both rows and columns in geometry D98. It might be just a coincidence, I wonder whether there might be an issue of definition of range somewhere (i.e. 1 -> 16 vs 0 -> 15). Although I do not immediately see where your code would possibly impact on this.

@fabiocos
Copy link
Contributor

sorry, I believe I misinterpreted the debugging output, row seems to be 7 (nrow and ncol are just the overall number). But the column is effectively 16.

@makortel
Copy link
Contributor

I've now ran workflow 25034.999 by reverting #43762 in CMSSW_14_1_ASAN_X_2024-02-12-2300 10 times without failures.

@fabiocos
Copy link
Contributor

I haven't managed to reproduce the failure even with multiple run trials, and the pixel ranges I see with internal dump printouts appear to be reasonable on the few inspected events. I am trying again with a dedicated sample where I try to cover as many pixels as possible in a single pad.

@dan131riley
Copy link

Crash in HGCalValidator with some preceding UBSAN hits. Log from

https://cmssdt.cern.ch/SDT/cgi-bin/logreader/el8_amd64_gcc12/CMSSW_14_1_UBSAN_X_2024-02-14-2300/pyRelValMatrixLogs/run/25034.999_TTbar_14TeV+2026D98PU_PMXS1S2PR/step4_TTbar_14TeV+2026D98PU_PMXS1S2PR.log#/1179-1179

src/Validation/HGCalValidation/src/HGVHistoProducerAlgo.cc:3565:61: runtime error: member access within null pointer of type 'const struct HGCRecHit'
    #0 0x14c1cc9e94cd in HGVHistoProducerAlgo::findmaxhit(reco::CaloCluster const&, std::unordered_map<DetId, HGCRecHit const*, std::hash<DetId>, std::equal_to<DetId>, std::allocator<std::pair<DetId const, HGCRecHit const*> > > const&) const src/Validation/HGCalValidation/src/HGVHistoProducerAlgo.cc:3565
    #1 0x14c1cca5acf4 in HGVHistoProducerAlgo::fill_generic_cluster_histos(HGVHistoProducerAlgoHistograms const&, int, edm::Handle<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> > >, std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> > const&, edm::Handle<std::vector<CaloParticle, std::allocator<CaloParticle> > >, std::vector<CaloParticle, std::allocator<CaloParticle> > const&, std::vector<unsigned long, std::allocator<unsigned long> > const&, std::vector<unsigned long, std::allocator<unsigned long> > const&, std::unordered_map<DetId, HGCRecHit const*, std::hash<DetId>, std::equal_to<DetId>, std::allocator<std::pair<DetId const, HGCRecHit const*> > > const&, std::map<double, double, std::less<double>, std::allocator<std::pair<double const, double> > >, unsigned int, std::vector<int, std::allocator<int> >, edm::AssociationMap<edm::OneToManyWithQualityGeneric<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> >, std::vector<CaloParticle, std::allocator<CaloParticle> >, float, unsigned int, edm::RefProd<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> > >, edm::RefProd<std::vector<CaloParticle, std::allocator<CaloParticle> > >, edm::Ref<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> >, reco::CaloCluster, edm::refhelper::FindUsingAdvance<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> >, reco::CaloCluster> >, edm::Ref<std::vector<CaloParticle, std::allocator<CaloParticle> >, CaloParticle, edm::refhelper::FindUsingAdvance<std::vector<CaloParticle, std::allocator<CaloParticle> >, CaloParticle> > > > const&, edm::AssociationMap<edm::OneToManyWithQualityGeneric<std::vector<CaloParticle, std::allocator<CaloParticle> >, std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> >, std::pair<float, float>, unsigned int, edm::RefProd<std::vector<CaloParticle, std::allocator<CaloParticle> > >, edm::RefProd<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> > >, edm::Ref<std::vector<CaloParticle, std::allocator<CaloParticle> >, CaloParticle, edm::refhelper::FindUsingAdvance<std::vector<CaloParticle, std::allocator<CaloParticle> >, CaloParticle> >, edm::Ref<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> >, reco::CaloCluster, edm::refhelper::FindUsingAdvance<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> >, reco::CaloCluster> > > > const&) const src/Validation/HGCalValidation/src/HGVHistoProducerAlgo.cc:2248
    #2 0x14c1ccdf4d99 in HGCalValidator::dqmAnalyze(edm::Event const&, edm::EventSetup const&, HGCalValidatorHistograms const&) const src/Validation/HGCalValidation/plugins/HGCalValidator.cc:380
    #3 0x14c30114954b in edm::global::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) src/FWCore/Framework/src/global/EDProducerBase.cc:69

src/Validation/HGCalValidation/src/HGVHistoProducerAlgo.cc:3565:61: runtime error: member call on null pointer of type 'const struct CaloRecHit'
    #0 0x14c1cc9e94a9 in HGVHistoProducerAlgo::findmaxhit(reco::CaloCluster const&, std::unordered_map<DetId, HGCRecHit const*, std::hash<DetId>, std::equal_to<DetId>, std::allocator<std::pair<DetId const, HGCRecHit const*> > > const&) const src/Validation/HGCalValidation/src/HGVHistoProducerAlgo.cc:3565
    #1 0x14c1cca5acf4 in HGVHistoProducerAlgo::fill_generic_cluster_histos(HGVHistoProducerAlgoHistograms const&, int, edm::Handle<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> > >, std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> > const&, edm::Handle<std::vector<CaloParticle, std::allocator<CaloParticle> > >, std::vector<CaloParticle, std::allocator<CaloParticle> > const&, std::vector<unsigned long, std::allocator<unsigned long> > const&, std::vector<unsigned long, std::allocator<unsigned long> > const&, std::unordered_map<DetId, HGCRecHit const*, std::hash<DetId>, std::equal_to<DetId>, std::allocator<std::pair<DetId const, HGCRecHit const*> > > const&, std::map<double, double, std::less<double>, std::allocator<std::pair<double const, double> > >, unsigned int, std::vector<int, std::allocator<int> >, edm::AssociationMap<edm::OneToManyWithQualityGeneric<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> >, std::vector<CaloParticle, std::allocator<CaloParticle> >, float, unsigned int, edm::RefProd<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> > >, edm::RefProd<std::vector<CaloParticle, std::allocator<CaloParticle> > >, edm::Ref<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> >, reco::CaloCluster, edm::refhelper::FindUsingAdvance<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> >, reco::CaloCluster> >, edm::Ref<std::vector<CaloParticle, std::allocator<CaloParticle> >, CaloParticle, edm::refhelper::FindUsingAdvance<std::vector<CaloParticle, std::allocator<CaloParticle> >, CaloParticle> > > > const&, edm::AssociationMap<edm::OneToManyWithQualityGeneric<std::vector<CaloParticle, std::allocator<CaloParticle> >, std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> >, std::pair<float, float>, unsigned int, edm::RefProd<std::vector<CaloParticle, std::allocator<CaloParticle> > >, edm::RefProd<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> > >, edm::Ref<std::vector<CaloParticle, std::allocator<CaloParticle> >, CaloParticle, edm::refhelper::FindUsingAdvance<std::vector<CaloParticle, std::allocator<CaloParticle> >, CaloParticle> >, edm::Ref<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> >, reco::CaloCluster, edm::refhelper::FindUsingAdvance<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> >, reco::CaloCluster> > > > const&) const src/Validation/HGCalValidation/src/HGVHistoProducerAlgo.cc:2248
    #2 0x14c1ccdf4d99 in HGCalValidator::dqmAnalyze(edm::Event const&, edm::EventSetup const&, HGCalValidatorHistograms const&) const src/Validation/HGCalValidation/plugins/HGCalValidator.cc:380
    #3 0x14c30114954b in edm::global::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) src/FWCore/Framework/src/global/EDProducerBase.cc:69

/data/cmsbld/jenkins/workspace/build-any-ib/w/tmp/BUILDROOT/c0a455c7a5b724be894a663904dad922/opt/cmssw/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_UBSAN_X_2024-02-14-2300/src/DataFormats/CaloRecHit/interface/CaloRecHit.h:29:43: runtime error: member access within null pointer of type 'const struct CaloRecHit'
    #0 0x14c1cc9e9485 in CaloRecHit::energy() const /data/cmsbld/jenkins/workspace/build-any-ib/w/tmp/BUILDROOT/c0a455c7a5b724be894a663904dad922/opt/cmssw/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_UBSAN_X_2024-02-14-2300/src/DataFormats/CaloRecHit/interface/CaloRecHit.h:29
    #1 0x14c1cc9e9485 in HGVHistoProducerAlgo::findmaxhit(reco::CaloCluster const&, std::unordered_map<DetId, HGCRecHit const*, std::hash<DetId>, std::equal_to<DetId>, std::allocator<std::pair<DetId const, HGCRecHit const*> > > const&) const src/Validation/HGCalValidation/src/HGVHistoProducerAlgo.cc:3565
    #2 0x14c1cca5acf4 in HGVHistoProducerAlgo::fill_generic_cluster_histos(HGVHistoProducerAlgoHistograms const&, int, edm::Handle<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> > >, std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> > const&, edm::Handle<std::vector<CaloParticle, std::allocator<CaloParticle> > >, std::vector<CaloParticle, std::allocator<CaloParticle> > const&, std::vector<unsigned long, std::allocator<unsigned long> > const&, std::vector<unsigned long, std::allocator<unsigned long> > const&, std::unordered_map<DetId, HGCRecHit const*, std::hash<DetId>, std::equal_to<DetId>, std::allocator<std::pair<DetId const, HGCRecHit const*> > > const&, std::map<double, double, std::less<double>, std::allocator<std::pair<double const, double> > >, unsigned int, std::vector<int, std::allocator<int> >, edm::AssociationMap<edm::OneToManyWithQualityGeneric<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> >, std::vector<CaloParticle, std::allocator<CaloParticle> >, float, unsigned int, edm::RefProd<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> > >, edm::RefProd<std::vector<CaloParticle, std::allocator<CaloParticle> > >, edm::Ref<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> >, reco::CaloCluster, edm::refhelper::FindUsingAdvance<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> >, reco::CaloCluster> >, edm::Ref<std::vector<CaloParticle, std::allocator<CaloParticle> >, CaloParticle, edm::refhelper::FindUsingAdvance<std::vector<CaloParticle, std::allocator<CaloParticle> >, CaloParticle> > > > const&, edm::AssociationMap<edm::OneToManyWithQualityGeneric<std::vector<CaloParticle, std::allocator<CaloParticle> >, std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> >, std::pair<float, float>, unsigned int, edm::RefProd<std::vector<CaloParticle, std::allocator<CaloParticle> > >, edm::RefProd<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> > >, edm::Ref<std::vector<CaloParticle, std::allocator<CaloParticle> >, CaloParticle, edm::refhelper::FindUsingAdvance<std::vector<CaloParticle, std::allocator<CaloParticle> >, CaloParticle> >, edm::Ref<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> >, reco::CaloCluster, edm::refhelper::FindUsingAdvance<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> >, reco::CaloCluster> > > > const&) const src/Validation/HGCalValidation/src/HGVHistoProducerAlgo.cc:2248
    #3 0x14c1ccdf4d99 in HGCalValidator::dqmAnalyze(edm::Event const&, edm::EventSetup const&, HGCalValidatorHistograms const&) const src/Validation/HGCalValidation/plugins/HGCalValidator.cc:380
    #4 0x14c30114954b in edm::global::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) src/FWCore/Framework/src/global/EDProducerBase.cc:69

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Thu Feb 15 15:11:33 CET 2024
Thread 5 (Thread 0x14c22e0ce700 (LWP 1801787) "cmsRun"):
#2  0x000014c2efc5cd7e in (anonymous namespace)::sig_pause_for_stacktrace (sig=<optimized out>) at src/FWCore/Services/plugins/InitRootHandlers.cc:453
#3  <signal handler called>
#4  0x000014bf49777852 in std::vector<float, std::allocator<float> >::operator[] (__n=212, this=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/bits/stl_vector.h:1124
#5  TSToSimTSHitLCAssociatorByEnergyScoreImpl::makeConnections (this=<optimized out>, tCH=..., lCCH=..., sCCH=..., cPCH=..., sTCH=...) at src/SimCalorimetry/HGCalAssociatorProducers/plugins/TSToSimTSHitLCAssociatorByEnergyScoreImpl.cc:151
#6  0x000014bf49784833 in TSToSimTSHitLCAssociatorByEnergyScoreImpl::associateRecoToSim (this=<optimized out>, tCH=..., lCCH=..., sCCH=..., cPCH=..., sTCH=...) at src/SimCalorimetry/HGCalAssociatorProducers/plugins/TSToSimTSHitLCAssociatorByEnergyScoreImpl.cc:216
#7  0x000014bf497b926a in hgcal::TracksterToSimTracksterHitLCAssociator::associateRecoToSim (sTCH=..., cPCH=..., sCCH=..., lCCH=..., tCH=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/tmp/BUILDROOT/c0a455c7a5b724be894a663904dad922/opt/cmssw/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_UBSAN_X_2024-02-14-2300/src/SimDataFormats/Associations/interface/TracksterToSimTracksterHitLCAssociator.h:39
#8  TSToSimTSHitLCAssociatorEDProducer::produce (this=<optimized out>, iEvent=..., iSetup=...) at src/SimCalorimetry/HGCalAssociatorProducers/plugins/TSToSimTSHitLCAssociatorEDProducer.cc:90
#9  0x000014c30114954c in edm::global::EDProducerBase::doEvent (this=this@entry=0x14bed0ffd430, info=..., act=0x14c2f48c7610, mcc=mcc@entry=0x14bed29a8ee8) at /data/cmsbld/jenkins/workspace/build-any-ib/w/tmp/BUILDROOT/c0a455c7a5b724be894a663904dad922/opt/cmssw/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_UBSAN_X_2024-02-14-2300/src/FWCore/Framework/interface/Event.h:98

Thread 4 (Thread 0x14c23057b700 (LWP 1801786) "cmsRun"):
#2  0x000014c2efc5cd7e in (anonymous namespace)::sig_pause_for_stacktrace (sig=<optimized out>) at src/FWCore/Services/plugins/InitRootHandlers.cc:453
#3  <signal handler called>
#4  0x000014c2aed4404c in PerigeeLinearizedTrackState::operator== (this=0x14bdf9879800, other=...) at src/RecoVertex/VertexTools/src/PerigeeLinearizedTrackState.cc:30
#5  0x000014c29c94921a in VertexTrack<5u>::operator== (this=<optimized out>, data=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/tmp/BUILDROOT/c0a455c7a5b724be894a663904dad922/opt/cmssw/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_UBSAN_X_2024-02-14-2300/src/RecoVertex/VertexPrimitives/interface/VertexTrack.h:94
#6  0x000014c29c94f4d8 in VertexTrackEqual<5u>::operator() (t=..., this=0x14c230573990) at /data/cmsbld/jenkins/workspace/build-any-ib/w/tmp/BUILDROOT/c0a455c7a5b724be894a663904dad922/opt/cmssw/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_UBSAN_X_2024-02-14-2300/src/RecoVertex/VertexPrimitives/interface/VertexTrack.h:118
#7  KalmanVertexTrackCompatibilityEstimator<5u>::estimate (this=0x14c1bebea980, vertex=..., tr=..., hint=<optimized out>) at src/RecoVertex/KalmanVertexFit/src/KalmanVertexTrackCompatibilityEstimator.cc:19
#8  0x000014c29c94770b in KalmanVertexTrackCompatibilityEstimator<5u>::estimate (this=0x14c1bebea980, vertex=..., track=..., hint=30) at src/RecoVertex/KalmanVertexFit/src/KalmanVertexTrackCompatibilityEstimator.cc:36
#9  0x000014c29cdd641e in AdaptiveVertexFitter::reWeightTracks (this=0x14c1bed39360, lTracks=..., vertex=...) at src/RecoVertex/AdaptiveVertexFit/src/AdaptiveVertexFitter.cc:361
#10 0x000014c29cdda458 in AdaptiveVertexFitter::reWeightTracks (this=0x14c1bed39360, tracks=..., seed=...) at src/RecoVertex/AdaptiveVertexFit/src/AdaptiveVertexFitter.cc:444
#11 0x000014c29cded16b in AdaptiveVertexFitter::fit (this=0x14c1bed39360, tracks=..., priorSeed=..., withPrior=<optimized out>) at src/RecoVertex/AdaptiveVertexFit/src/AdaptiveVertexFitter.cc:505
#12 0x000014c29cdf187c in AdaptiveVertexFitter::vertex (this=0x14c1bed39360, unstracks=...) at src/RecoVertex/AdaptiveVertexFit/src/AdaptiveVertexFitter.cc:172
#13 0x000014c1b9de9866 in PrimaryVertexResolution::Plots::calculateAndFillResolution (this=0x14c1bed396e8, tracks=..., nvertices=35, lumi=0, engine=..., fitter=...) at src/DQM/TrackingMonitor/src/PrimaryVertexResolution.cc:490
#14 0x000014c1b9deff08 in PrimaryVertexResolution::analyze (this=0x14c1bed39000, iEvent=..., iSetup=...) at src/DQM/TrackingMonitor/src/PrimaryVertexResolution.cc:435
#15 0x000014c30125b3ef in edm::stream::EDProducerAdaptorBase::doEvent (this=this@entry=0x14c1bed09bc0, info=..., act=0x14c2f48c7610, mcc=mcc@entry=0x14bed1132fa8) at src/FWCore/Framework/src/stream/EDProducerAdaptorBase.cc:82

Thread 3 (Thread 0x14c230f7c700 (LWP 1801785) "cmsRun"):
#2  0x000014c2efc5cd7e in (anonymous namespace)::sig_pause_for_stacktrace (sig=<optimized out>) at src/FWCore/Services/plugins/InitRootHandlers.cc:453
#3  <signal handler called>
#4  0x000014bf405e8801 in std::__array_traits<int, 289ul>::_S_ref (__n=<optimized out>, __t=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/array:61
#5  std::array<int, 289ul>::operator[] (__n=231, this=0x14bf408bad60 <ROOT::Math::MatRepSym<double, 17u>::off(int)::v>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/array:219
#6  ROOT::Math::MatRepSym<double, 17u>::off (i=231) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/MatrixRepresentationsStatic.h:309
#7  ROOT::Math::MatRepSym<double, 17u>::offset (j=10, i=13) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/MatrixRepresentationsStatic.h:316
#8  ROOT::Math::MatRepSym<double, 17u>::operator() (j=10, i=13, this=0x14c230f73280) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/MatrixRepresentationsStatic.h:226
#9  ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >::operator() (j=10, i=13, this=0x14c230f73280) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/SMatrix.icc:640
#10 ROOT::Math::meta_matrix_dot<13u>::g<ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> > > (j=10, i=<optimized out>, rhs=..., lhs=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/MatrixFunctions.h:309
#11 ROOT::Math::meta_matrix_dot<14u>::g<ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> > > (j=10, i=<optimized out>, rhs=..., lhs=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/MatrixFunctions.h:310
#12 ROOT::Math::meta_matrix_dot<15u>::g<ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> > > (j=10, i=<optimized out>, rhs=..., lhs=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/MatrixFunctions.h:310
#13 ROOT::Math::meta_matrix_dot<16u>::g<ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> > > (lhs=..., rhs=..., i=<optimized out>, j=10) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/MatrixFunctions.h:310
#14 0x000014bf405f6edb in ROOT::Math::MatrixMulOp<ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, double, 17u>::operator() (j=10, i=14, this=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/MatrixFunctions.h:362
#15 ROOT::Math::Expr<ROOT::Math::MatrixMulOp<ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, double, 17u>, double, 17u, 17u, ROOT::Math::MatRepStd<double, 17u, 17u> >::operator() (j=10, i=14, this=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/Expression.h:154
#16 ROOT::Math::meta_matrix_dot<10u>::g<ROOT::Math::Expr<ROOT::Math::MatrixMulOp<ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, double, 17u>, double, 17u, 17u, ROOT::Math::MatRepStd<double, 17u, 17u> >, ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> > > (j=1, i=14, rhs=..., lhs=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/MatrixFunctions.h:309
#17 ROOT::Math::meta_matrix_dot<11u>::g<ROOT::Math::Expr<ROOT::Math::MatrixMulOp<ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, double, 17u>, double, 17u, 17u, ROOT::Math::MatRepStd<double, 17u, 17u> >, ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> > > (j=1, i=14, rhs=..., lhs=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/MatrixFunctions.h:310
#18 ROOT::Math::meta_matrix_dot<12u>::g<ROOT::Math::Expr<ROOT::Math::MatrixMulOp<ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, double, 17u>, double, 17u, 17u, ROOT::Math::MatRepStd<double, 17u, 17u> >, ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> > > (j=1, i=14, rhs=..., lhs=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/MatrixFunctions.h:310
#19 ROOT::Math::meta_matrix_dot<13u>::g<ROOT::Math::Expr<ROOT::Math::MatrixMulOp<ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, double, 17u>, double, 17u, 17u, ROOT::Math::MatRepStd<double, 17u, 17u> >, ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> > > (j=1, i=14, rhs=..., lhs=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/MatrixFunctions.h:310
#20 ROOT::Math::meta_matrix_dot<14u>::g<ROOT::Math::Expr<ROOT::Math::MatrixMulOp<ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, double, 17u>, double, 17u, 17u, ROOT::Math::MatRepStd<double, 17u, 17u> >, ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> > > (j=1, i=14, rhs=..., lhs=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/MatrixFunctions.h:310
#21 ROOT::Math::meta_matrix_dot<15u>::g<ROOT::Math::Expr<ROOT::Math::MatrixMulOp<ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, double, 17u>, double, 17u, 17u, ROOT::Math::MatRepStd<double, 17u, 17u> >, ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> > > (j=1, i=14, rhs=..., lhs=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/MatrixFunctions.h:310
#22 ROOT::Math::meta_matrix_dot<16u>::g<ROOT::Math::Expr<ROOT::Math::MatrixMulOp<ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, double, 17u>, double, 17u, 17u, ROOT::Math::MatRepStd<double, 17u, 17u> >, ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> > > (j=1, i=14, rhs=..., lhs=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/MatrixFunctions.h:310
#23 ROOT::Math::MatrixMulOp<ROOT::Math::Expr<ROOT::Math::MatrixMulOp<ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, double, 17u>, double, 17u, 17u, ROOT::Math::MatRepStd<double, 17u, 17u> >, ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, double, 17u>::operator() (j=1, i=14, this=0x14c230f72f90) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/MatrixFunctions.h:363
#24 ROOT::Math::Expr<ROOT::Math::MatrixMulOp<ROOT::Math::Expr<ROOT::Math::MatrixMulOp<ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, double, 17u>, double, 17u, 17u, ROOT::Math::MatRepStd<double, 17u, 17u> >, ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, double, 17u>, double, 17u, 17u, ROOT::Math::MatRepStd<double, 17u, 17u> >::operator() (j=1, i=14, this=0x14c230f72f90) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/Expression.h:154
#25 ROOT::Math::AssignSym::Evaluate<double, 17u, ROOT::Math::MatrixMulOp<ROOT::Math::Expr<ROOT::Math::MatrixMulOp<ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, double, 17u>, double, 17u, 17u, ROOT::Math::MatRepStd<double, 17u, 17u> >, ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, double, 17u>, ROOT::Math::MatRepStd<double, 17u, 17u> > (lhs=..., rhs=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/HelperOps.h:163
#26 0x000014bf406a2916 in KinematicConstrainedVertexUpdatorT<2, 2>::update (this=0x14bdedbad200, inPar=..., inCov=..., lStates=..., lPoint=..., fieldValue=..., cs=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/MatrixFunctions.h:355
#27 0x000014bf406c81c2 in KinematicConstrainedVertexFitterT<2, 2>::fit (this=this@entry=0x14c230f74ac0, part=..., cs=cs@entry=0x14c230f74b00, pt=pt@entry=0x0) at /data/cmsbld/jenkins/workspace/build-any-ib/w/tmp/BUILDROOT/c0a455c7a5b724be894a663904dad922/opt/cmssw/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_UBSAN_X_2024-02-14-2300/src/RecoVertex/KinematicFit/interface/KinematicConstrainedVertexFitterT.h:215
#28 0x000014bf4048d7ef in KinematicConstrainedVertexFitterT<2, 2>::fit (cs=0x14c230f74b00, part=..., this=0x14c230f74ac0) at /data/cmsbld/jenkins/workspace/build-any-ib/w/tmp/BUILDROOT/c0a455c7a5b724be894a663904dad922/opt/cmssw/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_UBSAN_X_2024-02-14-2300/src/RecoVertex/KinematicFit/interface/KinematicConstrainedVertexFitterT.h:55
#29 ConversionVertexFinder::run (this=this@entry=0x14bf534e4598, _pair=..., the_vertex=...) at src/RecoEgamma/EgammaPhotonAlgos/src/ConversionVertexFinder.cc:69
#30 0x000014bf4179aaab in ConversionProducer::checkVertex (magField=..., the_vertex=..., ttk_r=..., ttk_l=..., this=0x14bf534e4200) at src/RecoEgamma/EgammaPhotonProducers/src/ConversionProducer.cc:165
#31 ConversionProducer::buildCollection (this=0x14bf534e4200, iEvent=..., iSetup=..., allTracks=..., superClusterPtrs=..., basicClusterPtrs=..., the_pvtx=..., outputConvPhotonCollection=...) at src/RecoEgamma/EgammaPhotonProducers/src/ConversionProducer.cc:535
#32 0x000014bf417b3e47 in ConversionProducer::produce (this=0x14bf534e4200, iEvent=..., iSetup=...) at src/RecoEgamma/EgammaPhotonProducers/src/ConversionProducer.cc:343
#33 0x000014c30125b3ef in edm::stream::EDProducerAdaptorBase::doEvent (this=this@entry=0x14bf53cda060, info=..., act=0x14c2f48c7610, mcc=mcc@entry=0x14bed1917de8) at src/FWCore/Framework/src/stream/EDProducerAdaptorBase.cc:82

Thread 1 (Thread 0x14c2f738f680 (LWP 1801360) "cmsRun"):
#3  0x000014c2efc7a786 in (anonymous namespace)::sig_dostack_then_abort (sig=<optimized out>) at src/FWCore/Services/plugins/InitRootHandlers.cc:546
#4  <signal handler called>
#5  0x000014c1cc9e92e7 in HGVHistoProducerAlgo::findmaxhit (this=this@entry=0x14c1dcdb1200, cluster=..., hitMap=...) at src/Validation/HGCalValidation/src/HGVHistoProducerAlgo.cc:3567
#6  0x000014c1cca5acf5 in HGVHistoProducerAlgo::fill_generic_cluster_histos (this=this@entry=0x14c1dcdb1200, histograms=..., count=count@entry=0, clusterHandle=..., clusters=..., caloParticleHandle=..., cP=..., cPIndices=..., cPSelectedIndices=..., hitMap=..., cummatbudg=..., layers=<optimized out>, thicknesses=..., cpsInLayerClusterMap=..., cPOnLayerMap=...) at src/Validation/HGCalValidation/src/HGVHistoProducerAlgo.cc:2248
#7  0x000014c1ccdf4d9a in HGCalValidator::dqmAnalyze (this=<optimized out>, event=..., setup=..., histograms=...) at src/Validation/HGCalValidation/plugins/HGCalValidator.cc:380
#8  0x000014c30114954c in edm::global::EDProducerBase::doEvent (this=this@entry=0x14c1dbb6cb60, info=..., act=0x14c2f48c7610, mcc=mcc@entry=0x14c1dd0c9d68) at /data/cmsbld/jenkins/workspace/build-any-ib/w/tmp/BUILDROOT/c0a455c7a5b724be894a663904dad922/opt/cmssw/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_UBSAN_X_2024-02-14-2300/src/FWCore/Framework/interface/Event.h:98

Current Modules:

Module: HGCalValidator:hgcalValidator (crashed)
Module: ConversionProducer:allConversions
Module: PrimaryVertexResolution:primaryVertexResolution
Module: TSToSimTSHitLCAssociatorEDProducer:tracksterSimTracksterAssociationLinkingPU

@dan131riley
Copy link

Stack trace from https://cmssdt.cern.ch/SDT/cgi-bin/logreader/el8_amd64_gcc12/CMSSW_14_1_ROOT626_X_2024-02-15-2300/pyRelValMatrixLogs/run/25034.999_TTbar_14TeV+2026D98PU_PMXS1S2PR/step4_TTbar_14TeV+2026D98PU_PMXS1S2PR.log#/151-151

I'm including this as a "guilt by association" trace. Many of the crashes in these WFs have TSToSimTSHitLCAssociatorEDProducer as a bystander. This one has a crash in TSToSimTSHitLCAssociatorEDProducer. Possibly still an innocent bystander, just collecting evidence.

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Fri Feb 16 08:26:02 CET 2024
Thread 5 (Thread 0x1554ba7ff700 (LWP 3421091) "cmsRun"):
#2  0x0000155503cdfcc0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02824/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ROOT626_X_2024-02-15-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00001554de3b3ae3 in ClosestApproachInRPhi::zCoord(Vector3DBase<float, GlobalTag> const&, Point3DBase<float, GlobalTag> const&, double, double, double, double, double) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02824/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ROOT626_X_2024-02-15-2300/lib/el8_amd64_gcc12/libTrackingToolsPatternTools.so
#5  0x00001554de3b3e0d in ClosestApproachInRPhi::compute(int const&, Vector3DBase<float, GlobalTag> const&, Point3DBase<float, GlobalTag> const&, int const&, Vector3DBase<float, GlobalTag> const&, Point3DBase<float, GlobalTag> const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02824/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ROOT626_X_2024-02-15-2300/lib/el8_amd64_gcc12/libTrackingToolsPatternTools.so
#6  0x00001554de3b431d in ClosestApproachInRPhi::calculate(FreeTrajectoryState const&, FreeTrajectoryState const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02824/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ROOT626_X_2024-02-15-2300/lib/el8_amd64_gcc12/libTrackingToolsPatternTools.so
#7  0x000015545e3a8d8b in V0Fitter::fitAll(edm::Event const&, edm::EventSetup const&, std::vector<reco::VertexCompositeCandidate, std::allocator<reco::VertexCompositeCandidate> >&, std::vector<reco::VertexCompositeCandidate, std::allocator<reco::VertexCompositeCandidate> >&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02824/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ROOT626_X_2024-02-15-2300/lib/el8_amd64_gcc12/pluginRecoVertexV0Producer.so
#8  0x000015545e3ac017 in V0Producer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02824/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ROOT626_X_2024-02-15-2300/lib/el8_amd64_gcc12/pluginRecoVertexV0Producer.so
#9  0x000015550b632e91 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02824/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ROOT626_X_2024-02-15-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so

Thread 4 (Thread 0x1554bb7a9700 (LWP 3421090) "cmsRun"):
#2  0x0000155503cdfcc0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02824/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ROOT626_X_2024-02-15-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000015547d3ddb87 in std::__detail::_Map_base<DetId, std::pair<DetId const, std::vector<std::pair<int, float>, std::allocator<std::pair<int, float> > > >, std::allocator<std::pair<DetId const, std::vector<std::pair<int, float>, std::allocator<std::pair<int, float> > > > >, std::__detail::_Select1st, std::equal_to<DetId>, std::hash<DetId>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true>, true>::operator[](DetId const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02824/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ROOT626_X_2024-02-15-2300/lib/el8_amd64_gcc12/pluginSimCalorimetryHGCalAssociatorProducersAuto.so
#5  0x000015547d3e169c in TSToSimTSHitLCAssociatorByEnergyScoreImpl::makeConnections(edm::Handle<std::vector<ticl::Trackster, std::allocator<ticl::Trackster> > > const&, edm::Handle<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> > > const&, edm::Handle<std::vector<SimCluster, std::allocator<SimCluster> > > const&, edm::Handle<std::vector<CaloParticle, std::allocator<CaloParticle> > > const&, edm::Handle<std::vector<ticl::Trackster, std::allocator<ticl::Trackster> > > const&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02824/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ROOT626_X_2024-02-15-2300/lib/el8_amd64_gcc12/pluginSimCalorimetryHGCalAssociatorProducersAuto.so
#6  0x000015547d3e418e in TSToSimTSHitLCAssociatorByEnergyScoreImpl::associateRecoToSim(edm::Handle<std::vector<ticl::Trackster, std::allocator<ticl::Trackster> > > const&, edm::Handle<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> > > const&, edm::Handle<std::vector<SimCluster, std::allocator<SimCluster> > > const&, edm::Handle<std::vector<CaloParticle, std::allocator<CaloParticle> > > const&, edm::Handle<std::vector<ticl::Trackster, std::allocator<ticl::Trackster> > > const&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02824/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ROOT626_X_2024-02-15-2300/lib/el8_amd64_gcc12/pluginSimCalorimetryHGCalAssociatorProducersAuto.so
#7  0x000015547d3dfe3f in TSToSimTSHitLCAssociatorEDProducer::produce(edm::StreamID, edm::Event&, edm::EventSetup const&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02824/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ROOT626_X_2024-02-15-2300/lib/el8_amd64_gcc12/pluginSimCalorimetryHGCalAssociatorProducersAuto.so
#8  0x000015550b620406 in edm::global::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02824/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ROOT626_X_2024-02-15-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so

Thread 3 (Thread 0x1554bc1aa700 (LWP 3421089) "cmsRun"):
#3  0x0000155503ce3720 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02824/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ROOT626_X_2024-02-15-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x000015547d3e1c8c in TSToSimTSHitLCAssociatorByEnergyScoreImpl::makeConnections(edm::Handle<std::vector<ticl::Trackster, std::allocator<ticl::Trackster> > > const&, edm::Handle<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> > > const&, edm::Handle<std::vector<SimCluster, std::allocator<SimCluster> > > const&, edm::Handle<std::vector<CaloParticle, std::allocator<CaloParticle> > > const&, edm::Handle<std::vector<ticl::Trackster, std::allocator<ticl::Trackster> > > const&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02824/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ROOT626_X_2024-02-15-2300/lib/el8_amd64_gcc12/pluginSimCalorimetryHGCalAssociatorProducersAuto.so
#6  0x000015547d3e418e in TSToSimTSHitLCAssociatorByEnergyScoreImpl::associateRecoToSim(edm::Handle<std::vector<ticl::Trackster, std::allocator<ticl::Trackster> > > const&, edm::Handle<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> > > const&, edm::Handle<std::vector<SimCluster, std::allocator<SimCluster> > > const&, edm::Handle<std::vector<CaloParticle, std::allocator<CaloParticle> > > const&, edm::Handle<std::vector<ticl::Trackster, std::allocator<ticl::Trackster> > > const&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02824/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ROOT626_X_2024-02-15-2300/lib/el8_amd64_gcc12/pluginSimCalorimetryHGCalAssociatorProducersAuto.so
#7  0x000015547d3dfe3f in TSToSimTSHitLCAssociatorEDProducer::produce(edm::StreamID, edm::Event&, edm::EventSetup const&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02824/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ROOT626_X_2024-02-15-2300/lib/el8_amd64_gcc12/pluginSimCalorimetryHGCalAssociatorProducersAuto.so
#8  0x000015550b620406 in edm::global::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02824/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ROOT626_X_2024-02-15-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so

Thread 1 (Thread 0x155508047640 (LWP 3420936) "cmsRun"):
#2  0x0000155503cdfcc0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02824/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ROOT626_X_2024-02-15-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  cache_bin_alloc_impl (adjust_low_water=false, success=<synthetic pointer>, bin=<optimized out>) at include/jemalloc/internal/cache_bin.h:361
#5  cache_bin_alloc_easy (success=<synthetic pointer>, bin=<optimized out>) at include/jemalloc/internal/cache_bin.h:399
#6  imalloc_fastpath (fallback_alloc=0x15550a1fb330 <fallback_impl<false>(std::size_t)>, size=3780) at include/jemalloc/internal/jemalloc_internal_inlines_c.h:326
#7  newImpl<false> (size=3780) at src/jemalloc_cpp.cpp:109
#8  operator new (size=3780) at src/jemalloc_cpp.cpp:114
#9  0x000015545e1b951f in void (anonymous namespace)::getSimTrackId<PixelDigiSimLink>(std::vector<std::pair<unsigned int, EncodedEventId>, std::allocator<std::pair<unsigned int, EncodedEventId> > >&, edm::Handle<edm::DetSetVector<PixelDigiSimLink> > const&, DetId const&, unsigned int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02824/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ROOT626_X_2024-02-15-2300/lib/el8_amd64_gcc12/pluginSimTrackerTrackerHitAssociationPlugins.so
#10 0x000015545e1bcb28 in ClusterTPAssociationProducer::produce(edm::StreamID, edm::Event&, edm::EventSetup const&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02824/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ROOT626_X_2024-02-15-2300/lib/el8_amd64_gcc12/pluginSimTrackerTrackerHitAssociationPlugins.so
#11 0x000015550b620406 in edm::global::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02824/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ROOT626_X_2024-02-15-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so

Current Modules:

Module: TSToSimTSHitLCAssociatorEDProducer:tracksterSimTracksterAssociationLinkingbyCLUE3D (crashed)
Module: V0Producer:generalV0Candidates
Module: TSToSimTSHitLCAssociatorEDProducer:tracksterSimTracksterAssociationLinkingPU
Module: ClusterTPAssociationProducer:tpClusterProducer

@makortel
Copy link
Contributor

Seems like the problems started in CMSSW_14_1_X_2024-02-09-2300

That IB added the following PRs (to document now before the information, at least in easily accessible form, disappears)

@dan131riley
Copy link

Three UBSAN crashes:

src/RecoLocalFastTime/FTLClusterizer/src/MTDThresholdClusterizer.cc:212:91: runtime error: member access within null pointer of type 'const struct TrackerGeomDet'
    #0 0x14eeac8dc3aa in MTDThresholdClusterizer::copy_to_buffer(__gnu_cxx::__normal_iterator<FTLRecHit const*, std::vector<FTLRecHit, std::allocator<FTLRecHit> > >, MTDGeometry const*, MTDTopology const*) src/RecoLocalFastTime/FTLClusterizer/src/MTDThresholdClusterizer.cc:212
    #1 0x14eeac8de506 in MTDThresholdClusterizer::clusterize(edm::SortedCollection<FTLRecHit, edm::StrictWeakOrdering<FTLRecHit> > const&, MTDGeometry const*, MTDTopology const*, edmNew::DetSetVector<FTLCluster>&) src/RecoLocalFastTime/FTLClusterizer/src/MTDThresholdClusterizer.cc:150
    #2 0x14eeacac5350 in MTDClusterProducer::produce(edm::Event&, edm::EventSetup const&) src/RecoLocalFastTime/FTLClusterizer/plugins/MTDClusterProducer.cc:135
    #3 0x14ef41c433ee in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) src/FWCore/Framework/src/stream/EDProducerAdaptorBase.cc:82

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Sat Feb 17 21:43:49 CET 2024
Thread 5 (Thread 0x14ee7ddff700 (LWP 3998355) "cmsRun"):
#2  0x000014ef30a9ad7e in (anonymous namespace)::sig_pause_for_stacktrace (sig=<optimized out>) at src/FWCore/Services/plugins/InitRootHandlers.cc:453
#3  <signal handler called>
#4  0x000014eeb81e5df0 in tensorflow::Tensor::CheckTypeAndIsAligned(tensorflow::DataType) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02824/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_UBSAN_X_2024-02-16-2300/external/el8_amd64_gcc12/lib/libtensorflow_framework.so.2
#5  0x000014eecad4ffe7 in tensorflow::Tensor::tensor<float, 4ul> (this=0x14ee7ddf8280) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/external/tensorflow/2.12.0-00a1cdfd30174f21df62ceb8138dbe97/include/tensorflow/core/framework/tensor.h:766
#6  0x000014ee1195c8b3 in ticl::PatternRecognitionbyCLUE3D<TICLGenericTile<std::array<TICLLayerTileT<ticl::TileConstants>, 104ul> > >::energyRegressionAndID (this=this@entry=0x14edf29a8e80, layerClusters=..., eidSession=0x14edacb72000, tracksters=...) at src/RecoHGCal/TICL/plugins/PatternRecognitionbyCLUE3D.cc:458
#7  0x000014ee11967339 in ticl::PatternRecognitionbyCLUE3D<TICLGenericTile<std::array<TICLLayerTileT<ticl::TileConstants>, 104ul> > >::makeTracksters (this=0x14edf29a8e80, input=..., result=..., seedToTracksterAssociation=...) at src/RecoHGCal/TICL/plugins/PatternRecognitionbyCLUE3D.cc:330
#8  0x000014ee11eb794b in TrackstersProducer::produce (this=<optimized out>, evt=..., es=...) at src/RecoHGCal/TICL/plugins/TrackstersProducer.cc:181
#9  0x000014ef41c433ef in edm::stream::EDProducerAdaptorBase::doEvent (this=this@entry=0x14edf58ebbe0, info=..., act=0x14ef33f10810, mcc=mcc@entry=0x14edff92f8e8) at src/FWCore/Framework/src/stream/EDProducerAdaptorBase.cc:82

Thread 4 (Thread 0x14ee7f585700 (LWP 3998354) "cmsRun"):
#2  0x000014ef30a9ad7e in (anonymous namespace)::sig_pause_for_stacktrace (sig=<optimized out>) at src/FWCore/Services/plugins/InitRootHandlers.cc:453
#3  <signal handler called>
#4  0x000014ef3801b015 in ZSTD_decompressSequences_bmi2.constprop.0 () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02824/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_UBSAN_X_2024-02-16-2300/external/el8_amd64_gcc12/lib/libzstd.so.1
#5  0x000014ef38022b6e in ZSTD_decompressBlock_internal () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02824/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_UBSAN_X_2024-02-16-2300/external/el8_amd64_gcc12/lib/libzstd.so.1
#6  0x000014ef38017e17 in ZSTD_decompressMultiFrame () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02824/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_UBSAN_X_2024-02-16-2300/external/el8_amd64_gcc12/lib/libzstd.so.1
#7  0x000014ef380181f0 in ZSTD_decompress_usingDDict () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02824/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_UBSAN_X_2024-02-16-2300/external/el8_amd64_gcc12/lib/libzstd.so.1
#8  0x000014ef3ab3bf18 in R__unzipZSTD () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02824/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_UBSAN_X_2024-02-16-2300/external/el8_amd64_gcc12/lib/libCore.so
#9  0x000014ef3ab3b5cc in R__unzip () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02824/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_UBSAN_X_2024-02-16-2300/external/el8_amd64_gcc12/lib/libCore.so
#10 0x000014ef3b514bf7 in TBasket::ReadBasketBuffers(long long, int, TFile*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02824/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_UBSAN_X_2024-02-16-2300/external/el8_amd64_gcc12/lib/libTree.so
#11 0x000014ef3b51fac4 in TBranch::GetBasketImpl(int, TBuffer*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02824/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_UBSAN_X_2024-02-16-2300/external/el8_amd64_gcc12/lib/libTree.so
#12 0x000014ef3b5211cd in TBranch::GetBasketAndFirst(TBasket*&, long long&, TBuffer*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02824/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_UBSAN_X_2024-02-16-2300/external/el8_amd64_gcc12/lib/libTree.so
#13 0x000014ef3b5213b7 in TBranch::GetEntry(long long, int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02824/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_UBSAN_X_2024-02-16-2300/external/el8_amd64_gcc12/lib/libTree.so
#14 0x000014ef3b533d74 in TBranchElement::GetEntry(long long, int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02824/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_UBSAN_X_2024-02-16-2300/external/el8_amd64_gcc12/lib/libTree.so
#15 0x000014ef3b533d2d in TBranchElement::GetEntry(long long, int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02824/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_UBSAN_X_2024-02-16-2300/external/el8_amd64_gcc12/lib/libTree.so
#16 0x000014ee7d02bac8 in edm::RootTree::getEntry (this=this@entry=0x14ef31eeb598, branch=branch@entry=0x14ee6ee0c400, entryNumber=2) at src/IOPool/Input/src/RootTree.cc:386
#17 0x000014ee7cdc2ea6 in edm::RootDelayedReader::getProduct_ (this=0x14ee6e934e30, k=..., ep=<optimized out>) at src/IOPool/Input/src/RootDelayedReader.cc:91
#18 0x000014ef40a0fc18 in edm::DelayedReader::getProduct (this=0x14ee6e934e30, k=..., ep=ep@entry=0x14ee6ff08b90, mcc=mcc@entry=0x14edff83ca28) at src/FWCore/Framework/src/DelayedReader.cc:33
#19 0x000014ef4144097f in operator() (__closure=0x14ee7f57eca0) at src/FWCore/Framework/src/ProductResolvers.cc:301
#20 0x000014ef41441b68 in edm::DataManagingProductResolver::resolveProductImpl<true, edm::DelayedReaderInputProductResolver::prefetchAsync_(edm::WaitingTaskHolder, const edm::Principal&, bool, const edm::ServiceToken&, edm::SharedResourcesAcquirer*, const edm::ModuleCallingContext*) const::<lambda()>::<lambda()> > (resolver=..., this=0x14ee6ffe4180) at src/FWCore/Framework/src/ProductResolvers.cc:66
#21 operator() (__closure=0x14edff0da518) at src/FWCore/Framework/src/ProductResolvers.cc:289
#22 0x000014ef41442f47 in edm::SerialTaskQueueChain::actionToRun<edm::DelayedReaderInputProductResolver::prefetchAsync_(edm::WaitingTaskHolder, const edm::Principal&, bool, const edm::ServiceToken&, edm::SharedResourcesAcquirer*, const edm::ModuleCallingContext*) const::<lambda()>&>(struct {...} &) (this=0x14eeaa4ea580, iAction=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/tmp/BUILDROOT/5fe42f26621cd4d5b00bd4943abd9965/opt/cmssw/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_UBSAN_X_2024-02-16-2300/src/FWCore/Concurrency/interface/SerialTaskQueueChain.h:113

Thread 3 (Thread 0x14ee7ff86700 (LWP 3998353) "cmsRun"):
#2  0x000014ef30a9ad7e in (anonymous namespace)::sig_pause_for_stacktrace (sig=<optimized out>) at src/FWCore/Services/plugins/InitRootHandlers.cc:453
#3  <signal handler called>
#4  0x000014eede5e9b6e in ROOT::Math::MatRepStd<double, 5u, 5u>::operator() (j=2, i=3, this=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/MatrixRepresentationsStatic.h:60
#5  ROOT::Math::SMatrix<double, 5u, 5u, ROOT::Math::MatRepStd<double, 5u, 5u> >::operator() (j=2, i=3, this=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/SMatrix.icc:640
#6  ROOT::Math::meta_matrix_dot<2u>::g<ROOT::Math::SMatrix<double, 5u, 5u, ROOT::Math::MatRepStd<double, 5u, 5u> >, ROOT::Math::Expr<ROOT::Math::TransposeOp<ROOT::Math::SMatrix<double, 5u, 5u, ROOT::Math::MatRepStd<double, 5u, 5u> >, double, 5u, 5u>, double, 5u, 5u, ROOT::Math::MatRepStd<double, 5u, 5u> > > (j=0, i=3, rhs=..., lhs=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/MatrixFunctions.h:309
#7  ROOT::Math::meta_matrix_dot<3u>::g<ROOT::Math::SMatrix<double, 5u, 5u, ROOT::Math::MatRepStd<double, 5u, 5u> >, ROOT::Math::Expr<ROOT::Math::TransposeOp<ROOT::Math::SMatrix<double, 5u, 5u, ROOT::Math::MatRepStd<double, 5u, 5u> >, double, 5u, 5u>, double, 5u, 5u, ROOT::Math::MatRepStd<double, 5u, 5u> > > (j=0, i=3, rhs=..., lhs=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/MatrixFunctions.h:310
#8  ROOT::Math::meta_matrix_dot<4u>::g<ROOT::Math::SMatrix<double, 5u, 5u, ROOT::Math::MatRepStd<double, 5u, 5u> >, ROOT::Math::Expr<ROOT::Math::TransposeOp<ROOT::Math::SMatrix<double, 5u, 5u, ROOT::Math::MatRepStd<double, 5u, 5u> >, double, 5u, 5u>, double, 5u, 5u, ROOT::Math::MatRepStd<double, 5u, 5u> > > (j=0, i=3, rhs=..., lhs=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/MatrixFunctions.h:310
#9  ROOT::Math::MatrixMulOp<ROOT::Math::SMatrix<double, 5u, 5u, ROOT::Math::MatRepStd<double, 5u, 5u> >, ROOT::Math::Expr<ROOT::Math::TransposeOp<ROOT::Math::SMatrix<double, 5u, 5u, ROOT::Math::MatRepStd<double, 5u, 5u> >, double, 5u, 5u>, double, 5u, 5u, ROOT::Math::MatRepStd<double, 5u, 5u> >, double, 5u>::operator() (j=0, i=3, this=0x14ee7ff7e540) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/MatrixFunctions.h:363
#10 ROOT::Math::Expr<ROOT::Math::MatrixMulOp<ROOT::Math::SMatrix<double, 5u, 5u, ROOT::Math::MatRepStd<double, 5u, 5u> >, ROOT::Math::Expr<ROOT::Math::TransposeOp<ROOT::Math::SMatrix<double, 5u, 5u, ROOT::Math::MatRepStd<double, 5u, 5u> >, double, 5u, 5u>, double, 5u, 5u, ROOT::Math::MatRepStd<double, 5u, 5u> >, double, 5u>, double, 5u, 5u, ROOT::Math::MatRepStd<double, 5u, 5u> >::operator() (j=0, i=3, this=0x14ee7ff7e540) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/Expression.h:154
#11 ROOT::Math::AssignSym::Evaluate<double, 5u, ROOT::Math::MatrixMulOp<ROOT::Math::SMatrix<double, 5u, 5u, ROOT::Math::MatRepStd<double, 5u, 5u> >, ROOT::Math::Expr<ROOT::Math::TransposeOp<ROOT::Math::SMatrix<double, 5u, 5u, ROOT::Math::MatRepStd<double, 5u, 5u> >, double, 5u, 5u>, double, 5u, 5u, ROOT::Math::MatRepStd<double, 5u, 5u> >, double, 5u>, ROOT::Math::MatRepStd<double, 5u, 5u> > (rhs=..., lhs=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/HelperOps.h:163
#12 ROOT::Math::Similarity<double, 5u, 5u, ROOT::Math::MatRepStd<double, 5u, 5u> > (lhs=..., rhs=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/MatrixFunctions.h:742
#13 0x000014eede5c0248 in (anonymous namespace)::lupdate<2> (aRecHit=..., tsos=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/MatrixFunctions.h:510
#14 KFUpdator::update (this=<optimized out>, this@entry=0x14ee7ff7f170, tsos=..., aRecHit=...) at src/TrackingTools/KalmanUpdators/src/KFUpdator.cc:182
#15 0x000014ee25b499c9 in SeedFromConsecutiveHitsCreator::buildSeed (this=0x14ee1723b608, seedCollection=..., hits=..., fts=...) at src/RecoTracker/TkSeedGenerator/plugins/SeedFromConsecutiveHitsCreator.cc:186
#16 0x000014ee25b4e772 in SeedFromConsecutiveHitsCreator::makeSeed (this=this@entry=0x14ee1723b608, seedCollection=..., hits=...) at src/RecoTracker/TkSeedGenerator/plugins/SeedFromConsecutiveHitsCreator.cc:111
#17 0x000014ee25b008fb in SeedCreatorFromRegionHitsEDProducerT<SeedFromConsecutiveHitsTripletOnlyCreator>::produce (this=0x14ee1723b300, iEvent=..., iSetup=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/tmp/BUILDROOT/5fe42f26621cd4d5b00bd4943abd9965/opt/cmssw/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_UBSAN_X_2024-02-16-2300/src/RecoTracker/TkSeedGenerator/interface/SeedCreatorFromRegionHitsEDProducerT.h:89
#18 0x000014ef41c433ef in edm::stream::EDProducerAdaptorBase::doEvent (this=this@entry=0x14ee17275ae0, info=..., act=0x14ef33f10810, mcc=mcc@entry=0x14edff85c8e8) at src/FWCore/Framework/src/stream/EDProducerAdaptorBase.cc:82

Thread 1 (Thread 0x14ef37d77680 (LWP 3997893) "cmsRun"):
#3  0x000014ef30ab8786 in (anonymous namespace)::sig_dostack_then_abort (sig=<optimized out>) at src/FWCore/Services/plugins/InitRootHandlers.cc:546
#4  <signal handler called>
#5  0x000014eeac8d9f6f in MTDThresholdClusterizer::copy_to_buffer (this=0x14ee50240140, itr=..., geom=<optimized out>, topo=<optimized out>) at src/RecoLocalFastTime/FTLClusterizer/src/MTDThresholdClusterizer.cc:212
#6  0x000014eeac8de507 in MTDThresholdClusterizer::clusterize (this=<optimized out>, input=..., geom=<optimized out>, topo=<optimized out>, output=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/bits/stl_iterator.h:1072
#7  0x000014eeacac5351 in MTDClusterProducer::produce (this=0x14ee0937b400, e=..., es=...) at src/RecoLocalFastTime/FTLClusterizer/plugins/MTDClusterProducer.cc:135
#8  0x000014ef41c433ef in edm::stream::EDProducerAdaptorBase::doEvent (this=this@entry=0x14ee03be5540, info=..., act=0x14ef33f10810, mcc=mcc@entry=0x14ee03bc25e8) at src/FWCore/Framework/src/stream/EDProducerAdaptorBase.cc:82

Current Modules:

Module: MTDClusterProducer:mtdClusters (crashed)
Module: TrackstersProducer:ticlTrackstersCLUE3DHigh
Module: SeedCreatorFromRegionConsecutiveHitsTripletOnlyEDProducer:initialStepSeeds
Module: none


src/Validation/HGCalValidation/src/HGVHistoProducerAlgo.cc:3565:61: runtime error: member access within null pointer of type 'const struct HGCRecHit'
    #0 0x14c1cc9e94cd in HGVHistoProducerAlgo::findmaxhit(reco::CaloCluster const&, std::unordered_map<DetId, HGCRecHit const*, std::hash<DetId>, std::equal_to<DetId>, std::allocator<std::pair<DetId const, HGCRecHit const*> > > const&) const src/Validation/HGCalValidation/src/HGVHistoProducerAlgo.cc:3565
    #1 0x14c1cca5acf4 in HGVHistoProducerAlgo::fill_generic_cluster_histos(HGVHistoProducerAlgoHistograms const&, int, edm::Handle<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> > >, std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> > const&, edm::Handle<std::vector<CaloParticle, std::allocator<CaloParticle> > >, std::vector<CaloParticle, std::allocator<CaloParticle> > const&, std::vector<unsigned long, std::allocator<unsigned long> > const&, std::vector<unsigned long, std::allocator<unsigned long> > const&, std::unordered_map<DetId, HGCRecHit const*, std::hash<DetId>, std::equal_to<DetId>, std::allocator<std::pair<DetId const, HGCRecHit const*> > > const&, std::map<double, double, std::less<double>, std::allocator<std::pair<double const, double> > >, unsigned int, std::vector<int, std::allocator<int> >, edm::AssociationMap<edm::OneToManyWithQualityGeneric<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> >, std::vector<CaloParticle, std::allocator<CaloParticle> >, float, unsigned int, edm::RefProd<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> > >, edm::RefProd<std::vector<CaloParticle, std::allocator<CaloParticle> > >, edm::Ref<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> >, reco::CaloCluster, edm::refhelper::FindUsingAdvance<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> >, reco::CaloCluster> >, edm::Ref<std::vector<CaloParticle, std::allocator<CaloParticle> >, CaloParticle, edm::refhelper::FindUsingAdvance<std::vector<CaloParticle, std::allocator<CaloParticle> >, CaloParticle> > > > const&, edm::AssociationMap<edm::OneToManyWithQualityGeneric<std::vector<CaloParticle, std::allocator<CaloParticle> >, std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> >, std::pair<float, float>, unsigned int, edm::RefProd<std::vector<CaloParticle, std::allocator<CaloParticle> > >, edm::RefProd<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> > >, edm::Ref<std::vector<CaloParticle, std::allocator<CaloParticle> >, CaloParticle, edm::refhelper::FindUsingAdvance<std::vector<CaloParticle, std::allocator<CaloParticle> >, CaloParticle> >, edm::Ref<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> >, reco::CaloCluster, edm::refhelper::FindUsingAdvance<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> >, reco::CaloCluster> > > > const&) const src/Validation/HGCalValidation/src/HGVHistoProducerAlgo.cc:2248
    #2 0x14c1ccdf4d99 in HGCalValidator::dqmAnalyze(edm::Event const&, edm::EventSetup const&, HGCalValidatorHistograms const&) const src/Validation/HGCalValidation/plugins/HGCalValidator.cc:380
    #3 0x14c30114954b in edm::global::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) src/FWCore/Framework/src/global/EDProducerBase.cc:69

src/Validation/HGCalValidation/src/HGVHistoProducerAlgo.cc:3565:61: runtime error: member call on null pointer of type 'const struct CaloRecHit'
    #0 0x14c1cc9e94a9 in HGVHistoProducerAlgo::findmaxhit(reco::CaloCluster const&, std::unordered_map<DetId, HGCRecHit const*, std::hash<DetId>, std::equal_to<DetId>, std::allocator<std::pair<DetId const, HGCRecHit const*> > > const&) const src/Validation/HGCalValidation/src/HGVHistoProducerAlgo.cc:3565
    #1 0x14c1cca5acf4 in HGVHistoProducerAlgo::fill_generic_cluster_histos(HGVHistoProducerAlgoHistograms const&, int, edm::Handle<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> > >, std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> > const&, edm::Handle<std::vector<CaloParticle, std::allocator<CaloParticle> > >, std::vector<CaloParticle, std::allocator<CaloParticle> > const&, std::vector<unsigned long, std::allocator<unsigned long> > const&, std::vector<unsigned long, std::allocator<unsigned long> > const&, std::unordered_map<DetId, HGCRecHit const*, std::hash<DetId>, std::equal_to<DetId>, std::allocator<std::pair<DetId const, HGCRecHit const*> > > const&, std::map<double, double, std::less<double>, std::allocator<std::pair<double const, double> > >, unsigned int, std::vector<int, std::allocator<int> >, edm::AssociationMap<edm::OneToManyWithQualityGeneric<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> >, std::vector<CaloParticle, std::allocator<CaloParticle> >, float, unsigned int, edm::RefProd<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> > >, edm::RefProd<std::vector<CaloParticle, std::allocator<CaloParticle> > >, edm::Ref<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> >, reco::CaloCluster, edm::refhelper::FindUsingAdvance<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> >, reco::CaloCluster> >, edm::Ref<std::vector<CaloParticle, std::allocator<CaloParticle> >, CaloParticle, edm::refhelper::FindUsingAdvance<std::vector<CaloParticle, std::allocator<CaloParticle> >, CaloParticle> > > > const&, edm::AssociationMap<edm::OneToManyWithQualityGeneric<std::vector<CaloParticle, std::allocator<CaloParticle> >, std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> >, std::pair<float, float>, unsigned int, edm::RefProd<std::vector<CaloParticle, std::allocator<CaloParticle> > >, edm::RefProd<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> > >, edm::Ref<std::vector<CaloParticle, std::allocator<CaloParticle> >, CaloParticle, edm::refhelper::FindUsingAdvance<std::vector<CaloParticle, std::allocator<CaloParticle> >, CaloParticle> >, edm::Ref<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> >, reco::CaloCluster, edm::refhelper::FindUsingAdvance<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> >, reco::CaloCluster> > > > const&) const src/Validation/HGCalValidation/src/HGVHistoProducerAlgo.cc:2248
    #2 0x14c1ccdf4d99 in HGCalValidator::dqmAnalyze(edm::Event const&, edm::EventSetup const&, HGCalValidatorHistograms const&) const src/Validation/HGCalValidation/plugins/HGCalValidator.cc:380
    #3 0x14c30114954b in edm::global::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) src/FWCore/Framework/src/global/EDProducerBase.cc:69

/data/cmsbld/jenkins/workspace/build-any-ib/w/tmp/BUILDROOT/c0a455c7a5b724be894a663904dad922/opt/cmssw/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_UBSAN_X_2024-02-14-2300/src/DataFormats/CaloRecHit/interface/CaloRecHit.h:29:43: runtime error: member access within null pointer of type 'const struct CaloRecHit'
    #0 0x14c1cc9e9485 in CaloRecHit::energy() const /data/cmsbld/jenkins/workspace/build-any-ib/w/tmp/BUILDROOT/c0a455c7a5b724be894a663904dad922/opt/cmssw/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_UBSAN_X_2024-02-14-2300/src/DataFormats/CaloRecHit/interface/CaloRecHit.h:29
    #1 0x14c1cc9e9485 in HGVHistoProducerAlgo::findmaxhit(reco::CaloCluster const&, std::unordered_map<DetId, HGCRecHit const*, std::hash<DetId>, std::equal_to<DetId>, std::allocator<std::pair<DetId const, HGCRecHit const*> > > const&) const src/Validation/HGCalValidation/src/HGVHistoProducerAlgo.cc:3565
    #2 0x14c1cca5acf4 in HGVHistoProducerAlgo::fill_generic_cluster_histos(HGVHistoProducerAlgoHistograms const&, int, edm::Handle<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> > >, std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> > const&, edm::Handle<std::vector<CaloParticle, std::allocator<CaloParticle> > >, std::vector<CaloParticle, std::allocator<CaloParticle> > const&, std::vector<unsigned long, std::allocator<unsigned long> > const&, std::vector<unsigned long, std::allocator<unsigned long> > const&, std::unordered_map<DetId, HGCRecHit const*, std::hash<DetId>, std::equal_to<DetId>, std::allocator<std::pair<DetId const, HGCRecHit const*> > > const&, std::map<double, double, std::less<double>, std::allocator<std::pair<double const, double> > >, unsigned int, std::vector<int, std::allocator<int> >, edm::AssociationMap<edm::OneToManyWithQualityGeneric<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> >, std::vector<CaloParticle, std::allocator<CaloParticle> >, float, unsigned int, edm::RefProd<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> > >, edm::RefProd<std::vector<CaloParticle, std::allocator<CaloParticle> > >, edm::Ref<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> >, reco::CaloCluster, edm::refhelper::FindUsingAdvance<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> >, reco::CaloCluster> >, edm::Ref<std::vector<CaloParticle, std::allocator<CaloParticle> >, CaloParticle, edm::refhelper::FindUsingAdvance<std::vector<CaloParticle, std::allocator<CaloParticle> >, CaloParticle> > > > const&, edm::AssociationMap<edm::OneToManyWithQualityGeneric<std::vector<CaloParticle, std::allocator<CaloParticle> >, std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> >, std::pair<float, float>, unsigned int, edm::RefProd<std::vector<CaloParticle, std::allocator<CaloParticle> > >, edm::RefProd<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> > >, edm::Ref<std::vector<CaloParticle, std::allocator<CaloParticle> >, CaloParticle, edm::refhelper::FindUsingAdvance<std::vector<CaloParticle, std::allocator<CaloParticle> >, CaloParticle> >, edm::Ref<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> >, reco::CaloCluster, edm::refhelper::FindUsingAdvance<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> >, reco::CaloCluster> > > > const&) const src/Validation/HGCalValidation/src/HGVHistoProducerAlgo.cc:2248
    #3 0x14c1ccdf4d99 in HGCalValidator::dqmAnalyze(edm::Event const&, edm::EventSetup const&, HGCalValidatorHistograms const&) const src/Validation/HGCalValidation/plugins/HGCalValidator.cc:380
    #4 0x14c30114954b in edm::global::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) src/FWCore/Framework/src/global/EDProducerBase.cc:69

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Thu Feb 15 15:11:33 CET 2024
Thread 5 (Thread 0x14c22e0ce700 (LWP 1801787) "cmsRun"):
#2  0x000014c2efc5cd7e in (anonymous namespace)::sig_pause_for_stacktrace (sig=<optimized out>) at src/FWCore/Services/plugins/InitRootHandlers.cc:453
#3  <signal handler called>
#4  0x000014bf49777852 in std::vector<float, std::allocator<float> >::operator[] (__n=212, this=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/bits/stl_vector.h:1124
#5  TSToSimTSHitLCAssociatorByEnergyScoreImpl::makeConnections (this=<optimized out>, tCH=..., lCCH=..., sCCH=..., cPCH=..., sTCH=...) at src/SimCalorimetry/HGCalAssociatorProducers/plugins/TSToSimTSHitLCAssociatorByEnergyScoreImpl.cc:151
#6  0x000014bf49784833 in TSToSimTSHitLCAssociatorByEnergyScoreImpl::associateRecoToSim (this=<optimized out>, tCH=..., lCCH=..., sCCH=..., cPCH=..., sTCH=...) at src/SimCalorimetry/HGCalAssociatorProducers/plugins/TSToSimTSHitLCAssociatorByEnergyScoreImpl.cc:216
#7  0x000014bf497b926a in hgcal::TracksterToSimTracksterHitLCAssociator::associateRecoToSim (sTCH=..., cPCH=..., sCCH=..., lCCH=..., tCH=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/tmp/BUILDROOT/c0a455c7a5b724be894a663904dad922/opt/cmssw/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_UBSAN_X_2024-02-14-2300/src/SimDataFormats/Associations/interface/TracksterToSimTracksterHitLCAssociator.h:39
#8  TSToSimTSHitLCAssociatorEDProducer::produce (this=<optimized out>, iEvent=..., iSetup=...) at src/SimCalorimetry/HGCalAssociatorProducers/plugins/TSToSimTSHitLCAssociatorEDProducer.cc:90
#9  0x000014c30114954c in edm::global::EDProducerBase::doEvent (this=this@entry=0x14bed0ffd430, info=..., act=0x14c2f48c7610, mcc=mcc@entry=0x14bed29a8ee8) at /data/cmsbld/jenkins/workspace/build-any-ib/w/tmp/BUILDROOT/c0a455c7a5b724be894a663904dad922/opt/cmssw/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_UBSAN_X_2024-02-14-2300/src/FWCore/Framework/interface/Event.h:98

Thread 4 (Thread 0x14c23057b700 (LWP 1801786) "cmsRun"):
#2  0x000014c2efc5cd7e in (anonymous namespace)::sig_pause_for_stacktrace (sig=<optimized out>) at src/FWCore/Services/plugins/InitRootHandlers.cc:453
#3  <signal handler called>
#4  0x000014c2aed4404c in PerigeeLinearizedTrackState::operator== (this=0x14bdf9879800, other=...) at src/RecoVertex/VertexTools/src/PerigeeLinearizedTrackState.cc:30
#5  0x000014c29c94921a in VertexTrack<5u>::operator== (this=<optimized out>, data=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/tmp/BUILDROOT/c0a455c7a5b724be894a663904dad922/opt/cmssw/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_UBSAN_X_2024-02-14-2300/src/RecoVertex/VertexPrimitives/interface/VertexTrack.h:94
#6  0x000014c29c94f4d8 in VertexTrackEqual<5u>::operator() (t=..., this=0x14c230573990) at /data/cmsbld/jenkins/workspace/build-any-ib/w/tmp/BUILDROOT/c0a455c7a5b724be894a663904dad922/opt/cmssw/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_UBSAN_X_2024-02-14-2300/src/RecoVertex/VertexPrimitives/interface/VertexTrack.h:118
#7  KalmanVertexTrackCompatibilityEstimator<5u>::estimate (this=0x14c1bebea980, vertex=..., tr=..., hint=<optimized out>) at src/RecoVertex/KalmanVertexFit/src/KalmanVertexTrackCompatibilityEstimator.cc:19
#8  0x000014c29c94770b in KalmanVertexTrackCompatibilityEstimator<5u>::estimate (this=0x14c1bebea980, vertex=..., track=..., hint=30) at src/RecoVertex/KalmanVertexFit/src/KalmanVertexTrackCompatibilityEstimator.cc:36
#9  0x000014c29cdd641e in AdaptiveVertexFitter::reWeightTracks (this=0x14c1bed39360, lTracks=..., vertex=...) at src/RecoVertex/AdaptiveVertexFit/src/AdaptiveVertexFitter.cc:361
#10 0x000014c29cdda458 in AdaptiveVertexFitter::reWeightTracks (this=0x14c1bed39360, tracks=..., seed=...) at src/RecoVertex/AdaptiveVertexFit/src/AdaptiveVertexFitter.cc:444
#11 0x000014c29cded16b in AdaptiveVertexFitter::fit (this=0x14c1bed39360, tracks=..., priorSeed=..., withPrior=<optimized out>) at src/RecoVertex/AdaptiveVertexFit/src/AdaptiveVertexFitter.cc:505
#12 0x000014c29cdf187c in AdaptiveVertexFitter::vertex (this=0x14c1bed39360, unstracks=...) at src/RecoVertex/AdaptiveVertexFit/src/AdaptiveVertexFitter.cc:172
#13 0x000014c1b9de9866 in PrimaryVertexResolution::Plots::calculateAndFillResolution (this=0x14c1bed396e8, tracks=..., nvertices=35, lumi=0, engine=..., fitter=...) at src/DQM/TrackingMonitor/src/PrimaryVertexResolution.cc:490
#14 0x000014c1b9deff08 in PrimaryVertexResolution::analyze (this=0x14c1bed39000, iEvent=..., iSetup=...) at src/DQM/TrackingMonitor/src/PrimaryVertexResolution.cc:435
#15 0x000014c30125b3ef in edm::stream::EDProducerAdaptorBase::doEvent (this=this@entry=0x14c1bed09bc0, info=..., act=0x14c2f48c7610, mcc=mcc@entry=0x14bed1132fa8) at src/FWCore/Framework/src/stream/EDProducerAdaptorBase.cc:82

Thread 3 (Thread 0x14c230f7c700 (LWP 1801785) "cmsRun"):
#2  0x000014c2efc5cd7e in (anonymous namespace)::sig_pause_for_stacktrace (sig=<optimized out>) at src/FWCore/Services/plugins/InitRootHandlers.cc:453
#3  <signal handler called>
#4  0x000014bf405e8801 in std::__array_traits<int, 289ul>::_S_ref (__n=<optimized out>, __t=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/array:61
#5  std::array<int, 289ul>::operator[] (__n=231, this=0x14bf408bad60 <ROOT::Math::MatRepSym<double, 17u>::off(int)::v>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/array:219
#6  ROOT::Math::MatRepSym<double, 17u>::off (i=231) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/MatrixRepresentationsStatic.h:309
#7  ROOT::Math::MatRepSym<double, 17u>::offset (j=10, i=13) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/MatrixRepresentationsStatic.h:316
#8  ROOT::Math::MatRepSym<double, 17u>::operator() (j=10, i=13, this=0x14c230f73280) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/MatrixRepresentationsStatic.h:226
#9  ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >::operator() (j=10, i=13, this=0x14c230f73280) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/SMatrix.icc:640
#10 ROOT::Math::meta_matrix_dot<13u>::g<ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> > > (j=10, i=<optimized out>, rhs=..., lhs=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/MatrixFunctions.h:309
#11 ROOT::Math::meta_matrix_dot<14u>::g<ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> > > (j=10, i=<optimized out>, rhs=..., lhs=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/MatrixFunctions.h:310
#12 ROOT::Math::meta_matrix_dot<15u>::g<ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> > > (j=10, i=<optimized out>, rhs=..., lhs=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/MatrixFunctions.h:310
#13 ROOT::Math::meta_matrix_dot<16u>::g<ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> > > (lhs=..., rhs=..., i=<optimized out>, j=10) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/MatrixFunctions.h:310
#14 0x000014bf405f6edb in ROOT::Math::MatrixMulOp<ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, double, 17u>::operator() (j=10, i=14, this=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/MatrixFunctions.h:362
#15 ROOT::Math::Expr<ROOT::Math::MatrixMulOp<ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, double, 17u>, double, 17u, 17u, ROOT::Math::MatRepStd<double, 17u, 17u> >::operator() (j=10, i=14, this=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/Expression.h:154
#16 ROOT::Math::meta_matrix_dot<10u>::g<ROOT::Math::Expr<ROOT::Math::MatrixMulOp<ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, double, 17u>, double, 17u, 17u, ROOT::Math::MatRepStd<double, 17u, 17u> >, ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> > > (j=1, i=14, rhs=..., lhs=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/MatrixFunctions.h:309
#17 ROOT::Math::meta_matrix_dot<11u>::g<ROOT::Math::Expr<ROOT::Math::MatrixMulOp<ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, double, 17u>, double, 17u, 17u, ROOT::Math::MatRepStd<double, 17u, 17u> >, ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> > > (j=1, i=14, rhs=..., lhs=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/MatrixFunctions.h:310
#18 ROOT::Math::meta_matrix_dot<12u>::g<ROOT::Math::Expr<ROOT::Math::MatrixMulOp<ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, double, 17u>, double, 17u, 17u, ROOT::Math::MatRepStd<double, 17u, 17u> >, ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> > > (j=1, i=14, rhs=..., lhs=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/MatrixFunctions.h:310
#19 ROOT::Math::meta_matrix_dot<13u>::g<ROOT::Math::Expr<ROOT::Math::MatrixMulOp<ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, double, 17u>, double, 17u, 17u, ROOT::Math::MatRepStd<double, 17u, 17u> >, ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> > > (j=1, i=14, rhs=..., lhs=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/MatrixFunctions.h:310
#20 ROOT::Math::meta_matrix_dot<14u>::g<ROOT::Math::Expr<ROOT::Math::MatrixMulOp<ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, double, 17u>, double, 17u, 17u, ROOT::Math::MatRepStd<double, 17u, 17u> >, ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> > > (j=1, i=14, rhs=..., lhs=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/MatrixFunctions.h:310
#21 ROOT::Math::meta_matrix_dot<15u>::g<ROOT::Math::Expr<ROOT::Math::MatrixMulOp<ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, double, 17u>, double, 17u, 17u, ROOT::Math::MatRepStd<double, 17u, 17u> >, ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> > > (j=1, i=14, rhs=..., lhs=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/MatrixFunctions.h:310
#22 ROOT::Math::meta_matrix_dot<16u>::g<ROOT::Math::Expr<ROOT::Math::MatrixMulOp<ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, double, 17u>, double, 17u, 17u, ROOT::Math::MatRepStd<double, 17u, 17u> >, ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> > > (j=1, i=14, rhs=..., lhs=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/MatrixFunctions.h:310
#23 ROOT::Math::MatrixMulOp<ROOT::Math::Expr<ROOT::Math::MatrixMulOp<ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, double, 17u>, double, 17u, 17u, ROOT::Math::MatRepStd<double, 17u, 17u> >, ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, double, 17u>::operator() (j=1, i=14, this=0x14c230f72f90) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/MatrixFunctions.h:363
#24 ROOT::Math::Expr<ROOT::Math::MatrixMulOp<ROOT::Math::Expr<ROOT::Math::MatrixMulOp<ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, double, 17u>, double, 17u, 17u, ROOT::Math::MatRepStd<double, 17u, 17u> >, ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, double, 17u>, double, 17u, 17u, ROOT::Math::MatRepStd<double, 17u, 17u> >::operator() (j=1, i=14, this=0x14c230f72f90) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/Expression.h:154
#25 ROOT::Math::AssignSym::Evaluate<double, 17u, ROOT::Math::MatrixMulOp<ROOT::Math::Expr<ROOT::Math::MatrixMulOp<ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, double, 17u>, double, 17u, 17u, ROOT::Math::MatRepStd<double, 17u, 17u> >, ROOT::Math::SMatrix<double, 17u, 17u, ROOT::Math::MatRepSym<double, 17u> >, double, 17u>, ROOT::Math::MatRepStd<double, 17u, 17u> > (lhs=..., rhs=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/HelperOps.h:163
#26 0x000014bf406a2916 in KinematicConstrainedVertexUpdatorT<2, 2>::update (this=0x14bdedbad200, inPar=..., inCov=..., lStates=..., lPoint=..., fieldValue=..., cs=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/MatrixFunctions.h:355
#27 0x000014bf406c81c2 in KinematicConstrainedVertexFitterT<2, 2>::fit (this=this@entry=0x14c230f74ac0, part=..., cs=cs@entry=0x14c230f74b00, pt=pt@entry=0x0) at /data/cmsbld/jenkins/workspace/build-any-ib/w/tmp/BUILDROOT/c0a455c7a5b724be894a663904dad922/opt/cmssw/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_UBSAN_X_2024-02-14-2300/src/RecoVertex/KinematicFit/interface/KinematicConstrainedVertexFitterT.h:215
#28 0x000014bf4048d7ef in KinematicConstrainedVertexFitterT<2, 2>::fit (cs=0x14c230f74b00, part=..., this=0x14c230f74ac0) at /data/cmsbld/jenkins/workspace/build-any-ib/w/tmp/BUILDROOT/c0a455c7a5b724be894a663904dad922/opt/cmssw/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_UBSAN_X_2024-02-14-2300/src/RecoVertex/KinematicFit/interface/KinematicConstrainedVertexFitterT.h:55
#29 ConversionVertexFinder::run (this=this@entry=0x14bf534e4598, _pair=..., the_vertex=...) at src/RecoEgamma/EgammaPhotonAlgos/src/ConversionVertexFinder.cc:69
#30 0x000014bf4179aaab in ConversionProducer::checkVertex (magField=..., the_vertex=..., ttk_r=..., ttk_l=..., this=0x14bf534e4200) at src/RecoEgamma/EgammaPhotonProducers/src/ConversionProducer.cc:165
#31 ConversionProducer::buildCollection (this=0x14bf534e4200, iEvent=..., iSetup=..., allTracks=..., superClusterPtrs=..., basicClusterPtrs=..., the_pvtx=..., outputConvPhotonCollection=...) at src/RecoEgamma/EgammaPhotonProducers/src/ConversionProducer.cc:535
#32 0x000014bf417b3e47 in ConversionProducer::produce (this=0x14bf534e4200, iEvent=..., iSetup=...) at src/RecoEgamma/EgammaPhotonProducers/src/ConversionProducer.cc:343
#33 0x000014c30125b3ef in edm::stream::EDProducerAdaptorBase::doEvent (this=this@entry=0x14bf53cda060, info=..., act=0x14c2f48c7610, mcc=mcc@entry=0x14bed1917de8) at src/FWCore/Framework/src/stream/EDProducerAdaptorBase.cc:82

Thread 1 (Thread 0x14c2f738f680 (LWP 1801360) "cmsRun"):
#3  0x000014c2efc7a786 in (anonymous namespace)::sig_dostack_then_abort (sig=<optimized out>) at src/FWCore/Services/plugins/InitRootHandlers.cc:546
#4  <signal handler called>
#5  0x000014c1cc9e92e7 in HGVHistoProducerAlgo::findmaxhit (this=this@entry=0x14c1dcdb1200, cluster=..., hitMap=...) at src/Validation/HGCalValidation/src/HGVHistoProducerAlgo.cc:3567
#6  0x000014c1cca5acf5 in HGVHistoProducerAlgo::fill_generic_cluster_histos (this=this@entry=0x14c1dcdb1200, histograms=..., count=count@entry=0, clusterHandle=..., clusters=..., caloParticleHandle=..., cP=..., cPIndices=..., cPSelectedIndices=..., hitMap=..., cummatbudg=..., layers=<optimized out>, thicknesses=..., cpsInLayerClusterMap=..., cPOnLayerMap=...) at src/Validation/HGCalValidation/src/HGVHistoProducerAlgo.cc:2248
#7  0x000014c1ccdf4d9a in HGCalValidator::dqmAnalyze (this=<optimized out>, event=..., setup=..., histograms=...) at src/Validation/HGCalValidation/plugins/HGCalValidator.cc:380
#8  0x000014c30114954c in edm::global::EDProducerBase::doEvent (this=this@entry=0x14c1dbb6cb60, info=..., act=0x14c2f48c7610, mcc=mcc@entry=0x14c1dd0c9d68) at /data/cmsbld/jenkins/workspace/build-any-ib/w/tmp/BUILDROOT/c0a455c7a5b724be894a663904dad922/opt/cmssw/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_UBSAN_X_2024-02-14-2300/src/FWCore/Framework/interface/Event.h:98

Current Modules:

Module: HGCalValidator:hgcalValidator (crashed)
Module: ConversionProducer:allConversions
Module: PrimaryVertexResolution:primaryVertexResolution
Module: TSToSimTSHitLCAssociatorEDProducer:tracksterSimTracksterAssociationLinkingPU

/data/cmsbld/jenkins/workspace/build-any-ib/w/tmp/BUILDROOT/5fe42f26621cd4d5b00bd4943abd9965/opt/cmssw/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_UBSAN_X_2024-02-16-2300/src/TrackingTools/TrajectoryParametrization/interface/PerigeeTrajectoryError.h:14:7: runtime error: load of value 28, which is not a valid value for type 'bool'
    #0 0x145b5b7e888c in PerigeeTrajectoryError::operator=(PerigeeTrajectoryError&&) /data/cmsbld/jenkins/workspace/build-any-ib/w/tmp/BUILDROOT/5fe42f26621cd4d5b00bd4943abd9965/opt/cmssw/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_UBSAN_X_2024-02-16-2300/src/TrackingTools/TrajectoryParametrization/interface/PerigeeTrajectoryError.h:14
    #1 0x145b5b7e888c in TrajectoryStateClosestToPoint::operator=(TrajectoryStateClosestToPoint&&) /data/cmsbld/jenkins/workspace/build-any-ib/w/tmp/BUILDROOT/5fe42f26621cd4d5b00bd4943abd9965/opt/cmssw/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_UBSAN_X_2024-02-16-2300/src/TrackingTools/TrajectoryState/interface/TrajectoryStateClosestToPoint.h:18
    #2 0x145b5b7e888c in PerigeeLinearizedTrackState::computeJacobians() const src/RecoVertex/VertexTools/src/PerigeeLinearizedTrackState.cc:12
    #3 0x145b5b8765be in PerigeeLinearizedTrackState::isValid() const /data/cmsbld/jenkins/workspace/build-any-ib/w/tmp/BUILDROOT/5fe42f26621cd4d5b00bd4943abd9965/opt/cmssw/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_UBSAN_X_2024-02-16-2300/src/RecoVertex/VertexTools/interface/PerigeeLinearizedTrackState.h:263
    #4 0x145b49691254 in KalmanVertexUpdator<5u>::positionUpdate(VertexState const&, ReferenceCountingPointer<LinearizedTrackState<5u> >, float, int) const src/RecoVertex/KalmanVertexFit/src/KalmanVertexUpdator.cc:76
    #5 0x145b496950ab in KalmanVertexUpdator<5u>::update(CachingVertex<5u> const&, ReferenceCountingPointer<VertexTrack<5u> >, float, int) const src/RecoVertex/KalmanVertexFit/src/KalmanVertexUpdator.cc:20
    #6 0x145b49699eeb in KalmanVertexUpdator<5u>::add(CachingVertex<5u> const&, ReferenceCountingPointer<VertexTrack<5u> >) const src/RecoVertex/KalmanVertexFit/src/KalmanVertexUpdator.cc:59
    #7 0x145b49992827 in AdaptiveVertexFitter::fit(std::vector<ReferenceCountingPointer<VertexTrack<5u> >, std::allocator<ReferenceCountingPointer<VertexTrack<5u> > > > const&, VertexState const&, bool) const src/RecoVertex/AdaptiveVertexFit/src/AdaptiveVertexFitter.cc:512
    #8 0x145b4999e0bc in AdaptiveVertexFitter::vertex(std::vector<reco::TransientTrack, std::allocator<reco::TransientTrack> > const&, Point3DBase<float, GlobalTag> const&) const src/RecoVertex/AdaptiveVertexFit/src/AdaptiveVertexFitter.cc:209
    #9 0x145ab7fd9327 in PFDisplacedVertexFinder::fitVertexFromSeed(reco::PFDisplacedVertexSeed const&, reco::PFDisplacedVertex&) src/RecoParticleFlow/PFTracking/src/PFDisplacedVertexFinder.cc:348
    #10 0x145ab7fe6d2a in PFDisplacedVertexFinder::findDisplacedVertices() src/RecoParticleFlow/PFTracking/src/PFDisplacedVertexFinder.cc:111
    #11 0x1457ad35caf6  (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02824/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_UBSAN_X_2024-02-16-2300/lib/el8_amd64_gcc12/pluginRecoParticleFlowPFTrackingPlugins.so+0x8d5af6)
    #12 0x145badd7e3ee in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) src/FWCore/Framework/src/stream/EDProducerAdaptorBase.cc:82

/data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/bits/hashtable_policy.h:456:39: runtime error: member access within null pointer of type 'struct __node_type'
    #0 0x145a80576168 in std::__detail::_Node_const_iterator<std::pair<DetId const, HGCRecHit const*>, false, false>::operator->() const /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/bits/hashtable_policy.h:456
    #1 0x145a80576168 in HGVHistoProducerAlgo::layerClusters_to_CaloParticles(HGVHistoProducerAlgoHistograms const&, edm::Handle<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> > >, std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> > const&, edm::Handle<std::vector<CaloParticle, std::allocator<CaloParticle> > >, std::vector<CaloParticle, std::allocator<CaloParticle> > const&, std::vector<unsigned long, std::allocator<unsigned long> > const&, std::vector<unsigned long, std::allocator<unsigned long> > const&, std::unordered_map<DetId, HGCRecHit const*, std::hash<DetId>, std::equal_to<DetId>, std::allocator<std::pair<DetId const, HGCRecHit const*> > > const&, unsigned int, edm::AssociationMap<edm::OneToManyWithQualityGeneric<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> >, std::vector<CaloParticle, std::allocator<CaloParticle> >, float, unsigned int, edm::RefProd<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> > >, edm::RefProd<std::vector<CaloParticle, std::allocator<CaloParticle> > >, edm::Ref<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> >, reco::CaloCluster, edm::refhelper::FindUsingAdvance<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> >, reco::CaloCluster> >, edm::Ref<std::vector<CaloParticle, std::allocator<CaloParticle> >, CaloParticle, edm::refhelper::FindUsingAdvance<std::vector<CaloParticle, std::allocator<CaloParticle> >, CaloParticle> > > > const&, edm::AssociationMap<edm::OneToManyWithQualityGeneric<std::vector<CaloParticle, std::allocator<CaloParticle> >, std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> >, std::pair<float, float>, unsigned int, edm::RefProd<std::vector<CaloParticle, std::allocator<CaloParticle> > >, edm::RefProd<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> > >, edm::Ref<std::vector<CaloParticle, std::allocator<CaloParticle> >, CaloParticle, edm::refhelper::FindUsingAdvance<std::vector<CaloParticle, std::allocator<CaloParticle> >, CaloParticle> >, edm::Ref<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> >, reco::CaloCluster, edm::refhelper::FindUsingAdvance<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> >, reco::CaloCluster> > > > const&) const src/Validation/HGCalValidation/src/HGVHistoProducerAlgo.cc:1808
    #2 0x145a8057dcfd in HGVHistoProducerAlgo::fill_generic_cluster_histos(HGVHistoProducerAlgoHistograms const&, int, edm::Handle<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> > >, std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> > const&, edm::Handle<std::vector<CaloParticle, std::allocator<CaloParticle> > >, std::vector<CaloParticle, std::allocator<CaloParticle> > const&, std::vector<unsigned long, std::allocator<unsigned long> > const&, std::vector<unsigned long, std::allocator<unsigned long> > const&, std::unordered_map<DetId, HGCRecHit const*, std::hash<DetId>, std::equal_to<DetId>, std::allocator<std::pair<DetId const, HGCRecHit const*> > > const&, std::map<double, double, std::less<double>, std::allocator<std::pair<double const, double> > >, unsigned int, std::vector<int, std::allocator<int> >, edm::AssociationMap<edm::OneToManyWithQualityGeneric<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> >, std::vector<CaloParticle, std::allocator<CaloParticle> >, float, unsigned int, edm::RefProd<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> > >, edm::RefProd<std::vector<CaloParticle, std::allocator<CaloParticle> > >, edm::Ref<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> >, reco::CaloCluster, edm::refhelper::FindUsingAdvance<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> >, reco::CaloCluster> >, edm::Ref<std::vector<CaloParticle, std::allocator<CaloParticle> >, CaloParticle, edm::refhelper::FindUsingAdvance<std::vector<CaloParticle, std::allocator<CaloParticle> >, CaloParticle> > > > const&, edm::AssociationMap<edm::OneToManyWithQualityGeneric<std::vector<CaloParticle, std::allocator<CaloParticle> >, std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> >, std::pair<float, float>, unsigned int, edm::RefProd<std::vector<CaloParticle, std::allocator<CaloParticle> > >, edm::RefProd<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> > >, edm::Ref<std::vector<CaloParticle, std::allocator<CaloParticle> >, CaloParticle, edm::refhelper::FindUsingAdvance<std::vector<CaloParticle, std::allocator<CaloParticle> >, CaloParticle> >, edm::Ref<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> >, reco::CaloCluster, edm::refhelper::FindUsingAdvance<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> >, reco::CaloCluster> > > > const&) const src/Validation/HGCalValidation/src/HGVHistoProducerAlgo.cc:2214
    #3 0x145a80918d99 in HGCalValidator::dqmAnalyze(edm::Event const&, edm::EventSetup const&, HGCalValidatorHistograms const&) const src/Validation/HGCalValidation/plugins/HGCalValidator.cc:380
    #4 0x145badc6c54b in edm::global::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) src/FWCore/Framework/src/global/EDProducerBase.cc:69

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Sat Feb 17 22:48:38 CET 2024
Thread 5 (Thread 0x145ada600700 (LWP 2840550) "cmsRun"):
#2  0x0000145b9c78cd7e in (anonymous namespace)::sig_pause_for_stacktrace (sig=<optimized out>) at src/FWCore/Services/plugins/InitRootHandlers.cc:453
#3  <signal handler called>
#4  0x0000145b16cd1f6d in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (this=0x145ada5f8ef8, __in_chrg=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/bits/shared_ptr_base.h:1071
#5  std::__shared_ptr<BasicTrajectoryState, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr (this=0x145ada5f8ef0, __in_chrg=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/bits/shared_ptr_base.h:1524
#6  std::shared_ptr<BasicTrajectoryState>::~shared_ptr (this=0x145ada5f8ef0, __in_chrg=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/bits/shared_ptr.h:175
#7  ProxyBase11<BasicTrajectoryState>::~ProxyBase11 (this=0x145ada5f8ef0, __in_chrg=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/tmp/BUILDROOT/5fe42f26621cd4d5b00bd4943abd9965/opt/cmssw/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_UBSAN_X_2024-02-16-2300/src/TrackingTools/TrajectoryState/interface/ProxyBase11.h:40
#8  TrajectoryStateOnSurface::~TrajectoryStateOnSurface (this=0x145ada5f8ef0, __in_chrg=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/tmp/BUILDROOT/5fe42f26621cd4d5b00bd4943abd9965/opt/cmssw/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_UBSAN_X_2024-02-16-2300/src/TrackingTools/TrajectoryState/interface/TrajectoryStateOnSurface.h:30
#9  TrajectoryMeasurement::~TrajectoryMeasurement (this=0x145ada5f8ed0, __in_chrg=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/tmp/BUILDROOT/5fe42f26621cd4d5b00bd4943abd9965/opt/cmssw/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_UBSAN_X_2024-02-16-2300/src/TrackingTools/PatternTools/interface/TrajectoryMeasurement.h:25
#10 0x0000145b16ce08bd in KFTrajectorySmoother::trajectory (this=0x145680b54e00, aTraj=...) at src/TrackingTools/TrackFitters/src/KFTrajectorySmoother.cc:186
#11 0x0000145b16cfefbf in TrajectorySmoother::trajectories (this=0x145680b54e00, traj=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/tmp/BUILDROOT/5fe42f26621cd4d5b00bd4943abd9965/opt/cmssw/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_UBSAN_X_2024-02-16-2300/src/TrackingTools/PatternTools/interface/TrajectorySmoother.h:19
#12 0x0000145aa3b53d10 in TrackTransformer::transform (this=<optimized out>, track=..., recHitsForReFit=...) at src/TrackingTools/TrackRefitter/src/TrackTransformer.cc:291
#13 0x0000145aa3b5850c in TrackTransformer::transform (this=this@entry=0x145a727ece18, newTrack=...) at src/TrackingTools/TrackRefitter/src/TrackTransformer.cc:160
#14 0x0000145a6dd3f9b7 in TrackingRecoMaterialAnalyser::analyze (this=<optimized out>, event=..., setup=...) at src/DQM/TrackingMonitor/src/TrackingRecoMaterialAnalyser.cc:249
#15 0x0000145badd7e3ef in edm::stream::EDProducerAdaptorBase::doEvent (this=this@entry=0x145a72d7e880, info=..., act=0x145ba15f3610, mcc=mcc@entry=0x1458068f26e8) at src/FWCore/Framework/src/stream/EDProducerAdaptorBase.cc:82

Thread 4 (Thread 0x145add09c700 (LWP 2840549) "cmsRun"):
#2  0x0000145b9c78cd7e in (anonymous namespace)::sig_pause_for_stacktrace (sig=<optimized out>) at src/FWCore/Services/plugins/InitRootHandlers.cc:453
#3  <signal handler called>
#4  0x0000145b49687eee in ROOT::Math::TransposeOp<ROOT::Math::SMatrix<double, 5u, 3u, ROOT::Math::MatRepStd<double, 5u, 3u> >, double, 5u, 3u>::apply (i=3, this=0x145add093fa8) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/MatrixFunctions.h:514
#5  ROOT::Math::Expr<ROOT::Math::TransposeOp<ROOT::Math::SMatrix<double, 5u, 3u, ROOT::Math::MatRepStd<double, 5u, 3u> >, double, 5u, 3u>, double, 3u, 5u, ROOT::Math::MatRepStd<double, 3u, 5u> >::apply (i=3, this=0x145add093fa8) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/Expression.h:151
#6  ROOT::Math::meta_matrix_dot<3u>::f<ROOT::Math::Expr<ROOT::Math::TransposeOp<ROOT::Math::SMatrix<double, 5u, 3u, ROOT::Math::MatRepStd<double, 5u, 3u> >, double, 5u, 3u>, double, 3u, 5u, ROOT::Math::MatRepStd<double, 3u, 5u> >, ROOT::Math::SMatrix<double, 5u, 5u, ROOT::Math::MatRepSym<double, 5u> > > (lhs=..., rhs=..., offset=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/MatrixFunctions.h:298
#7  0x0000145b4968ae67 in ROOT::Math::meta_matrix_dot<4u>::f<ROOT::Math::Expr<ROOT::Math::TransposeOp<ROOT::Math::SMatrix<double, 5u, 3u, ROOT::Math::MatRepStd<double, 5u, 3u> >, double, 5u, 3u>, double, 3u, 5u, ROOT::Math::MatRepStd<double, 3u, 5u> >, ROOT::Math::SMatrix<double, 5u, 5u, ROOT::Math::MatRepSym<double, 5u> > > (offset=<optimized out>, rhs=..., lhs=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/MatrixFunctions.h:300
#8  ROOT::Math::MatrixMulOp<ROOT::Math::Expr<ROOT::Math::TransposeOp<ROOT::Math::SMatrix<double, 5u, 3u, ROOT::Math::MatRepStd<double, 5u, 3u> >, double, 5u, 3u>, double, 3u, 5u, ROOT::Math::MatRepStd<double, 3u, 5u> >, ROOT::Math::SMatrix<double, 5u, 5u, ROOT::Math::MatRepSym<double, 5u> >, double, 5u>::apply (i=<optimized out>, this=0x145add093fb0) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/MatrixFunctions.h:359
#9  ROOT::Math::Expr<ROOT::Math::MatrixMulOp<ROOT::Math::Expr<ROOT::Math::TransposeOp<ROOT::Math::SMatrix<double, 5u, 3u, ROOT::Math::MatRepStd<double, 5u, 3u> >, double, 5u, 3u>, double, 3u, 5u, ROOT::Math::MatRepStd<double, 3u, 5u> >, ROOT::Math::SMatrix<double, 5u, 5u, ROOT::Math::MatRepSym<double, 5u> >, double, 5u>, double, 3u, 5u, ROOT::Math::MatRepStd<double, 3u, 5u> >::apply (i=3, this=0x145add093fb0) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/Expression.h:151
#10 ROOT::Math::meta_row_dot<3u>::f<ROOT::Math::Expr<ROOT::Math::MatrixMulOp<ROOT::Math::Expr<ROOT::Math::TransposeOp<ROOT::Math::SMatrix<double, 5u, 3u, ROOT::Math::MatRepStd<double, 5u, 3u> >, double, 5u, 3u>, double, 3u, 5u, ROOT::Math::MatRepStd<double, 3u, 5u> >, ROOT::Math::SMatrix<double, 5u, 5u, ROOT::Math::MatRepSym<double, 5u> >, double, 5u>, double, 3u, 5u, ROOT::Math::MatRepStd<double, 3u, 5u> >, ROOT::Math::VecExpr<ROOT::Math::BinaryOp<ROOT::Math::MinOp<double>, ROOT::Math::SVector<double, 5u>, ROOT::Math::SVector<double, 5u>, double>, double, 5u> > (offset=0, rhs=..., lhs=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/MatrixFunctions.h:88
#11 ROOT::Math::meta_row_dot<4u>::f<ROOT::Math::Expr<ROOT::Math::MatrixMulOp<ROOT::Math::Expr<ROOT::Math::TransposeOp<ROOT::Math::SMatrix<double, 5u, 3u, ROOT::Math::MatRepStd<double, 5u, 3u> >, double, 5u, 3u>, double, 3u, 5u, ROOT::Math::MatRepStd<double, 3u, 5u> >, ROOT::Math::SMatrix<double, 5u, 5u, ROOT::Math::MatRepSym<double, 5u> >, double, 5u>, double, 3u, 5u, ROOT::Math::MatRepStd<double, 3u, 5u> >, ROOT::Math::VecExpr<ROOT::Math::BinaryOp<ROOT::Math::MinOp<double>, ROOT::Math::SVector<double, 5u>, ROOT::Math::SVector<double, 5u>, double>, double, 5u> > (offset=0, rhs=..., lhs=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/MatrixFunctions.h:88
#12 ROOT::Math::VectorMatrixRowOp<ROOT::Math::Expr<ROOT::Math::MatrixMulOp<ROOT::Math::Expr<ROOT::Math::TransposeOp<ROOT::Math::SMatrix<double, 5u, 3u, ROOT::Math::MatRepStd<double, 5u, 3u> >, double, 5u, 3u>, double, 3u, 5u, ROOT::Math::MatRepStd<double, 3u, 5u> >, ROOT::Math::SMatrix<double, 5u, 5u, ROOT::Math::MatRepSym<double, 5u> >, double, 5u>, double, 3u, 5u, ROOT::Math::MatRepStd<double, 3u, 5u> >, ROOT::Math::VecExpr<ROOT::Math::BinaryOp<ROOT::Math::MinOp<double>, ROOT::Math::SVector<double, 5u>, ROOT::Math::SVector<double, 5u>, double>, double, 5u>, 5u>::apply (i=0, this=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/MatrixFunctions.h:123
#13 ROOT::Math::VecExpr<ROOT::Math::VectorMatrixRowOp<ROOT::Math::Expr<ROOT::Math::MatrixMulOp<ROOT::Math::Expr<ROOT::Math::TransposeOp<ROOT::Math::SMatrix<double, 5u, 3u, ROOT::Math::MatRepStd<double, 5u, 3u> >, double, 5u, 3u>, double, 3u, 5u, ROOT::Math::MatRepStd<double, 3u, 5u> >, ROOT::Math::SMatrix<double, 5u, 5u, ROOT::Math::MatRepSym<double, 5u> >, double, 5u>, double, 3u, 5u, ROOT::Math::MatRepStd<double, 3u, 5u> >, ROOT::Math::VecExpr<ROOT::Math::BinaryOp<ROOT::Math::MinOp<double>, ROOT::Math::SVector<double, 5u>, ROOT::Math::SVector<double, 5u>, double>, double, 5u>, 5u>, double, 3u>::apply (i=0, this=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/Expression.h:78
#14 ROOT::Math::BinaryOpCopyL<ROOT::Math::MulOp<double>, ROOT::Math::Constant<float>, ROOT::Math::VecExpr<ROOT::Math::VectorMatrixRowOp<ROOT::Math::Expr<ROOT::Math::MatrixMulOp<ROOT::Math::Expr<ROOT::Math::TransposeOp<ROOT::Math::SMatrix<double, 5u, 3u, ROOT::Math::MatRepStd<double, 5u, 3u> >, double, 5u, 3u>, double, 3u, 5u, ROOT::Math::MatRepStd<double, 3u, 5u> >, ROOT::Math::SMatrix<double, 5u, 5u, ROOT::Math::MatRepSym<double, 5u> >, double, 5u>, double, 3u, 5u, ROOT::Math::MatRepStd<double, 3u, 5u> >, ROOT::Math::VecExpr<ROOT::Math::BinaryOp<ROOT::Math::MinOp<double>, ROOT::Math::SVector<double, 5u>, ROOT::Math::SVector<double, 5u>, double>, double, 5u>, 5u>, double, 3u>, double>::apply (i=0, this=0x145add093fe0) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/Expression.h:288
#15 ROOT::Math::VecExpr<ROOT::Math::BinaryOpCopyL<ROOT::Math::MulOp<double>, ROOT::Math::Constant<float>, ROOT::Math::VecExpr<ROOT::Math::VectorMatrixRowOp<ROOT::Math::Expr<ROOT::Math::MatrixMulOp<ROOT::Math::Expr<ROOT::Math::TransposeOp<ROOT::Math::SMatrix<double, 5u, 3u, ROOT::Math::MatRepStd<double, 5u, 3u> >, double, 5u, 3u>, double, 3u, 5u, ROOT::Math::MatRepStd<double, 3u, 5u> >, ROOT::Math::SMatrix<double, 5u, 5u, ROOT::Math::MatRepSym<double, 5u> >, double, 5u>, double, 3u, 5u, ROOT::Math::MatRepStd<double, 3u, 5u> >, ROOT::Math::VecExpr<ROOT::Math::BinaryOp<ROOT::Math::MinOp<double>, ROOT::Math::SVector<double, 5u>, ROOT::Math::SVector<double, 5u>, double>, double, 5u>, 5u>, double, 3u>, double>, double, 3u>::apply (i=0, this=0x145add093fe0) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/Expression.h:78
#16 ROOT::Math::BinaryOp<ROOT::Math::AddOp<double>, ROOT::Math::SVector<double, 3u>, ROOT::Math::VecExpr<ROOT::Math::BinaryOpCopyL<ROOT::Math::MulOp<double>, ROOT::Math::Constant<float>, ROOT::Math::VecExpr<ROOT::Math::VectorMatrixRowOp<ROOT::Math::Expr<ROOT::Math::MatrixMulOp<ROOT::Math::Expr<ROOT::Math::TransposeOp<ROOT::Math::SMatrix<double, 5u, 3u, ROOT::Math::MatRepStd<double, 5u, 3u> >, double, 5u, 3u>, double, 3u, 5u, ROOT::Math::MatRepStd<double, 3u, 5u> >, ROOT::Math::SMatrix<double, 5u, 5u, ROOT::Math::MatRepSym<double, 5u> >, double, 5u>, double, 3u, 5u, ROOT::Math::MatRepStd<double, 3u, 5u> >, ROOT::Math::VecExpr<ROOT::Math::BinaryOp<ROOT::Math::MinOp<double>, ROOT::Math::SVector<double, 5u>, ROOT::Math::SVector<double, 5u>, double>, double, 5u>, 5u>, double, 3u>, double>, double, 3u>, double>::apply (this=this@entry=0x145add093ff0, i=i@entry=0) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/Expression.h:245
#17 0x0000145b496909d0 in ROOT::Math::VecExpr<ROOT::Math::BinaryOp<ROOT::Math::AddOp<double>, ROOT::Math::SVector<double, 3u>, ROOT::Math::VecExpr<ROOT::Math::BinaryOpCopyL<ROOT::Math::MulOp<double>, ROOT::Math::Constant<float>, ROOT::Math::VecExpr<ROOT::Math::VectorMatrixRowOp<ROOT::Math::Expr<ROOT::Math::MatrixMulOp<ROOT::Math::Expr<ROOT::Math::TransposeOp<ROOT::Math::SMatrix<double, 5u, 3u, ROOT::Math::MatRepStd<double, 5u, 3u> >, double, 5u, 3u>, double, 3u, 5u, ROOT::Math::MatRepStd<double, 3u, 5u> >, ROOT::Math::SMatrix<double, 5u, 5u, ROOT::Math::MatRepSym<double, 5u> >, double, 5u>, double, 3u, 5u, ROOT::Math::MatRepStd<double, 3u, 5u> >, ROOT::Math::VecExpr<ROOT::Math::BinaryOp<ROOT::Math::MinOp<double>, ROOT::Math::SVector<double, 5u>, ROOT::Math::SVector<double, 5u>, double>, double, 5u>, 5u>, double, 3u>, double>, double, 3u>, double>, double, 3u>::apply (i=0, this=0x145add093ff0) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/Expression.h:77
#18 ROOT::Math::SVector<double, 3u>::operator=<ROOT::Math::BinaryOp<ROOT::Math::AddOp<double>, ROOT::Math::SVector<double, 3u>, ROOT::Math::VecExpr<ROOT::Math::BinaryOpCopyL<ROOT::Math::MulOp<double>, ROOT::Math::Constant<float>, ROOT::Math::VecExpr<ROOT::Math::VectorMatrixRowOp<ROOT::Math::Expr<ROOT::Math::MatrixMulOp<ROOT::Math::Expr<ROOT::Math::TransposeOp<ROOT::Math::SMatrix<double, 5u, 3u, ROOT::Math::MatRepStd<double, 5u, 3u> >, double, 5u, 3u>, double, 3u, 5u, ROOT::Math::MatRepStd<double, 3u, 5u> >, ROOT::Math::SMatrix<double, 5u, 5u, ROOT::Math::MatRepSym<double, 5u> >, double, 5u>, double, 3u, 5u, ROOT::Math::MatRepStd<double, 3u, 5u> >, ROOT::Math::VecExpr<ROOT::Math::BinaryOp<ROOT::Math::MinOp<double>, ROOT::Math::SVector<double, 5u>, ROOT::Math::SVector<double, 5u>, double>, double, 5u>, 5u>, double, 3u>, double>, double, 3u>, double> > (rhs=..., this=0x145add094000) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/SVector.icc:210
#19 ROOT::Math::SVector<double, 3u>::SVector<ROOT::Math::BinaryOp<ROOT::Math::AddOp<double>, ROOT::Math::SVector<double, 3u>, ROOT::Math::VecExpr<ROOT::Math::BinaryOpCopyL<ROOT::Math::MulOp<double>, ROOT::Math::Constant<float>, ROOT::Math::VecExpr<ROOT::Math::VectorMatrixRowOp<ROOT::Math::Expr<ROOT::Math::MatrixMulOp<ROOT::Math::Expr<ROOT::Math::TransposeOp<ROOT::Math::SMatrix<double, 5u, 3u, ROOT::Math::MatRepStd<double, 5u, 3u> >, double, 5u, 3u>, double, 3u, 5u, ROOT::Math::MatRepStd<double, 3u, 5u> >, ROOT::Math::SMatrix<double, 5u, 5u, ROOT::Math::MatRepSym<double, 5u> >, double, 5u>, double, 3u, 5u, ROOT::Math::MatRepStd<double, 3u, 5u> >, ROOT::Math::VecExpr<ROOT::Math::BinaryOp<ROOT::Math::MinOp<double>, ROOT::Math::SVector<double, 5u>, ROOT::Math::SVector<double, 5u>, double>, double, 5u>, 5u>, double, 3u>, double>, double, 3u>, double> > (rhs=..., this=0x145add094000) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/lcg/root/6.30.05-d0417d1d3c55ded4cc9aa75628166a77/include/Math/SVector.icc:61
#20 KalmanVertexUpdator<5u>::positionUpdate (this=this@entry=0x1456ae2f60a0, oldVertex=..., linearizedTrack=..., weight=weight@entry=9.99999968e-21, sign=sign@entry=1) at src/RecoVertex/KalmanVertexFit/src/KalmanVertexUpdator.cc:113
#21 0x0000145b496950ac in KalmanVertexUpdator<5u>::update (this=this@entry=0x1456ae2f60a0, oldVertex=..., track=..., weight=9.99999968e-21, sign=sign@entry=1) at src/RecoVertex/KalmanVertexFit/src/KalmanVertexUpdator.cc:20
#22 0x0000145b49699eec in KalmanVertexUpdator<5u>::add (this=0x1456ae2f60a0, oldVertex=..., track=...) at src/RecoVertex/KalmanVertexFit/src/KalmanVertexUpdator.cc:59
#23 0x0000145b49992828 in AdaptiveVertexFitter::fit (this=0x145add094d10, tracks=..., priorSeed=..., withPrior=<optimized out>) at src/RecoVertex/AdaptiveVertexFit/src/AdaptiveVertexFitter.cc:512
#24 0x0000145b4999e0bd in AdaptiveVertexFitter::vertex (this=this@entry=0x145add094d10, tracks=..., linPoint=...) at src/RecoVertex/AdaptiveVertexFit/src/AdaptiveVertexFitter.cc:209
#25 0x0000145ab7fd9328 in PFDisplacedVertexFinder::fitVertexFromSeed (this=0x1457a5279358, displacedVertexSeed=..., displacedVertex=...) at src/RecoParticleFlow/PFTracking/src/PFDisplacedVertexFinder.cc:348
#26 0x0000145ab7fe6d2b in PFDisplacedVertexFinder::findDisplacedVertices (this=this@entry=0x1457a5279358) at src/RecoParticleFlow/PFTracking/src/PFDisplacedVertexFinder.cc:111
#27 0x00001457ad35caf7 in PFDisplacedVertexProducer::produce (this=0x1457a5279000, iEvent=..., iSetup=...) at src/RecoParticleFlow/PFTracking/plugins/PFDisplacedVertexProducer.cc:146
#28 0x0000145badd7e3ef in edm::stream::EDProducerAdaptorBase::doEvent (this=this@entry=0x1457a5269be0, info=..., act=0x145ba15f3610, mcc=mcc@entry=0x1457860d7228) at src/FWCore/Framework/src/stream/EDProducerAdaptorBase.cc:82

Thread 3 (Thread 0x145adda9d700 (LWP 2840548) "cmsRun"):
#3  0x0000145b9c7aa786 in (anonymous namespace)::sig_dostack_then_abort (sig=<optimized out>) at src/FWCore/Services/plugins/InitRootHandlers.cc:546
#4  <signal handler called>
#5  0x0000145a80560418 in HGVHistoProducerAlgo::layerClusters_to_CaloParticles (this=<optimized out>, histograms=..., clusterHandle=..., clusters=..., caloParticleHandle=..., cP=..., cPIndices=..., cPSelectedIndices=..., hitMap=..., layers=<optimized out>, cpsInLayerClusterMap=..., cPOnLayerMap=...) at src/Validation/HGCalValidation/src/HGVHistoProducerAlgo.cc:1808
#6  0x0000145a8057dcfe in HGVHistoProducerAlgo::fill_generic_cluster_histos (this=this@entry=0x145a92e8e200, histograms=..., count=count@entry=0, clusterHandle=..., clusters=..., caloParticleHandle=..., cP=..., cPIndices=..., cPSelectedIndices=..., hitMap=..., cummatbudg=..., layers=<optimized out>, thicknesses=..., cpsInLayerClusterMap=..., cPOnLayerMap=...) at src/Validation/HGCalValidation/src/HGVHistoProducerAlgo.cc:2214
#7  0x0000145a80918d9a in HGCalValidator::dqmAnalyze (this=<optimized out>, event=..., setup=..., histograms=...) at src/Validation/HGCalValidation/plugins/HGCalValidator.cc:380
#8  0x0000145badc6c54c in edm::global::EDProducerBase::doEvent (this=this@entry=0x145a92db1b60, info=..., act=0x145ba15f3610, mcc=mcc@entry=0x14579a24e9a8) at /data/cmsbld/jenkins/workspace/build-any-ib/w/tmp/BUILDROOT/5fe42f26621cd4d5b00bd4943abd9965/opt/cmssw/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_UBSAN_X_2024-02-16-2300/src/FWCore/Framework/interface/Event.h:98

Thread 1 (Thread 0x145ba3eb2680 (LWP 2838888) "cmsRun"):
#2  0x0000145b9c78cd7e in (anonymous namespace)::sig_pause_for_stacktrace (sig=<optimized out>) at src/FWCore/Services/plugins/InitRootHandlers.cc:453
#3  <signal handler called>
#4  __gnu_cxx::__aligned_buffer<std::pair<DetId const, std::vector<std::pair<int, float>, std::allocator<std::pair<int, float> > > > >::_M_addr (this=0x1456b6927c68) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/ext/aligned_buffer.h:102
#5  __gnu_cxx::__aligned_buffer<std::pair<DetId const, std::vector<std::pair<int, float>, std::allocator<std::pair<int, float> > > > >::_M_ptr (this=0x1456b6927c68) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/ext/aligned_buffer.h:115
#6  std::__detail::_Hash_node_value_base<std::pair<DetId const, std::vector<std::pair<int, float>, std::allocator<std::pair<int, float> > > > >::_M_valptr (this=0x1456b6927c68) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/bits/hashtable_policy.h:296
#7  std::__detail::_Hash_node_value_base<std::pair<DetId const, std::vector<std::pair<int, float>, std::allocator<std::pair<int, float> > > > >::_M_v (this=0x1456b6927c68) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/bits/hashtable_policy.h:304
#8  std::__detail::_Map_base<DetId, std::pair<DetId const, std::vector<std::pair<int, float>, std::allocator<std::pair<int, float> > > >, std::allocator<std::pair<DetId const, std::vector<std::pair<int, float>, std::allocator<std::pair<int, float> > > > >, std::__detail::_Select1st, std::equal_to<DetId>, std::hash<DetId>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true>, true>::operator[] (this=this@entry=0x7ffdb33bb9b0, __k=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/bits/hashtable_policy.h:776
#9  0x00001457fd2c04fe in std::unordered_map<DetId, std::vector<std::pair<int, float>, std::allocator<std::pair<int, float> > >, std::hash<DetId>, std::equal_to<DetId>, std::allocator<std::pair<DetId const, std::vector<std::pair<int, float>, std::allocator<std::pair<int, float> > > > > >::operator[] (__k=..., this=0x7ffdb33bb9b0) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/bits/unordered_map.h:979
#10 TSToSimTSHitLCAssociatorByEnergyScoreImpl::makeConnections (this=<optimized out>, tCH=..., lCCH=..., sCCH=..., cPCH=..., sTCH=...) at src/SimCalorimetry/HGCalAssociatorProducers/plugins/TSToSimTSHitLCAssociatorByEnergyScoreImpl.cc:185
#11 0x00001457fd2cde48 in TSToSimTSHitLCAssociatorByEnergyScoreImpl::associateSimToReco (this=<optimized out>, tCH=..., lCCH=..., sCCH=..., cPCH=..., sTCH=...) at src/SimCalorimetry/HGCalAssociatorProducers/plugins/TSToSimTSHitLCAssociatorByEnergyScoreImpl.cc:244
#12 0x00001457fd3003bc in hgcal::TracksterToSimTracksterHitLCAssociator::associateSimToReco (sTCH=..., cPCH=..., sCCH=..., lCCH=..., tCH=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/tmp/BUILDROOT/5fe42f26621cd4d5b00bd4943abd9965/opt/cmssw/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_UBSAN_X_2024-02-16-2300/src/SimDataFormats/Associations/interface/TracksterToSimTracksterHitLCAssociator.h:49
#13 TSToSimTSHitLCAssociatorEDProducer::produce (this=<optimized out>, iEvent=..., iSetup=...) at src/SimCalorimetry/HGCalAssociatorProducers/plugins/TSToSimTSHitLCAssociatorEDProducer.cc:94
#14 0x0000145badc6c54c in edm::global::EDProducerBase::doEvent (this=this@entry=0x14578502cdb0, info=..., act=0x145ba15f3610, mcc=mcc@entry=0x1457857515a8) at /data/cmsbld/jenkins/workspace/build-any-ib/w/tmp/BUILDROOT/5fe42f26621cd4d5b00bd4943abd9965/opt/cmssw/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_UBSAN_X_2024-02-16-2300/src/FWCore/Framework/interface/Event.h:98

Current Modules:

Module: HGCalValidator:hgcalValidator (crashed)
Module: PFDisplacedVertexProducer:particleFlowDisplacedVertex
Module: TSToSimTSHitLCAssociatorEDProducer:tracksterSimTracksterAssociationLinking
Module: TrackingRecoMaterialAnalyser:materialDumperAnalyzer

@fabiocos
Copy link
Contributor

We (on the MTD side) have been so far unable to reproduce the crashes in standalone tests. The simplest possible way to advance I see is to temporarily revert #43762 and see the effect on a couple of IB

@fabiocos fabiocos mentioned this issue Feb 20, 2024
Closed
@makortel
Copy link
Contributor

@fabiocos I have a step3.root from the DIGI step of workflow 25034.999 in
https://mkortela.web.cern.ch/mkortela/issue43942/wf25034.999_step3.root
with which the RECO step of that workflow fails quite reproducibly in ASAN IB.
(I produced the file in CMSSW_14_1_ASAN_X_2024-02-12-2300)

@makortel
Copy link
Contributor

makortel commented Feb 20, 2024 •
edited
Loading

@Dr15Jones and I traced where the row and column are originally set. As far as we can tell, they are first set in ETLDeviceSim::getHitsResponse() that is called by the MixingModule during classical mixing / premixed pileup library creation

cmssw/SimFastTiming/FastTimingCommon/src/ETLDeviceSim.cc

Lines 101 to 123 in db6029c

Local3DPoint simscaled(convertMmToCm(position.x()), convertMmToCm(position.y()), convertMmToCm(position.z()));
const auto& global_point = thedet->toGlobal(simscaled);
radius[0] = global_point.perp();
fluence[0] = integratedLum_ * fluence_.evaluate(radius, emptyV);
gain[0] = lgadGain_.evaluate(fluence, emptyV);
//The following lines check whether the pixel point is actually out of the active area.
if (topo.isInPixel(simscaled)) {
charge *= gain[0];
MPV_charge *= gain[0];
} else {
if (applyDegradation_) {
double dGapCenter = TMath::Max(TMath::Abs(simscaled.x()), TMath::Abs(simscaled.y()));
param[0] = gain[0];
param[1] = dGapCenter;
gain[0] = lgadGainDegradation_.evaluate(param, emptyV);
charge *= gain[0];
MPV_charge *= gain[0];
}
}
const auto& thepixel = topo.pixel(simscaled);

Before #43762 the getHisResponse() skipped a SimHit if the simscaled point didn't satisfy topo.isInPixel(simscaled).

@makortel
Copy link
Contributor

I added a printout to the ETLDeviceSim::getHitsResponse() after

cmssw/SimFastTiming/FastTimingCommon/src/ETLDeviceSim.cc

Lines 123 to 124 in db6029c

const auto& thepixel = topo.pixel(simscaled);
const uint8_t row(thepixel.first), col(thepixel.second);

and ran the step 2 (SingleNuE10_cfi_GEN_SIM_DIGI_PU.py) of the workflow 25034.999 with 4 threads. The printout showed that for detid 1661345329 (that has 32 rows and 16 columns) there is a SimHit for which the column is 16 (also in floating point, i.e. thepixel.second). The isInPixel() was false for the SimHit, so it would have been filtered out earlier.

@fabiocos
Copy link
Contributor

ok, I managed to reproduce the ASAN crash, I was trying with multithreaded jobs on a gun illuminating a whole sensor to probe all possible positions but without result on 1k tracks. I see that all the problems happen in PU workflows.

@makortel
Copy link
Contributor

The condition appears in the step2 (SingleNuE10_cfi_GEN_SIM_DIGI_PU.py) of the workflow 25034.999 also on a single thread. The position (simscaled) of the particular hit is (0.884143,1.06,0.000226099).

@fabiocos fabiocos mentioned this issue Feb 21, 2024
Merged
@fabiocos
Copy link
Contributor

as far as I can see from the integration build after merging #44051 , the ETL-related problem in the ASAN build disappears as expected from my check. There are still a couple of crashes left among various architectures of the wf 25234.911 in HGCal modules.

@makortel
Copy link
Contributor

Thanks @fabiocos. The other HGCal-related failures (so far) seem to be related to already known problems #42025 and #42470 .

@smuzaffar
Copy link
Contributor

Looks like this was fixed by #44051 ... should we close this issue?

@iarspider
Copy link
Contributor Author

@cmsbuild please close

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
dqm-pending pending-signatures reconstruction-pending upgrade-pending
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@makortel @iarspider @dan131riley @fabiocos @smuzaffar @cmsbuild