Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add wfs with HLT as separate step #37603

Closed
wants to merge 7 commits into from
Closed
Show file tree
Hide file tree
Changes from 6 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions Configuration/PyReleaseValidation/python/relval_steps.py
Original file line number Diff line number Diff line change
Expand Up @@ -3585,6 +3585,14 @@ def gen2021HiMix(fragment,howMuch):
'--eventcontent':'FEVTDEBUGHLT',
'--geometry' : geom
}

upgradeStepDict['DigiNoHLT'][k] = {'-s':'DIGI:pdigi_valid,L1,DIGI2RAW',
'--conditions':gt,
'--datatier':'GEN-SIM-DIGI-RAW',
'-n':'10',
'--eventcontent':'FEVTDEBUGHLT',
'--geometry' : geom
}

# Adding Track trigger step in step2
upgradeStepDict['DigiTrigger'][k] = {'-s':'DIGI:pdigi_valid,L1TrackTrigger,L1,DIGI2RAW,HLT:%s'%(hltversion),
Expand All @@ -3594,6 +3602,15 @@ def gen2021HiMix(fragment,howMuch):
'--eventcontent':'FEVTDEBUGHLT',
'--geometry' : geom
}

upgradeStepDict['HLT'][k] = {'-s':'HLT:%s'%(hltversion),
'--conditions':gt,
'--datatier':'GEN-SIM-DIGI-RAW',
'-n':'10',
'--eventcontent':'FEVTDEBUGHLT',
'--geometry' : geom,
'--outputCommands' : '"drop *_*_*_GEN,drop *_*_*_DIGI2RAW"'
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @jfernan2 ! You can reproduce the DQM crash by replacing the drop statements here with:
"drop *_*_*_GEN,drop *_*_*_SIM,drop *_*_*_DIGI2RAW" and running 12424.0. The error message at the last step is:

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Sun Apr 17 20:58:01 CEST 2022
Thread 2 (Thread 0x7f071393e700 (LWP 1782) "cmsRun"):
#0 0x00007f073c5e41d9 in waitpid () from /lib64/libpthread.so.0
#1 0x00007f07362665e7 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms.cern.ch/slc7_amd64_gcc11/cms/cmssw/CMSSW_12_3_0/lib/slc7_amd64_gcc11/pluginFWCoreServicesPlugins.so
#2 0x00007f073626712a in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms.cern.ch/slc7_amd64_gcc11/cms/cmssw/CMSSW_12_3_0/lib/slc7_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3 0x00007f073cbe2bf4 in std::execute_native_thread_routine (__p=0x7f07323e4600) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82
#4 0x00007f073c5dcea5 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f073c305b0d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f073a458540 (LWP 1577) "cmsRun"):
#0 0x00007f073c2faddd in poll () from /lib64/libc.so.6
#1 0x00007f073626689f in full_read.constprop () from /cvmfs/cms.cern.ch/slc7_amd64_gcc11/cms/cmssw/CMSSW_12_3_0/lib/slc7_amd64_gcc11/pluginFWCoreServicesPlugins.so
#2 0x00007f07362671fc in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms.cern.ch/slc7_amd64_gcc11/cms/cmssw/CMSSW_12_3_0/lib/slc7_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3 0x00007f0736269a3b in sig_dostack_then_abort () from /cvmfs/cms.cern.ch/slc7_amd64_gcc11/cms/cmssw/CMSSW_12_3_0/lib/slc7_amd64_gcc11/pluginFWCoreServicesPlugins.so
#4
#5 0x00007f06ee55dcbc in EcalCondObjectContainer::find(unsigned int) const () from /cvmfs/cms.cern.ch/slc7_amd64_gcc11/cms/cmssw/CMSSW_12_3_0/lib/slc7_amd64_gcc11/libDQMEcalMonitorClient.so
#6 0x00007f06ee55ba65 in ecaldqm::IntegrityClient::producePlots(ecaldqm::DQWorkerClient::ProcessType) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc11/cms/cmssw/CMSSW_12_3_0/lib/slc7_amd64_gcc11/libDQMEcalMonitorClient.so
#7 0x00007f06ee5bb5b4 in EcalDQMonitorClient::runWorkers(dqm::implementation::IGetter&, ecaldqm::DQWorkerClient::ProcessType) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc11/cms/cmssw/CMSSW_12_3_0/lib/slc7_amd64_gcc11/pluginDQMEcalMonitorClientPlugins.so
#8 0x00007f06ee5bbf8d in EcalDQMonitorClient::dqmEndJob(dqm::implementation::IBooker&, dqm::implementation::IGetter&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc11/cms/cmssw/CMSSW_12_3_0/lib/slc7_amd64_gcc11/pluginDQMEcalMonitorClientPlugins.so
#9 0x00007f06ee5bea34 in non-virtual thunk to DQMEDHarvester::endProcessBlockProduce(edm::ProcessBlock&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc11/cms/cmssw/CMSSW_12_3_0/lib/slc7_amd64_gcc11/pluginDQMEcalMonitorClientPlugins.so
#10 0x00007f073ed9f1e0 in edm::one::EDProducerBase::doEndProcessBlock(edm::ProcessBlockPrincipal const&, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc11/cms/cmssw/CMSSW_12_3_0/lib/slc7_amd64_gcc11/libFWCoreFramework.so
#11 0x00007f073ed88a80 in edm::WorkerTedm::one::EDProducerBase::implDoEndProcessBlock(edm::ProcessBlockPrincipal const&, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc11/cms/cmssw/CMSSW_12_3_0/lib/slc7_amd64_gcc11/libFWCoreFramework.so
#12 0x00007f073ec95567 in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::ProcessBlockPrincipal, (edm::BranchActionType)3> >(edm::OccurrenceTraits<edm::ProcessBlockPrincipal, (edm::BranchActionType)3>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::ProcessBlockPrincipal, (edm::BranchActionType)3>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::ProcessBlockPrincipal, (edm::BranchActionType)3> >(edm::OccurrenceTraits<edm::ProcessBlockPrincipal, (edm::BranchActionType)3>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::ProcessBlockPrincipal, (edm::BranchActionType)3>::Context const*)::{lambda()#1}) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc11/cms/cmssw/CMSSW_12_3_0/lib/slc7_amd64_gcc11/libFWCoreFramework.so
#13 0x00007f073ec95960 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::ProcessBlockPrincipal, (edm::BranchActionType)3> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::ProcessBlockPrincipal, (edm::BranchActionType)3>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::ProcessBlockPrincipal, (edm::BranchActionType)3>::Context const*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc11/cms/cmssw/CMSSW_12_3_0/lib/slc7_amd64_gcc11/libFWCoreFramework.so
#14 0x00007f073ec95f0a in void edm::SerialTaskQueueChain::actionToRun<edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::ProcessBlockPrincipal, (edm::BranchActionType)3> >::execute()::{lambda()#1}&>(edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::ProcessBlockPrincipal, (edm::BranchActionType)3> >::execute()::{lambda()#1}&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc11/cms/cmssw/CMSSW_12_3_0/lib/slc7_amd64_gcc11/libFWCoreFramework.so
#15 0x00007f073ec95fe1 in edm::SerialTaskQueue::QueuedTask<edm::SerialTaskQueueChain::push<edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::ProcessBlockPrincipal, (edm::BranchActionType)3> >::execute()::{lambda()#1}&>(tbb::detail::d1::task_group&, edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::ProcessBlockPrincipal, (edm::BranchActionType)3> >::execute()::{lambda()#1}&)::{lambda()#1}>::execute() () from /cvmfs/cms.cern.ch/slc7_amd64_gcc11/cms/cmssw/CMSSW_12_3_0/lib/slc7_amd64_gcc11/libFWCoreFramework.so
#16 0x00007f073eee6055 in tbb::detail::d1::function_taskedm::SerialTaskQueue::spawn(edm::SerialTaskQueue::TaskBase&)::{lambda()#1}::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc11/cms/cmssw/CMSSW_12_3_0/lib/slc7_amd64_gcc11/libFWCoreConcurrency.so
#17 0x00007f073d444a59 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=0x7f06b7054c00, this=0x7f0738eafe00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_12_3_0-slc7_amd64_gcc11/build/CMSSW_12_3_0-build/BUILD/slc7_amd64_gcc11/external/tbb/v2021.4.0-0929d4245541a9360696e439234c1bfc/tbb-v2021.4.0/src/tbb/task_dispatcher.h:322
#18 tbb::detail::r1::task_dispatcher::local_wait_for_alltbb::detail::r1::external_waiter (waiter=..., t=, this=0x7f0738eafe00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_12_3_0-slc7_amd64_gcc11/build/CMSSW_12_3_0-build/BUILD/slc7_amd64_gcc11/external/tbb/v2021.4.0-0929d4245541a9360696e439234c1bfc/tbb-v2021.4.0/src/tbb/task_dispatcher.h:463
#19 tbb::detail::r1::task_dispatcher::execute_and_wait (t=, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_12_3_0-slc7_amd64_gcc11/build/CMSSW_12_3_0-build/BUILD/slc7_amd64_gcc11/external/tbb/v2021.4.0-0929d4245541a9360696e439234c1bfc/tbb-v2021.4.0/src/tbb/task_dispatcher.cpp:168
#20 0x00007f073ec624c3 in edm::EventProcessor::endProcessBlock(bool, bool) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc11/cms/cmssw/CMSSW_12_3_0/lib/slc7_amd64_gcc11/libFWCoreFramework.so
#21 0x00007f073ec667f9 in edm::EventProcessor::runToCompletion() () from /cvmfs/cms.cern.ch/slc7_amd64_gcc11/cms/cmssw/CMSSW_12_3_0/lib/slc7_amd64_gcc11/libFWCoreFramework.so
#22 0x000000000040a18d in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#23 0x00007f073d432898 in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_12_3_0-slc7_amd64_gcc11/build/CMSSW_12_3_0-build/BUILD/slc7_amd64_gcc11/external/tbb/v2021.4.0-0929d4245541a9360696e439234c1bfc/tbb-v2021.4.0/src/tbb/arena.cpp:698
#24 0x000000000040afd9 in main::{lambda()#1}::operator()() const ()
#25 0x00000000004096fc in main ()

Current Modules:

Module: EcalDQMonitorClient:ecalMonitorClient (crashed)

}

upgradeStepDict['Reco'][k] = {'-s':'RAW2DIGI,L1Reco,RECO,RECOSIM,PAT,VALIDATION:@standardValidation+@miniAODValidation,DQM:@standardDQM+@ExtraHLT+@miniAODDQM',
'--conditions':gt,
Expand Down Expand Up @@ -3702,6 +3719,9 @@ def gen2021HiMix(fragment,howMuch):
'--geometry' : geom
}

upgradeStepDict['RecoNanoHLT'][k] = upgradeStepDict['RecoNano'][k]
upgradeStepDict['RecoNanoHLT'][k].update({'--secondfilein': 'file:step2.root', '--filein': 'file:step3.root'})

# setup baseline and variations
for specialType,specialWF in upgradeWFs.items():
specialWF.setup(upgradeStepDict, k, upgradeProperties[year][k])
Expand Down
4 changes: 2 additions & 2 deletions Configuration/PyReleaseValidation/python/relval_upgrade.py
Original file line number Diff line number Diff line change
Expand Up @@ -46,9 +46,9 @@ def makeStepName(key,frag,step,suffix):
# hack to add an extra step
if 'ProdLike' in specialType:
if 'Reco' in step: # handles both Reco and RecoGlobal
stepList[specialType].append(stepMaker(key,frag[:-4],step.replace('RecoGlobal','MiniAOD').replace('RecoNano','MiniAOD').replace('Reco','MiniAOD'),specialWF.suffix))
stepList[specialType].append(stepMaker(key,frag[:-4],step.replace('RecoGlobal','MiniAOD').replace('RecoNanoHLT','MiniAOD').replace('RecoNano','MiniAOD').replace('Reco','MiniAOD'),specialWF.suffix))
if 'RecoNano' in step:
stepList[specialType].append(stepMaker(key,frag[:-4],step.replace('RecoNano','Nano'),specialWF.suffix))
stepList[specialType].append(stepMaker(key,frag[:-4],step.replace('RecoNanoHLT','Nano').replace('RecoNano','Nano'),specialWF.suffix))
# similar hacks for premixing
if 'PMX' in specialType:
if 'GenSim' in step:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,10 @@
'2021PU',
'2021Design',
'2021DesignPU',
'2021HLT',
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should not change the order here. New workflow should go at the end, i.e. after 2024.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, will append it to the end.

'2021HLTPU',
'2021DesignHLT',
'2021DesignHLTPU',
'2023',
'2023PU',
'2024',
Expand Down Expand Up @@ -171,11 +175,14 @@ def condition(self, fragment, stepList, key, hasHarvest):
'GenSimHLBeamSpotHGCALCloseBy',
'Digi',
'DigiTrigger',
'DigiNoHLT',
'HLT',
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HLT seems to be very common name. Can it be more specific, i.e. HLTRun3?

'RecoLocal',
'Reco',
'RecoFakeHLT',
'RecoGlobal',
'RecoNano',
'RecoNanoHLT',
'HARVEST',
'HARVESTFakeHLT',
'HARVESTNano',
Expand All @@ -188,12 +195,15 @@ def condition(self, fragment, stepList, key, hasHarvest):
],
PU = [
'DigiTrigger',
'DigiNoHLT',
'HLT',
'RecoLocal',
'RecoGlobal',
'Digi',
'Reco',
'RecoFakeHLT',
'RecoNano',
'RecoNanoHLT',
'HARVEST',
'HARVESTFakeHLT',
'HARVESTNano',
Expand Down Expand Up @@ -1868,6 +1878,22 @@ def condition(self, fragment, stepList, key, hasHarvest):
'BeamSpot': 'GaussSigmaZ4cm',
'ScenToRun' : ['GenSim','Digi','RecoNano','HARVESTNano'],
},
'2021HLT' : {
'Geom' : 'DB:Extended',
'GT' : 'auto:phase1_2021_realistic',
'HLTmenu': '@relval2021',
'Era' : 'Run3',
'BeamSpot': 'Run3RoundOptics25ns13TeVLowSigmaZ',
'ScenToRun' : ['GenSim','DigiNoHLT','HLT','RecoNanoHLT','HARVESTNano','ALCA'],
},
'2021DesignHLT' : {
'Geom' : 'DB:Extended',
'GT' : 'auto:phase1_2021_design',
'HLTmenu': '@relval2021',
'Era' : 'Run3',
'BeamSpot': 'GaussSigmaZ4cm',
'ScenToRun' : ['GenSim','DigiNoHLT','HLT','RecoNanoHLT','HARVESTNano'],
},
'2023' : {
'Geom' : 'DB:Extended',
'GT' : 'auto:phase1_2023_realistic',
Expand Down