Investigate endLuminosityBlock transition behavior in DAQ

[makortel]

See cms-sw/cmssw#42931 for details

Issue 1

• Check again if the delayed "finish end" of lumi 1 repeats, switch to StallMonitor #690
• Craft a reproducer of the reported behavior (possible with a "sleeping source") #691
• Allow streamEndLumi() to be called outside of readNextEventForStream() #701
• Add ability for Source to tell that a new LuminosityBlock is the only occurrence of that LuminosityBlock #716

Issue 2

• Allow a stream to skip a lumi if all events processed #742
• Understand if DQMStore can handle streams not seeing all lumis (ask DQM) #702
• Explore if anything else than DQM could break because of not all streams seeing all lumis #703
• Improve GenWeightsTableProducer #785
• Improve ExternalGeneratorFilter #799
• Work around L1RCTProducer #800
• Improve ConcurrentGeneratorFilter and ConcurrentHadronizerFilter #801

Tracer improvements

• Add run and lumi transitions to StallMonitor #692
• Extend Tracer to produce a compact trace log to a separate file #693
• Develop a rudimentary web viewer for the compact Tracer data #709
• Add Source::getNextType callbacks to ActivityRegistry #711
• Add between Event callbacks to Tracer log file #717
• Fix tracer log viewer issue with callID #725
• Add callID to ESModuleCallingContext #729
• Add module centric view to the Tracer viewer #736
• Add transform info to Tracer Log #743

Documentation

• Talk in core software meeting #820
• Clarify behavior in twiki #821

Aftermath

• Investigate IB failures #854
• Initialize statsGuard_ on LogErrorEventFilter #856
• Fix two spinlocks #862

[makortel]

Some notes on discussion today on Issue 1

Could we make https://github.com/cms-sw/cmssw/blob/c85a467052fc4101cbcea854b2158afcbf7ee0a4/FWCore/Framework/src/LuminosityBlockProcessingStatus.h#L109C1-L110C1 atomic, and move this block
https://github.com/cms-sw/cmssw/blob/c85a467052fc4101cbcea854b2158afcbf7ee0a4/FWCore/Framework/src/EventProcessor.cc#L2161-L2164
(and possibly https://github.com/cms-sw/cmssw/blob/c85a467052fc4101cbcea854b2158afcbf7ee0a4/FWCore/Framework/src/EventProcessor.cc#L2151-L2159) out of the readNextEventForStream()?

If moved outside, we would need to keep the if blocks inside readNextEventForStream() as well.

In the end, we should make the "event processing state" atomic, and copy both if blocks also outside of readNextEventForStream(), to beginning of handleNextEventForStreamAsync() https://github.com/cms-sw/cmssw/blob/c85a467052fc4101cbcea854b2158afcbf7ee0a4/FWCore/Framework/src/EventProcessor.cc#L2204.

Need to be sure that the beginLumi of the next lumi has already been called, before calling streamEndLumiAsync().

The handleNextEventForStreamAsync() would probably be better call along handleNextTransitionForStreamAsync().

[makortel]

Add ability for Source to tell that a new LuminosityBlock is the only occurrence of that LuminosityBlock

• Try to make InputSource::ItemType an enum class
• Make InputSource::ItemType explicitly a char
• Add a new struct that holds the InputSource::ItemType and an additional bool to tell if that LuminosityBlock is the only occurrence. The struct would have explicit conversion function to InputSource::ItemType
• Pass the new struct from concrete Source to the framework via InputSource::getNextItemType()