DQM visualization-live and visualization-live-secondInstance crash during HI cosmics

[nothingface0]

We noticed that the DQM visualization clients, visualization-live and visualization-live-secondInstance crash during some of the latest cosmic runs, namely: 387579,387557,387556,387555,387552,387548,387546,387544,387541,387539,387531,387338,387240,387235,387212,387209,387207

The exception is:

----- Begin Fatal Exception 29-Oct-2024 15:06:19 CET-----------------------
An exception of category 'NoProductResolverException' occurred while
   [0] Processing  Event run: 387552 lumi: 9 event: 19781392 stream: 3
   [1] Running path 'FEVToutput_step'
   [2] Prefetching for module JsonWritingTimeoutPoolOutputModule/'FEVToutput'
   [3] Calling method for module DeDxHitInfoProducer/'dedxHitInfoCosmicTF'
Exception Message:
No data of type "ClusterShapeHitFilter" with label "ClusterShapeHitFilter" in record "CkfComponentsRecord"
 Please add an ESSource or ESProducer to your job which can deliver this data.
----- End Fatal Exception -------------------------------------------------

More logs here.

First instance was during run 387207 (24/10/2024). Not all cosmic runs lead to this behavior, however. There was no such crash during 387559, for example.

We were using CMSSW_14_1_1 and CMSSW_14_1_4_patch1 at the time of the crashes, with Global Tag 141X_dataRun3_Express_v3. We have not tested yet if this is reproducible with 14_0_X.

Any input is appreciated.

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @nothingface0.

@Dr15Jones, @antoniovilela, @makortel, @mandrenguyen, @rappoccio, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[makortel]

assign dqm

[makortel]

@cms-sw/trk-dpg-l2

[cmsbuild]

New categories assigned: dqm

@antoniovagnerini,@nothingface0,@rvenditti,@syuvivida,@tjavaid you have been requested to review this Pull request/Issue and eventually sign? Thanks

[nothingface0]

This does not seem to be reproducible with 14_0_15_patch1 and the 140X_dataRun3_Express_v3 Global Tag.

We do get lots of the following however:

%MSG                                                                                                                      
Begin processing the 14th record. Run 387552, Event 48791634, LumiSection 20 on stream 7 at 30-Oct-2024 16:02:51.222 CET  
%MSG-e TooManyClusters:  CosmicSeedGenerator:cosmicseedfinderP5  30-Oct-2024 16:02:51 CET Run: 387552 Event: 48791634     
Found too many clusters (379), bailing out.                                                                               
                                                                                                                          
%MSG                                                                                                                      
%MSG-e TooManyClusters:  SimpleCosmicBONSeeder:simpleCosmicBONSeeds  30-Oct-2024 16:02:51 CET Run: 387552 Event: 48791634 
Found too many clusters (379), bailing out.                                                                               
                                                                                                                          
%MSG                                                                                                                      

[mmusich]

The exception is:

It's very likely the problem is due to #45016 @stahlleiton FYI

We do get lots of the following however:

alas that's normal, see #46283 for details.

[mmusich]

#46563 offers a trivial fix.