Issue message if DB IOV updated during job

[Dr15Jones]

PR description:

This adds a LogSystem output each time the CondDBESSource updates an IOV during the job. The output is avoided the first IOV check as the number of lines added to the log file is quite large.

This is meant to help diagnose a problem being seen in the HLT.

PR validation:

Code compiles. Forcing the report to always be on gave the expected output.

[Dr15Jones]

please test

[cmsbuild]

cms-bot internal usage

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-46393/42239

[cmsbuild]

A new Pull Request was created by @Dr15Jones for master.

It involves the following packages:

• CondCore/CondDB (db)

@atpathak, @consuegs, @francescobrivio, @perrotta can you please review it and eventually sign? Thanks.
@JanChyczynski, @PonIlya, @mmusich, @rsreds, @yuanchao this is something you requested to watch as well.
@antoniovilela, @mandrenguyen, @rappoccio, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

[cmsbuild]

+1

Size: This PR adds an extra 20KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-1e78aa/42207/summary.html
COMMIT: f1d5a12
CMSSW: CMSSW_14_2_X_2024-10-15-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/46393/42207/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially added 152 lines to the logs
• Reco comparison results: 21 differences found in the comparisons
• DQMHistoTests: Total files compared: 44
• DQMHistoTests: Total histograms compared: 3335065
• DQMHistoTests: Total failures: 2106
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3332939
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 43 files compared)
• Checked 193 log files, 163 edm output root files, 44 DQM output files
• TriggerResults: no differences found

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-46393/42246

[cmsbuild]

Pull request #46393 was updated. @atpathak, @cmsbuild, @consuegs, @francescobrivio, @perrotta can you please check and sign again.

[Dr15Jones]

please test

[Dr15Jones]

So from the previous PR tests, it does appear that fetching during the job can happen for offline jobs processing data. My hypothesis is some conditions data have such large IOV lists that they are not fetched all at once, instead they are fetched in pages. At construction time CondDBESSource just fetches the first page of IOVs for all conditions objects. Then at begin run, if the IOVs page containing that particular run was not pre-fetched, it goes and fetches it. This appears to only be true for a couple of conditions objects, not all of them.

[mmusich]

My hypothesis is some conditions data have such large IOV lists that they are not fetched all at once, instead they are fetched in pages. At construction time CondDBESSource just fetches the first page of IOVs for all conditions objects. Then at begin run, if the IOVs page containing that particular run was not pre-fetched, it goes and fetches it. This appears to only be true for a couple of conditions objects, not all of them.

that was the "concern" I was referring to earlier today. Few time-stamp based conditions can have tens or hundreds of thousands of IoVs and cannot fit in a single read.

[cmsbuild]

+1

Size: This PR adds an extra 20KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-1e78aa/42220/summary.html
COMMIT: d798eec
CMSSW: CMSSW_14_2_X_2024-10-15-1400/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/46393/42220/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially added 151 lines to the logs
• Reco comparison results: 23 differences found in the comparisons
• DQMHistoTests: Total files compared: 44
• DQMHistoTests: Total histograms compared: 3335065
• DQMHistoTests: Total failures: 2124
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3332921
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 43 files compared)
• Checked 193 log files, 163 edm output root files, 44 DQM output files
• TriggerResults: no differences found

[perrotta]

Just for my understanding: does this update need to be merged, or it is just for your private tests? I.e. do you need the db signature, here and in the backport PRs?

[mmusich]

does this update need to be merged, or it is just for your private tests? I.e. do you need the db signature, here and in the backport PRs?

if we want it to be useful for debugging #45555 we'll need a release to be deployed at HLT with this PR in it, so I think unless we want to make branches diverge, yes db signature is needed here and in backports.

[perrotta]

+db

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @rappoccio, @mandrenguyen, @antoniovilela, @sextonkennedy (and backports should be raised in the release meeting by the corresponding L2)

[mandrenguyen]

+1