Set numberOfConcurrentLuminosityBlocks to 1 for GEN steps that have EDModules that are not going to support concurrent lumis

[makortel]

PR description:

As reported in #25090 several generator EDModules are not planned to be made support concurrent lumis (e.g. too difficult, so little overal CPU time spent so that it is not worth it, or with ExternalGeneratorFilter desire to continue testing generators also directly).

In order to silence the warning message on multithreaded jobs, and to later allow changing that warning to an exception (to better prevent any modules that do not support concurrent lumis to creep in workflow steps that are expected to support concurrent lumis), following the example of #35073 this PR suggest the ConfigBuilder to set the number of concurrent lumis (and IOVs) explicitly to 1 if the job has GEN step with any of the generator EDModules that do not support concurrent lumis. In this PR I added a new file Configuration/Generator/python/concurrentLumisDisable.py to contain the list of the EDModules (so that it would be explicitly maintained by @cms-sw/generators-l2), in principle they could be listed elsewhere too.

This PR is very similar to

PR validation:

Workflows 5.2,140.0,521.0,7.0,300.0,140.0,5.5,511.0,281.0,8.1,534.0,281.0,132.0,280.0,120.0 set their GEN step to use 1 concurrent lumi, and workflow 10024.0 continues to set 2 concurrent lumis (when run multithreaded).

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-35182/25119

• This PR adds an extra 52KB to repository

• There are other open Pull requests which might conflict with changes you have proposed:

  • File Configuration/Applications/python/ConfigBuilder.py modified in PR(s): Add SiStripCalCosmics ALCANANO #34985

[cmsbuild]

A new Pull Request was created by @makortel (Matti Kortelainen) for master.

It involves the following packages:

• Configuration/Applications (operations)
• Configuration/Generator (generators)

@SiewYan, @perrotta, @mkirsano, @Saptaparna, @cmsbuild, @GurpreetSinghChahal, @agrohsje, @alberto-sanchez, @qliphy, @fabiocos, @davidlange6 can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @fabiocos this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

[makortel]

test parameters:

• workflows_threading = 5.2,140.0,521.0,7.0,300.0,140.0,5.5,511.0,281.0,8.1,534.0,281.0,132.0,280.0,120.0
• enable_tests = threading

[makortel]

@cmsbuild, please test

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f5d47e/18369/summary.html
COMMIT: b777388
CMSSW: CMSSW_12_1_X_2021-09-07-1100/slc7_amd64_gcc900
Additional Tests: THREADING
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/35182/18369/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

The workflows 140.53 have different files in step1_dasquery.log than the ones found in the baseline. You may want to check and retrigger the tests if necessary. You can check it in the "files" directory in the results of the comparisons

Summary:

• No significant changes to the logs found
• Reco comparison results: 1303 differences found in the comparisons
• DQMHistoTests: Total files compared: 39
• DQMHistoTests: Total histograms compared: 3001001
• DQMHistoTests: Total failures: 3677
• DQMHistoTests: Total nulls: 19
• DQMHistoTests: Total successes: 2997283
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 45.703 KiB( 38 files compared)
• DQMHistoSizes: changed ( 140.53 ): 44.531 KiB Hcal/DigiRunHarvesting
• DQMHistoSizes: changed ( 140.53 ): 1.172 KiB RPC/DCSInfo
• Checked 165 log files, 37 edm output root files, 39 DQM output files
• TriggerResults: no differences found

[makortel]

I inspected the GEN step configuration files, and the following workflows set the numberOfConcurrentLuminosityBlocks to

• 1: 5.2, 5.5, 7.0, 7.3, 8.1, 101.0, 120.0, 132.0, 140.0, 280.0, 281.0, 300.0, 312.0, 511.0, 534.0
• 2: 5.1, 135.4, 521.0, 11634.911, 11634.912, 28234.0

[agrohsje]

Thanks @makortel. There is also the pt and dxy gun we could add: Pythia8PtAndDxyGun .

[makortel]

There is also the pt and dxy gun we could add: Pythia8PtAndDxyGun .

Thanks, added.

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-35182/25231

• This PR adds an extra 28KB to repository

• There are other open Pull requests which might conflict with changes you have proposed:

  • File Configuration/Applications/python/ConfigBuilder.py modified in PR(s): Add SiStripCalCosmics ALCANANO #34985

[cmsbuild]

Pull request #35182 was updated. @SiewYan, @perrotta, @mkirsano, @Saptaparna, @cmsbuild, @GurpreetSinghChahal, @agrohsje, @alberto-sanchez, @qliphy, @fabiocos, @davidlange6 can you please check and sign again.

[makortel]

@cmsbuild, please test

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f5d47e/18558/summary.html
COMMIT: ab8c522
CMSSW: CMSSW_12_1_X_2021-09-13-1100/slc7_amd64_gcc900
Additional Tests: THREADING
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/35182/18558/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 39
• DQMHistoTests: Total histograms compared: 3001001
• DQMHistoTests: Total failures: 0
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3000979
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 38 files compared)
• Checked 165 log files, 37 edm output root files, 39 DQM output files
• TriggerResults: no differences found

[agrohsje]

+generators

[qliphy]

+1

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will be automatically merged.