introduced a python script for Phase-2 HLT integration tests

[mmusich]

PR description:

As there has been a recent surge of activities concerning the phase-2 HLT menu, it's important to start engineering some tests to ensure that the basic functionalities of the menu are preserved when configuration updates are introduced.
The goal of this PR is to propose a initial version of a driver script for Phase-2 HLT integration tests to be discussed with the phase2 HLT upgrade community.
The hltIntegrationTests scripts we do have for the Run-3 menu are not well suited for this use case, due to the structure of the phase-2 menu and the necessity to run the L1P2GT step upfront.
This script:

• creates the configuration for the phase-2 menu from cmssw;
• for each of the N paths in the menu it creates a copy of the menu with only that one path in the schedule;
• it runs ten (configurable) GEN-SIM-DIGI-RAW TTbar event for the current phase-2 default geometry (as the time of this PR that would be D110);
• runs on those events first the full menu and then N jobs for the N single-path versions;
• finally it loops over the output N root files and compares for each one of the standalone path outputs the trigger results with the full menu.

In case any of the cmsRun process fails, the script fails. In case one of the trigger results comparison fail, the script fails.
Run single-threaded takes about 40 minutes on lxplus. I provide options to run multi-threaded, but it might be too demanding in terms of requested resources if there are parallel cmsRun executions (4 by default).
I think this kind of test should be able to spot early on the kind of issue that were corrected at #46019 or #46054.
As the execution time is quite long, I would propose to run this test in IBs as we do for the Run-3 menus here

PR validation:

In a recent CMSSW_14_2_X IB:

cd HLTrigger/Configuration/python/HLT_75e33/test/
python3 hltPhase2UpgradeIntegrationTests.py

runs.

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

Not a backport, not meant to be backported.

[cmsbuild]

cms-bot internal usage

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-46110/41899

[cmsbuild]

A new Pull Request was created by @mmusich for master.

It involves the following packages:

• HLTrigger/Configuration (hlt)

@Martin-Grunewald, @cmsbuild, @mmusich can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @SohamBhattacharya, @VourMa, @missirol, @rovere, @silviodonato this is something you requested to watch as well.
@antoniovilela, @mandrenguyen, @rappoccio, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

[mmusich]

@cmsbuild, please test

• even though there's nothing to test

[cmsbuild]

+1

Size: This PR adds an extra 12KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-52d644/41728/summary.html
COMMIT: 5f13b1c
CMSSW: CMSSW_14_2_X_2024-09-24-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/46110/41728/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially removed 4 lines from the logs
• Reco comparison results: 88 differences found in the comparisons
• DQMHistoTests: Total files compared: 44
• DQMHistoTests: Total histograms compared: 3331070
• DQMHistoTests: Total failures: 5328
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3325722
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 43 files compared)
• Checked 193 log files, 163 edm output root files, 44 DQM output files
• TriggerResults: no differences found

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-46110/41912

[cmsbuild]

Pull request #46110 was updated. @Martin-Grunewald, @cmsbuild, @mmusich can you please check and sign again.

[mmusich]

@cmsbuild, please test

[cmsbuild]

+1

Size: This PR adds an extra 20KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-52d644/41743/summary.html
COMMIT: ac50279
CMSSW: CMSSW_14_2_X_2024-09-24-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/46110/41743/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially added 1 lines to the logs
• Reco comparison results: 88 differences found in the comparisons
• DQMHistoTests: Total files compared: 44
• DQMHistoTests: Total histograms compared: 3331070
• DQMHistoTests: Total failures: 2874
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3328176
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 43 files compared)
• Checked 193 log files, 163 edm output root files, 44 DQM output files
• TriggerResults: no differences found

[mmusich]

+hlt

• we'll follow-up with changes to cms-bot once this is integrated

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @sextonkennedy, @mandrenguyen, @rappoccio, @antoniovilela (and backports should be raised in the release meeting by the corresponding L2)

[mandrenguyen]

+1