TkAl: fix PV validation and PV resolution submission scripts

[mmusich]

PR description:

The goal of this PR is to fix the scripts submitPVResolutionJobs.py and submitPVValidationJobs.py such that they can run in recent releases and with recent (Run 3) data.
The related unit tests are improved such that, in addition to technically creating the configuration to be run with HTCondor, said configuration is also tested for at least one job (running locally).

PR validation:

Run successfully:

• scram b runtests_SubmitPVsplit
• scram b runtests_SubmitPVrbr

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

Not a backport, but it will be backported to CMSSW_14_0_X for 2024 data-taking purposes.

[cmsbuild]

cms-bot internal usage

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-44467/39551

• This PR adds an extra 36KB to repository

• There are other open Pull requests which might conflict with changes you have proposed:

  • File Alignment/OfflineValidation/scripts/submitPVResolutionJobs.py modified in PR(s): [Dummy] Try creating big-enough cache to cause splitting #44368, Dummy commit #44401
  • File Alignment/OfflineValidation/scripts/submitPVValidationJobs.py modified in PR(s): [Dummy] Try creating big-enough cache to cause splitting #44368, Dummy commit #44401
  • File Alignment/OfflineValidation/test/PrimaryVertexResolution_templ_cfg.py modified in PR(s): [Dummy] Try creating big-enough cache to cause splitting #44368, Dummy commit #44401

[cmsbuild]

A new Pull Request was created by @mmusich for master.

It involves the following packages:

• Alignment/OfflineValidation (alca)

@perrotta, @consuegs, @cmsbuild, @saumyaphor4252 can you please review it and eventually sign? Thanks.
@mmusich, @tlampen, @yuanchao, @tocheng, @rsreds, @adewit this is something you requested to watch as well.
@rappoccio, @antoniovilela, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

[mmusich]

@cmsbuild, please test

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-44467/39557

• This PR adds an extra 36KB to repository

• There are other open Pull requests which might conflict with changes you have proposed:

  • File Alignment/OfflineValidation/scripts/submitPVResolutionJobs.py modified in PR(s): [Dummy] Try creating big-enough cache to cause splitting #44368, Dummy commit #44401
  • File Alignment/OfflineValidation/scripts/submitPVValidationJobs.py modified in PR(s): [Dummy] Try creating big-enough cache to cause splitting #44368, Dummy commit #44401
  • File Alignment/OfflineValidation/test/PrimaryVertexResolution_templ_cfg.py modified in PR(s): [Dummy] Try creating big-enough cache to cause splitting #44368, Dummy commit #44401

[cmsbuild]

Pull request #44467 was updated. @perrotta, @saumyaphor4252, @consuegs, @cmsbuild can you please check and sign again.

[mmusich]

please test

[cmsbuild]

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-dbc1c1/38270/summary.html
COMMIT: 8687f99
CMSSW: CMSSW_14_1_X_2024-03-19-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/44467/38270/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found 1 errors in the following unit tests:

---> test SubmitPVsplit had ERRORS

Comparison Summary

Summary:

• You potentially added 85 lines to the logs
• Reco comparison results: 45 differences found in the comparisons
• DQMHistoTests: Total files compared: 48
• DQMHistoTests: Total histograms compared: 3297369
• DQMHistoTests: Total failures: 3
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3297346
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
• Checked 202 log files, 165 edm output root files, 48 DQM output files
• TriggerResults: no differences found

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-44467/39568

• This PR adds an extra 36KB to repository

• There are other open Pull requests which might conflict with changes you have proposed:

  • File Alignment/OfflineValidation/scripts/submitPVResolutionJobs.py modified in PR(s): [Dummy] Try creating big-enough cache to cause splitting #44368, Dummy commit #44401
  • File Alignment/OfflineValidation/scripts/submitPVValidationJobs.py modified in PR(s): [Dummy] Try creating big-enough cache to cause splitting #44368, Dummy commit #44401
  • File Alignment/OfflineValidation/test/PrimaryVertexResolution_templ_cfg.py modified in PR(s): [Dummy] Try creating big-enough cache to cause splitting #44368, Dummy commit #44401

[cmsbuild]

Pull request #44467 was updated. @cmsbuild, @saumyaphor4252, @consuegs, @perrotta can you please check and sign again.

[mmusich]

@cmsbuild, please test

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-dbc1c1/38276/summary.html
COMMIT: 6f66382
CMSSW: CMSSW_14_1_X_2024-03-19-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/44467/38276/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially added 84 lines to the logs
• Reco comparison results: 54 differences found in the comparisons
• DQMHistoTests: Total files compared: 48
• DQMHistoTests: Total histograms compared: 3297369
• DQMHistoTests: Total failures: 3
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3297346
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
• Checked 202 log files, 165 edm output root files, 48 DQM output files
• TriggerResults: no differences found

[perrotta]

+1

• The updated unit test succeeds

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @sextonkennedy, @rappoccio, @antoniovilela (and backports should be raised in the release meeting by the corresponding L2)

[antoniovilela]

+1

[aandvalenzuela]

Hello @mmusich,

I noticed both unit tests SubmitPVsplit and SubmitPVrbr fail in IBs when the number of runs is 0. It has happened two times:

• In CMSSW_14_1_MULTIARCHS_X_2024-03-31-2300 for el9_amd64_gcc12. See log.

• In CMSSW_14_1_MULTIARCHS_X_2024-03-28-2300 for el8_amd64_gcc12. See log.

In both cases, the error appears when trying to copy a file that I think is only generated if there is at least one run:

Will run on 1 workflows
>>>> This is Data!
>>>> Doing run based selection
Will run on  0 runs: 
 []
first run: 1 last run: 999999


======================================================================
|| WARNING: won't run on any run, probably DAS returned an empty query,
|| but that's fine because this is a unit test!
======================================================================




 TESTING Primary Vertex Validation script execution ...
cp: cannot stat './BASH/PVValidation_testingOfflineGT_HLTPhysics_Run2023D_0.sh': No such file or directory
/data/cmsbld/jenkins/workspace/ib-run-qa/CMSSW_14_1_MULTIARCHS_X_2024-03-31-2300/src/Alignment/OfflineValidation/test/testingScripts/test_unitSubmitPVrbr.sh: line 28: ./PVValidation_testingOfflineGT_HLTPhysics_Run2023D_0.sh: No such file or directory
Failure running PVValidation script: status 127

[mmusich]

@aandvalenzuela I think that's fine - that's only a symptom. It should be investigated instead why the das query was empty.

[aandvalenzuela]

Thanks @mmusich, I thought it was fine to have an empty query from DAS (as reported in the warning message) and that it should be handled by the unit test. In any case, I am trying to see if there is a way to prevent DAS from returning an empty query.

[mmusich]

@aandvalenzuela

thought it was fine to have an empty query from DAS (as reported in the warning message) and that it should be handled by the unit test.

It's fine in the sense that when I created the test, since DAS query failures are fairly frequent in IB tests (for reasons not clear to me), I thought of avoiding the test to fail every time. In any case here's a patch for that #44588. Still it would be nice to understand why the query failed specifically for MULTIARCHS and not elsewhere.

[smuzaffar]

@aandvalenzuela @mmusich , I think this das error has nothing to do with these unit tests. Normally when the voms proxy is in bad state [a] then all unit tests which run das queries fail. We had tried to fix this by creating a dedicated proxy for each jenkins job but still do not understand why proxy file is either deleted or overridden

[a]

2024/04/01 10:40:46 ERROR failed to parse X509 proxy: crypto/tls: failed to parse key: asn1: syntax error: data truncated