Adding PNET scores to offline HLT DQM #44754

aniketkhanal · 2024-04-16T20:35:53Z

PR Description

This PR aims to replace ParticleNet(PNET) as the default offline HLT b-tagging algorithm updating #42937 , while creating new folders for each path inside HLT/BTV to keep the current DeepJet histograms for groups that still use it.

It also updates the titles and x-axis labels for the b-tagging efficiency histograms (currently mislabeled as CSV).

The expected changes are as follows:

The already existing folders inside HLT/BTV will contain PNET input plots instead of DeepJet.
For each path, we now have a separate '*_DeepJet' folder that contains plots with DeepJet inputs (note: some paths malfunction with PNET, so they only contain DeepJet folders)
The histograms with name effic_bjetCSV now have appropriate titles and x-axis labels for both PNET and DeepJet.

PR Validation

The changes pass the basic tests suggested in https://cms-sw.github.io/PRWorkflow.html.
12834.0 workflow runs without error and all the changes are visible in local GUI.

cmsbuild · 2024-04-16T20:36:15Z

cms-bot internal usage

cmsbuild · 2024-04-16T20:43:17Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-44754/39976

This PR adds an extra 24KB to repository

cmsbuild · 2024-04-16T20:43:43Z

A new Pull Request was created by @aniketkhanal for master.

It involves the following packages:

DQMOffline/Trigger (dqm)

@tjavaid, @cmsbuild, @syuvivida, @nothingface0, @rvenditti, @antoniovagnerini can you please review it and eventually sign? Thanks.
@Fedespring, @HuguesBrun, @missirol, @trocino, @mtosi, @jhgoh, @cericeci, @rociovilar this is something you requested to watch as well.
@sextonkennedy, @antoniovilela, @rappoccio you are the release manager for this.

cms-bot commands are listed here

tjavaid · 2024-04-17T05:09:36Z

please test

cmsbuild · 2024-04-17T06:19:18Z

-1

Failed Tests: RelVals RelVals-INPUT
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c8339d/38883/summary.html
COMMIT: a13c5b6
CMSSW: CMSSW_14_1_X_2024-04-16-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/44754/38883/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals

136.7611

----- Begin Fatal Exception 17-Apr-2024 07:35:19 CEST-----------------------
An exception of category 'ConfigFileReadError' occurred while
   [0] Processing the python configuration file named step2_PAT_DQM.py
Exception Message:
 unknown python problem occurred.
RuntimeError: An exception of category 'FileInPathError' occurred.
Exception Message:
edm::FileInPath unable to find file RecoBTag/Combined/data/UParTAK4/PUPPI/V00/UParTAK4.onnx anywhere in the search path.
The search path is defined by: CMSSW_SEARCH_PATH
${CMSSW_SEARCH_PATH} is: /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/44754/38883/CMSSW_14_1_X_2024-04-16-2300/poison:/cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/44754/38883/CMSSW_14_1_X_2024-04-16-2300/src:/cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/44754/38883/CMSSW_14_1_X_2024-04-16-2300/external/el8_amd64_gcc12/data:/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-16-2300/poison:/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-16-2300/src:/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-16-2300/external/el8_amd64_gcc12/data
Current directory is: /data/cmsbld/jenkins/workspace/ib-run-pr-relvals/matrix-results/136.7611_RunJetHT2016EreMINIAOD


At:
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-16-2300/src/FWCore/ParameterSet/python/Types.py(887): insertInto
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-16-2300/src/FWCore/ParameterSet/python/Mixins.py(385): insertContentsInto
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-16-2300/src/FWCore/ParameterSet/python/Mixins.py(520): insertInto
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-16-2300/src/FWCore/ParameterSet/python/Modules.py(161): insertInto
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-16-2300/src/FWCore/ParameterSet/python/Config.py(1217): _insertManyInto
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-16-2300/src/FWCore/ParameterSet/python/Config.py(1490): fillProcessDesc
  <string>(2): <module>

----- End Fatal Exception -------------------------------------------------

136.88811

----- Begin Fatal Exception 17-Apr-2024 07:35:19 CEST-----------------------
An exception of category 'ConfigFileReadError' occurred while
   [0] Processing the python configuration file named step2_PAT_DQM.py
Exception Message:
 unknown python problem occurred.
RuntimeError: An exception of category 'FileInPathError' occurred.
Exception Message:
edm::FileInPath unable to find file RecoBTag/Combined/data/UParTAK4/PUPPI/V00/UParTAK4.onnx anywhere in the search path.
The search path is defined by: CMSSW_SEARCH_PATH
${CMSSW_SEARCH_PATH} is: /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/44754/38883/CMSSW_14_1_X_2024-04-16-2300/poison:/cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/44754/38883/CMSSW_14_1_X_2024-04-16-2300/src:/cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/44754/38883/CMSSW_14_1_X_2024-04-16-2300/external/el8_amd64_gcc12/data:/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-16-2300/poison:/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-16-2300/src:/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-16-2300/external/el8_amd64_gcc12/data
Current directory is: /data/cmsbld/jenkins/workspace/ib-run-pr-relvals/matrix-results/136.88811_RunJetHT2018DreMINIAODUL


At:
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-16-2300/src/FWCore/ParameterSet/python/Types.py(887): insertInto
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-16-2300/src/FWCore/ParameterSet/python/Mixins.py(385): insertContentsInto
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-16-2300/src/FWCore/ParameterSet/python/Mixins.py(520): insertInto
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-16-2300/src/FWCore/ParameterSet/python/Modules.py(161): insertInto
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-16-2300/src/FWCore/ParameterSet/python/Config.py(1217): _insertManyInto
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-16-2300/src/FWCore/ParameterSet/python/Config.py(1490): fillProcessDesc
  <string>(2): <module>

----- End Fatal Exception -------------------------------------------------

136.8311

----- Begin Fatal Exception 17-Apr-2024 07:35:20 CEST-----------------------
An exception of category 'ConfigFileReadError' occurred while
   [0] Processing the python configuration file named step2_PAT_DQM.py
Exception Message:
 unknown python problem occurred.
RuntimeError: An exception of category 'FileInPathError' occurred.
Exception Message:
edm::FileInPath unable to find file RecoBTag/Combined/data/UParTAK4/PUPPI/V00/UParTAK4.onnx anywhere in the search path.
The search path is defined by: CMSSW_SEARCH_PATH
${CMSSW_SEARCH_PATH} is: /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/44754/38883/CMSSW_14_1_X_2024-04-16-2300/poison:/cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/44754/38883/CMSSW_14_1_X_2024-04-16-2300/src:/cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/44754/38883/CMSSW_14_1_X_2024-04-16-2300/external/el8_amd64_gcc12/data:/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-16-2300/poison:/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-16-2300/src:/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-16-2300/external/el8_amd64_gcc12/data
Current directory is: /data/cmsbld/jenkins/workspace/ib-run-pr-relvals/matrix-results/136.8311_RunJetHT2017FreMINIAOD


At:
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-16-2300/src/FWCore/ParameterSet/python/Types.py(887): insertInto
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-16-2300/src/FWCore/ParameterSet/python/Mixins.py(385): insertContentsInto
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-16-2300/src/FWCore/ParameterSet/python/Mixins.py(520): insertInto
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-16-2300/src/FWCore/ParameterSet/python/Modules.py(161): insertInto
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-16-2300/src/FWCore/ParameterSet/python/Config.py(1217): _insertManyInto
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-16-2300/src/FWCore/ParameterSet/python/Config.py(1490): fillProcessDesc
  <string>(2): <module>

----- End Fatal Exception -------------------------------------------------

Expand to see more relval errors ...

RelVals-INPUT

4.64.6_MinimumBias2010A/step2_MinimumBias2010A.log
136.72411136.72411_RunJetHT2016B_reminiaodUL/step2_RunJetHT2016B_reminiaodUL.log
136.72412136.72412_RunJetHT2016B_reminiaodUL/step2_RunJetHT2016B_reminiaodUL.log

Expand to see more relval errors ...

mmusich · 2024-04-17T17:38:42Z

@cmsbuild, please test

cmsbuild · 2024-04-17T22:39:37Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c8339d/38902/summary.html
COMMIT: a13c5b6
CMSSW: CMSSW_14_1_X_2024-04-17-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/44754/38902/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

There are some workflows for which there are errors in the baseline:
24834.78 step 2
The results for the comparisons for these workflows could be incomplete
This means most likely that the IB is having errors in the relvals.The error does NOT come from this pull request

Summary:

You potentially removed 56 lines from the logs
Reco comparison results: 2992 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3319355
DQMHistoTests: Total failures: 6410
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3312925
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 6931.257 KiB( 47 files compared)
DQMHistoSizes: changed ( 12834.0,... ): 1035.999 KiB HLT/BTV
DQMHistoSizes: changed ( 141.042,... ): 929.087 KiB HLT/BTV
Checked 202 log files, 165 edm output root files, 48 DQM output files
TriggerResults: found differences in 1 / 46 workflows

tjavaid · 2024-04-25T04:44:01Z

Hi @aniketkhanal , could you please confirm whether the warnings e.g. for potential memory leaks are safe?
src/DQMOffline/Trigger/src/HLTMuonMatchAndPlot.cc:463:1: warning: Potential leak of memory pointed to by 'edgesX' [cplusplus.NewDeleteLeaks] 463 | } | ^ src/DQMOffline/Trigger/src/HLTMuonMatchAndPlot.cc:463:1: warning: Potential leak of memory pointed to by 'edgesY' [cplusplus.NewDeleteLeaks] 2 warnings generated.

Thanks!

mmusich · 2024-04-29T19:00:20Z

@tjavaid

could you please confirm whether the warnings e.g. for potential memory leaks are safe?

Sorry, but isn't this already present in the IB static analyzer logs: https://cmssdt.cern.ch/SDT/jenkins-artifacts/ib-static-analysis/CMSSW_14_1_X_2024-04-29-1100/el8_amd64_gcc12/llvm-analysis/report-0bf3b2.html#EndPath ?

Why is this blocking this particular PR? as far as I can tell this has nothing to do with it.

tjavaid · 2024-04-30T12:38:50Z

+1

cmsbuild · 2024-04-30T12:39:15Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @rappoccio, @antoniovilela, @sextonkennedy (and backports should be raised in the release meeting by the corresponding L2)

rappoccio · 2024-05-01T19:59:02Z

+1

mmusich · 2024-05-02T07:24:48Z

@aniketkhanal do you plan a 14.0.x backport for the current data-taking?

Martin-Grunewald · 2024-05-02T11:39:10Z

It seems this PR creates problems for HIon tests in TSG as well as standard relval workflow 142.0:

----- Begin Fatal Exception 02-May-2024 09:40:49 CEST-----------------------
An exception of category 'ProductNotFound' occurred while
   [0] Processing  Event run: 1 lumi: 3 event: 606 stream: 0
   [1] Running path 'dqmoffline_step'
   [2] Prefetching for module TopMonitor/'BTagMu_AK8Jet300_Mu5'
   [3] Prefetching for module BTagProbabilityToDiscriminator/'pfMassDecorrelatedParticleNetDiscriminatorsJetTags'
   [4] Prefetching for module BoostedJetONNXJetTagsProducer/'pfMassDecorrelatedParticleNetJetTags'
   [5] Calling method for module DeepBoostedJetTagInfoProducer/'pfParticleNetTagInfos'
Exception Message:
Principal::getByToken: Found zero products matching all criteria
Looking for a container with elements of type: reco::Jet
Looking for module label: ak8PFJetsPuppi
Looking for productInstanceName: 

   Additional Info:
      [a] If you wish to continue processing events after a ProductNotFound exception,
add "TryToContinue = cms.untracked.vstring('ProductNotFound')" to the "options" PSet in the configuration.

----- End Fatal Exception -------------------------------------------------

See here: https://cmssdt.cern.ch/SDT/jenkins-artifacts/HLT-Validation/CMSSW_14_1_X_2024-05-01-2300/el8_amd64_gcc12/RelVal_RECO_HIon_MC.log
Presumably, ak8PFJetsPuppi jets are not produced in HIon? The dqm code however should not crash like this upon a missing collection!

See also standard relval workflow 142.0:
https://cmssdt.cern.ch/SDT/cgi-bin/logreader/el8_amd64_gcc12/CMSSW_14_1_X_2024-05-01-2300/pyRelValMatrixLogs/run/142.0_RunHIPhysicsRawPrime2023A/step3_RunHIPhysicsRawPrime2023A.log#/

mmusich · 2024-05-02T11:57:45Z

The dqm code however should not crash like this upon a missing collection!

From the exception is does not look like that's DQM code crashing, but rather:

Calling method for module DeepBoostedJetTagInfoProducer/'pfParticleNetTagInfos'

makortel · 2024-05-03T14:00:00Z

Now also in an issue #44899

aniketkhanal · 2024-05-08T17:18:21Z

@aniketkhanal do you plan a 14.0.x backport for the current data-taking?

@mmusich Yes we do

mmusich · 2024-05-08T17:21:34Z

Yes we do

Ok. But then you also need to backport #44926 together to avoid relval failures.

aniketkhanal added 3 commits April 5, 2024 16:55

making PNet default b-tagging algorithm

d98c7fe

adding PNet scores while keeping DeepJet

a6b333b

Add PNET and keep DeepJet

a13c5b6

cmsbuild added this to the CMSSW_14_1_X milestone Apr 16, 2024

cmsbuild added dqm-pending pending-signatures tests-pending orp-pending code-checks-pending labels Apr 16, 2024

cmsbuild added code-checks-approved and removed code-checks-pending labels Apr 16, 2024

cmsbuild added tests-started and removed tests-pending labels Apr 17, 2024

cmsbuild added tests-rejected and removed tests-started labels Apr 17, 2024

cmsbuild added tests-started and removed tests-rejected labels Apr 17, 2024

cmsbuild added tests-approved and removed tests-started labels Apr 17, 2024

mmusich mentioned this pull request Apr 18, 2024

fixup eta lookup in absEtaLowEdges_ in miscellaneous RecoEgamma/EgammaHLTProducers plugins #44764

Merged

makortel mentioned this pull request Apr 18, 2024

Non-reproducibility in workflow 11634.0 #44779

Open

cmsbuild added dqm-approved fully-signed and removed dqm-pending pending-signatures labels Apr 30, 2024

cmsbuild added orp-approved and removed orp-pending labels May 1, 2024

cmsbuild merged commit 6e9205a into cms-sw:master May 1, 2024
11 checks passed

makortel mentioned this pull request May 3, 2024

ProductNotFound exception in workflow 142.0 #44899

Closed

aniketkhanal mentioned this pull request May 20, 2024

[14_0_X] Adding PNET scores to offline HLT DQM #45003

Merged

aniketkhanal mentioned this pull request Dec 19, 2024

Adding UParT scores to offline HLT DQM #47012

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding PNET scores to offline HLT DQM #44754

Adding PNET scores to offline HLT DQM #44754

aniketkhanal commented Apr 16, 2024

cmsbuild commented Apr 16, 2024 •

edited

Loading

cmsbuild commented Apr 16, 2024

cmsbuild commented Apr 16, 2024

tjavaid commented Apr 17, 2024

cmsbuild commented Apr 17, 2024

mmusich commented Apr 17, 2024

cmsbuild commented Apr 17, 2024

tjavaid commented Apr 25, 2024

mmusich commented Apr 29, 2024

tjavaid commented Apr 30, 2024

cmsbuild commented Apr 30, 2024

rappoccio commented May 1, 2024

mmusich commented May 2, 2024

Martin-Grunewald commented May 2, 2024 •

edited

Loading

mmusich commented May 2, 2024

makortel commented May 3, 2024

aniketkhanal commented May 8, 2024

mmusich commented May 8, 2024

Adding PNET scores to offline HLT DQM #44754

Adding PNET scores to offline HLT DQM #44754

Conversation

aniketkhanal commented Apr 16, 2024

cmsbuild commented Apr 16, 2024 • edited Loading

cmsbuild commented Apr 16, 2024

cmsbuild commented Apr 16, 2024

tjavaid commented Apr 17, 2024

cmsbuild commented Apr 17, 2024

RelVals

RelVals-INPUT

mmusich commented Apr 17, 2024

cmsbuild commented Apr 17, 2024

Comparison Summary

tjavaid commented Apr 25, 2024

mmusich commented Apr 29, 2024

tjavaid commented Apr 30, 2024

cmsbuild commented Apr 30, 2024

rappoccio commented May 1, 2024

mmusich commented May 2, 2024

Martin-Grunewald commented May 2, 2024 • edited Loading

mmusich commented May 2, 2024

makortel commented May 3, 2024

aniketkhanal commented May 8, 2024

mmusich commented May 8, 2024

cmsbuild commented Apr 16, 2024 •

edited

Loading

Martin-Grunewald commented May 2, 2024 •

edited

Loading