Update Muon MVA ID model and working points in MiniAOD and NanoAOD #41903

andrea21z · 2023-06-08T12:37:02Z

PR description:

This PR includes the update of the Muon MVA ID working points in MiniAOD and NanoAOD for the latest training.
This training is the final and has been published in CMS-PAS-MUO-22-001.
Related PR: to include the new MVA model in data repository, in .onnx format -> Updating Muon MVA ID model cms-data/RecoMuon-MuonIdentification#9

PR validation:

We check the output of the new branch and Working Points running runTheMatrix and everything looks fine.
We execute the basic tests suggested in the CMSSW PR instructions.

cmsbuild · 2023-06-08T12:43:38Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-41903/35850

This PR adds an extra 32KB to repository

cmsbuild · 2023-06-08T12:44:08Z

A new Pull Request was created by @andrea21z for master.

It involves the following packages:

PhysicsTools/NanoAOD (xpog)
PhysicsTools/PatAlgos (xpog, reconstruction)

@cmsbuild, @simonepigazzini, @mandrenguyen, @clacaputo, @vlimant can you please review it and eventually sign? Thanks.
@AlexDeMoor, @rappoccio, @gouskos, @jdolen, @JyothsnaKomaragiri, @ahinzmann, @AnnikaStein, @schoef, @emilbols, @jdamgov, @mbluj, @nhanvtran, @gkasieczka, @hatakeyamak, @gpetruc, @azotz, @mariadalfonso, @demuller, @andrzejnovak, @seemasharmafnal, @mmarionncern this is something you requested to watch as well.
@perrotta, @dpiparo, @rappoccio you are the release manager for this.

cms-bot commands are listed here

clacaputo · 2023-06-12T10:04:28Z

type muon

clacaputo · 2023-06-12T10:04:51Z

enable nano

clacaputo · 2023-06-12T10:05:33Z

please test with cms-data/RecoMuon-MuonIdentification#9

cmsbuild · 2023-06-12T13:30:56Z

-1

Failed Tests: RelVals-INPUT
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e5036b/33092/summary.html
COMMIT: 01ab854
CMSSW: CMSSW_13_2_X_2023-06-12-1100/el8_amd64_gcc11
Additional Tests: NANO
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/41903/33092/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals-INPUT

250200.17DAS Error

Comparison Summary

Summary:

You potentially removed 1 lines from the logs
Reco comparison results: 105 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3190910
DQMHistoTests: Total failures: 2262
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3188626
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
Checked 207 log files, 159 edm output root files, 48 DQM output files
TriggerResults: no differences found

NANO Comparison Summary

Summary:

You potentially added 4 lines to the logs
Reco comparison results: 18 differences found in the comparisons
DQMHistoTests: Total files compared: 12
DQMHistoTests: Total histograms compared: 12351
DQMHistoTests: Total failures: 46
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 12305
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 11 files compared)
Checked 25 log files, 11 edm output root files, 12 DQM output files

Nano size comparison Summary:

Sample	kb/ev	ref kb/ev	diff kb/ev	ev/s/thd	ref ev/s/thd	diff rate	mem/thd	ref mem/thd
2500.31	2.438	2.438	0.000 ( +0.0% )	5.46	5.49	-0.4%	2.107	2.116
2500.311	2.546	2.546	0.000 ( +0.0% )	4.89	4.90	-0.1%	2.539	2.541
2500.312	2.491	2.491	0.000 ( +0.0% )	5.09	5.06	+0.5%	2.521	2.525
2500.33	1.240	1.240	0.000 ( +0.0% )	10.21	10.27	-0.6%	2.227	2.233
2500.331	1.604	1.603	0.000 ( +0.0% )	5.45	5.41	+0.8%	2.416	2.418
2500.332	1.485	1.485	0.000 ( +0.0% )	7.90	7.82	+1.0%	2.327	2.332
2500.401	2.153	2.153	0.000 ( +0.0% )	5.53	5.49	+0.9%	1.970	1.965
2500.501	1.753	1.753	0.000 ( +0.0% )	9.65	9.57	+0.8%	1.851	1.850
2500.511	1.145	1.145	0.000 ( +0.0% )	4.46	4.49	-0.8%	2.167	2.170
2500.5111	1.503	1.503	0.000 ( +0.0% )	3.93	3.95	-0.4%	2.245	2.251
2500.601	2.015	2.015	0.000 ( +0.0% )	13.72	13.74	-0.1%	1.728	1.727

andrea21z · 2023-06-14T14:16:03Z

Hi @clacaputo, it seems to have failed due to an 'input DAS' error, should I do something else?

clacaputo · 2023-06-19T14:40:45Z

test parameters:

enable = nano
pull_request = Updating Muon MVA ID model cms-data/RecoMuon-MuonIdentification#9

clacaputo · 2023-06-19T14:42:21Z

please test

cmsbuild · 2023-06-19T18:17:57Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e5036b/33241/summary.html
COMMIT: 01ab854
CMSSW: CMSSW_13_2_X_2023-06-19-1100/el8_amd64_gcc11
Additional Tests: NANO
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/41903/33241/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e5036b/33241/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e5036b/33241/git-merge-result

Comparison Summary

Summary:

You potentially removed 85 lines from the logs
Reco comparison results: 112 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3196062
DQMHistoTests: Total failures: 293
DQMHistoTests: Total nulls: 16
DQMHistoTests: Total successes: 3195731
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.6559999999999999 KiB( 47 files compared)
DQMHistoSizes: changed ( 11634.0,... ): -0.164 KiB L1T/L1TStage2uGT
DQMHistoSizes: changed ( 12434.0,... ): 0.492 KiB L1T/L1TStage2uGT
DQMHistoSizes: changed ( 312.0 ): 0.164 KiB L1T/L1TStage2uGT
Checked 207 log files, 159 edm output root files, 48 DQM output files
TriggerResults: no differences found

NANO Comparison Summary

Summary:

You potentially removed 1 lines from the logs
Reco comparison results: 21 differences found in the comparisons
DQMHistoTests: Total files compared: 15
DQMHistoTests: Total histograms compared: 15505
DQMHistoTests: Total failures: 51
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 15454
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 14 files compared)
Checked 31 log files, 14 edm output root files, 15 DQM output files

Nano size comparison Summary:

Sample	kb/ev	ref kb/ev	diff kb/ev	ev/s/thd	ref ev/s/thd	diff rate	mem/thd	ref mem/thd
2500.0	2.436	2.436	0.000 ( +0.0% )	5.39	5.22	+3.2%	2.108	2.046
2500.001	2.546	2.546	0.000 ( +0.0% )	4.87	4.76	+2.4%	2.548	2.473
2500.002	2.491	2.491	0.000 ( +0.0% )	5.04	4.89	+3.1%	2.535	2.465
2500.01	1.240	1.240	0.000 ( +0.0% )	10.00	9.78	+2.2%	2.229	2.168
2500.011	1.604	1.603	0.000 ( +0.0% )	5.37	5.30	+1.2%	2.427	2.339
2500.012	1.485	1.485	0.000 ( +0.0% )	7.76	7.60	+2.1%	2.331	2.257
2500.1	2.163	2.163	0.000 ( +0.0% )	5.41	5.40	+0.2%	1.982	1.978
2500.2	2.260	2.260	0.000 ( +0.0% )	6.31	6.22	+1.5%	1.881	1.883
2500.21	1.145	1.145	0.000 ( +0.0% )	4.44	4.39	+1.3%	2.181	2.153
2500.211	1.503	1.503	0.000 ( +0.0% )	3.86	3.73	+3.4%	2.250	2.231
2500.3	1.998	1.998	0.000 ( +0.0% )	13.65	13.15	+3.8%	1.685	1.689
2500.31	1.207	1.207	-0.000 ( -0.0% )	21.39	20.19	+6.0%	2.094	2.056
2500.311	1.584	1.584	0.000 ( +0.0% )	14.98	15.19	-1.4%	2.163	2.128
2500.4	1.998	1.998	0.000 ( +0.0% )	13.67	13.12	+4.2%	1.693	1.690

cmsbuild · 2023-07-06T09:01:52Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-41903/36203

This PR adds an extra 36KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File PhysicsTools/NanoAOD/python/muons_cff.py modified in PR(s): Custom_Muon_POG_NanoAOD #42039

cmsbuild · 2023-07-06T09:02:19Z

Pull request #41903 was updated. @cmsbuild, @simonepigazzini, @mandrenguyen, @clacaputo, @vlimant can you please check and sign again.

clacaputo · 2023-07-07T13:56:15Z

please test

cmsbuild · 2023-07-07T18:18:01Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e5036b/33599/summary.html
COMMIT: 4443ff6
CMSSW: CMSSW_13_2_X_2023-07-07-1100/el8_amd64_gcc11
Additional Tests: NANO
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/41903/33599/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

@lathomas IsoPhotonEB RAW-RECO skim #42150

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e5036b/33599/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e5036b/33599/git-merge-result

Comparison Summary

There are some workflows for which there are errors in the baseline:
136.897 step 2
140.025 step 2
The results for the comparisons for these workflows could be incomplete
This means most likely that the IB is having errors in the relvals.The error does NOT come from this pull request

Summary:

You potentially added 17 lines to the logs
Reco comparison results: 50 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3193892
DQMHistoTests: Total failures: 30
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3193840
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
Checked 207 log files, 159 edm output root files, 48 DQM output files
TriggerResults: no differences found

NANO Comparison Summary

Summary:

You potentially added 8 lines to the logs
Reco comparison results: 21 differences found in the comparisons
DQMHistoTests: Total files compared: 15
DQMHistoTests: Total histograms compared: 15505
DQMHistoTests: Total failures: 51
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 15454
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 14 files compared)
Checked 31 log files, 14 edm output root files, 15 DQM output files

Nano size comparison Summary:

Sample	kb/ev	ref kb/ev	diff kb/ev	ev/s/thd	ref ev/s/thd	diff rate	mem/thd	ref mem/thd
2500.0	2.443	2.443	0.000 ( +0.0% )	5.35	5.46	-1.9%	2.047	2.081
2500.001	2.552	2.552	0.000 ( +0.0% )	4.79	4.89	-2.1%	2.461	2.122
2500.002	2.497	2.497	0.000 ( +0.0% )	4.97	5.06	-1.9%	2.464	2.134
2500.01	1.240	1.240	0.000 ( +0.0% )	9.74	10.21	-4.6%	2.143	1.959
2500.011	1.604	1.604	0.000 ( +0.0% )	5.26	5.40	-2.7%	2.288	1.998
2500.012	1.485	1.485	0.000 ( +0.0% )	7.52	7.82	-3.9%	2.222	2.028
2500.1	2.168	2.169	-0.000 ( -0.0% )	5.42	5.46	-0.7%	1.905	1.957
2500.2	2.271	2.271	0.000 ( +0.0% )	6.35	6.34	+0.2%	1.796	1.837
2500.21	1.146	1.146	0.000 ( +0.0% )	4.47	4.41	+1.3%	2.150	1.850
2500.211	1.504	1.504	0.000 ( +0.0% )	3.92	3.90	+0.5%	2.122	1.945
2500.3	2.008	2.008	0.000 ( +0.0% )	13.74	13.78	-0.3%	1.689	1.664
2500.31	1.209	1.209	0.000 ( +0.0% )	21.43	21.95	-2.4%	2.062	2.086
2500.311	1.585	1.585	0.000 ( +0.0% )	15.53	15.90	-2.3%	2.021	2.155
2500.4	2.008	2.008	0.000 ( +0.0% )	13.81	13.78	+0.2%	1.694	1.683

clacaputo · 2023-07-10T14:51:07Z

+reconstruction

Update Muon MVA ID model and working points in MiniAOD and NanoAOD #41903 (comment)

vlimant · 2023-07-11T07:28:50Z

+1

cmsbuild · 2023-07-11T07:29:15Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

perrotta · 2023-07-11T09:45:28Z

+1

Andrea Trapote added 2 commits June 8, 2023 14:21

updating Muon MVA ID working points for the new training

709d8b4

Updating working points of the Muon MVA ID for the new model in NanoAOD

01ab854

cmsbuild added this to the CMSSW_13_2_X milestone Jun 8, 2023

cmsbuild added reconstruction-pending pending-signatures tests-pending orp-pending code-checks-pending xpog-pending labels Jun 8, 2023

cmsbuild added code-checks-approved and removed code-checks-pending labels Jun 8, 2023

cmsbuild added the muon label Jun 12, 2023

cmsbuild added tests-started requires-external and removed tests-pending labels Jun 12, 2023

cmsbuild added tests-rejected and removed tests-started labels Jun 12, 2023

cmsbuild added tests-started and removed tests-rejected labels Jun 19, 2023

cmsbuild removed the tests-started label Jun 19, 2023

code formats

4443ff6

cmsbuild added code-checks-pending and removed code-checks-rejected labels Jul 6, 2023

cmsbuild added code-checks-approved and removed code-checks-pending labels Jul 6, 2023

cmsbuild added tests-started and removed tests-pending labels Jul 7, 2023

cmsbuild added tests-approved and removed tests-started labels Jul 7, 2023

cmsbuild added reconstruction-approved and removed reconstruction-pending labels Jul 10, 2023

cmsbuild added fully-signed xpog-approved and removed pending-signatures xpog-pending labels Jul 11, 2023

cmsbuild added orp-approved and removed orp-pending labels Jul 11, 2023

cmsbuild merged commit 546d4a7 into cms-sw:master Jul 11, 2023

This was referenced Jul 11, 2023

Update TF to 2.12 (backport from IB/CMSSW_13_2_X/g13 branch) cms-sw/cmsdist#8565

Merged

fix off-by-one error in TauNNIdHW::compute #42233

Merged

[ASAN] Fix runtime error by moving std::regex code in anonymous namespace #42077

Merged

cramonal mentioned this pull request Feb 15, 2024

Fix on muon_mvaID in NanoAOD #43976

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update Muon MVA ID model and working points in MiniAOD and NanoAOD #41903

Update Muon MVA ID model and working points in MiniAOD and NanoAOD #41903

andrea21z commented Jun 8, 2023

cmsbuild commented Jun 8, 2023

cmsbuild commented Jun 8, 2023

clacaputo commented Jun 12, 2023

clacaputo commented Jun 12, 2023

clacaputo commented Jun 12, 2023

cmsbuild commented Jun 12, 2023

andrea21z commented Jun 14, 2023

clacaputo commented Jun 19, 2023

clacaputo commented Jun 19, 2023

cmsbuild commented Jun 19, 2023

cmsbuild commented Jul 6, 2023

cmsbuild commented Jul 6, 2023

clacaputo commented Jul 7, 2023

cmsbuild commented Jul 7, 2023

clacaputo commented Jul 10, 2023

vlimant commented Jul 11, 2023

cmsbuild commented Jul 11, 2023

perrotta commented Jul 11, 2023

Update Muon MVA ID model and working points in MiniAOD and NanoAOD #41903

Update Muon MVA ID model and working points in MiniAOD and NanoAOD #41903

Conversation

andrea21z commented Jun 8, 2023

PR description:

PR validation:

cmsbuild commented Jun 8, 2023

cmsbuild commented Jun 8, 2023

clacaputo commented Jun 12, 2023

clacaputo commented Jun 12, 2023

clacaputo commented Jun 12, 2023

cmsbuild commented Jun 12, 2023

RelVals-INPUT

Comparison Summary

NANO Comparison Summary

andrea21z commented Jun 14, 2023

clacaputo commented Jun 19, 2023

clacaputo commented Jun 19, 2023

cmsbuild commented Jun 19, 2023

Comparison Summary

NANO Comparison Summary

cmsbuild commented Jul 6, 2023

cmsbuild commented Jul 6, 2023

clacaputo commented Jul 7, 2023

cmsbuild commented Jul 7, 2023

Comparison Summary

NANO Comparison Summary

clacaputo commented Jul 10, 2023

vlimant commented Jul 11, 2023

cmsbuild commented Jul 11, 2023

perrotta commented Jul 11, 2023