Reduce symbol search #193

smuzaffar · 2023-12-24T13:57:30Z

No description provided.

smuzaffar · 2023-12-24T13:57:46Z

please test for CMSSW_14_0_ROOT6_X

cmsbuild · 2023-12-24T13:58:02Z

A new Pull Request was created by @smuzaffar (Malik Shahzad Muzaffar) for branch cms/master/329fb5ae0c.

@aandvalenzuela, @iarspider, @smuzaffar can you please review it and eventually sign? Thanks.
@sextonkennedy, @rappoccio, @antoniovilela you are the release manager for this.
cms-bot commands are listed here

cmsbuild · 2023-12-24T13:58:03Z

cms-bot internal usage

cmsbuild · 2023-12-24T23:24:59Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-78f86a/36678/summary.html
COMMIT: e2af2f9
CMSSW: CMSSW_14_0_ROOT6_X_2023-12-21-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/root/193/36678/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-78f86a/36678/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-78f86a/36678/git-merge-result

Comparison Summary

Summary:

You potentially removed 3 lines from the logs
Reco comparison results: 3879 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3247277
DQMHistoTests: Total failures: 8009
DQMHistoTests: Total nulls: 20
DQMHistoTests: Total successes: 3239226
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
Checked 200 log files, 161 edm output root files, 48 DQM output files
TriggerResults: found differences in 1 / 46 workflows

cmsbuild · 2024-01-08T10:26:42Z

Pull request #193 was updated.

smuzaffar · 2024-01-08T11:08:46Z

please test for CMSSW_14_0_ROOT6_X

cmsbuild · 2024-01-08T18:43:28Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-78f86a/36736/summary.html
COMMIT: 8f4c26d
CMSSW: CMSSW_14_0_ROOT6_X_2024-01-07-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/root/193/36736/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially added 87 lines to the logs
Reco comparison results: 993 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3247277
DQMHistoTests: Total failures: 6270
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3240985
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
Checked 200 log files, 161 edm output root files, 48 DQM output files
TriggerResults: no differences found

The llvm9 JIT issued callbacks when a symbol was missing and we reacted on it by loading the relevant library. In root-project/root@9b2041e3 we have kept the logic but now the JIT started querying more often even for symbols which are okay to be missing. In turn that leads to scanning all libraries causing performance issues. This patch tries to limit this functionality only in contexts where automatic loading is allowed.

Co-authored-by: Jonas Hahnfeld <jonas.hahnfeld@cern.ch>

Symbol lookup is a quite expensive operation and might result in JIT compilation and library loading. Co-authored-by: Jonas Hahnfeld <jonas.hahnfeld@cern.ch>

This reverts commit 329fb5a.

cmsbuild · 2024-01-11T11:43:34Z

Pull request #193 was updated.

smuzaffar · 2024-01-15T15:50:53Z

please test for CMSSW_14_0_ROOT6_X

smuzaffar · 2024-01-15T16:00:36Z

please test with cms-sw/cmsdist#8936 for CMSSW_14_0_ROOT6_X

cmsbuild · 2024-01-15T21:36:06Z

-1

Failed Tests: UnitTests RelVals
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-78f86a/36826/summary.html
COMMIT: 55145f0
CMSSW: CMSSW_14_0_ROOT6_X_2024-01-14-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/root/193/36826/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found 3 errors in the following unit tests:

---> test runtestRecoEgammaElectronIdentification had ERRORS
---> test testTauEmbeddingProducers had ERRORS
---> test testAlcaBeamMonitor had ERRORS

RelVals

136.7611

----- Begin Fatal Exception 15-Jan-2024 21:58:19 CET-----------------------
An exception of category 'FileInPathError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing module: class=ElectronMVAValueMapProducer label='electronMVAValueMapProducer'
Exception Message:
edm::FileInPath unable to find file RecoEgamma/ElectronIdentification/data/MVAWeightFiles/Winter22HZZV1/EB1_5.weights.xml.gz anywhere in the search path.
The search path is defined by: CMSSW_SEARCH_PATH
${CMSSW_SEARCH_PATH} is: /cvmfs/cms-ci.cern.ch/week0/cms-sw/root/193/36826/CMSSW_14_0_ROOT6_X_2024-01-14-2300/poison:/cvmfs/cms-ci.cern.ch/week0/cms-sw/root/193/36826/CMSSW_14_0_ROOT6_X_2024-01-14-2300/src:/cvmfs/cms-ci.cern.ch/week0/cms-sw/root/193/36826/CMSSW_14_0_ROOT6_X_2024-01-14-2300/external/el8_amd64_gcc12/data:/cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_0_ROOT6_X_2024-01-14-2300/src
Current directory is: /data/cmsbld/jenkins/workspace/ib-run-pr-relvals/matrix-results/136.7611_RunJetHT2016EreMINIAOD
----- End Fatal Exception -------------------------------------------------

136.8311

----- Begin Fatal Exception 15-Jan-2024 21:58:19 CET-----------------------
An exception of category 'FileInPathError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing module: class=ElectronMVAValueMapProducer label='electronMVAValueMapProducer'
Exception Message:
edm::FileInPath unable to find file RecoEgamma/ElectronIdentification/data/MVAWeightFiles/Winter22HZZV1/EB1_5.weights.xml.gz anywhere in the search path.
The search path is defined by: CMSSW_SEARCH_PATH
${CMSSW_SEARCH_PATH} is: /cvmfs/cms-ci.cern.ch/week0/cms-sw/root/193/36826/CMSSW_14_0_ROOT6_X_2024-01-14-2300/poison:/cvmfs/cms-ci.cern.ch/week0/cms-sw/root/193/36826/CMSSW_14_0_ROOT6_X_2024-01-14-2300/src:/cvmfs/cms-ci.cern.ch/week0/cms-sw/root/193/36826/CMSSW_14_0_ROOT6_X_2024-01-14-2300/external/el8_amd64_gcc12/data:/cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_0_ROOT6_X_2024-01-14-2300/src
Current directory is: /data/cmsbld/jenkins/workspace/ib-run-pr-relvals/matrix-results/136.8311_RunJetHT2017FreMINIAOD
----- End Fatal Exception -------------------------------------------------

136.88811

----- Begin Fatal Exception 15-Jan-2024 21:58:20 CET-----------------------
An exception of category 'FileInPathError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing module: class=ElectronMVAValueMapProducer label='electronMVAValueMapProducer'
Exception Message:
edm::FileInPath unable to find file RecoEgamma/ElectronIdentification/data/MVAWeightFiles/Winter22HZZV1/EB1_5.weights.xml.gz anywhere in the search path.
The search path is defined by: CMSSW_SEARCH_PATH
${CMSSW_SEARCH_PATH} is: /cvmfs/cms-ci.cern.ch/week0/cms-sw/root/193/36826/CMSSW_14_0_ROOT6_X_2024-01-14-2300/poison:/cvmfs/cms-ci.cern.ch/week0/cms-sw/root/193/36826/CMSSW_14_0_ROOT6_X_2024-01-14-2300/src:/cvmfs/cms-ci.cern.ch/week0/cms-sw/root/193/36826/CMSSW_14_0_ROOT6_X_2024-01-14-2300/external/el8_amd64_gcc12/data:/cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_0_ROOT6_X_2024-01-14-2300/src
Current directory is: /data/cmsbld/jenkins/workspace/ib-run-pr-relvals/matrix-results/136.88811_RunJetHT2018DreMINIAODUL
----- End Fatal Exception -------------------------------------------------

Expand to see more relval errors ...

smuzaffar · 2024-01-16T10:19:08Z

please test with cms-sw/cmsdist#8936 for CMSSW_14_0_ROOT6_X

cmsbuild · 2024-01-16T16:58:40Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-78f86a/36833/summary.html
COMMIT: 55145f0
CMSSW: CMSSW_14_0_ROOT6_X_2024-01-15-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/root/193/36833/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-78f86a/36833/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-78f86a/36833/git-merge-result

Unit Tests

I found 1 errors in the following unit tests:

---> test testAlcaBeamMonitor had ERRORS

Comparison Summary

Summary:

You potentially added 14080 lines to the logs
Reco comparison results: 58 differences found in the comparisons
DQMHistoTests: Total files compared: 10
DQMHistoTests: Total histograms compared: 327017
DQMHistoTests: Total failures: 225
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 326784
DQMHistoTests: Total skipped: 8
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 9 files compared)
Checked 149 log files, 71 edm output root files, 10 DQM output files
TriggerResults: no differences found

cmsbuild added orp-pending pending-signatures externals-pending tests-started labels Dec 24, 2023

smuzaffar mentioned this pull request Dec 24, 2023

[core] Reduce symbol search only to when autoloading is enabled. root-project/root#14287

Merged

cmsbuild added tests-approved and removed tests-started labels Dec 24, 2023

vgvassilev force-pushed the reduce-symbol-search branch from e2af2f9 to 8f4c26d Compare January 8, 2024 10:26

cmsbuild added tests-pending and removed tests-approved labels Jan 8, 2024

smuzaffar changed the base branch from cms/master/329fb5ae0c to cms/master/3ef0acc January 8, 2024 11:08

cmsbuild added tests-started and removed tests-pending labels Jan 8, 2024

cmsbuild added tests-approved and removed tests-started labels Jan 8, 2024

vgvassilev and others added 4 commits January 11, 2024 10:27

[core] Evaluate initializer of constant variables

c22be55

Co-authored-by: Jonas Hahnfeld <jonas.hahnfeld@cern.ch>

[core] Prefer variable initializer over symbol search

566e73c

Symbol lookup is a quite expensive operation and might result in JIT compilation and library loading. Co-authored-by: Jonas Hahnfeld <jonas.hahnfeld@cern.ch>

Revert "[core] Materialize symbols for TError variables"

55145f0

This reverts commit 329fb5a.

hahnjo force-pushed the reduce-symbol-search branch from 8f4c26d to 55145f0 Compare January 11, 2024 11:43

cmsbuild added tests-pending and removed tests-approved labels Jan 11, 2024

cmsbuild added tests-started and removed tests-pending labels Jan 15, 2024

smuzaffar changed the base branch from cms/master/3ef0acc to cms/master/8e8380f3d3 January 15, 2024 16:00

cmsbuild added the requires-external label Jan 15, 2024

cmsbuild added tests-rejected and removed tests-started labels Jan 15, 2024

cmsbuild added tests-started and removed tests-rejected labels Jan 16, 2024

cmsbuild added tests-rejected and removed tests-started labels Jan 16, 2024

smuzaffar closed this Jan 26, 2024

iarspider mentioned this pull request Feb 8, 2024

Script to mass-update bot cache in all issues cms-sw/cms-bot#2165

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reduce symbol search #193

Reduce symbol search #193

smuzaffar commented Dec 24, 2023

smuzaffar commented Dec 24, 2023

cmsbuild commented Dec 24, 2023

cmsbuild commented Dec 24, 2023 •

edited

Loading

cmsbuild commented Dec 24, 2023

cmsbuild commented Jan 8, 2024

smuzaffar commented Jan 8, 2024

cmsbuild commented Jan 8, 2024

cmsbuild commented Jan 11, 2024

smuzaffar commented Jan 15, 2024

smuzaffar commented Jan 15, 2024

cmsbuild commented Jan 15, 2024

smuzaffar commented Jan 16, 2024

cmsbuild commented Jan 16, 2024

Reduce symbol search #193

Reduce symbol search #193

Conversation

smuzaffar commented Dec 24, 2023

smuzaffar commented Dec 24, 2023

cmsbuild commented Dec 24, 2023

cmsbuild commented Dec 24, 2023 • edited Loading

cmsbuild commented Dec 24, 2023

Comparison Summary

cmsbuild commented Jan 8, 2024

smuzaffar commented Jan 8, 2024

cmsbuild commented Jan 8, 2024

Comparison Summary

cmsbuild commented Jan 11, 2024

smuzaffar commented Jan 15, 2024

smuzaffar commented Jan 15, 2024

cmsbuild commented Jan 15, 2024

Unit Tests

RelVals

smuzaffar commented Jan 16, 2024

cmsbuild commented Jan 16, 2024

Unit Tests

Comparison Summary

cmsbuild commented Dec 24, 2023 •

edited

Loading