[MkFit] Fix UBSAN error - skip processing of layers with no hits. #37146

osschar · 2022-03-04T19:28:35Z

When processing layers with no hits the LayerOfHits vector was in default state with storage pointer set to nullptr.

Loaders of hits for vector processing were still being initialized, resulting in doing pointer arithmetic with nullptr -- though they were never actually used.

This should fix #37136. Note that the wf there was 10004.0, single photon 10GeV and there were indeed several occurrences of empty mkfit hit containers.

cmsbuild · 2022-03-04T19:34:19Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-37146/28688

This PR adds an extra 24KB to repository

cmsbuild · 2022-03-04T19:34:38Z

A new Pull Request was created by @osschar (Matevž Tadel) for master.

It involves the following packages:

RecoTracker/MkFitCore (reconstruction)

@jpata, @cmsbuild, @clacaputo, @slava77 can you please review it and eventually sign? Thanks.
@makortel, @felicepantaleo, @GiacomoSguazzoni, @JanFSchulte, @rovere, @VinInn, @ebrondol, @gpetruc, @mmusich, @mtosi, @dgulhan this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

makortel · 2022-03-04T20:16:02Z

test parameters:

workflow = 10004.0

makortel · 2022-03-04T20:16:09Z

@cmsbuild, please test

mmusich · 2022-03-07T15:58:01Z

@smuzaffar this has been running tests for 3 days (!?).
Is there something stuck?

smuzaffar · 2022-03-07T18:04:40Z

@mmusich , UBSAN IB comparison does not work. The job timedout after 16 hours ( https://cmssdt.cern.ch/jenkins/job/compare-root-files-short-matrix/48727/ ) . Check the relvals for wf 10004.0 to see if USBAN issue is fixed or not?

I have restarted the comparison job now but I am afraid it will timeout again.

smuzaffar · 2022-03-07T18:05:01Z

please test

cmsbuild · 2022-03-07T21:51:22Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-cdca02/22911/summary.html
COMMIT: 81e7728
CMSSW: CMSSW_12_3_X_2022-03-07-1100/slc7_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/37146/22911/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

@slava77 comparisons for the following workflows were not done due to missing matrix map:

/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-cdca02/10004.0_SingleGammaPt10+2017+SingleGammaPt10_pythia8_GenSim+Digi+RecoFakeHLT+HARVESTFakeHLT+ALCA+Nano

Summary:

No significant changes to the logs found
Reco comparison results: 27 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3984447
DQMHistoTests: Total failures: 864
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3983561
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
Checked 204 log files, 45 edm output root files, 49 DQM output files
TriggerResults: no differences found

clacaputo · 2022-03-07T22:37:28Z

@mmusich , UBSAN IB comparison does not work. The job timedout after 16 hours ( https://cmssdt.cern.ch/jenkins/job/compare-root-files-short-matrix/48727/ ) . Check the relvals for wf 10004.0 to see if USBAN issue is fixed or not?

I have restarted the comparison job now but I am afraid it will timeout again.

Hi @smuzaffar, where can I check wf 10004.0 relvals for UBSAN?

smuzaffar · 2022-03-08T06:59:02Z

URL (i.e. https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-cdca02/22856) for cms/37146/UBSAN/slc7_amd64_gcc11 context points to UBSAN results. Go there and then click on the summary.html page to see the UBSAN results

cmsbuild · 2022-03-08T07:07:01Z

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-cdca02/22856/summary.html
COMMIT: 81e7728
CMSSW: CMSSW_12_3_UBSAN_X_2022-03-04-1100/slc7_amd64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/37146/22856/install.sh to create a dev area with all the needed externals and cmssw changes.

smuzaffar · 2022-03-08T07:07:22Z

ok, I have forced set the status for ubsan comparison check, but should add the comment for ubsan now

clacaputo · 2022-03-08T10:25:16Z

If I check the output in 10004.0 step3 log, the runtime error is still there

/cvmfs/cms-ib.cern.ch/nweek-02722/slc7_amd64_gcc11/external/gcc/11.2.1-f478fee2760dbd22aaabb4e3a8fe1640/include/c++/11.2.1/bits/stl_vector.h:1046:34: runtime error: reference binding to null pointer of type 'value_type'
    #0 0x2abc3b477a03  (/cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/37146/22856/CMSSW_12_3_UBSAN_X_2022-03-04-1100/lib/slc7_amd64_gcc11/libRecoTrackerMkFitCore.so+0x730a03)
    #1 0x2abc3b49126b  (/cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/37146/22856/CMSSW_12_3_UBSAN_X_2022-03-04-1100/lib/slc7_amd64_gcc11/libRecoTrackerMkFitCore.so+0x74a26b)
    #2 0x2abc3ad1aefc  (/cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc11/cms/cmssw/CMSSW_12_3_UBSAN_X_2022-03-04-1100/lib/slc7_amd64_gcc11/libRecoTrackerMkFitCMS.so+0x50efc)
    #3 0x2abc3a82fd8e  (/cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc11/cms/cmssw/CMSSW_12_3_UBSAN_X_2022-03-04-1100/lib/slc7_amd64_gcc11/pluginRecoTrackerMkFitPlugins.so+0x65cd8e)

mmusich · 2022-03-08T12:41:10Z

If I check the output in 10004.0 step3 log, the runtime error is still there

I observe the same locally:

cmsrel CMSSW_12_3_UBSAN_X_2022-03-04-1100
cd CMSSW_12_3_UBSAN_X_2022-03-04-1100/src/
cmsenv
git cms-merge-topic 37146
scramv1 b -j 20
runTheMatrix.py -l 10004.0 -t 4 -j 8
more 10004.0_SingleGammaPt10+2017+SingleGammaPt10_pythia8_GenSim+Digi+RecoFakeHLT+HARVESTFakeHLT+ALCA+Nano/step3_SingleGammaPt10+2017+SingleGammaPt10_pythia8_GenSim+Digi+RecoFakeHLT+HARVESTFakeHLT+ALCA+Nano.log | grep libRecoTrackerMkFitCore | wc -l 
2

osschar · 2022-03-08T16:25:21Z

Oh rats. How does one setup the build environment for this?
Reading gcc man page ... is USER_CXXFLAGS="-fsanitize=undefined -fsanitize=null" sufficient?

…eeds Shortcut MkFitProducer to produce an empty track vector when input seed vector is empty. In MkBuilder::import_seeds() assert that the input vector is not empty.

cmsbuild · 2022-03-09T22:54:23Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-37146/28767

This PR adds an extra 44KB to repository

cmsbuild · 2022-03-09T22:54:49Z

Pull request #37146 was updated. @jpata, @cmsbuild, @clacaputo, @slava77 can you please check and sign again.

slava77 · 2022-03-10T00:02:43Z

@cmsbuild please test

cmsbuild · 2022-03-10T04:19:17Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-cdca02/22994/summary.html
COMMIT: b548ebf
CMSSW: CMSSW_12_3_X_2022-03-09-1100/slc7_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/37146/22994/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

@slava77 comparisons for the following workflows were not done due to missing matrix map:

/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-cdca02/10004.0_SingleGammaPt10+2017+SingleGammaPt10_pythia8_GenSim+Digi+RecoFakeHLT+HARVESTFakeHLT+ALCA+Nano

Summary:

No significant changes to the logs found
Reco comparison results: 6 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 3806002
DQMHistoTests: Total failures: 15
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 3805964
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.004 KiB( 49 files compared)
DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
Checked 210 log files, 45 edm output root files, 50 DQM output files
TriggerResults: no differences found

clacaputo · 2022-03-10T11:24:48Z

+reconstruction

technical PR, fix for UBSAN
checked locally

cmsrel CMSSW_12_3_UBSAN_X_2022-03-04-1100
cd CMSSW_12_3_UBSAN_X_2022-03-04-1100/src/
cmsenv
git cms-merge-topic 37146
scramv1 b -j 20
runTheMatrix.py -l 10004.0 -t 4 -j 8
more 10004.0_SingleGammaPt10+2017+SingleGammaPt10_pythia8_GenSim+Digi+RecoFakeHLT+HARVESTFakeHLT+ALCA+Nano/step3_SingleGammaPt10+2017+SingleGammaPt10_pythia8_GenSim+Digi+RecoFakeHLT+HARVESTFakeHLT+ALCA+Nano.log | grep libRecoTrackerMkFitCore | wc -l 
0

no reco changes expected or observed after b548ebf

cmsbuild · 2022-03-10T11:25:06Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

perrotta · 2022-03-11T16:16:29Z

+1

slava77 · 2022-03-15T16:05:29Z

@cms-sw/core-l2 @perrotta @qliphy
do you need a backport to 12_3_X?

makortel · 2022-03-15T16:17:14Z

I suppose it would be useful (to minimize UB in 12_3).

…-1100_mkFit_X [MkFit] Fix UBSAN error in (almost) empty events (backport of #37146)

[MkFit] Fix UBSAN error - skip processing of layers with no hits.

81e7728

cmsbuild added this to the CMSSW_12_3_X milestone Mar 4, 2022

cmsbuild added code-checks-pending orp-pending pending-signatures reconstruction-pending tests-pending labels Mar 4, 2022

cmsbuild added code-checks-approved and removed code-checks-pending labels Mar 4, 2022

cmsbuild added tests-started and removed tests-pending labels Mar 4, 2022

cmsbuild added tests-approved and removed tests-started labels Mar 7, 2022

[MkFit] Fix UBSAN error II. - skip processing of iterations with no s…

1d83b15

…eeds Shortcut MkFitProducer to produce an empty track vector when input seed vector is empty. In MkBuilder::import_seeds() assert that the input vector is not empty.

cmsbuild removed the tests-approved label Mar 8, 2022

cmsbuild added code-checks-pending tests-pending and removed code-checks-approved labels Mar 9, 2022

cmsbuild added code-checks-approved and removed code-checks-pending labels Mar 9, 2022

cmsbuild added tests-started and removed tests-pending labels Mar 10, 2022

cmsbuild added tests-approved and removed tests-started labels Mar 10, 2022

cmsbuild added fully-signed reconstruction-approved and removed reconstruction-pending pending-signatures labels Mar 10, 2022

smuzaffar modified the milestones: CMSSW_12_3_X, CMSSW_12_4_X Mar 11, 2022

cmsbuild added orp-approved and removed orp-pending labels Mar 11, 2022

cmsbuild merged commit 3faf37c into cms-sw:master Mar 11, 2022

cmsbuild mentioned this pull request Mar 11, 2022

introduce alignment campaign-like unit test for pede #37211

Merged

slava77 mentioned this pull request Mar 15, 2022

[MkFit] Fix UBSAN error in (almost) empty events (backport of #37146) #37254

Merged

cmsbuild added a commit that referenced this pull request Mar 19, 2022

Merge pull request #37254 from trackreco/from-CMSSW_12_3_X_2022-02-28…

462cc82

…-1100_mkFit_X [MkFit] Fix UBSAN error in (almost) empty events (backport of #37146)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[MkFit] Fix UBSAN error - skip processing of layers with no hits. #37146

[MkFit] Fix UBSAN error - skip processing of layers with no hits. #37146

osschar commented Mar 4, 2022

cmsbuild commented Mar 4, 2022

cmsbuild commented Mar 4, 2022

makortel commented Mar 4, 2022 •

edited by smuzaffar

Loading

makortel commented Mar 4, 2022

mmusich commented Mar 7, 2022

smuzaffar commented Mar 7, 2022 •

edited

Loading

smuzaffar commented Mar 7, 2022

cmsbuild commented Mar 7, 2022

clacaputo commented Mar 7, 2022

smuzaffar commented Mar 8, 2022

cmsbuild commented Mar 8, 2022

smuzaffar commented Mar 8, 2022

clacaputo commented Mar 8, 2022

mmusich commented Mar 8, 2022

osschar commented Mar 8, 2022

cmsbuild commented Mar 9, 2022

cmsbuild commented Mar 9, 2022

slava77 commented Mar 10, 2022

cmsbuild commented Mar 10, 2022

clacaputo commented Mar 10, 2022

cmsbuild commented Mar 10, 2022

perrotta commented Mar 11, 2022

slava77 commented Mar 15, 2022

makortel commented Mar 15, 2022

[MkFit] Fix UBSAN error - skip processing of layers with no hits. #37146

[MkFit] Fix UBSAN error - skip processing of layers with no hits. #37146

Conversation

osschar commented Mar 4, 2022

cmsbuild commented Mar 4, 2022

cmsbuild commented Mar 4, 2022

makortel commented Mar 4, 2022 • edited by smuzaffar Loading

makortel commented Mar 4, 2022

mmusich commented Mar 7, 2022

smuzaffar commented Mar 7, 2022 • edited Loading

smuzaffar commented Mar 7, 2022

cmsbuild commented Mar 7, 2022

Comparison Summary

clacaputo commented Mar 7, 2022

smuzaffar commented Mar 8, 2022

cmsbuild commented Mar 8, 2022

smuzaffar commented Mar 8, 2022

clacaputo commented Mar 8, 2022

mmusich commented Mar 8, 2022

osschar commented Mar 8, 2022

cmsbuild commented Mar 9, 2022

cmsbuild commented Mar 9, 2022

slava77 commented Mar 10, 2022

cmsbuild commented Mar 10, 2022

Comparison Summary

clacaputo commented Mar 10, 2022

cmsbuild commented Mar 10, 2022

perrotta commented Mar 11, 2022

slava77 commented Mar 15, 2022

makortel commented Mar 15, 2022

makortel commented Mar 4, 2022 •

edited by smuzaffar

Loading

smuzaffar commented Mar 7, 2022 •

edited

Loading