Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

libunwind master with fix for uninitialized access #9639

Merged

Conversation

gartung
Copy link
Member

@gartung gartung commented Jan 28, 2025

Testing with ASAN build showed heap buffer overflow in libunwind. Fixed by libunwind master commit
libunwind/libunwind@e63e024

==3053803==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x63200dc35bdf at pc 0x7fea9fc73a71 bp 0x63200dc35bb0 sp 0x63200dc35370
WRITE of size 1 at 0x63200dc35bdf thread T0
    #0 0x7fea9fc73a70 in __interceptor_read ../../../../libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:1025
    #1 0x7fea9d3b38b7 in _write_validate mi/Gaddress_validator.c:118
    #2 0x7fea9d3b38b7 in _UIx86_64_address_is_valid mi/Gaddress_validator.c:291
    #3 0x7fea9d3b45f1 in access_mem x86_64/Ginit.c:90
    #4 0x7fea9d3b576a in is_plt_entry x86_64/Gstep.c:44
    #5 0x7fea9d3b576a in _ULx86_64_step x86_64/Gstep.c:142
    #6 0x7fea9d3b6344 in trace_init_addr x86_64/Gtrace.c:249
    #7 0x7fea9d3b6344 in trace_lookup x86_64/Gtrace.c:331
    #8 0x7fea9d3b6344 in _ULx86_64_tdep_trace x86_64/Gtrace.c:449
    #9 0x7fea9d3b350b in unw_backtrace mi/backtrace.c:70
    #10 0x7feaa04de6c7 in profileSignalHandler /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/igprof/5.9.16-885cac9f391726eb03a40073a814536b/igprof-16da627e12a806cd8ab072e7288223c91086ea25/src/profile-perf.cc:66
    #11 0x7fea9ddc85af  (/lib64/libc.so.6+0x4e5af)
    #12 0x7fe98e88e9f4  (<unknown module>)

0x63200dc35bdf is located 2531 bytes to the right of 84476-byte region [0x63200dc20800,0x63200dc351fc)
allocated by thread T0 here:
    #0 0x7fea9fce0838 in operator new[](unsigned long) ../../../../libsanitizer/asan/asan_new_delete.cpp:98
    #1 0x7fea9e625257 in TStorage::ReAllocChar(char*, unsigned long, unsigned long) (/cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_15_0_ASAN_X_2025-01-27-2300/external/el8_amd64_gcc12/lib/libCore.so+0x266257)

SUMMARY: AddressSanitizer: heap-buffer-overflow ../../../../libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:1025 in __interceptor_read
Shadow bytes around the buggy address:
  0x0c6481b7eb20: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c6481b7eb30: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c6481b7eb40: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c6481b7eb50: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c6481b7eb60: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0c6481b7eb70: fa fa fa fa fa fa fa fa fa fa fa[fa]fa fa fa fa
  0x0c6481b7eb80: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c6481b7eb90: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c6481b7eba0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c6481b7ebb0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c6481b7ebc0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==3053803==ABORTING

resolves cms-sw/framework-team#1204

@gartung
Copy link
Member Author

gartung commented Jan 28, 2025

please test

@cmsbuild
Copy link
Contributor

A new Pull Request was created by @gartung for branch IB/CMSSW_15_0_X/master.

@iarspider, @smuzaffar can you please review it and eventually sign? Thanks.
@antoniovilela, @mandrenguyen, @rappoccio, @sextonkennedy you are the release manager for this.
cms-bot commands are listed here

@cmsbuild
Copy link
Contributor

cmsbuild commented Jan 28, 2025

cms-bot internal usage

@cmsbuild
Copy link
Contributor

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-eaf0ee/44002/summary.html
COMMIT: 6cd9693
CMSSW: CMSSW_15_0_X_2025-01-28-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/9639/44002/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

  • You potentially removed 1 lines from the logs
  • Reco comparison results: 8 differences found in the comparisons
  • DQMHistoTests: Total files compared: 50
  • DQMHistoTests: Total histograms compared: 3994126
  • DQMHistoTests: Total failures: 48
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 3994058
  • DQMHistoTests: Total skipped: 20
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 7332.6720000000005 KiB( 49 files compared)
  • DQMHistoSizes: changed ( 10224.0,... ): 24.940 KiB Btag/TagInfo_pfParticleNetFromMiniAODAK4PuppiCentral_Inc
  • DQMHistoSizes: changed ( 10224.0,... ): 24.940 KiB Btag/TagInfo_pfParticleNetFromMiniAODAK4PuppiForward_Inc
  • DQMHistoSizes: changed ( 10224.0,... ): 24.790 KiB Btag/TagInfo_pfParticleNetFromMiniAODAK4PuppiCentral_B
  • DQMHistoSizes: changed ( 10224.0,... ): 24.790 KiB Btag/TagInfo_pfParticleNetFromMiniAODAK4PuppiCentral_C
  • DQMHistoSizes: changed ( 10224.0,... ): 24.790 KiB Btag/TagInfo_pfParticleNetFromMiniAODAK4PuppiCentral_L
  • DQMHistoSizes: changed ( 10224.0,... ): 23.873 KiB Btag/TagInfo_pfUnifiedParticleTransformerAK4_Inc
  • DQMHistoSizes: changed ( 10224.0,... ): 23.738 KiB Btag/TagInfo_pfUnifiedParticleTransformerAK4_B
  • DQMHistoSizes: changed ( 10224.0,... ): 23.738 KiB Btag/TagInfo_pfUnifiedParticleTransformerAK4_C
  • DQMHistoSizes: changed ( 10224.0,... ): 23.738 KiB Btag/TagInfo_pfUnifiedParticleTransformerAK4_L
  • DQMHistoSizes: changed ( 10224.0,... ): 11.008 KiB Btag/TagInfo_pfDeepFlavour_Inc
  • DQMHistoSizes: changed ( 10224.0 ): ...
  • Checked 218 log files, 189 edm output root files, 50 DQM output files
  • TriggerResults: no differences found

@cmsbuild
Copy link
Contributor

Pull request #9639 was updated.

@gartung gartung changed the title libunwind with --enable-debug libunwind master with fix for uninitialized access Jan 28, 2025
@gartung
Copy link
Member Author

gartung commented Jan 28, 2025

please test

@cmsbuild
Copy link
Contributor

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-eaf0ee/44008/summary.html
COMMIT: 420fff5
CMSSW: CMSSW_15_0_X_2025-01-28-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/9639/44008/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

  • You potentially added 16 lines to the logs
  • Reco comparison results: 5 differences found in the comparisons
  • DQMHistoTests: Total files compared: 50
  • DQMHistoTests: Total histograms compared: 4016770
  • DQMHistoTests: Total failures: 74
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 4016676
  • DQMHistoTests: Total skipped: 20
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
  • Checked 218 log files, 189 edm output root files, 50 DQM output files
  • TriggerResults: no differences found

@cmsbuild
Copy link
Contributor

Pull request #9639 was updated.

@gartung
Copy link
Member Author

gartung commented Jan 29, 2025

please test

@cmsbuild
Copy link
Contributor

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-eaf0ee/44029/summary.html
COMMIT: 501efd0
CMSSW: CMSSW_15_0_X_2025-01-29-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/9639/44029/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

  • You potentially added 3 lines to the logs
  • Reco comparison results: 10 differences found in the comparisons
  • DQMHistoTests: Total files compared: 50
  • DQMHistoTests: Total histograms compared: 4016770
  • DQMHistoTests: Total failures: 51
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 4016699
  • DQMHistoTests: Total skipped: 20
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
  • Checked 218 log files, 189 edm output root files, 50 DQM output files
  • TriggerResults: no differences found

@gartung
Copy link
Member Author

gartung commented Feb 4, 2025

please test with cms-sw/cmssw#47259, cms-sw/cmssw#47261 for CMSSW_15_0_ASAN_X/el8_amd64_gcc13

@gartung
Copy link
Member Author

gartung commented Feb 4, 2025

abort

@gartung
Copy link
Member Author

gartung commented Feb 4, 2025

please test

@cmsbuild
Copy link
Contributor

cmsbuild commented Feb 4, 2025

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-eaf0ee/44193/summary.html
COMMIT: 501efd0
CMSSW: CMSSW_15_0_X_2025-02-04-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/9639/44193/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-eaf0ee/44193/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-eaf0ee/44193/git-merge-result

Comparison Summary

Summary:

  • You potentially added 12 lines to the logs
  • ROOTFileChecks: Some differences in event products or their sizes found
  • Reco comparison results: 2979 differences found in the comparisons
  • DQMHistoTests: Total files compared: 50
  • DQMHistoTests: Total histograms compared: 4016938
  • DQMHistoTests: Total failures: 32660
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 3984258
  • DQMHistoTests: Total skipped: 20
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 2.838 KiB( 49 files compared)
  • DQMHistoSizes: changed ( 11634.0,... ): 0.129 KiB Physics/NanoAODDQM
  • Checked 218 log files, 189 edm output root files, 50 DQM output files
  • TriggerResults: found differences in 7 / 48 workflows

@gartung
Copy link
Member Author

gartung commented Feb 4, 2025

@smuzaffar this is ready to merge

@smuzaffar
Copy link
Contributor

+externals

@smuzaffar smuzaffar merged commit 5d3e8a0 into cms-sw:IB/CMSSW_15_0_X/master Feb 4, 2025
11 of 13 checks passed
@cmsbuild
Copy link
Contributor

cmsbuild commented Feb 4, 2025

This pull request is fully signed and it will be integrated in one of the next IB/CMSSW_15_0_X/master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @sextonkennedy, @antoniovilela, @rappoccio, @mandrenguyen (and backports should be raised in the release meeting by the corresponding L2)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

libunwind master with fix for uninitialized access
3 participants