Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

root fwlite interactive plot for PackedCandidates segfaults #42234

Closed
slava77 opened this issue Jul 11, 2023 · 34 comments
Closed

root fwlite interactive plot for PackedCandidates segfaults #42234

slava77 opened this issue Jul 11, 2023 · 34 comments

Comments

@slava77
Copy link
Contributor

slava77 commented Jul 11, 2023

this was initially noticed in CMSSW_13_0_9, but is also reproduced in CMSSW_13_2_0_pre2

local version on cmsdev34:

FWLiteEnabler::enable();
e = new TChain("Events");
e->Add("/build/slava77/RawReference/Muon0/Run2023D-PromptReco-v1/MINIAOD/11189a7c-8a53-4ecf-b1bd-2351a0d1e91b.root");
e->SetAlias("p", "patPackedCandidates_packedPFCandidates__RECO.obj");
e->Draw("p.phi():p.eta()", "p.fromPV()>0&&p.hasTrackDetails()&&p.pt()>1", "", 10);

crashes

adding more or less variables or reading a remote copy of the file root://cms-xrd-global.cern.ch//store/data/Run2023D/Muon0/MINIAOD/PromptReco-v1/000/369/956/00000/11189a7c-8a53-4ecf-b1bd-2351a0d1e91b.root sometimes succeeds

There are clang/cling warnings/errors preceding the crash, but some of them are present also in non-crashing variant.
The crashing variant seems to most consistently have this right before the crash

In file included from TrackingToolsPatternTools_xr dictionary payload:86:
/cvmfs/cms.cern.ch/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_0_9/src/TrackingTools/PatternTools/interface/TrackConstraintAssociation.h:19:45: error: typedef redefinition with different types ('pair<...>' vs 'pair<...>')
typedef std::pair<GlobalPoint, GlobalError> VertexConstraint;
                                            ^
TrackingToolsPatternTools_xr dictionary forward declarations' payload:30:45: note: previous definition is here
typedef std::pair<GlobalPoint, GlobalError> VertexConstraint __attribute__((annotate("$clingAutoload$TrackingTools/PatternTools/interface/TrackConstraintAssociation.h"))) ;
                                            ^

 *** Break *** segmentation violation

The stack trace follows:

===========================================================
There was a crash.
This is the entire stack trace of all threads:
===========================================================
#0  0x00007f4e4df9e60c in waitpid () from /lib64/libc.so.6
#1  0x00007f4e4df1bf62 in do_system () from /lib64/libc.so.6
#2  0x00007f4e4ecc605c in TUnixSystem::StackTrace() () from /cvmfs/cms.cern.ch/slc7_amd64_gcc11/lcg/root/6.26.11-fec5b250e1cd56b91a094709de26e5b8/bin/../lib/libCore.so
#3  0x00007f4e4ecc3755 in TUnixSystem::DispatchSignals(ESignals) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc11/lcg/root/6.26.11-fec5b250e1cd56b91a094709de26e5b8/bin/../lib/libCore.so
#4  <signal handler called>
#5  0x00007f4e4b7feeec in clang::CXXRecordDecl::getLambdaCallOperator() const () from /cvmfs/cms.cern.ch/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_0_9/external/slc7_amd64_gcc11/lib/libCling.so
#6  0x00007f4e4935be18 in clang::RecursiveASTVisitor<cling::(anonymous namespace)::StaticVarCollector>::TraverseLambdaExpr(clang::LambdaExpr*, llvm::SmallVectorImpl<llvm::PointerIntPair<clang::Stmt*, 1u, bool, llvm::PointerLikeTypeTraits<clang::Stmt*>, llvm::PointerIntPairInfo<clang::Stmt*, 1u, llvm::PointerLikeTypeTraits<clang::Stmt*> > > >*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_0_9/external/slc7_amd64_gcc11/lib/libCling.so
#7  0x00007f4e493582c5 in clang::RecursiveASTVisitor<cling::(anonymous namespace)::StaticVarCollector>::TraverseStmt(clang::Stmt*, llvm::SmallVectorImpl<llvm::PointerIntPair<clang::Stmt*, 1u, bool, llvm::PointerLikeTypeTraits<clang::Stmt*>, llvm::PointerIntPairInfo<clang::Stmt*, 1u, llvm::PointerLikeTypeTraits<clang::Stmt*> > > >*) [clone .constprop.0] () from /cvmfs/cms.cern.ch/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_0_9/external/slc7_amd64_gcc11/lib/libCling.so
#8  0x00007f4e49354415 in clang::RecursiveASTVisitor<cling::(anonymous namespace)::StaticVarCollector>::TraverseDecl(clang::Decl*) [clone .part.0] () from /cvmfs/cms.cern.ch/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_0_9/external/slc7_amd64_gcc11/lib/libCling.so
#9  0x00007f4e49354cce in clang::RecursiveASTVisitor<cling::(anonymous namespace)::StaticVarCollector>::TraverseDeclStmt(clang::DeclStmt*, llvm::SmallVectorImpl<llvm::PointerIntPair<clang::Stmt*, 1u, bool, llvm::PointerLikeTypeTraits<clang::Stmt*>, llvm::PointerIntPairInfo<clang::Stmt*, 1u, llvm::PointerLikeTypeTraits<clang::Stmt*> > > >*) [clone .constprop.0] () from /cvmfs/cms.cern.ch/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_0_9/external/slc7_amd64_gcc11/lib/libCling.so
#10 0x00007f4e493582c5 in clang::RecursiveASTVisitor<cling::(anonymous namespace)::StaticVarCollector>::TraverseStmt(clang::Stmt*, llvm::SmallVectorImpl<llvm::PointerIntPair<clang::Stmt*, 1u, bool, llvm::PointerLikeTypeTraits<clang::Stmt*>, llvm::PointerIntPairInfo<clang::Stmt*, 1u, llvm::PointerLikeTypeTraits<clang::Stmt*> > > >*) [clone .constprop.0] () from /cvmfs/cms.cern.ch/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_0_9/external/slc7_amd64_gcc11/lib/libCling.so
#11 0x00007f4e4935859b in cling::DeclUnloader::VisitFunctionDecl(clang::FunctionDecl*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_0_9/external/slc7_amd64_gcc11/lib/libCling.so
#12 0x00007f4e4932ac74 in cling::TransactionUnloader::unloadDeserializedDeclarations(cling::Transaction*, cling::DeclUnloader&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_0_9/external/slc7_amd64_gcc11/lib/libCling.so
#13 0x00007f4e4932b14e in cling::TransactionUnloader::RevertTransaction(cling::Transaction*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_0_9/external/slc7_amd64_gcc11/lib/libCling.so
#14 0x00007f4e4931079e in cling::Interpreter::unload(cling::Transaction&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_0_9/external/slc7_amd64_gcc11/lib/libCling.so
#15 0x00007f4e4939203f in cling::IncrementalParser::commitTransaction(llvm::PointerIntPair<cling::Transaction*, 2u, cling::IncrementalParser::EParseResult, llvm::PointerLikeTypeTraits<cling::Transaction*>, llvm::PointerIntPairInfo<cling::Transaction*, 2u, llvm::PointerLikeTypeTraits<cling::Transaction*> > >&, bool) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_0_9/external/slc7_amd64_gcc11/lib/libCling.so
#16 0x00007f4e493933a9 in cling::IncrementalParser::Compile(llvm::StringRef, cling::CompilationOptions const&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_0_9/external/slc7_amd64_gcc11/lib/libCling.so
#17 0x00007f4e4930d93b in cling::Interpreter::parseForModule(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_0_9/external/slc7_amd64_gcc11/lib/libCling.so
#18 0x00007f4e49242b36 in ExecAutoParse(char const*, bool, cling::Interpreter*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_0_9/external/slc7_amd64_gcc11/lib/libCling.so
#19 0x00007f4e49250d96 in TCling::AutoParseImplRecurse(char const*, bool) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_0_9/external/slc7_amd64_gcc11/lib/libCling.so
#20 0x00007f4e49250bc5 in TCling::AutoParseImplRecurse(char const*, bool) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_0_9/external/slc7_amd64_gcc11/lib/libCling.so
#21 0x00007f4e49255b35 in TCling::AutoParse(char const*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_0_9/external/slc7_amd64_gcc11/lib/libCling.so
#22 0x00007f4e49231a0a in TClingLookupHelper__AutoParse(char const*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_0_9/external/slc7_amd64_gcc11/lib/libCling.so
#23 0x00007f4e49198a17 in ROOT::TMetaUtils::TClingLookupHelper::GetPartiallyDesugaredNameWithScopeHandling(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, bool) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_0_9/external/slc7_amd64_gcc11/lib/libCling.so
#24 0x00007f4e4ec4c6ec in TClassEdit::TSplitType::ShortType(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, int) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc11/lcg/root/6.26.11-fec5b250e1cd56b91a094709de26e5b8/bin/../lib/libCore.so
#25 0x00007f4e4ec4d4a1 in TClassEdit::GetNormalizedName(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::basic_string_view<char, std::char_traits<char> >) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc11/lcg/root/6.26.11-fec5b250e1cd56b91a094709de26e5b8/bin/../lib/libCore.so
#26 0x00007f4e4ec68f3c in TClass::GetClass(char const*, bool, bool, unsigned long, unsigned long) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc11/lcg/root/6.26.11-fec5b250e1cd56b91a094709de26e5b8/bin/../lib/libCore.so
#27 0x00007f4e45817799 in edm::TypeWithDict::byName(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, long) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_0_9/lib/slc7_amd64_gcc11/libFWCoreReflection.so
#28 0x00007f4e45818045 in edm::TypeWithDict::byName(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_0_9/lib/slc7_amd64_gcc11/libFWCoreReflection.so
#29 0x00007f4e334d8dd6 in edm::BranchDescription::initFromDictionary() () from /cvmfs/cms.cern.ch/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_0_9/lib/slc7_amd64_gcc11/libDataFormatsProvenance.so
#30 0x00007f4e471890c1 in fwlite::internal::BranchMapReaderStrategyV17::updateFile(TFile*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_0_9/lib/slc7_amd64_gcc11/libFWCoreFWLite.so
#31 0x00007f4e4718956d in fwlite::internal::BranchMapReaderStrategyV17::BranchMapReaderStrategyV17(TFile*, int) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_0_9/lib/slc7_amd64_gcc11/libFWCoreFWLite.so
#32 0x00007f4e47189753 in fwlite::BranchMapReader::newStrategy(TFile*, int) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_0_9/lib/slc7_amd64_gcc11/libFWCoreFWLite.so
#33 0x00007f4e471898d5 in fwlite::BranchMapReader::updateFile(TFile*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_0_9/lib/slc7_amd64_gcc11/libFWCoreFWLite.so
#34 0x00007f4e47189a22 in BareRootProductGetter::getIt(edm::ProductID const&) const () from /cvmfs/cms.cern.ch/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_0_9/lib/slc7_amd64_gcc11/libFWCoreFWLite.so
#35 0x00007f4e3357dfc9 in edm::RefCore::tryToGetProductPtr(std::type_info const&, edm::EDProductGetter const*) const () from /cvmfs/cms.cern.ch/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_0_9/lib/slc7_amd64_gcc11/libDataFormatsCommon.so

I know that historically calling the TTree::Draw on PackedCandidates data that needs unpacking was pretty brittle, but perhaps there can be a resolution.

@cmsbuild
Copy link
Contributor

A new Issue was created by @slava77 Slava Krutelyov.

@Dr15Jones, @perrotta, @dpiparo, @rappoccio, @makortel, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

@Dr15Jones
Copy link
Contributor

@slava77 could you perhaps also run valgrind on it?

@Dr15Jones
Copy link
Contributor

assign core

@cmsbuild
Copy link
Contributor

New categories assigned: core

@Dr15Jones,@smuzaffar,@makortel you have been requested to review this Pull request/Issue and eventually sign? Thanks

@slava77
Copy link
Contributor Author

slava77 commented Jul 11, 2023

running valgrind --tool=memcheck --suppressions=$CMSSW_RELEASE_BASE/src/Utilities/ReleaseScripts/data/cms-valgrind-memcheck.supp --num-callers=20 --xml=yes --xml-file=valgrind-%p.xml --malloc-fill=0xa5 --free-fill=0x5a root.exe

output is available on cmsdev34:/build/slava77/reltest/CMSSW_13_0_9-orig/issue42234/valgrind-29364
The main output is valgrind-29364.xml

ignoring the memleak reports there are a couple of pointers to cmssw libs with UninitCondition (looking at a few the last call is TStorage::UpdateIsOnHeap, not sure if it's some fake positive)

grep -B 546535 FINISHED valgrind-29364.xml | grep cms/cmssw | sort | uniq -c
      1       <obj>/cvmfs/cms.cern.ch/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_0_9/lib/slc7_amd64_gcc11/libCondFormatsL1TObjects.so</obj>
      1       <obj>/cvmfs/cms.cern.ch/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_0_9/lib/slc7_amd64_gcc11/libDataFormatsCommon.so</obj>
     32       <obj>/cvmfs/cms.cern.ch/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_0_9/lib/slc7_amd64_gcc11/libDataFormatsProvenance.so</obj>
      3       <obj>/cvmfs/cms.cern.ch/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_0_9/lib/slc7_amd64_gcc11/libDataFormatsStdDictionaries.so</obj>

There are also a couple of InvalidRead; the second is more likely the crash itself with the last call clang::CXXRecordDecl::getLambdaCallOperator() reading at 0x0

@slava77
Copy link
Contributor Author

slava77 commented Jul 17, 2023

@Dr15Jones
I'm not sure if the valgrind report was useful to you already.
Is there anything else to check or is someone in the Core group already working on this?

@wddgit
Copy link
Contributor

wddgit commented Jul 17, 2023

I am looking at this now. Started about an hour ago. I don't see anything obviously wrong yet, but I'm still looking. As far as I know, no one was working on this problem. Thanks for reporting it.

@wddgit
Copy link
Contributor

wddgit commented Jul 17, 2023

I'm not sure what I am doing that is different, but I didn't see a seg fault on cmsdev34. For me it ran to the end and I see a plot displayed. I did see multiple cling/clang warnings/errors. I do not know if those are serious issues we should be worried about.

@Dr15Jones Do you want me to keep looking tomorrow? Any suggestions at what to look at? I'm done for today.

@dan131riley
Copy link

I managed to reproduce it (on a local system at Cornell), so I tried in a ROOT6 master IB and got something that might be informative to a cling expert. I'll poke at it some more tomorrow:

Processing draw_crash.C...
Info in <TCanvas::MakeDefCanvas>:  created default TCanvas with name c1
CondFormatsL1TObjects_xr dictionary forward declarations' payload:8:192: error: enumeration previously declared with nonfixed underlying type
enum  __attribute__((annotate("$clingAutoload$DataFormats/L1GlobalTrigger/interface/L1GtObject.h")))  __attribute__((annotate("$clingAutoload$CondFormats/L1TObjects/interface/L1GtBoard.h"))) L1GtObject : unsigned int;
                                                                                                                                                                                               ^
/cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_2_ROOT6_X_2023-07-17-2300/src/DataFormats/L1GlobalTrigger/interface/L1GtObject.h:28:6: note: previous declaration is here
enum L1GtObject {
     ^
root.exe: /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-e50d353bab9bcfd48d79a72b6af74b45/root-6.29.01/core/metacling/src/TCling.cxx:2198: virtual void TCling::RegisterModule(const char*, const char**, const char**, const char*, const char*, void (*)(), const FwdDeclArgsToKeepCollection_t&, const char**, Bool_t, Bool_t): Assertion `cling::Interpreter::kSuccess == compRes && "The forward declarations could not be compiled"' failed.

@slava77
Copy link
Contributor Author

slava77 commented Jul 18, 2023

I managed to reproduce it (on a local system at Cornell), so I tried in a ROOT6 master IB and got something that might be informative to a cling expert. I'll poke at it some more tomorrow:

Processing draw_crash.C...
Info in <TCanvas::MakeDefCanvas>:  created default TCanvas with name c1
CondFormatsL1TObjects_xr dictionary forward declarations' payload:8:192: error: enumeration previously declared with nonfixed underlying type

...

Thank you Dan,
I noticed the L1TObjects error as well, but it shows up both in working an not working cases. So, I guess it's somewhat harmless.

But I suppose the other part root-6.29.01/core/metacling/src/TCling.cxx:2198: virtual void TCling::RegisterModule ... is more relevant.

@dan131riley
Copy link

Thank you Dan, I noticed the L1TObjects error as well, but it shows up both in working an not working cases. So, I guess it's somewhat harmless.

But I suppose the other part root-6.29.01/core/metacling/src/TCling.cxx:2198: virtual void TCling::RegisterModule ... is more relevant.

My suspicion is that the L1TObjects error is where the trouble starts. In 6.26 it sometimes leads to a crash (seg faults can be finicky), but in ROOT master it triggers an assertion, so I guess someone felt it was a serious error. To test, I made the change to make L1GtObject explicitly unsigned int:

--- a/DataFormats/L1GlobalTrigger/interface/L1GtObject.h
+++ b/DataFormats/L1GlobalTrigger/interface/L1GtObject.h
@@ -25,7 +25,7 @@

 /// L1 GT objects
 ///    ObjNull catch all errors
-enum L1GtObject {
+enum L1GtObject : unsigned int {
   Mu,
   NoIsoEG,
   IsoEG,

and both the crashes in 6.26 and the assertion failure in master went away. This seems like a bug in ROOT?

@wddgit
Copy link
Contributor

wddgit commented Jul 18, 2023

This is a question because I don't understand how this works. Is it relevant that the enum is defined in a DataFormats package and the usage where the unsigned int type is specified is in a CondFormats package that uses boost serialization?

@dan131riley
Copy link

This is a question because I don't understand how this works.

Me neither. It looks like when cling processes the CondFormats package it inferred the type for L1GtObject : unsigned int as a forward declaration--it needs a size since the type gets stored--and then got unhappy when the actual declaration for the enum didn't include the type specifier. I find this confusing since cling inferred the type in the first place, since you can't forward declare an enum without a storage type specification. I don't see any explicit forward declaration in CondFormats, and I doubt boost serialization is relevant.

@wddgit
Copy link
Contributor

wddgit commented Jul 18, 2023

Maybe not exactly the same issue, but similar from 2018.

https://sft.its.cern.ch/jira/si/jira.issueviews:issue-html/ROOT-9660/ROOT-9660.html

@dan131riley
Copy link

root-project/root#5059 looks like it got lost when LLVM was upgraded, as I don't see that code in the current tree.

@wddgit
Copy link
Contributor

wddgit commented Jul 18, 2023

Just to confirm what we already knew, I can see in my working area the enum declaration modified to explicitly specify the type in _xr.cc file generated for the dictionary:

Singularity> grep enum tmp/el8_amd64_gcc11/src/CondFormats/L1TObjects/src/CondFormatsL1TObjects/a/CondFormatsL1TObjects_xr.cc
enum  __attribute__((annotate("$clingAutoload$DataFormats/L1GlobalTrigger/interface/L1GtObject.h")))  __attribute__((annotate("$clingAutoload$CondFormats/L1TObjects/interface/L1GtBoard.h"))) L1GtObject : unsigned int;

@dan131riley
Copy link

The workaround is harmless, so might as well implement it, but would be good to get this fixed in ROOT (again?). @vgvassilev @pcanal

@pcanal
Copy link
Contributor

pcanal commented Jul 18, 2023

Also @hahnjo

@vgvassilev
Copy link
Contributor

root-project/root#5059 looks like it got lost when LLVM was upgraded, as I don't see that code in the current tree.

Nice catch! That’s probably it. I know @hahnjo was making some patch cleanup and we might hit a case where we dropped the patch as we don’t have a test case. Now we do.

However, on a broader perspective that’s one of the reasons we want C++ modules in cmssw.

Cc: @davidlange6

@wddgit
Copy link
Contributor

wddgit commented Jul 21, 2023

@vgvassilev @pcanal Is there a plan for getting this fixed? Do we need to do any more from our end?

@vgvassilev
Copy link
Contributor

@hahnjo ping..

hahnjo added a commit to hahnjo/roottest that referenced this issue Jul 25, 2023
This is a heavily reduced case that shows the problem originally
reported in cms-sw/cmssw#42234.
hahnjo added a commit to hahnjo/root that referenced this issue Jul 25, 2023
There can be multiple attributes in the forward declaration, see the
added test in roottest/cling/dict/enum (reduced from a case reported
by CMS in cms-sw/cmssw#42234), so we have
to look for the last closing parentheses.
hahnjo added a commit to root-project/roottest that referenced this issue Jul 26, 2023
This is a heavily reduced case that shows the problem originally
reported in cms-sw/cmssw#42234.
hahnjo added a commit to hahnjo/roottest that referenced this issue Jul 26, 2023
This is a heavily reduced case that shows the problem originally
reported in cms-sw/cmssw#42234.

(cherry picked from commit 09f72c3)
hahnjo added a commit to hahnjo/root that referenced this issue Jul 26, 2023
There can be multiple attributes in the forward declaration, see the
added test in roottest/cling/dict/enum (reduced from a case reported
by CMS in cms-sw/cmssw#42234), so we have
to look for the last closing parentheses.

(cherry picked from commit 9d2f761)
hahnjo added a commit to hahnjo/roottest that referenced this issue Jul 26, 2023
This is a heavily reduced case that shows the problem originally
reported in cms-sw/cmssw#42234.

(cherry picked from commit 09f72c3)
hahnjo added a commit to hahnjo/root that referenced this issue Jul 26, 2023
There can be multiple attributes in the forward declaration, see the
added test in roottest/cling/dict/enum (reduced from a case reported
by CMS in cms-sw/cmssw#42234), so we have
to look for the last closing parentheses.

(cherry picked from commit 9d2f761)
@hahnjo
Copy link
Contributor

hahnjo commented Jul 26, 2023

Ok, backport PRs for 6.28 and 6.28 are up: root-project/root#13326 root-project/root#13327

@smuzaffar can I run this through CMS testing by opening a PR to https://github.com/cms-sw/root/ ?

@smuzaffar
Copy link
Contributor

@hahnjo , yes PR tests can be run by using the correct cms/v6.2x.... branches. @aandvalenzuela , can you please open cms-sw/root PRs using root-project/root#13326 and root-project/root#13327 for root6.26 and root 6.28 branches of 13.3.X release cycles

@smuzaffar
Copy link
Contributor

smuzaffar commented Jul 27, 2023

I can reproduce the crash in 13.2.0.pre2 [a] and though 13.2.0.pre3 does not crash but does dump some extra messages [b]. The only difference I see is that 13.2.0.pre3 is using GCC 11.4 as compare to GCC 11.2 for 13.2.0.pre2.

I can not reproduce this issue for CMSSW_13_3 release cycle [c] (which is also using GCC 11.4). Note that root commit used for 13.2.0.pre2 and 13.3.X IBs is exactly the same. So could it be that this has already been fixed via some cmssw PR and do we need the root-project/root#13326 root-project/root#13327 changes?

[a] CMSSW_13_2_0_pre2 (with root 6.26 and GCC 11.2.1)

Singularity> root                    
   ------------------------------------------------------------------
  | Welcome to ROOT 6.26/11                        https://root.cern |
  | (c) 1995-2021, The ROOT Team; conception: R. Brun, F. Rademakers |
  | Built for linuxx8664gcc on Jun 15 2023, 17:01:00                 |
  | From tag , 16 November 2022                                      |
  | With g++ (GCC) 11.2.1 20210817                                   |
  | Try '.help', '.demo', '.license', '.credits', '.quit'/'.q'       |
   ------------------------------------------------------------------

root [0] FWLiteEnabler::enable();
root [1] e = new TChain("Events");
root [2] e->Add("/build/muz/test.root");
root [3] e->SetAlias("p", "patPackedCandidates_packedPFCandidates__RECO.obj");
root [4] e->Draw("p.phi():p.eta()", "p.fromPV()>0&&p.hasTrackDetails()&&p.pt()>1", "", 10);
Info in <TCanvas::MakeDefCanvas>:  created default TCanvas with name c1
CondFormatsL1TObjects_xr dictionary forward declarations' payload:8:192: error: enumeration previously declared with nonfixed underlying type
enum  __attribute__((annotate("$clingAutoload$DataFormats/L1GlobalTrigger/interface/L1GtObject.h")))  __attribute__((annotate("$clingAutoload$CondFormats/L1TObjects/interface/L1GtBoard.h"))) L1GtObject : unsigned int;
                                                                                                                                                                                               ^
/cvmfs/cms.cern.ch/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_0_pre2/src/DataFormats/L1GlobalTrigger/interface/L1GtObject.h:28:6: note: previous declaration is here
enum L1GtObject {
     ^
Warning in <TInterpreter::TCling::RegisterModule>: Problems in compiling forward declarations for module CondFormatsL1TObjects_xr: '
#line 1 "CondFormatsL1TObjects_xr dictionary forward declarations' payload"
#pragma clang diagnostic ignored "-Wkeyword-compat"
#pragma clang diagnostic ignored "-Wignored-attributes"
#pragma clang diagnostic ignored "-Wreturn-type-c-linkage"
extern int __Cling_AutoLoading_Map;
class __attribute__((annotate("$clingAutoload$CondFormats/L1TObjects/interface/L1MuDTEtaPattern.h")))  __attribute__((annotate("$clingAutoload$CondFormats/L1TObjects/interface/L1MuDTEtaPatternLut.h")))  L1MuDTEtaPattern;
namespace std{template <typename _T1, typename _T2> struct __attribute__((annotate("$clingAutoload$bits/stl_iterator.h")))  __attribute__((annotate("$clingAutoload$string")))  pair;
...
....
In file included from TrackingToolsPatternTools_xr dictionary payload:86:
/cvmfs/cms.cern.ch/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_0_pre2/src/TrackingTools/PatternTools/interface/TrackConstraintAssociation.h:19:45: error: typedef redefinition with different types ('pair<...>' vs 'pair<...>')
typedef std::pair<GlobalPoint, GlobalError> VertexConstraint;
                                            ^
TrackingToolsPatternTools_xr dictionary forward declarations' payload:30:45: note: previous definition is here
typedef std::pair<GlobalPoint, GlobalError> VertexConstraint __attribute__((annotate("$clingAutoload$TrackingTools/PatternTools/interface/TrackConstraintAssociation.h"))) ;
                                            ^

 *** Break *** segmentation violation

[b] CMSSW_13_2_0_pre3 (with root 6.26)

Singularity> root
   ------------------------------------------------------------------
  | Welcome to ROOT 6.26/11                        https://root.cern |
  | (c) 1995-2021, The ROOT Team; conception: R. Brun, F. Rademakers |
  | Built for linuxx8664gcc on Jul 17 2023, 09:53:00                 |
  | From tag , 16 November 2022                                      |
  | With g++ (GCC) 11.4.1 20230601                                   |
  | Try '.help', '.demo', '.license', '.credits', '.quit'/'.q'       |
   ------------------------------------------------------------------

root [0] FWLiteEnabler::enable();
root [1] e = new TChain("Events");
root [2] e->Add("/build/muz/test.root");
root [3] e->SetAlias("p", "patPackedCandidates_packedPFCandidates__RECO.obj");
root [4] e->Draw("p.phi():p.eta()", "p.fromPV()>0&&p.hasTrackDetails()&&p.pt()>1", "", 10);
Info in <TCanvas::MakeDefCanvas>:  created default TCanvas with name c1
CondFormatsL1TObjects_xr dictionary forward declarations' payload:8:192: error: enumeration previously declared with nonfixed underlying type
enum  __attribute__((annotate("$clingAutoload$DataFormats/L1GlobalTrigger/interface/L1GtObject.h")))  __attribute__((annotate("$clingAutoload$CondFormats/L1TObjects/interface/L1GtBoard.h"))) L1GtObject : unsigned int;
                                                                                                                                                                                               ^
/cvmfs/cms.cern.ch/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_0_pre3/src/DataFormats/L1GlobalTrigger/interface/L1GtObject.h:28:6: note: previous declaration is here
enum L1GtObject {
     ^
Warning in <TInterpreter::TCling::RegisterModule>: Problems in compiling forward declarations for module CondFormatsL1TObjects_xr: '
#line 1 "CondFormatsL1TObjects_xr dictionary forward declarations' payload"
#pragma clang diagnostic ignored "-Wkeyword-compat"
#pragma clang diagnostic ignored "-Wignored-attributes"
#pragma clang diagnostic ignored "-Wreturn-type-c-linkage"
extern int __Cling_AutoLoading_Map;
class __attribute__((annotate("$clingAutoload$CondFormats/L1TObjects/interface/L1MuDTEtaPattern.h")))  __attribute__((annotate("$clingAutoload$CondFormats/L1TObjects/interface/L1MuDTEtaPatternLut.h")))  L1MuDTEtaPattern;
namespace std{template <typename _T1, typename _T2> struct __attribute__((annotate("$clingAutoload$bits/stl_iterator.h")))  __attribute__((annotate("$clingAutoload$string")))  pair;
}
enum  __attribute__((annotate("$clingAutoload$DataFormats/L1GlobalTrigger/interface/L1GtObject.h")))  __attribute__((annotate("$clingAutoload$CondFormats/L1TObjects/interface/L1GtBoard.h"))) L1GtObject : unsigned int;
namespace std{template <typename _Tp> class __attribute__((annotate("$clingAutoload$bits/allocator.h")))  __attribute__((annotate("$clingAutoload$string")))  allocator;
}
namespace std{template <class _CharT> struct __attribute__((annotate("$clingAutoload$bits/char_traits.h")))  __attribute__((annotate("$clingAutoload$string")))  char_traits;
}
namespace std{template <typename _Tp = void> struct __attribute__((annotate("$clingAutoload$bits/stl_function.h")))  __attribute__((annotate("$clingAutoload$string")))  less;
....
...
#include "DataFormats/ParticleFlowReco/interface/PreId.h"
#include "DataFormats/ParticleFlowReco/interface/PreIdFwd.h"
#include "DataFormats/ParticleFlowReco/interface/RecoPFClusterRefCandidate.h"
#include "DataFormats/ParticleFlowReco/interface/RecoPFClusterRefCandidateFwd.h"

#include "DataFormats/ParticleFlowReco/interface/HGCalMultiCluster.h"

#include <map>

#undef  _BACKWARD_BACKWARD_WARNING_H

root [5] .q
Singularity>

[c]

  • CMSSW 13.3.X (with root 6.26)
Singularity> cmsenv
Singularity> root
   ------------------------------------------------------------------
  | Welcome to ROOT 6.26/11                        https://root.cern |
  | (c) 1995-2021, The ROOT Team; conception: R. Brun, F. Rademakers |
  | Built for linuxx8664gcc on Jul 23 2023, 01:37:00                 |
  | From tag , 16 November 2022                                      |
  | With g++ (GCC) 11.4.1 20230601                                   |
  | Try '.help', '.demo', '.license', '.credits', '.quit'/'.q'       |
   ------------------------------------------------------------------

root [0] FWLiteEnabler::enable();
root [1] e = new TChain("Events");
root [2] e->Add("/build/muz/test.root");
root [3] e->SetAlias("p", "patPackedCandidates_packedPFCandidates__RECO.obj");
root [4] e->Draw("p.phi():p.eta()", "p.fromPV()>0&&p.hasTrackDetails()&&p.pt()>1", "", 10);
Info in <TCanvas::MakeDefCanvas>:  created default TCanvas with name c1
root [5] .q
Singularity> 
  • CMSSW_13_3_ROOT628 with root 6.28
Singularity> root
   ------------------------------------------------------------------
  | Welcome to ROOT 6.28/05                        https://root.cern |
  | (c) 1995-2023, The ROOT Team; conception: R. Brun, F. Rademakers |
  | Built for linuxx8664gcc on Jul 24 2023, 23:57:00                 |
  | From tag , 8 May 2023                                            |
  | With g++ (GCC) 11.4.1 20230601                                   |
  | Try '.help'/'.?', '.demo', '.license', '.credits', '.quit'/'.q'  |
   ------------------------------------------------------------------

root [0] FWLiteEnabler::enable();
root [1] e = new TChain("Events");
root [2] e->Add("/build/muz/test.root");
root [3] e->SetAlias("p", "patPackedCandidates_packedPFCandidates__RECO.obj");
root [4] e->Draw("p.phi():p.eta()", "p.fromPV()>0&&p.hasTrackDetails()&&p.pt()>1", "", 10);
Info in <TCanvas::MakeDefCanvas>:  created default TCanvas with name c1
root [5] .q
Singularity>
  • CMSSW_13_3_ROOT6 with root maser
Singularity> root
   ------------------------------------------------------------------
  | Welcome to ROOT 6.29/01                        https://root.cern |
  | (c) 1995-2022, The ROOT Team; conception: R. Brun, F. Rademakers |
  | Built for linuxx8664gcc on Jul 24 2023, 00:19:00                 |
  | From tag , 21 December 2022                                      |
  | With g++ (GCC) 11.4.1 20230601                                   |
  | Try '.help'/'.?', '.demo', '.license', '.credits', '.quit'/'.q'  |
   ------------------------------------------------------------------

root [0] FWLiteEnabler::enable();
root [1] e = new TChain("Events");
root [2] e->Add("/build/muz/test.root");
root [3] e->SetAlias("p", "patPackedCandidates_packedPFCandidates__RECO.obj");
root [4] e->Draw("p.phi():p.eta()", "p.fromPV()>0&&p.hasTrackDetails()&&p.pt()>1", "", 10);
Info in <TCanvas::MakeDefCanvas>:  created default TCanvas with name c1
root [5] .q
Singularity>

@hahnjo
Copy link
Contributor

hahnjo commented Jul 27, 2023

So could it be that this has already been fixed via some cmssw PR

Ah right, this was worked around by #42300. Sorry, I missed that there is no easy reproducer in CMSSW master anymore...

do we need the root-project/root#13326 root-project/root#13327 changes?

Not for L1GtObject in master then, but it could in principle affect other enums. And I'm not sure if you prefer backporting the workaround or update ROOT in older release cycles?

Anyway, I'll take that CMSSW built fine with the patches as a sign that the fix isn't horribly breaking something and merge them into ROOT's release branches. Then it's up to you to decide which route you want to go. Sorry again that your testing was defied by the workaround already put in place...

hahnjo added a commit to root-project/root that referenced this issue Jul 27, 2023
There can be multiple attributes in the forward declaration, see the
added test in roottest/cling/dict/enum (reduced from a case reported
by CMS in cms-sw/cmssw#42234), so we have
to look for the last closing parentheses.

(cherry picked from commit 9d2f761)
hahnjo added a commit to root-project/root that referenced this issue Jul 27, 2023
There can be multiple attributes in the forward declaration, see the
added test in roottest/cling/dict/enum (reduced from a case reported
by CMS in cms-sw/cmssw#42234), so we have
to look for the last closing parentheses.

(cherry picked from commit 9d2f761)
hahnjo added a commit to root-project/roottest that referenced this issue Jul 27, 2023
This is a heavily reduced case that shows the problem originally
reported in cms-sw/cmssw#42234.

(cherry picked from commit 09f72c3)
hahnjo added a commit to root-project/roottest that referenced this issue Jul 27, 2023
This is a heavily reduced case that shows the problem originally
reported in cms-sw/cmssw#42234.

(cherry picked from commit 09f72c3)
@slava77
Copy link
Contributor Author

slava77 commented Jul 27, 2023

I tried an even more complicated Draw command in CMSSW_13_3_X_2023-07-27-1100 and it also worked.

So, from my point of view, the issue is resolved.
Thanks.

I'm not going to close myself, in case the solution in root still needs to be tracked.

@hahnjo
Copy link
Contributor

hahnjo commented Jul 28, 2023

I tried an even more complicated Draw command in CMSSW_13_3_X_2023-07-27-1100 and it also worked.

Just to be clear, this is because of the workaround in #42300 because the proper ROOT fix isn't in that IB yet, right?

So, from my point of view, the issue is resolved. Thanks.

I'm not going to close myself, in case the solution in root still needs to be tracked.

We should be good from the ROOT side, I merged the two backports to 6.26 and 6.28. The fixes will appear in the next point releases.

@makortel
Copy link
Contributor

makortel commented Aug 4, 2023

+core

Seems like there is nothing more to be done

@cmsbuild
Copy link
Contributor

cmsbuild commented Aug 4, 2023

This issue is fully signed and ready to be closed.

@makortel
Copy link
Contributor

makortel commented Aug 4, 2023

@cmsbuild, please close

@cmsbuild cmsbuild closed this as completed Aug 4, 2023
maksgraczyk pushed a commit to maksgraczyk/root that referenced this issue Jan 12, 2024
There can be multiple attributes in the forward declaration, see the
added test in roottest/cling/dict/enum (reduced from a case reported
by CMS in cms-sw/cmssw#42234), so we have
to look for the last closing parentheses.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

10 participants