Integration to CMSSW master

[makortel]

This issue is to track progress towards integrating mkFit to CMSSW master:

Overall (to get mkFit running within offline reco)

• merge Fix compiler error with GCC 7 #235
• maybe include Update README for mictest->mkFit #237 as well?
• tag a new release
• create mkFit external to cmsdist, merged as Add mkFit as an external cms-sw/cmsdist#5150
  • finalize mkFit external, merged as Add mkfit to cmssw-tool-conf cms-sw/cmsdist#5169
• tag a new release following the merge of Change found hit bonus to 7.5 #241
• update mkFit external in cmsdist, merged as Update mkFit to 2.0.1 cms-sw/cmsdist#5213
• integrate RecoTracker/MkFit package from makortel:mkfit_1040p1 to CMSSW master, merged as Add EDProducers for mkFit cms-sw/cmssw#27895
• update mkFit with interface towards CMSSW to enable duplicate removal, done in Allow enabling duplicate removal from CMSSW #244
• enable duplicate removal in MkFitProducer
• Deal with non-x86 architectures Put conditions for headers depending on the architecture #240
• Deal with OpenMP runtime lib Can OpenMP runtime be dropped? #257
• Deal with SMatrix Remove SMatrix copy #258 (done in Pick SMatrix from CMSSW if building for CMSSW #264)
• Fix backward fit for triplet seeds (Fix backward fit for tracks without a hit on innermost pixel layer #265)
• update mkFit external

Getting mkFit to run within HLT reconstruction some additional changes may be wanted

• either re-open and merge Use also the fourth hit of pixel track in SeedGeneratorFromProtoTracksEDProducer cms-sw/cmssw#26322, or remove or allow chancing the minimum hit cut for seeds in the seed cleaning, merged as Make it possible to use the fourth hit of a pixel track in SeedGeneratorFromProtoTracksEDProducer cms-sw/cmssw#28473 with a configuration flag

  mkFit/Event.cc

  Line 855 in 9afc251

  if (nHits[ts] < minNHits) continue;

  • the seeds from quadruplet pixel tracks (as in HLT iter0 at the moment) get their last hit dropped, so the seed cleaning will drop all seeds in that case
  • of course all this will change for run3
• make PR for makortel/cmssw@ddd42d7 (Add a flag to disable strip rechit matching), merged as Add a flag to disable matching in SiStripRecHitConverter cms-sw/cmssw#28471
  • will become close-to-unnecessary once we get the GPU version of strip local reco
• add customize function to use mkFit in HLT iter0 pattern recognition

[slava77]

either re-open and merge cms-sw/cmssw#26322, or remove or allow chancing the minimum hit cut for seeds in the seed cleaning

I'm not sure that the CMSSW PR reopening will go so easily, because the original issue there is not addressed. Pattern reco turns out to deal with the outer hits better compared to the seeding. It didn't look like the POG was willing to commit to making things slightly worse under an assumption that it will all be done better anyways for the final Run 3 tracking.
Perhaps that regression goes away with the patatrack seeds with refit, but this will apparently not show up in CMSSW for some time.

In the current iterative tracking that we are using for tests all seeds are of the same length. So, this selection doesn't really make sense on a seed by seed level.
I propose to drop this cut.

[slava77]

either re-open and merge cms-sw/cmssw#26322,

Oh, if you can percolate this addition as configurable, there should be no problem to get this PR in.

[makortel]

@slava77 tagged a new release https://github.com/trackreco/mkFit/releases/tag/V2.0.0-1+pr237

I have the full chain tested locally up to a runTheMatrix workflow running with a customize function changing the usual initialStep pattern recognition with mkFit.

[makortel]

The cmsdist PR is here cms-sw/cmsdist#5150

[makortel]

Apparently cms-sw/cmsdist#5150 (merged on last Friday) was not sufficient to get the mkfit external usable from CMSSW, so I made a new PR to add the (hopefully last) missing piece cms-sw/cmsdist#5169.

[slava77]

we should also add some "native" support to replace patches prepared in cms-sw/cmsdist#5174
@osschar you are probably the most familiar with the intrinsics to get it right.

[makortel]

we should also add some "native" support to replace patches prepared in cms-sw/cmsdist#5174
@osschar you are probably the most familiar with the intrinsics to get it right.

Let's follow up those in #240.

[osschar]

Well, it's not just what's the subject of #240 is (headers) ... we also need to:

1. massage the makefile, have some architecture detection in there
2. figure out how much time we want to spend on getting Matriplex intrinsic vectorization to work on arm and ppc (my guess is zero at this point) ... so the easiest would be to simply fall back to GenMul generating standard c++ and let the compilers try to vectorize that.
   In any case, we need access to those machines with compilers cms uses to try things out.

[makortel]

The CMSSW PR is here cms-sw/cmssw#27895

[makortel]

Just to document here that in light of the presentation at TRK POG on Monday showing the performance with #241, we just decided to cut a new release after its merge (now), and update the version in cmsdist.

[makortel]

Just to document here that in light of the presentation at TRK POG on Monday showing the performance with #241, we just decided to cut a new release after its merge (now), and update the version in cmsdist.

We have a new release https://github.com/trackreco/mkFit/releases/tag/V2.0.1-0+pr241

[makortel]

Just to document here that in light of the presentation at TRK POG on Monday showing the performance with #241, we just decided to cut a new release after its merge (now), and update the version in cmsdist.

We have a new release https://github.com/trackreco/mkFit/releases/tag/V2.0.1-0+pr241

cmsdist PR is here cms-sw/cmsdist#5213

[makortel]

#27895 was merged, the added matrix workflow 11634.7 is visible in IBs and running succesfully
https://cmssdt.cern.ch/SDT/cgi-bin/logreader/slc7_amd64_gcc820/CMSSW_11_0_X_2019-09-18-2300/pyRelValMatrixLogs/run/11634.7_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2021_GenSimFullINPUT+DigiFull_2021+RecoFull_trackingMkFit_2021+HARVESTFull_2021/step3_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2021_GenSimFullINPUT+DigiFull_2021+RecoFull_trackingMkFit_2021+HARVESTFull_2021.log#/

[makortel]

I'm leaving the issue open though until the HLT part is resolved as well.

[makortel]

This issue is effectively resolved