[GPU] Unit test testEigenGPUNoFit_t failing when testing inverse 5x5

[aandvalenzuela]

Hello,

Unit test testEigenGPUNoFit_t (module RecoTracker/PixelTrackFitting) is failing when running TEST INVERSE 5x5 with:

===== Test "testEigenGPUNoFit_t" ====
TEST EIGENVALUES
TEST INVERSE 3x3
TEST INVERSE 4x4
TEST INVERSE 5x5
testEigenGPUNoFit_t: /tmp/avalenzu/CMSSW_13_3_GPU_X_2023-08-02-2300/src/RecoTracker/PixelTrackFitting/test/test_common.h:24: bool isEqualFuzzy(C1, C2, double) [with C1 = Eigen::Matrix<double, 5, 5>; C2 = Eigen::Matrix<double, 5, 5>]: Assertion `std::abs(a(i, j) - b(i, j)) < std::min(std::abs(a(i, j)), std::abs(b(i, j))) * epsilon' failed.
/bin/sh: line 1: 1044014 Aborted                 (core dumped) sh -c 'testEigenGPUNoFit_t '

---> test testEigenGPUNoFit_t had ERRORS
TestTime:4
^^^^ End Test testEigenGPUNoFit_t ^^^^                                         

We already had a similar issue when testing the 5x5 inverse in this unit test (See #33797).

I can reproduce it in lxplus-gpu nodes.

[cmsbuild]

A new Issue was created by @aandvalenzuela Andrea Valenzuela.

@Dr15Jones, @perrotta, @dpiparo, @rappoccio, @makortel, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[makortel]

assign reconstruction, heterogeneous

FYI @cms-sw/tracking-pog-l2 @VinInn

[cmsbuild]

New categories assigned: heterogeneous,reconstruction

@mandrenguyen,@fwyzard,@clacaputo,@makortel you have been requested to review this Pull request/Issue and eventually sign? Thanks

[fwyzard]

Unit test testEigenGPUNoFit_t (module RecoTracker/PixelTrackFitting) is failing when running TEST INVERSE 5x5

When did it start to fail ?

[aandvalenzuela]

It failed yesterday for the first time CMSSW_13_3_GPU_X_2023-08-01-2300 and today again CMSSW_13_3_GPU_X_2023-08-02-2300

[fwyzard]

The only PRs merged between CMSSW_13_3_GPU_X_2023-07-31-2300 and CMSSW_13_3_GPU_X_2023-08-01-2300 are:

• Code changes for Protobuf 3.21.9 and TensorFlow 2.12 #42228
• Removal of StringCutObjectSelector from Muon trigger DQM #42437
• [DBG] Fix comparison of integer expressions of different signedness #42405
• Maybe remove build warning in TrackerMap, but at least clean it up #42373
• Phase2-sim145 Study the difference in SD's between DDD and DD4hep for Phase2 scenarios #42420
• Replace the SFINAE check with static_assert #42425

None of them looks like it should affect testEigenGPUNoFit_t.

There was a change in Eigen as part of the update to TensorFlow:

• Update TF to 2.12 (backport from IB/CMSSW_13_2_X/g13 branch) cmsdist#8565

Maybe that is the reason ?

[iarspider]

@fwyzard are there any steps that we (externals) need to do to help triage this?

[fwyzard]

Is this still present after the latest Eigen update ?

[iarspider]

@fwyzard yes: https://cmssdt.cern.ch/SDT/cgi-bin/logreader/el8_amd64_gcc11/CMSSW_13_3_GPU_X_2023-10-01-2300/unitTestLogs/RecoTracker/PixelTrackFitting#/471-471

[fwyzard]

OK.

I will try to set aside some time to look into it.

[fwyzard]

Should be fixed by cms-sw/cmsdist#8782 .