Fix weights to be identical to original commit

[ssrothman]

When updating the model files in the previous PR I accidentally used weights from a different training. The physics performance between the two trainings is identical, but for consistency this PR reverts the weights to be identical with the original training.

[cmsbuild]

A new Pull Request was created by @ssrothman for branch main.

@smuzaffar, @aandvalenzuela, @iarspider, @clacaputo, @cmsbuild, @mandrenguyen can you please review it and eventually sign? Thanks.
@perrotta, @dpiparo, @rappoccio you are the release manager for this.
cms-bot commands are listed here

[kpedro88]

test parameters

• workflows = 10805.31

[kpedro88]

please test

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-eec557/31014/summary.html
COMMIT: 38d5cd0
CMSSW: CMSSW_13_1_X_2023-03-02-1100/el8_amd64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-data/RecoEgamma-EgammaPhotonProducers/3/31014/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

• @cms-sw Update tag for HeterogeneousCore-SonicTriton to V00-02-00 cms-sw/cmsdist#8348
• @kpedro88 Update Triton client to version 2.25 cms-sw/cmsdist#8324
• @cms-sw [SimG4CMS-Calo] Update HFShowerLibrary_run3_v7 cms-sw/cmsdist#8292
• @cms-sw Update tag for RecoEcal-EgammaClusterProducers to V00-03-00 cms-sw/cmsdist#8350
• @cms-sw Update tag for RecoEgamma-EgammaPhotonProducers to V00-02-00 cms-sw/cmsdist#8351

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-eec557/31014/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-eec557/31014/git-merge-result

Comparison Summary

Summary:

• You potentially added 10 lines to the logs
• Reco comparison results: 2 differences found in the comparisons
• DQMHistoTests: Total files compared: 50
• DQMHistoTests: Total histograms compared: 3632285
• DQMHistoTests: Total failures: 2
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3632261
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
• Checked 219 log files, 169 edm output root files, 50 DQM output files
• TriggerResults: no differences found

[perrotta]

please test

• wf 10805.31 shows now differences wrt CMSSW_13_1_X_2023-03-02-1100 plus the original PRs that this one is supposed to fix: I would have expected none...
• Let try to test once again on top of a clean CMSSW_13_1_X_2023-03-02-2300: here I would expect to see reverted the differences observed for #40814

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-eec557/31022/summary.html
COMMIT: 38d5cd0
CMSSW: CMSSW_13_1_X_2023-03-02-2300/el8_amd64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-data/RecoEgamma-EgammaPhotonProducers/3/31022/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

• @fwyzard [BuildRules] Enable alpaka ROCm backend cms-sw/cmsdist#8346

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-eec557/31022/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-eec557/31022/git-merge-result

Comparison Summary

Summary:

• You potentially added 8 lines to the logs
• Reco comparison results: 14 differences found in the comparisons
• DQMHistoTests: Total files compared: 50
• DQMHistoTests: Total histograms compared: 3632285
• DQMHistoTests: Total failures: 6
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3632257
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
• Checked 219 log files, 169 edm output root files, 50 DQM output files
• TriggerResults: no differences found

[perrotta]

@ssrothman @kpedro88

• the first tests included both the original cms-data PR (Updated configs to play better with GPU backend #2) and this one, which should kind of revert it, since it reverts the weights to be identical with the original training, as you write. However the test results showed that the previous figures weren't restored as expected:
  
  

• To cross check I run some further tests on keeping CMSSW_13_1_X_2023-03-02-2300 as baseline, which contains Updated configs to play better with GPU backend #2. The results of that comparison should have shown the opposite of the modifications reported for the original PR: this is not the case, instead.

• Let me add here the configurations (from Ragged batching interface for SonicTriton cms-sw/cmssw#40814 (comment)) which we are expected to revert would this PR really revert the weights to be identical with the original training, so that they can be saved even when the test results will disappear
  
  

• Evidently, this fix does not reverts the weights to be pretty identical with the original training. Perhaps also for the other cms-data PRs merged together with Ragged batching interface for SonicTriton cms-sw/cmssw#40814 weights from a different training were used? E.g.

  • Add model for ragged batching unit test HeterogeneousCore-SonicTriton#2
  • Updated configs to play better with GPU backend #2
  • Updated model files to play nicely with GPU backend RecoEcal-EgammaClusterProducers#3

• Or maybe the new vesion of the Triton client, Update Triton client to version 2.25 cms-sw/cmsdist#8324, does originate some (kind of random) modification in the results with respect to the previous one?

[perrotta]

@ssrothman @kpedro88 any news on this? None of you was present at the today's ORP meeting, where this issue was addressed...
Is there some better understanding of the issue by now?

[kpedro88]

@perrotta apologies, I'm away at a workshop this week.

We also saw this behavior in our private tests last week and realized that there is some randomness inherent to the network itself. The random behavior has been there all along, but http://github.com/cms-sw/cmssw/pull/40814 may actually have been the first time that comparison tests were run for 10805.31 (since it is not part of the short matrix), so it wasn't noticed before. This PR does correctly restore the original weights, but we need to make some more changes to make the network deterministic (this is a work in progress right now).

[perrotta]

@ssrothman @kpedro88
As a cross-check, I made bot running the comparisons for the wf 10805.31 in another PR which is completely uncorrelated from the update of these weights. And the outcome is exactly as what you say here, that is there is some randomness in the results which is inherent to the network itself. The "randomness" seems quite significant, i.e. not just a numerical fluctuation in the last digi somewhere.

Therefore:

• Let merge this PR with the correct weights, as you verified independently
• Open a github issue to take record of the progresses in the solution of this "randomness" problem

[perrotta]

+1

[perrotta]

merge