Implement AlpakaBackendProducer and AlpakaBackendFilter #44387

fwyzard · 2024-03-13T10:50:09Z

PR description:

Implement AlpakaBackendProducer, an empty alpaka-based EDProducer whose only purpose is to save in the event what alpaka backend has been used.

Implement AlpakaBackendFilter, an EDFilter that selects events based on the alpaka backend used to run a previous producer.

Implement a unit test for both modules.

The aim is to replace this SwitchProducer in the HLT menu

process.statusOnGPU = SwitchProducerCUDA(
   cpu = cms.EDProducer( "BooleanProducer",
       value = cms.bool( False )
   ),
  cuda = cms.EDProducer( "BooleanProducer",
       value = cms.bool( True )
   ),
 )

process.statusOnGPUFilter = cms.EDFilter( "BooleanFilter",
    src = cms.InputTag( "statusOnGPU" )
)

process.Status_OnCPU = cms.Path( process.statusOnGPU + ~process.statusOnGPUFilter )
process.Status_OnGPU = cms.Path( process.statusOnGPU + process.statusOnGPUFilter )

with this alpaka-based solution

process.hltBackend = cms.EDProducer('AlpakaBackendProducer@alpaka')

process.hltStatusOnGPUFilter = cms.EDFilter('AlpakaBackendFilter',
  producer = cms.InputTag('hltBackend', 'backend'),
  backends = cms.vstring('CudaAsync', 'ROCmAsync')
)

process.Status_OnCPU = cms.Path( process.hltBackend + ~process.hltStatusOnGPUFilter )
process.Status_OnGPU = cms.Path( process.hltBackend + process.hltStatusOnGPUFilter )

PR validation:

The new unit test passes.

Backport status

To be backported to 14.0.x for data taking.

cmsbuild · 2024-03-13T10:50:34Z

cms-bot internal usage

fwyzard · 2024-03-13T10:50:38Z

@makortel could you have a look an let me know if you have any suggestions ?

fwyzard · 2024-03-13T10:50:54Z

@mmusich @Martin-Grunewald @missirol FYI

fwyzard · 2024-03-13T10:50:57Z

enable gpu

fwyzard · 2024-03-13T10:54:11Z

please test

cmsbuild · 2024-03-13T10:56:26Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-44387/39449

This PR adds an extra 16KB to repository

cmsbuild · 2024-03-13T10:56:54Z

A new Pull Request was created by @fwyzard for master.

It involves the following packages:

HeterogeneousCore/AlpakaCore (heterogeneous)

@makortel, @fwyzard can you please review it and eventually sign? Thanks.
@missirol, @makortel, @rovere this is something you requested to watch as well.
@rappoccio, @antoniovilela, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

cmsbuild · 2024-03-13T13:42:53Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-4d4957/38097/summary.html
COMMIT: 1f3f9fe
CMSSW: CMSSW_14_1_X_2024-03-12-2300/el8_amd64_gcc12
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/44387/38097/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially removed 81 lines from the logs
Reco comparison results: 44 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3297383
DQMHistoTests: Total failures: 0
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3297363
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
Checked 202 log files, 165 edm output root files, 48 DQM output files
TriggerResults: no differences found

GPU Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 11 differences found in the comparisons
DQMHistoTests: Total files compared: 3
DQMHistoTests: Total histograms compared: 39740
DQMHistoTests: Total failures: 503
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 39237
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 2 files compared)
Checked 8 log files, 10 edm output root files, 3 DQM output files
TriggerResults: no differences found

Implement AlpakaBackendProducer, an empty alpaka-based EDProducer whose only purpose is to save in the event what alpaka backend has been used. Implement AlpakaBackendFilter, an EDFilter that selects events based on the alpaka backend used to run a previous producer. Implement a unit test for both modules.

fwyzard · 2024-03-15T16:11:28Z

enable gpu

fwyzard · 2024-03-15T16:11:32Z

please test

cmsbuild · 2024-03-15T16:16:06Z

Pull request #44387 was updated. can you please check and sign again.

antoniovilela · 2024-03-15T18:07:35Z

+1

smuzaffar · 2024-03-15T21:38:48Z

@smuzaffar I've updated the test to use the SCRAM_ALPAKA_BACKEND environment variable, if set.

@fwyzard , looks like build rule change worked. I see SCRAM_ALPAKA_BACKEND was properly set for testAlpakaBackendFilterCudaAsync

Testing the alpaka backend CudaAsync using the process accelerator gpu-nvidia

antoniovilela · 2024-03-15T22:17:43Z

-orp

Signed before tests run by mistake. Will re-sign later.

cmsbuild · 2024-03-15T22:39:46Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-4d4957/38177/summary.html
COMMIT: d51e58e
CMSSW: CMSSW_14_1_X_2024-03-15-1100/el8_amd64_gcc12
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/44387/38177/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially added 108 lines to the logs
Reco comparison results: 46 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3297383
DQMHistoTests: Total failures: 9
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3297354
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
Checked 202 log files, 165 edm output root files, 48 DQM output files
TriggerResults: no differences found

GPU Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 19 differences found in the comparisons
DQMHistoTests: Total files compared: 3
DQMHistoTests: Total histograms compared: 39740
DQMHistoTests: Total failures: 670
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 39070
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 2 files compared)
Checked 8 log files, 10 edm output root files, 3 DQM output files
TriggerResults: no differences found

fwyzard · 2024-03-15T23:22:51Z

@fwyzard , looks like build rule change worked.

Yep, I also tested it locally with a recent IB, and it worked correctly.

antoniovilela · 2024-03-16T18:59:58Z

+1

antoniovilela · 2024-03-18T05:29:33Z

Something strange in IB: https://cmssdt.cern.ch/SDT/cgi-bin/logreader/el8_amd64_gcc12/CMSSW_14_1_X_2024-03-17-2300/unitTestLogs/HeterogeneousCore/AlpakaTest#/

makortel · 2024-03-18T06:53:49Z

HeterogeneousCore/AlpakaCore/plugins/BuildFile.xml

+  <flags EDM_PLUGIN="1"/>
+</library>
+
+<library file="alpaka/*.cc" name="HeterogeneousCoreAlpakaTestPluginsPortable">


Problem lies here, duplicate plugin name wrt

cmssw/HeterogeneousCore/AlpakaTest/plugins/BuildFile.xml

Line 16 in ec85b46

<library file="alpaka/*.cc" name="HeterogeneousCoreAlpakaTestPluginsPortable">

makortel · 2024-03-18T06:56:53Z

#44445 should fix

antoniovilela · 2024-03-19T15:13:07Z

@fwyzard @makortel
Do you understand why the unit test is still failing?

smuzaffar · 2024-03-19T15:16:07Z

@antoniovilela , unit tests only fail in patch IBs. This is because we still have old duplicates in the full IB. Errors should go away when we will full IB (hopefully tonight as I want to merge cms-sw/cmsdist#9080 for tonight IB)

antoniovilela · 2024-03-19T15:18:51Z

@antoniovilela , unit tests only fail in patch IBs. This is because we still have old duplicates in the full IB. Errors should go away when we will full IB (hopefully tonight as I want to merge cms-sw/cmsdist#9080 for tonight IB)

Thanks.

cmsbuild added this to the CMSSW_14_1_X milestone Mar 13, 2024

cmsbuild added pending-signatures tests-pending orp-pending code-checks-pending heterogeneous-pending labels Mar 13, 2024

cmsbuild added tests-started and removed tests-pending labels Mar 13, 2024

fwyzard mentioned this pull request Mar 13, 2024

Implement AlpakaBackendProducer and AlpakaBackendFilter [14.0.x] #44388

Merged

cmsbuild added code-checks-approved and removed code-checks-pending labels Mar 13, 2024

cmsbuild added tests-approved and removed tests-started labels Mar 13, 2024

fwyzard force-pushed the implement_AlpakaBackendFilter_14_1_x branch from 1f3f9fe to d51e58e Compare March 15, 2024 16:11

cmsbuild added tests-pending code-checks-pending and removed tests-approved code-checks-approved labels Mar 15, 2024

cmsbuild removed the tests-pending label Mar 15, 2024

cmsbuild added the code-checks-approved label Mar 15, 2024

cmsbuild added orp-approved and removed orp-pending labels Mar 15, 2024

cmsbuild added orp-rejected and removed orp-approved labels Mar 15, 2024

cmsbuild added tests-approved and removed tests-started labels Mar 15, 2024

cmsbuild added orp-approved and removed orp-rejected labels Mar 16, 2024

cmsbuild merged commit ec85b46 into cms-sw:master Mar 16, 2024
14 checks passed

cmsbuild mentioned this pull request Mar 17, 2024

[PyROOT] Update cppyy to a recent version cms-sw/root#205

Closed

fwyzard mentioned this pull request Mar 17, 2024

Update the HLT alpaka customisation #44439

Merged

makortel reviewed Mar 18, 2024

View reviewed changes

makortel mentioned this pull request Mar 18, 2024

Fix plugin library name AlpakaCore/plugins/BuildFile.xml #44445

Merged

This was referenced Mar 18, 2024

[Do not merge] Testing Tensorflow 2.15 GPU tests #44449

Closed

[CPP20] Remove deprecated implicit capture of this #44451

Merged

[AArch64, PowerPC] Fix deprecated-copy warnings in extBasic{2,3}DVector #44448

Merged

fwyzard deleted the implement_AlpakaBackendFilter_14_1_x branch March 19, 2024 17:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement AlpakaBackendProducer and AlpakaBackendFilter #44387

Implement AlpakaBackendProducer and AlpakaBackendFilter #44387

fwyzard commented Mar 13, 2024 •

edited

Loading

cmsbuild commented Mar 13, 2024 •

edited

Loading

fwyzard commented Mar 13, 2024

fwyzard commented Mar 13, 2024

fwyzard commented Mar 13, 2024

fwyzard commented Mar 13, 2024

cmsbuild commented Mar 13, 2024

cmsbuild commented Mar 13, 2024

cmsbuild commented Mar 13, 2024

fwyzard commented Mar 15, 2024

fwyzard commented Mar 15, 2024

cmsbuild commented Mar 15, 2024

antoniovilela commented Mar 15, 2024

smuzaffar commented Mar 15, 2024

antoniovilela commented Mar 15, 2024

cmsbuild commented Mar 15, 2024

fwyzard commented Mar 15, 2024

antoniovilela commented Mar 16, 2024

antoniovilela commented Mar 18, 2024

makortel Mar 18, 2024

makortel commented Mar 18, 2024

antoniovilela commented Mar 19, 2024

smuzaffar commented Mar 19, 2024 •

edited

Loading

antoniovilela commented Mar 19, 2024

Implement AlpakaBackendProducer and AlpakaBackendFilter #44387

Implement AlpakaBackendProducer and AlpakaBackendFilter #44387

Conversation

fwyzard commented Mar 13, 2024 • edited Loading

PR description:

PR validation:

Backport status

cmsbuild commented Mar 13, 2024 • edited Loading

fwyzard commented Mar 13, 2024

fwyzard commented Mar 13, 2024

fwyzard commented Mar 13, 2024

fwyzard commented Mar 13, 2024

cmsbuild commented Mar 13, 2024

cmsbuild commented Mar 13, 2024

cmsbuild commented Mar 13, 2024

Comparison Summary

GPU Comparison Summary

fwyzard commented Mar 15, 2024

fwyzard commented Mar 15, 2024

cmsbuild commented Mar 15, 2024

antoniovilela commented Mar 15, 2024

smuzaffar commented Mar 15, 2024

antoniovilela commented Mar 15, 2024

cmsbuild commented Mar 15, 2024

Comparison Summary

GPU Comparison Summary

fwyzard commented Mar 15, 2024

antoniovilela commented Mar 16, 2024

antoniovilela commented Mar 18, 2024

makortel Mar 18, 2024

Choose a reason for hiding this comment

makortel commented Mar 18, 2024

antoniovilela commented Mar 19, 2024

smuzaffar commented Mar 19, 2024 • edited Loading

antoniovilela commented Mar 19, 2024

fwyzard commented Mar 13, 2024 •

edited

Loading

cmsbuild commented Mar 13, 2024 •

edited

Loading

smuzaffar commented Mar 19, 2024 •

edited

Loading