Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SYCL Track Finding in the Example Executables, main branch (2025.01.09.) #812

Merged

Conversation

krasznaa
Copy link
Member

@krasznaa krasznaa commented Jan 9, 2025

This is building on top of #811. That one has to be sorted out / merged first.

This PR:

  • Adds traccc::sycl::measurement_sorting_algorithm, to make it possible to run track finding on top of our cell CSV files;
  • Teaches traccc_seq_example_sycl how to run the track finding algorithm;
  • Teaches the SYCL throughput applications to include the track finding algorithm.

I didn't bother about traccc_seeding_example_sycl. 🤔 Now that track finding is (almost) in place, I'll want to merge the code of these examples finally. Similar to how the throughput executables are set up. 🤔 I just wanted to test traccc_seq_example_sycl even before that.

I only tested the example executables on an NVIDIA backend for now, but that at least seems to be working. 😄 For example:

[bash][Legolas]:out > ./build/sycl-fp32/bin/traccc_throughput_mt_sycl --detector-file=geometries/odd/odd-detray_geometry_detray.json --material-file=geometries/odd/odd-detray_material_detray.json --grid-file=geometries/odd/odd-detray_surface_grids_detray.json --use-detray-detector --digitization-file=geometries/odd/odd-digi-geometric-config.json --use-acts-geom-source --input-directory=odd/geant4_10muon_100GeV/ --input-events=10 --processed-events=1000 --cpu-threads=4

Running Multi-threaded SYCL GPU throughput tests

>>> Detector Options <<<
  Detector file       : geometries/odd/odd-detray_geometry_detray.json
  Material file       : geometries/odd/odd-detray_material_detray.json
  Surface grid file   : geometries/odd/odd-detray_surface_grids_detray.json
  Use detray::detector: yes
  Digitization file   : geometries/odd/odd-digi-geometric-config.json
>>> Input Data Options <<<
  Use ACTS geometry source      : yes
  Input data format             : csv
  Input directory               : odd/geant4_10muon_100GeV/
  Number of input events        : 10
  Number of input events to skip: 0
>>> Clusterization Options <<<
  Threads per partition:      256
  Target cells per thread:    8
  Max cells per thread:       16
  Scratch space size mult.:   256
>>> Track Seeding Options <<<
  None
>>> Track Finding Options <<<
  Max number of branches per seed: 10
  Max number of branches per surface: 10
  Track candidates range   : 3:100
  Minimum step length for the next surface: 0.5 [mm] 
  Maximum step counts for the next surface: 100
  Maximum Chi2             : 10
  Maximum branches per step: 10
  Maximum number of skipped steps per candidates: 3
  PDG Number: 13
>>> Track Propagation Options <<<
Navigation
----------------------------
  Min. mask tolerance   : 1e-05 [mm]
  Max. mask tolerance   : 1 [mm]
  Mask tolerance scalor : 0.05
  Path tolerance        : 1 [um]
  Overstep tolerance    : -100 [um]
  Search window         : 0 x 0

Parameter Transport
----------------------------
  Min. Stepsize         : 0.0001 [mm]
  Runge-Kutta tolerance : 0.0001 [mm]
  Max. step updates     : 10000
  Stepsize  constraint  : 3.40282e+38 [mm]
  Path limit            : 5 [m]
  Use Bethe energy loss : true
  Do cov. transport     : true
  Use eloss gradient    : false
  Use B-field gradient  : false

Geometry Context
----------------------------
0

>>> Throughput Measurement Options <<<
  Cold run event(s) : 10
  Processed event(s): 1000
  Log file          : 
>>> Multi-Threading Options <<<
  CPU threads: 4

WARNING: No material in detector

WARNING: No entries in volume finder

Detector check: OK
WARNING: No entries in volume finder

Detector check: OK
WARNING: @traccc::io::csv::read_cells: 34 duplicate cells found in /data/ssd-1tb/projects/traccc/traccc/data/odd/geant4_10muon_100GeV/event000000000-cells.csv
WARNING: @traccc::io::csv::read_cells: 55 duplicate cells found in /data/ssd-1tb/projects/traccc/traccc/data/odd/geant4_10muon_100GeV/event000000002-cells.csv
WARNING: @traccc::io::csv::read_cells: 58 duplicate cells found in /data/ssd-1tb/projects/traccc/traccc/data/odd/geant4_10muon_100GeV/event000000003-cells.csv
WARNING: @traccc::io::csv::read_cells: 53 duplicate cells found in /data/ssd-1tb/projects/traccc/traccc/data/odd/geant4_10muon_100GeV/event000000008-cells.csv
WARNING: @traccc::io::csv::read_cells: 41 duplicate cells found in /data/ssd-1tb/projects/traccc/traccc/data/odd/geant4_10muon_100GeV/event000000001-cells.csv
WARNING: @traccc::io::csv::read_cells: 155 duplicate cells found in /data/ssd-1tb/projects/traccc/traccc/data/odd/geant4_10muon_100GeV/event000000005-cells.csv
WARNING: @traccc::io::csv::read_cells: 87 duplicate cells found in /data/ssd-1tb/projects/traccc/traccc/data/odd/geant4_10muon_100GeV/event000000006-cells.csv
WARNING: @traccc::io::csv::read_cells: 28 duplicate cells found in /data/ssd-1tb/projects/traccc/traccc/data/odd/geant4_10muon_100GeV/event000000009-cells.csv
WARNING: @traccc::io::csv::read_cells: 271 duplicate cells found in /data/ssd-1tb/projects/traccc/traccc/data/odd/geant4_10muon_100GeV/event000000007-cells.csv
WARNING: @traccc::io::csv::read_cells: 154 duplicate cells found in /data/ssd-1tb/projects/traccc/traccc/data/odd/geant4_10muon_100GeV/event000000004-cells.csv
Using SYCL device: NVIDIA GeForce RTX 3080
Using SYCL device: NVIDIA GeForce RTX 3080
Using SYCL device: NVIDIA GeForce RTX 3080
Using SYCL device: NVIDIA GeForce RTX 3080
Using SYCL device: NVIDIA GeForce RTX 3080
Warm-up processing [==================================================] 100% [00m:00s]                                                                                                                                                                                                                                                                                                                  
Event processing   [==================================================] 100% [00m:00s]                                                                                                                                                                                                                                                                                                                  
Reconstructed track parameters: 180367
Time totals:
                  File reading  25 ms
            Warm-up processing  187 ms
              Event processing  8141 ms
Throughput:
            Warm-up processing  18.7812 ms/event, 53.2448 events/s
              Event processing  8.1414 ms/event, 122.829 events/s
[bash][Legolas]:out >

@krasznaa krasznaa added feature New feature or request sycl Changes related to SYCL examples Changes to the examples labels Jan 9, 2025
@krasznaa krasznaa force-pushed the SYCLTrackFindingExample-main-20250109 branch from 22771c5 to e3fe90c Compare January 10, 2025 08:40
@krasznaa krasznaa force-pushed the SYCLTrackFindingExample-main-20250109 branch from e3fe90c to 4e26907 Compare January 10, 2025 15:13
@krasznaa krasznaa marked this pull request as ready for review January 10, 2025 15:13
Copy link

@krasznaa
Copy link
Member Author

Note that I updated the README a little.

  • Changed the table with algorithm readiness to show that the SYCL CKF is alive and running, while the SYCL KF is more in an "under work" state.
  • Updated some of the text a little.
  • Updated the Medmaid diagram to look like this:

image

(The orange lines belong to the Alpaka algorithms.)

As I just now tested, oneAPI-2025.0 is once again able to build our project at the same time for AMD and NVIDIA GPUs with the following setting for SYCLFLAGS:

export SYCLFLAGS="-fsycl -fsycl-targets=nvidia_gpu_sm_75,amd_gpu_gfx1031 -Wno-unknown-cuda-version"

See:

[bash][pcadp04]:traccc > ONEAPI_DEVICE_SELECTOR=cuda:gpu ./build-sycl-amdnvidia/bin/traccc_seq_example_sycl --detector-file=geometries/odd/odd-detray_geometry_detray.json --material-file=geometries/odd/odd-detray_material_detray.json --grid-file=geometries/odd/odd-detray_surface_grids_detray.json --use-detray-detector --digitization-file=geometries/odd/odd-digi-geometric-config.json --use-acts-geom-source --input-directory=/data/Acts/odd-simulations-20240509/geant4_ttbar_mu140/ --input-events=2 --compare-with-cpu

Running Full Tracking Chain Using SYCL

>>> Detector Options <<<
  Detector file       : geometries/odd/odd-detray_geometry_detray.json
  Material file       : geometries/odd/odd-detray_material_detray.json
  Surface grid file   : geometries/odd/odd-detray_surface_grids_detray.json
  Use detray::detector: yes
  Digitization file   : geometries/odd/odd-digi-geometric-config.json
>>> Input Data Options <<<
  Use ACTS geometry source      : yes
  Input data format             : csv
  Input directory               : /data/Acts/odd-simulations-20240509/geant4_ttbar_mu140/
  Number of input events        : 2
  Number of input events to skip: 0
>>> Clusterization Options <<<
  Threads per partition:      256
  Target cells per thread:    8
  Max cells per thread:       16
  Scratch space size mult.:   256
>>> Track Seeding Options <<<
  None
>>> Track Finding Options <<<
  Max number of branches per seed: 10
  Max number of branches per surface: 10
  Track candidates range   : 3:100
  Minimum step length for the next surface: 0.5 [mm] 
  Maximum step counts for the next surface: 100
  Maximum Chi2             : 10
  Maximum branches per step: 10
  Maximum number of skipped steps per candidates: 3
  PDG Number: 13
>>> Track Propagation Options <<<
Navigation
----------------------------
  Min. mask tolerance   : 1e-05 [mm]
  Max. mask tolerance   : 1 [mm]
  Mask tolerance scalor : 0.05
  Path tolerance        : 1 [um]
  Overstep tolerance    : -100 [um]
  Search window         : 0 x 0

Parameter Transport
----------------------------
  Min. Stepsize         : 0.0001 [mm]
  Runge-Kutta tolerance : 0.0001 [mm]
  Max. step updates     : 10000
  Stepsize  constraint  : 3.40282e+38 [mm]
  Path limit            : 5 [m]
  Use Bethe energy loss : true
  Do cov. transport     : true
  Use eloss gradient    : false
  Use B-field gradient  : false

Geometry Context
----------------------------
0

>>> Performance Measurement Options <<<
  Run performance checks: no
>>> Accelerator Options <<<
  Compare with CPU results: yes

Running on device: NVIDIA RTX A5000
WARNING: No material in detector

WARNING: No entries in volume finder

Detector check: OK
WARNING: No entries in volume finder

Detector check: OK
WARNING: @traccc::io::csv::read_cells: 12177 duplicate cells found in /data/Acts/odd-simulations-20240509/geant4_ttbar_mu140/event000000000-cells.csv
===>>> Event 0 <<<===
Number of measurements: 65269 (host), 65269 (device)
Running comparison [==================================================] 100% [00m:00s]                                            
  Matching rate(s):
    - 98.8816% at 0.01% uncertainty
    - 99.9525% at 0.1% uncertainty
    - 99.9556% at 1% uncertainty
    - 99.9556% at 5% uncertainty
Number of spacepoints: 51560 (host), 51560 (device)
Running comparison [==================================================] 100% [00m:00s]                                            
  Matching rate(s):
    - 98.5842% at 0.01% uncertainty
    - 99.9399% at 0.1% uncertainty
    - 99.9438% at 1% uncertainty
    - 99.9438% at 5% uncertainty
Number of seeds: 20923 (host), 20923 (device)
Running comparison [==================================================] 100% [00m:00s]                                            
  Matching rate(s):
    - 89.0647% at 0.01% uncertainty
    - 99.3213% at 0.1% uncertainty
    - 99.8518% at 1% uncertainty
    - 99.8949% at 5% uncertainty
Number of track parameters: 20923 (host), 20923 (device)
Running comparison [==================================================] 100% [00m:00s]                                            
  Matching rate(s):
    - 99.7945% at 0.01% uncertainty
    - 99.8996% at 0.1% uncertainty
    - 99.9426% at 1% uncertainty
    - 99.9904% at 5% uncertainty
Number of track candidates (header): 37325 (host), 37308 (device)
Running comparison [==================================================] 100% [00m:00s]                                            
  Matching rate(s):
    - 99.7133% at 0.01% uncertainty
    - 99.7964% at 0.1% uncertainty
    - 99.8768% at 1% uncertainty
    - 99.933% at 5% uncertainty
  Track candidates (item) matching rate: 97.5995%
WARNING: @traccc::io::csv::read_cells: 15013 duplicate cells found in /data/Acts/odd-simulations-20240509/geant4_ttbar_mu140/event000000001-cells.csv
===>>> Event 1 <<<===
Number of measurements: 82027 (host), 82027 (device)
Running comparison [==================================================] 100% [00m:00s]                                            
  Matching rate(s):
    - 98.8211% at 0.01% uncertainty
    - 99.9695% at 0.1% uncertainty
    - 99.9732% at 1% uncertainty
    - 99.9744% at 5% uncertainty
Number of spacepoints: 64114 (host), 64114 (device)
Running comparison [==================================================] 100% [00m:00s]                                            
  Matching rate(s):
    - 98.4917% at 0.01% uncertainty
    - 99.961% at 0.1% uncertainty
    - 99.9657% at 1% uncertainty
    - 99.9672% at 5% uncertainty
Number of seeds: 27622 (host), 27621 (device)
Running comparison [==================================================] 100% [00m:00s]                                            
  Matching rate(s):
    - 88.3607% at 0.01% uncertainty
    - 99.3121% at 0.1% uncertainty
    - 99.8154% at 1% uncertainty
    - 99.8697% at 5% uncertainty
Number of track parameters: 27622 (host), 27621 (device)
Running comparison [==================================================] 100% [00m:00s]                                            
  Matching rate(s):
    - 99.7647% at 0.01% uncertainty
    - 99.895% at 0.1% uncertainty
    - 99.9566% at 1% uncertainty
    - 99.9928% at 5% uncertainty
Number of track candidates (header): 53469 (host), 53389 (device)
Running comparison [==================================================] 100% [00m:00s]                                            
  Matching rate(s):
    - 99.5006% at 0.01% uncertainty
    - 99.5773% at 0.1% uncertainty
    - 99.6409% at 1% uncertainty
    - 99.697% at 5% uncertainty
  Track candidates (item) matching rate: 95.8294%
==> Statistics ... 
- read    530244 cells
- created (cpu)  147296 measurements
- created (cpu)  115674 spacepoints
- created (sycl) 115674 spacepoints     
- created  (cpu) 48545 seeds
- created (sycl) 48544 seeds
- found (cpu)    90794 tracks
- found (sycl)   90697 tracks
==>Elapsed times...
           File reading  (cpu)  759 ms
         Clusterization (sycl)  1678 ms
         Clusterization  (cpu)  22 ms
   Spacepoint formation (sycl)  273 ms
   Spacepoint formation  (cpu)  1 ms
                Seeding (sycl)  926 ms
                Seeding  (cpu)  2799 ms
           Track params (sycl)  338 ms
           Track params  (cpu)  10 ms
          Track finding (sycl)  31476 ms
          Track finding  (cpu)  6753 ms
                     Wall time  45054 ms
[bash][pcadp04]:traccc > ONEAPI_DEVICE_SELECTOR=hip:gpu ./build-sycl-amdnvidia/bin/traccc_seq_example_sycl --detector-file=geometries/odd/odd-detray_geometry_detray.json --material-file=geometries/odd/odd-detray_material_detray.json --grid-file=geometries/odd/odd-detray_surface_grids_detray.json --use-detray-detector --digitization-file=geometries/odd/odd-digi-geometric-config.json --use-acts-geom-source --input-directory=/data/Acts/odd-simulations-20240509/geant4_ttbar_mu140/ --input-events=2 --compare-with-cpu

Running Full Tracking Chain Using SYCL

>>> Detector Options <<<
  Detector file       : geometries/odd/odd-detray_geometry_detray.json
  Material file       : geometries/odd/odd-detray_material_detray.json
  Surface grid file   : geometries/odd/odd-detray_surface_grids_detray.json
  Use detray::detector: yes
  Digitization file   : geometries/odd/odd-digi-geometric-config.json
>>> Input Data Options <<<
  Use ACTS geometry source      : yes
  Input data format             : csv
  Input directory               : /data/Acts/odd-simulations-20240509/geant4_ttbar_mu140/
  Number of input events        : 2
  Number of input events to skip: 0
>>> Clusterization Options <<<
  Threads per partition:      256
  Target cells per thread:    8
  Max cells per thread:       16
  Scratch space size mult.:   256
>>> Track Seeding Options <<<
  None
>>> Track Finding Options <<<
  Max number of branches per seed: 10
  Max number of branches per surface: 10
  Track candidates range   : 3:100
  Minimum step length for the next surface: 0.5 [mm] 
  Maximum step counts for the next surface: 100
  Maximum Chi2             : 10
  Maximum branches per step: 10
  Maximum number of skipped steps per candidates: 3
  PDG Number: 13
>>> Track Propagation Options <<<
Navigation
----------------------------
  Min. mask tolerance   : 1e-05 [mm]
  Max. mask tolerance   : 1 [mm]
  Mask tolerance scalor : 0.05
  Path tolerance        : 1 [um]
  Overstep tolerance    : -100 [um]
  Search window         : 0 x 0

Parameter Transport
----------------------------
  Min. Stepsize         : 0.0001 [mm]
  Runge-Kutta tolerance : 0.0001 [mm]
  Max. step updates     : 10000
  Stepsize  constraint  : 3.40282e+38 [mm]
  Path limit            : 5 [m]
  Use Bethe energy loss : true
  Do cov. transport     : true
  Use eloss gradient    : false
  Use B-field gradient  : false

Geometry Context
----------------------------
0

>>> Performance Measurement Options <<<
  Run performance checks: no
>>> Accelerator Options <<<
  Compare with CPU results: yes

Running on device: AMD Radeon RX 6700 XT
WARNING: No material in detector

WARNING: No entries in volume finder

Detector check: OK
WARNING: No entries in volume finder

Detector check: OK
WARNING: @traccc::io::csv::read_cells: 12177 duplicate cells found in /data/Acts/odd-simulations-20240509/geant4_ttbar_mu140/event000000000-cells.csv
===>>> Event 0 <<<===
Number of measurements: 65269 (host), 65269 (device)
Running comparison [==================================================] 100% [00m:00s]                                            
  Matching rate(s):
    - 98.8816% at 0.01% uncertainty
    - 99.9525% at 0.1% uncertainty
    - 99.9556% at 1% uncertainty
    - 99.9556% at 5% uncertainty
Number of spacepoints: 51560 (host), 51560 (device)
Running comparison [==================================================] 100% [00m:00s]                                            
  Matching rate(s):
    - 98.5842% at 0.01% uncertainty
    - 99.9399% at 0.1% uncertainty
    - 99.9438% at 1% uncertainty
    - 99.9438% at 5% uncertainty
Number of seeds: 20923 (host), 20923 (device)
Running comparison [==================================================] 100% [00m:00s]                                            
  Matching rate(s):
    - 89.1937% at 0.01% uncertainty
    - 99.35% at 0.1% uncertainty
    - 99.8518% at 1% uncertainty
    - 99.8949% at 5% uncertainty
Number of track parameters: 20923 (host), 20923 (device)
Running comparison [==================================================] 100% [00m:00s]                                            
  Matching rate(s):
    - 99.8232% at 0.01% uncertainty
    - 99.8949% at 0.1% uncertainty
    - 99.9379% at 1% uncertainty
    - 99.9904% at 5% uncertainty
Number of track candidates (header): 37325 (host), 37289 (device)
Running comparison [==================================================] 100% [00m:00s]                                            
  Matching rate(s):
    - 99.6785% at 0.01% uncertainty
    - 99.7428% at 0.1% uncertainty
    - 99.8687% at 1% uncertainty
    - 99.9277% at 5% uncertainty
  Track candidates (item) matching rate: 97.5673%
WARNING: @traccc::io::csv::read_cells: 15013 duplicate cells found in /data/Acts/odd-simulations-20240509/geant4_ttbar_mu140/event000000001-cells.csv
===>>> Event 1 <<<===
Number of measurements: 82027 (host), 82027 (device)
Running comparison [==================================================] 100% [00m:00s]                                            
  Matching rate(s):
    - 98.8211% at 0.01% uncertainty
    - 99.9695% at 0.1% uncertainty
    - 99.9732% at 1% uncertainty
    - 99.9744% at 5% uncertainty
Number of spacepoints: 64114 (host), 64114 (device)
Running comparison [==================================================] 100% [00m:00s]                                            
  Matching rate(s):
    - 98.4917% at 0.01% uncertainty
    - 99.961% at 0.1% uncertainty
    - 99.9657% at 1% uncertainty
    - 99.9672% at 5% uncertainty
Number of seeds: 27622 (host), 27621 (device)
Running comparison [==================================================] 100% [00m:00s]                                            
  Matching rate(s):
    - 88.4802% at 0.01% uncertainty
    - 99.3411% at 0.1% uncertainty
    - 99.8298% at 1% uncertainty
    - 99.8769% at 5% uncertainty
Number of track parameters: 27622 (host), 27621 (device)
Running comparison [==================================================] 100% [00m:00s]                                            
  Matching rate(s):
    - 99.8081% at 0.01% uncertainty
    - 99.8986% at 0.1% uncertainty
    - 99.9602% at 1% uncertainty
    - 100% at 5% uncertainty
Number of track candidates (header): 53469 (host), 53380 (device)
Running comparison [==================================================] 100% [00m:00s]                                            
  Matching rate(s):
    - 99.5231% at 0.01% uncertainty
    - 99.5848% at 0.1% uncertainty
    - 99.6634% at 1% uncertainty
    - 99.7494% at 5% uncertainty
  Track candidates (item) matching rate: 95.878%
==> Statistics ... 
- read    530244 cells
- created (cpu)  147296 measurements
- created (cpu)  115674 spacepoints
- created (sycl) 115674 spacepoints     
- created  (cpu) 48545 seeds
- created (sycl) 48544 seeds
- found (cpu)    90794 tracks
- found (sycl)   90669 tracks
==>Elapsed times...
           File reading  (cpu)  760 ms
         Clusterization (sycl)  80 ms
         Clusterization  (cpu)  22 ms
   Spacepoint formation (sycl)  4 ms
   Spacepoint formation  (cpu)  1 ms
                Seeding (sycl)  66 ms
                Seeding  (cpu)  2752 ms
           Track params (sycl)  7 ms
           Track params  (cpu)  9 ms
          Track finding (sycl)  313 ms
          Track finding  (cpu)  6706 ms
                     Wall time  10734 ms
[bash][pcadp04]:traccc >

(Look for the "Running on device" lines. 😉)

We should merge this PR in to allow for easier debugging on the fitting algorithm, and all the other developments that I have in mind. 😉

Copy link
Contributor

@beomki-yeo beomki-yeo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me

@krasznaa krasznaa merged commit 269704d into acts-project:main Jan 16, 2025
29 checks passed
@krasznaa krasznaa deleted the SYCLTrackFindingExample-main-20250109 branch January 16, 2025 08:08
@@ -28,137 +28,141 @@ hidden.

```mermaid
flowchart LR
subgraph clusterization [<a href='https://github.com/acts-project/traccc/blob/main/core/include/traccc/clusterization/clusterization_algorithm.hpp'>Clusterization</a>]
subgraph clusterization [<a href='https://github.com/acts-project/traccc/blob/main/core/include/traccc/clusterization/'>Clusterization</a>]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI you forgot to update the paragraph above this. 😉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
examples Changes to the examples feature New feature or request sycl Changes related to SYCL
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants