Update event_map for detray detector

[beomki-yeo]

Some breaking changes

1. test_clsterization_resolution is disabled now. I don't know exactly why but there is a mismatch between modules from spacepoint_formation and truth one. I believe @krasznaa can follow the logics in read_spacepoint and fix later
2. Some mapper tests with tml geometry are deleted. csv geometry is not supported anymore...
3. test_cca is also disabled because of missing measurement_id header in the old cell csv data format. @stephenswat can follow up this later as well.
4. For the same reason, all tml data test with cell csv are deleted due to missing measurement_id header
5. I believe mapper is pretty error-prone right now because of the switch between measurement_id and hit_id, and all the fixes that I made are under the assumption that two IDs are identical, which may not be true. Thus, @krasznaa if you find something wrong with the performance, I suggest investigating hit_map in mapper at the first place

In the near future we will need to wipe out all old data format and stop supporting them in traccc. mapper or event_map also needs to be rewritten from scratch for better maintenance.

[krasznaa]

While doing this, could you also extend seq_example_cuda.cpp with the track finding and fitting performance writers? 🤔 I thought I already did, but apparently not.

[beomki-yeo]

@krasznaa The seeding and track finding performance check are working now (See the results at below). You can change the condition for particle filtering at

traccc/performance/src/efficiency/finding_performance_writer.cpp

Line 141 in 9cf6eb1

if (ptc.charge == 0 || getter::perp(ptc.mom) < m_cfg.pT_cut) {

Note that event_map currently does not support the fitting performance check. To make it work out of box, you might need to add meas_xp_map as for event_map2.

---

Benchmark Log

[beomki@pc-3 traccc_build]$ ./bin/traccc_seq_example_cuda --detector-file=geometries/odd/odd-detray_geometry_detray.json --grid-file=geometries/odd/odd-detray_surface_grids_detray.json --use-detray-detector --digitization-file=geometries/odd/odd-digi-geometric-config.json --input-directory=odd/geant4_10muon_1GeV/ --input-events=1 --check-performance --compare-with-cpu

Running Full Tracking Chain Using CUDA

>>> Detector Options <<<
  Detector file       : geometries/odd/odd-detray_geometry_detray.json
  Material file       : 
  Surface rid file    : geometries/odd/odd-detray_surface_grids_detray.json
  Use detray::detector: yes
  Digitization file   : geometries/odd/odd-digi-geometric-config.json
>>> Input Data Options <<<
  Input data format             : csv
  Input directory               : odd/geant4_10muon_1GeV/
  Number of input events        : 1
  Number of input events to skip: 0
>>> Clusterization Options <<<
  Target cells per partition: 1024
>>> Track Seeding Options <<<
  None
>>> Track Finding Options <<<
  Track candidates range   : 3:100
  Minimum step length for the next surface: 0.5 [mm] 
  Maximum step counts for the next surface: 100
  Maximum Chi2             : 30
  Maximum branches per step: 4294967295
  Maximum number of skipped steps per candidates: 3
>>> Track Propagation Options <<<
  Constraint step size  : 3.40282e+38 [mm]
  Overstep tolerance    : -100 [um]
  Minimum mask tolerance: 1e-05 [mm]
  Maximum mask tolerance: 1 [mm]
  Search window         : 0 x 0
  Runge-Kutta tolerance : 0.0001
>>> Performance Measurement Options <<<
  Run performance checks: yes
>>> Accelerator Options <<<
  Compare with CPU results: yes

WARNING: No material in detector
WARNING: No entries in volume finder
Detector check: OK
WARNING: No material in detector
WARNING: No entries in volume finder
Detector check: OK
WARNING: @traccc::io::csv::read_cells: 230 duplicate cells found in /home/beomki/projects/traccc/traccc/data/odd/geant4_10muon_1GeV/event000000000-cells.csv
===>>> Event 0 <<<===
Number of measurements: 595 (host), 595 (device)
  Matching rate(s):
    - 100% at 0.01% uncertainty
    - 100% at 0.1% uncertainty
    - 100% at 1% uncertainty
    - 100% at 5% uncertainty
Number of spacepoints: 595 (host), 595 (device)
  Matching rate(s):
    - 100% at 0.01% uncertainty
    - 100% at 0.1% uncertainty
    - 100% at 1% uncertainty
    - 100% at 5% uncertainty
Number of seeds: 119 (host), 119 (device)
  Matching rate(s):
    - 100% at 0.01% uncertainty
    - 100% at 0.1% uncertainty
    - 100% at 1% uncertainty
    - 100% at 5% uncertainty
Number of track parameters: 119 (host), 119 (device)
  Matching rate(s):
    - 100% at 0.01% uncertainty
    - 100% at 0.1% uncertainty
    - 100% at 1% uncertainty
    - 100% at 5% uncertainty
Number of track candidates (header): 128 (host), 128 (device)
  Matching rate(s):
    - 100% at 0.01% uncertainty
    - 100% at 0.1% uncertainty
    - 100% at 1% uncertainty
    - 100% at 5% uncertainty
  Track candidates (item) matching rate: 100%
Number of track states: 128 (host), 128 (device)
  Matching rate(s):
    - 5.46875% at 0.01% uncertainty
    - 18.75% at 0.1% uncertainty
    - 47.6562% at 1% uncertainty
    - 71.0938% at 5% uncertainty
WARNING: No material in detector
WARNING: No entries in volume finder
Detector check: OK
WARNING: @traccc::io::csv::read_cells: 230 duplicate cells found in /home/beomki/projects/traccc/traccc/data/odd/geant4_10muon_1GeV/event000000000-cells.csv
==> Statistics ... 
- read    1531 cells from 520 modules
- created (cpu)  595 measurements     
- created (cuda)  595 measurements     
- created (cpu)  595 spacepoints     
- created (cuda) 595 spacepoints     
- created  (cpu) 119 seeds
- created (cuda) 119 seeds
- found (cpu)    128 tracks
- found (cuda)   128 tracks
- fitted (cpu)   128 tracks
- fitted (cuda)  128 tracks
==>Elapsed times...
           File reading  (cpu)  9 ms
         Clusterization (cuda)  36 ms
   Spacepoint formation (cuda)  1 ms
         Clusterization  (cpu)  0 ms
   Spacepoint formation  (cpu)  0 ms
                Seeding (cuda)  14 ms
                Seeding  (cpu)  1 ms
           Track params (cuda)  6 ms
           Track params  (cpu)  0 ms
          Track finding (cuda)  3292 ms
           Track finding (cpu)  329 ms
          Track fitting (cuda)  2124 ms
           Track fitting (cpu)  237 ms
                     Wall time  6056 ms

Track finding efficiency

[krasznaa]

I can overlook the discomfort of making the I/O code yet more complicated, since there I see a path for simplification in the future. But I'm really uncomfortable with switching off the clusterization unit tests.

Especially since the CUDA and SYCL tests are still in place. I imagine those would still fail at the moment. 🤔

[krasznaa]

This is pretty bad. 😦 How come we can't read both cell data types now? Then why does the current main branch read the new CSV cell files without errors? (Even if one of the columns changed names.)

[beomki-yeo]

I don't know. Feel free to fix the PR if you find the reason.
This is what I get when running the test in this PR

[  FAILED  ] SparseCclAlgorithm/ConnectedComponentAnalysisTests.Run/trackml_like_0000000028, where GetParam() = (32-byte object <10-FE 29-01 00-00 00-00 70-DB 2B-01 00-00 00-00 E0-51 46-00 00-00 00-00 20-59 46-00 00-00 00-00>, "trackml_like_0000000028") (0 ms)
[ RUN      ] SparseCclAlgorithm/ConnectedComponentAnalysisTests.Run/trackml_like_0000000029
unknown file: Failure
C++ exception with description "Missing header column 'measurement_id'" thrown in the test body.

[beomki-yeo]

I also don't see the error in the main branch, but the hit_id is always zero for ODD data which drops the efficiency to zero as you saw in the mattermost.

[krasznaa]

Let's keep this one open for the moment. 🤔 I'll try it out shortly from your branch.

Worst case scenario, we make some measurements using this branch as a temporary solution, and then we discuss how to implement all of these things in a slightly more pleasing way. 🤔

[beomki-yeo]

Sure

[beomki-yeo]

Don't remember what it was about, but it seems overlapping with #692