Add ROOT-based event exporter #900

stognini · 2023-08-17T20:56:26Z

This PR adds the option to export Celeritas' primaries to a ROOT file and read those primaries back into Celeritas.

RootOffloadWriter writes data to a ROOT file using basic types to avoid the use of dictionaries.
RootOffloadReader reads back the data in an event-by-event basis.

The reader can be used by a future class RootPrimaryGenerator final : public G4VPrimaryGenerator { };.

The use of a ROOT file considerably decreases the output file size, as expected. For the standard celer-g4 test using celeritas/app/data/ttbarbits-12evt-118prim.hepmc3:

1.9M  simple-cms.offloaded.hepmc3
357K  simple-cms.offloaded.root

- Missing PDG from writer; EventReader has to be updated for that - Revert back to 1 root entry == 1 primary - celer-g4 has test code that has to be cleaned

- Refactor class names - Move to accel/detail - Writer uses a vector<Primary> as input - WIP: refactor reader to return vector<Primary> from ROOT

sethrj · 2023-08-17T20:59:53Z

src/accel/detail/RootOffloadReader.hh

+// See the top-level COPYRIGHT file for details.
+// SPDX-License-Identifier: (Apache-2.0 OR MIT)
+//---------------------------------------------------------------------------//
+//! \file accel/detail/RootOffloadReader.hh


This should be celeritas/io/EventReader since we will need to use it from the celer-sim app.

sethrj · 2023-08-17T21:00:50Z

src/accel/detail/RootOffloadWriter.hh

+ *
+ * One TTree entry represents one primary.
+ */
+class RootOffloadWriter


Likewise since this class doesn't depend on anything in accel and we can use it for writing out whatever primaries, put this in celeritas/io as RootEventWriter. The only thing that OffloadWriter did was to wrap a class for thread safety.

sethrj · 2023-08-17T21:02:50Z

src/accel/detail/RootOffloadWriter.cc

+{
+    CELER_EXPECT(!primaries.empty());
+
+    std::scoped_lock{write_mutex_};


You should remove this. The OffloadWriter is the only thing that needs to be thread safe. If we need thread safety outside what ROOT provides we can use the existing OffloadWriter and make its EventWriter a generic std::function<void(vector<Primary>)>. Also note that the way the reader is structured, you MUST lock out the file when writing an event, otherwise primaries from different events could be interleaved.

Just to be sure I got your point: so you're going to lock the operator() operation itself, whether that's a HepMC3 or ROOT writer?

Right now the OffloadWriter in accel has a single purpose: to wrap an EventWriter with a thread safe lock. I think we'll use that same mechanism to wrap the RootWriter, since for the reading to work, we have to write all tracks consecutively for a single event.

src/celeritas/ext/RootUniquePtr.root.cc

sethrj · 2023-08-17T21:06:23Z

src/accel/SharedParams.hh

+    {
+        return root_offload_writer_;
+    }
+


Delete all this code; we should be using the same offload writer interface: either HepMC3 or ROOT, we don't need separate copies of both. I just wanted you to add the RootEvent{Reader,Writer} class and I was going to add in a separate PR the extra bit of std::functional and "make a different writer based on the output extension" code needed to hook it in.

- Refactor RootOffload{Writer, Reader} -> RootEvent{Writer, Reader} - Move to celeritas/io - Remove root offload option from SharedParams/SetupOptionMessenger

sethrj · 2023-08-19T11:43:48Z

src/accel/CMakeLists.txt

+
+  if (CELERITAS_USE_ROOT)
+    list(APPEND PRIVATE_DEPS ROOT::Tree)
+  endif()


This can be reverted since the ROOT code is no longer in accel.

@stognini You might have missed this comment earlier?

sethrj · 2023-08-19T11:44:15Z

src/accel/SetupOptions.hh

@@ -112,6 +112,8 @@ struct SetupOptions
    std::string physics_output_file;
    //! Filename to dump a HepMC3 copy of offloaded tracks as events
    std::string offload_output_file;
+    //! Filename to dump a ROOT copy of offloaded tracks as events
+    std::string offload_root_output_file;


Remove this; we'll reuse offload_output_file and switch format based on the file extension.

Also this one?

Ahh crap. Thanks for noticing this. I checked out back a few files from a previous commit just to test a celer-g4 run and undid my fixes. I'll clean this up.

sethrj · 2023-08-19T11:45:06Z

src/celeritas/ext/RootUniquePtr.root.cc

@@ -16,8 +16,7 @@ namespace celeritas
 {
 //---------------------------------------------------------------------------//
 /*!
- * Call `TObject->Write()` before deletion. Used by TFile and TTree writer
- * classes.
+ * Call `TObject->Write()` before deletion.


This would be a good place to mention that Write doesn't do what one would expect 😒

src/celeritas/ext/RootUniquePtr.root.cc

sethrj · 2023-08-19T11:49:11Z

src/celeritas/io/EventReader.cc

-
-        // Set the registered ID of the particle
-        primary.particle_id = particle_id;
-


I don't understand the use case for having an event reader that can't read the particle type. What can we do with primaries that are unknown particles? In all of the applications I can imagine, we'll be loading events into a simulation (or a test) that defines particle types already. I think the changes to EventReader.* should be reverted.

@stognini I think you missed this comment too.

src/celeritas/io/RootEventReader.cc

sethrj · 2023-08-19T11:53:13Z

src/celeritas/io/RootEventReader.cc

+ */
+auto RootEventReader::operator()() -> result_type
+{
+    CELER_EXPECT(entry_count_ <= num_entries_);


Add another ScopedRootErrorHandler here too.

src/celeritas/io/RootEventReader.cc

sethrj · 2023-08-19T12:05:17Z

src/celeritas/io/RootEventReader.cc

+//---------------------------------------------------------------------------//
+/*!
+ * Helper function to fetch leaves.
+ */
+template<class T>
+auto RootEventReader::from_leaf(char const* leaf_name) -> T
+{
+    CELER_EXPECT(ttree_);
+    auto const leaf = ttree_->GetLeaf(leaf_name);
+    CELER_ASSERT(leaf);
+    return static_cast<T>(leaf->GetValue());
+}
+
+//---------------------------------------------------------------------------//
+/*!
+ * Helper function to fetch leaves containing `std::array<double, 3>`.
+ */
+Real3 RootEventReader::from_array_leaf(char const* leaf_name)
+{
+    CELER_EXPECT(ttree_);
+    auto const leaf = ttree_->GetLeaf(leaf_name);
+    CELER_ASSERT(leaf);
+    CELER_ASSERT(leaf->GetLen() == 3);
+    return {leaf->GetValue(0), leaf->GetValue(1), leaf->GetValue(2)};
+}


Depending on how often we write a non-dictionary ROOT file it might be h

class RootLeafReader { explicit RootLeafReader(TTree* tree); template<class T> void operator()(char const* leaf_name, T* value) { /* default implementation */ } template<class T, class N> void from_leaf(char const* leaf_name, Array<T, N>* value) { /* call GetValue on each item */ } template<class T> void from_leaf(char const* leaf_name, OpaqueId<T>* value) {...}

so that you can make a local RootLeafReader read_leaf{ttree_.get()}; read_leaf("pos", p.pos); etc.

Probably not worth it for this small file but something to keep in mind.

sethrj · 2023-08-19T12:58:49Z

@stognini I refactored the event IO test (originally used for HepMC3 only) so that it can be reused for testing the ROOT reader and writer. Please take a look at the output: I think we're missing (1) the check to ensure that all primaries being written at once have the same event ID and (2) resetting the track IDs when reading back in. In fact I think we can entirely delete the track IDs from the ROOT output since we have to reset them sequentially anyway.

stognini · 2023-08-21T17:16:50Z

I refactored the event IO test (originally used for HepMC3 only) so that it can be reused for testing the ROOT reader and write

Thanks!

In fact I think we can entirely delete the track IDs from the ROOT output since we have to reset them sequentially anyway.

Agreed. removed track_id branch from ROOT and manually set them during reading.

Now there is one detail that I am not sure how to proceed with the test: The event_id in EventWriter is set manually and sequentially, whereas the RootEventWriter initially simply assigned whatever event_id is stored in celeritas::Primary at the time, leading to the different results shown in the test. I have added an event id counter to the RootEventWriter to manually match the exact behavior of EventWriter. Let me know if that is the way you prefer, or if I should amend the test and keep the Root{reader/writer} as previously (i.e. simply accepting that whatever information in vector<Primary> is correct to begin with).

- Update RootEventWriter constructor to use a SPRootFileManager, as the RootFileManager needs to be initialized or have a default constructor. - Fix missing bits when CELERITAS_USE_ROOT=OFF

sethrj · 2023-08-22T16:36:12Z

The event_id in EventWriter is set manually and sequentially, whereas the RootEventWriter initially simply assigned whatever event_id is stored in celeritas::Primary at the time, leading to the different results shown in the test. I have added an event id counter to the RootEventWriter to manually match the exact behavior of EventWriter. Let me know if that is the way you prefer, or if I should amend the test and keep the Root{reader/writer} as previously (i.e. simply accepting that whatever information in vector is correct to begin with).

One big problem with the previous method is that interleaving the event IDs breaks the reader. Keeping the exact same behavior as the HepMC3 writer limits us, but it ensures no surprises when switching between offload_output_file = "foo.hepmc3"; and offload_output_file = "foo.root";

sethrj · 2023-08-23T01:47:45Z

src/accel/CMakeLists.txt

+
+  if (CELERITAS_USE_ROOT)
+    list(APPEND PRIVATE_DEPS ROOT::Tree)
+  endif()


@stognini You might have missed this comment earlier?

sethrj · 2023-08-23T01:47:51Z

src/accel/SetupOptions.hh

@@ -112,6 +112,8 @@ struct SetupOptions
    std::string physics_output_file;
    //! Filename to dump a HepMC3 copy of offloaded tracks as events
    std::string offload_output_file;
+    //! Filename to dump a ROOT copy of offloaded tracks as events
+    std::string offload_root_output_file;


Also this one?

src/celeritas/io/EventReader.hh

sethrj · 2023-08-23T01:50:28Z

src/celeritas/io/EventReader.cc

-
-        // Set the registered ID of the particle
-        primary.particle_id = particle_id;
-


@stognini I think you missed this comment too.

src/celeritas/io/RootEventReader.cc

src/celeritas/io/RootEventReader.hh

sethrj

Looks good to me, sorry this took so long. I'll hook it up to accel next.

stognini added 10 commits August 10, 2023 09:29

Draft dump to root option to HepMC3PrimaryGenerator.cc

3eff886

Move functionality to a separate class

a591721

Change to per-event tree entries

6a919c6

Add HepMC3RootReader; WIP on other fronts

c7039fa

- Missing PDG from writer; EventReader has to be updated for that - Revert back to 1 root entry == 1 primary - celer-g4 has test code that has to be cleaned

Update HepMC3RootReader to return events

389d5cb

Reorg writer/reader

9fdd0c5

- Refactor class names - Move to accel/detail - Writer uses a vector<Primary> as input - WIP: refactor reader to return vector<Primary> from ROOT

Complete reader

b514630

Add UPRootAutoSave custom deleter; Clean up code

58a32be

Add offloadRootOutputFile macro cmd option

5365993

Merge branch 'develop' into dump-primaries

bc4d486

stognini requested review from sethrj and pcanal August 17, 2023 20:56

stognini added enhancement New feature or request io labels Aug 17, 2023

sethrj requested changes Aug 17, 2023

View reviewed changes

stognini added 5 commits August 17, 2023 20:12

Address comments

d822921

- Refactor RootOffload{Writer, Reader} -> RootEvent{Writer, Reader} - Move to celeritas/io - Remove root offload option from SharedParams/SetupOptionMessenger

Use template specialization for UPRootWritable<TTree>

c14d2fb

Remove mutex from writer/reader

b0e9f43

Fix types

8db8f19

Merge branch 'develop' into dump-primaries

2ba1132

sethrj requested changes Aug 19, 2023

View reviewed changes

sethrj added 3 commits August 19, 2023 08:51

Refactor event IO test for reusability

2b68f96

Add root IO test

473b5c8

Prevent move/copy for event reader and writer

95d6885

stognini added 4 commits August 19, 2023 14:55

Fix missing symbols error

e0bd0ab

Update root unique ptrs

5b5340e

Improve reader; Add ScopedRootErrorHandler; Remove TrackId branch

0bfede8

Add scoped error handler; clean up

3ac3ced

stognini added 3 commits August 21, 2023 13:17

Update RootEventWriter to store contiguous event ids

9f99cff

Fixup

798b407

- Update RootEventWriter constructor to use a SPRootFileManager, as the RootFileManager needs to be initialized or have a default constructor. - Fix missing bits when CELERITAS_USE_ROOT=OFF

Fixup; Minor improvements

47c7618

Merge branch 'develop' into dump-primaries

ea48c6a

sethrj mentioned this pull request Aug 23, 2023

Refactor event IO test for reusability #902

Merged

sethrj requested changes Aug 23, 2023

View reviewed changes

sethrj and others added 2 commits August 23, 2023 09:38

Merge remote-tracking branch 'upstream/develop' into dump-primaries

b5fa3d9

Revert EventReader; Address comments

1ec6e33

sethrj approved these changes Aug 24, 2023

View reviewed changes

sethrj merged commit 5e0e42c into celeritas-project:develop Aug 24, 2023

sethrj changed the title ~~Add capability to export Celeritas primaries as a ROOT file~~ Add ROOT-based event exporter Aug 24, 2023

stognini deleted the dump-primaries branch August 24, 2023 13:16

sethrj added external Dependencies and framework-oriented features and removed io labels Nov 23, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add ROOT-based event exporter #900

Add ROOT-based event exporter #900

stognini commented Aug 17, 2023 •

edited

Loading

sethrj Aug 17, 2023

sethrj Aug 17, 2023

sethrj Aug 17, 2023

stognini Aug 18, 2023

sethrj Aug 18, 2023

sethrj Aug 17, 2023

sethrj Aug 19, 2023

sethrj Aug 23, 2023

sethrj Aug 19, 2023

sethrj Aug 23, 2023

stognini Aug 23, 2023 •

edited

Loading

sethrj Aug 19, 2023

sethrj Aug 19, 2023

sethrj Aug 23, 2023

sethrj Aug 19, 2023

sethrj Aug 19, 2023

sethrj commented Aug 19, 2023

stognini commented Aug 21, 2023 •

edited

Loading

sethrj commented Aug 22, 2023

sethrj Aug 23, 2023

sethrj Aug 23, 2023

sethrj Aug 23, 2023

sethrj left a comment


		// Set the registered ID of the particle
		primary.particle_id = particle_id;

Add ROOT-based event exporter #900

Add ROOT-based event exporter #900

Conversation

stognini commented Aug 17, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stognini Aug 23, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sethrj commented Aug 19, 2023

stognini commented Aug 21, 2023 • edited Loading

sethrj commented Aug 22, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sethrj left a comment

Choose a reason for hiding this comment

stognini commented Aug 17, 2023 •

edited

Loading

stognini Aug 23, 2023 •

edited

Loading

stognini commented Aug 21, 2023 •

edited

Loading