Starting work on pre-processing and data reader

[vuillaut]

Hi.
I just open a thread to discuss the developments to be done for the reading and pre-processing of the produced files.
As shown in a ASWG conf call, I have a prototype reader for LST that I use with the standard analysis chain.
Also as discussed in Madrid, @mikael10j has done a lot of work for the pre-processing of images.

I think for a start, we should try to list the developments that we want to see done and order them by priorities.
I will start with:

• develop a reader for produced HDF5 files into ctapipe containers
• this reader should be set as a plug-in for ctapipe
• this has to be related to use more ctapipe functionality #31 from early stages to decide where we go

[nietootein]

Thank you, Thomas, for opening this issue. For your convenience I'm pasting here Thomas' developments on github (please, correct me if the branch is wrong):

https://github.com/vuillaut/cta-lstchain/tree/read_hdf5_dl1

[aribrill]

I have started developing a reader. The idea is for loading the data into a format usable by Tensorflow/PyTorch, as discussed in Madrid. The data loading is basically complete, pending some final testing. The image mapping is a placeholder intended to be replaced with the functions currently in CTLearn and the preprocessing is essentially a placeholder since I only implemented a few trivial functions.

This seems a bit different though. To clarify, you are discussing a reader of the DL1 files into a ctapipe container, correct? So there would be two different types of reader/data loader? However, they could share the same pre-processing and image mapping functions.

[vuillaut]

This seems a bit different though. To clarify, you are discussing a reader of the DL1 files into a ctapipe container, correct? So there would be two different types of reader/data loader? However, they could share the same pre-processing and image mapping functions.

You are right.
There should be different tasks or even project there:

• pre-processing
• data reader for Tensorlow/Pytorch
• data reader for ctapipe

[mikael10j]

I have started developing a reader. The idea is for loading the data into a format usable by Tensorflow/PyTorch, as discussed in Madrid. The data loading is basically complete, pending some final testing. The image mapping is a placeholder intended to be replaced with the functions currently in CTLearn and the preprocessing is essentially a placeholder since I only implemented a few trivial functions.

How can I help ?

[aribrill]

@mikael10j I will make a PR for the reader in the next couple days, after I finish up some more testing on it. Then the next steps will be to:

1. Implement preprocessing methods (starting with those already in GammaLearn/CTLearn)
2. Merge image mapping methods from CTLearn
3. Add image shifting mapping method (i.e. for hexagonal convolution)
4. Integrate the DL1DH reader into CTLearn
5. Integrate the DL1DH reader into GammaLearn

I would much appreciate your help for tasks 1, 3, and 4 especially, in addition to (of course) review, revision, testing, and debugging in general.

[bryankim96]

Work on #2 has started in the shift_imagemapper branch. I'll continue working on it this week and hopefully make a PR soon. I think it would be fine to begin work on #4 (and #5) already in branches in the respective repositories, as the final API of DL1DH is already settled and the remaining work on #2 is mostly fixing bugs. I'm not exactly sure what #3 entails, but it seems like it would also not require holding back #4 or #5.

[TjarkMiener]

Solved even before v0.10.0.