Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

reader: event identifers building with multithreading #59

Open
mikael10j opened this issue Jun 11, 2019 · 1 comment
Open

reader: event identifers building with multithreading #59

mikael10j opened this issue Jun 11, 2019 · 1 comment
Assignees

Comments

@mikael10j
Copy link

During the building of event identifiers we may apply some intense filters. It would be nice to do it in a multithreaded way to save time.

mikael10j pushed a commit to mikael10j/dl1-data-handler that referenced this issue Jun 20, 2019
@mikael10j mikael10j self-assigned this Jun 20, 2019
@TjarkMiener
Copy link
Member

TjarkMiener commented Dec 10, 2021

Hi @mikael10j
Thanks a lot for the issue and the suggested solution. Some points: With v0.10.0, the DL1b data level is included in both the official CTA DL1 data model and the deprecated DL1DH data format (supports multiple data of the DL1b level). The building of the example identifiers is rather quick and can be even bypassed by saving the example identifiers to disk when running more than one random seed. Besides this, Johan had some bad experiences, when working with the deprecated DL1DHWriter (using 'parallel' mode via Python multithreading) on the grid back in the days #51. Since most of our future experiments will be run on GPU clusters, we would probably face similar problems, even though one can also select more supporting CPUs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants