Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Catalogs handler #3

Open
8 of 11 tasks
aguinot opened this issue Jan 22, 2023 · 0 comments
Open
8 of 11 tasks

Catalogs handler #3

aguinot opened this issue Jan 22, 2023 · 0 comments

Comments

@aguinot
Copy link
Owner

aguinot commented Jan 22, 2023

Build a Catalog class to deal with everything related to catalog call. The catalog will be handle through dask. This will require to write high-level method to make it easier to use and more transparent for the user.
Using dask will allow to use large catalogs transparently. Also, dask does more than catalog handling. The computation on the catalog are done either on a LocalCluster or a user-defined one. Some computations are automatically done in parallel. For more specific things it will be necessary to write wrapper around existing functions to make use of all the capability of dask.
The catalogs are read making use of the vaex library to handle .fits files. Also, vaex use dask under the hood which make this choice very easy. Also, vaex can open multiple catalogs at once as long as they follow the same format. We will make use of this feature.

Note: by using dask all the computation are "lazy". That means that until you call .compute() method nothing actually happen. Only the tree of the computation is done.

  • Read catalog using vaex
    • Allow the convertion to .hdf5 format to allow memory mapping (not possible from .fits format)
    • Store the converted catalog in the workspace directory
    • Make it possible to remove the converted file to save space on disk
    • Keep track of the catalog of origin for each objects
  • Read catalog from config file
  • Make it possible to instantiate a catalog from a path or a list of path
  • Make a parent class
    • Build a children class for galaxy catalog
    • Build a children class for star catalog
  • Handle star catalogs from MCCD (-> convert global positions to local, maybe this should be done in MCCD)
@aguinot aguinot changed the title Catalog handler Catalogs handler Jan 22, 2023
This was referenced Jan 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant