-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Catalogs handler #3
Comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Build a Catalog class to deal with everything related to catalog call. The catalog will be handle through
dask
. This will require to write high-level method to make it easier to use and more transparent for the user.Using
dask
will allow to use large catalogs transparently. Also,dask
does more than catalog handling. The computation on the catalog are done either on aLocalCluster
or a user-defined one. Some computations are automatically done in parallel. For more specific things it will be necessary to write wrapper around existing functions to make use of all the capability ofdask
.The catalogs are read making use of the
vaex
library to handle.fits
files. Also,vaex
usedask
under the hood which make this choice very easy. Also,vaex
can open multiple catalogs at once as long as they follow the same format. We will make use of this feature.Note: by using
dask
all the computation are "lazy". That means that until you call.compute()
method nothing actually happen. Only the tree of the computation is done.vaex
.hdf5
format to allow memory mapping (not possible from.fits
format)The text was updated successfully, but these errors were encountered: