Enable distributed cube computing #58
Labels
enhancement
New feature or request
important
This is very important for the project
urgent
High external pressure to address this ASAP
xcube gen
This is related to data cube generation, CLI "xcube gen"
xcube serve
This is related to server component, CLI "xcube serve"
Is your feature request related to a problem? Please describe.
xcube serve
supports datasets that are computed on-the-fly, e.g. applying temporal aggregation. This requires high data throughput and CPU resources, therefore requests based on computed datasets often respond too slow and then time out. Such computations may be much faster if computation is distributed on a dedicated cluster.xcube gen
may be parallelized so that individual input files (usually spatial time slices) are transformed in a distributed way and then results are combined chronologically into the desired cube.The same may apply to other xcube commands that perform heavy computing on chunks of data.
Describe the solution you'd like
Add option to xcube CLI commands that configure how the command is executed in a distributed manner. The option is TBD.
Describe alternatives you've considered
None, besides using larger machines.
Additional context
As we use xarray and xarray uses dask, the solution should be based on Dask Distributed.
The text was updated successfully, but these errors were encountered: