Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable distributed cube computing #58

Closed
forman opened this issue May 8, 2019 · 2 comments · Fixed by #60
Closed

Enable distributed cube computing #58

forman opened this issue May 8, 2019 · 2 comments · Fixed by #60
Assignees
Labels
enhancement New feature or request important This is very important for the project urgent High external pressure to address this ASAP xcube gen This is related to data cube generation, CLI "xcube gen" xcube serve This is related to server component, CLI "xcube serve"

Comments

@forman
Copy link
Member

forman commented May 8, 2019

Is your feature request related to a problem? Please describe.

xcube serve supports datasets that are computed on-the-fly, e.g. applying temporal aggregation. This requires high data throughput and CPU resources, therefore requests based on computed datasets often respond too slow and then time out. Such computations may be much faster if computation is distributed on a dedicated cluster.

xcube gen may be parallelized so that individual input files (usually spatial time slices) are transformed in a distributed way and then results are combined chronologically into the desired cube.

The same may apply to other xcube commands that perform heavy computing on chunks of data.

Describe the solution you'd like

Add option to xcube CLI commands that configure how the command is executed in a distributed manner. The option is TBD.

Describe alternatives you've considered

None, besides using larger machines.

Additional context

As we use xarray and xarray uses dask, the solution should be based on Dask Distributed.

@forman forman added enhancement New feature or request urgent High external pressure to address this ASAP important This is very important for the project xcube serve This is related to server component, CLI "xcube serve" xcube gen This is related to data cube generation, CLI "xcube gen" labels May 8, 2019
@forman forman self-assigned this May 8, 2019
@forman forman added the in progress The assignee is working on it label May 8, 2019
@forman
Copy link
Member Author

forman commented May 8, 2019

Started this so we can start using Dask.distributed for first tests.

@forman
Copy link
Member Author

forman commented May 8, 2019

forman added a commit that referenced this issue May 8, 2019
@forman forman mentioned this issue May 8, 2019
@forman forman removed the in progress The assignee is working on it label May 13, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request important This is very important for the project urgent High external pressure to address this ASAP xcube gen This is related to data cube generation, CLI "xcube gen" xcube serve This is related to server component, CLI "xcube serve"
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant