Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unify xcube dataset i/o #516

Open
4 of 6 tasks
TonioF opened this issue Sep 8, 2021 · 2 comments
Open
4 of 6 tasks

Unify xcube dataset i/o #516

TonioF opened this issue Sep 8, 2021 · 2 comments
Assignees
Labels
code Issue is related to code refactoring or optimisation enhancement New feature or request in progress The assignee is working on it xcube data stores This is related to data store framework or data store implementations xcube gen This is related to data cube generation, CLI "xcube gen" xcube serve This is related to server component, CLI "xcube serve"

Comments

@TonioF
Copy link
Contributor

TonioF commented Sep 8, 2021

Currently there are parts in the xcube dataset i/o that do not use the data store framework but try to access data on their own. This leads to redundant code and an increased maintenance effort. These places should be changed so that the data store framework is used throughout and there is a single source of truth. This will result in less code, less maintenance, and a more comprehensible xcube API

  • Use store framework consistently in xcube gen2
  • Use store framework consistently in xcube serve
  • Use store framework consistently in other xcube cli tools (gen, chunk, level, ...)
  • Remove xcube.core.dsio.DatasetIO and derivates

EDIT

  • Deprecate xcube.core.mldataset.FileStorageMultiLevelDataset
  • Deprecate xcube.core.mldataset.ObjectStorageMultiLevelDataset
@TonioF TonioF self-assigned this Sep 8, 2021
@forman forman added code Issue is related to code refactoring or optimisation enhancement New feature or request in progress The assignee is working on it labels Sep 9, 2021
@TonioF TonioF mentioned this issue Sep 14, 2021
4 tasks
@TonioF TonioF mentioned this issue Sep 21, 2021
4 tasks
@forman
Copy link
Member

forman commented Oct 4, 2021

See #458

@forman forman added xcube serve This is related to server component, CLI "xcube serve" xcube gen This is related to data cube generation, CLI "xcube gen" xcube data stores This is related to data store framework or data store implementations labels Mar 15, 2022
@forman
Copy link
Member

forman commented Mar 15, 2022

The only usage of xcube.core.mldataset.open_ml_dataset() seems to be the xcube level tool, see #617.

If we can replace this by data store API calls, we also can get rid of

  • xcube.core.mldataset.open_ml_dataset_from_object_storage() and xcube.core.mldataset.ObjectStorageMultiLevelDataset
  • xcube.core.mldataset.open_ml_dataset_from_local_fs() and xcube.core.mldataset.FileStorageMultiLevelDataset

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
code Issue is related to code refactoring or optimisation enhancement New feature or request in progress The assignee is working on it xcube data stores This is related to data store framework or data store implementations xcube gen This is related to data cube generation, CLI "xcube gen" xcube serve This is related to server component, CLI "xcube serve"
Projects
None yet
Development

No branches or pull requests

2 participants