Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Toniof 516 unify datasetio #527

Merged
merged 14 commits into from
Sep 21, 2021
Merged

Toniof 516 unify datasetio #527

merged 14 commits into from
Sep 21, 2021

Conversation

TonioF
Copy link
Contributor

@TonioF TonioF commented Sep 14, 2021

This PR addresses #516. It ensures that, if a Configuration with Datasets is passed to a ServiceContext, each dataset is assigned to a datastore that will be used to open the data (unless the dataset is in-memory). It is attempted to find common roots for data stores to group datasets located at same/similar locations together.

When reviewing this, please consider especially the function get_data_store_pool of the ServerContext: This function used to remove a data store pool if there were no datastores explicitly configured. I had to change this, but I don't fully understand what was the original reasoning behind this.

Checklist:

  • Add unit tests and/or doctests in docstrings
  • [ ] Add docstrings and API docs for any new/modified user-facing classes and functions
  • [ ] New/modified features documented in docs/source/*
  • Changes documented in CHANGES.md
  • AppVeyor CI passes
  • Test coverage remains or increases (target 100%)

Remember to close associated issues after merge!

@TonioF TonioF requested a review from forman September 14, 2021 06:54
@TonioF TonioF self-assigned this Sep 14, 2021
@codecov-commenter
Copy link

codecov-commenter commented Sep 14, 2021

Codecov Report

Merging #527 (9f49034) into master (1b53e69) will increase coverage by 0.17%.
The diff coverage is 92.48%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #527      +/-   ##
==========================================
+ Coverage   91.51%   91.69%   +0.17%     
==========================================
  Files         288      288              
  Lines       26271    26426     +155     
==========================================
+ Hits        24043    24230     +187     
+ Misses       2228     2196      -32     
Impacted Files Coverage Δ
test/webapi/test_config.py 100.00% <ø> (ø)
xcube/webapi/config.py 100.00% <ø> (ø)
xcube/webapi/context.py 76.99% <85.57%> (+7.66%) ⬆️
xcube/core/store/storepool.py 97.56% <91.66%> (+0.17%) ⬆️
test/core/store/test_storepool.py 100.00% <100.00%> (ø)
test/webapi/test_context.py 100.00% <100.00%> (ø)
xcube/webapi/handlers.py 92.01% <100.00%> (+0.29%) ⬆️
xcube/core/mldataset.py 78.61% <0.00%> (-0.87%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1b53e69...9f49034. Read the comment docs.

Copy link
Member

@forman forman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe, this is very good!

However I have some trouble understanding the base algorithm implemented here. The code urgently needs some explanation, may it be docstrings, comments, or by refactorings. Please have a look at the comments.

return self._dataset_configs

def _maybe_assign_store_instance_ids(self):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Long, complex method. Please refactor and/or add comments so one can understand the algorithm implementation. It takes me too much time to reverse-engineer the intentions coded here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, I tried.

Comment on lines 265 to 269
def get_store_instance_id(self,
store_config: DataStoreConfig,
strict_check: bool = False
) \
-> Optional[str]:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code style:

Suggested change
def get_store_instance_id(self,
store_config: DataStoreConfig,
strict_check: bool = False
) \
-> Optional[str]:
def get_store_instance_id(self,
store_config: DataStoreConfig,
strict_check: bool = False
) \
-> Optional[str]:
Suggested change
def get_store_instance_id(self,
store_config: DataStoreConfig,
strict_check: bool = False
) \
-> Optional[str]:
def get_store_instance_id(
self,
store_config: DataStoreConfig,
strict_check: bool = False
) -> Optional[str]:

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GitHub has a bug here: it doubles the suggestions! Only the second is valid!

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It even doubles every comment!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just wanted to check our code style for this issue. I did not find it, I just recall we already talked about this before.

But are you certain that the suggestion below is right? Going to the lowest indention level with the method header?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did not find it, I just recall we already talked about this before.

PEP-8. And yes, we talked about this before, see https://www.python.org/dev/peps/pep-0008/#indentation

Going to the lowest indention level with the method header?

No that's not right, it happened per accident.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, changed it according to pep8

@TonioF TonioF requested a review from forman September 14, 2021 13:22
Copy link
Member

@forman forman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good, let's talk about _maybe_assign_store_instance_ids(self) tomorrow!

@dzelge dzelge merged commit 61815c3 into master Sep 21, 2021
dzelge added a commit that referenced this pull request Sep 21, 2021
TonioF added a commit that referenced this pull request Sep 21, 2021
@TonioF TonioF mentioned this pull request Sep 21, 2021
4 tasks
@TonioF TonioF deleted the toniof-516-unify-datasetio branch March 14, 2022 14:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants