Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor dependencies in pyproject.toml #32

Merged
merged 5 commits into from
May 6, 2024
Merged

Refactor dependencies in pyproject.toml #32

merged 5 commits into from
May 6, 2024

Conversation

andersy005
Copy link
Collaborator

@andersy005 andersy005 commented May 6, 2024

  • Refactored dependencies in pyproject.toml. The original monolithic list has been split into optional dependencies under 'pangeo-forge' and 'catalog'.
  • Added 'complete' directive under optional dependencies for encapsulating all dependencies.

I believe these changes will facilitate better dependecies management (for e.g. i would like to use the cataloging features without needing to install apache-beam, etc...)

@andersy005 andersy005 requested a review from jbusecke May 6, 2024 18:07
@jbusecke
Copy link
Collaborator

jbusecke commented May 6, 2024

Ah nice! Thanks @andersy005. Looks good to me!

@jbusecke jbusecke merged commit cd86889 into leap-stc:main May 6, 2024
6 checks passed
@andersy005 andersy005 deleted the group-dependencies-into-optional-dependencies branch May 6, 2024 20:56
@andersy005
Copy link
Collaborator Author

andersy005 commented May 6, 2024

it turns out that the imports in

from .cmip_transforms import CMIPBQInterface, IIDEntry, LogCMIPToBigQuery
from .data_management_transforms import BQInterface, RegisterDatasetToCatalog
__all__ = (
need to be updated for this to work. currently,

from leap_data_management_utils.catalog import main

goes through init.py and this triggers import of all dependencies (includding those i wanted to avoid (apache-beam, etc))

@jbusecke
Copy link
Collaborator

jbusecke commented May 6, 2024

? I am not quite sure I understand.

@andersy005
Copy link
Collaborator Author

? I am not quite sure I understand.

when you execute the command pip install leap-data-management["catalog"], it installs both the base dependencies and the additional catalog dependencies. however, attempting to import the package with the following code:

from leap_data_management_utils import catalog

triggers the invocation of imports within the leap_data_management_utils.__init__ file, which currently references objects from subpackages, such as cmip_transforms.py and data_management_transforms.py.

@andersy005
Copy link
Collaborator Author

andersy005 commented May 6, 2024

what is desired is the ability to install leap-data-management["catalog"] and subsequently import the catalog subpackage without initiating imports from any other subpackages. does this make sense? :0

@jbusecke
Copy link
Collaborator

jbusecke commented May 8, 2024

Ahh thanks for clarifying. I think we can take those imports out of the init.py. It will break some of my stuff downstream, but should be super easy to fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants