-
Notifications
You must be signed in to change notification settings - Fork 910
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add hook example to access metadata
#2998
Changes from 5 commits
f10bede
fe72ced
da45b9d
8338928
0108fac
389ce36
91e5e48
976016d
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
|
@@ -200,3 +200,22 @@ HOOKS = (AzureSecretsHook(),) | |||||
```{note} | ||||||
Note: `DefaultAzureCredential()` is Azure's recommended approach to authorise access to data in your storage accounts. For more information, consult the [documentation about how to authenticate to Azure and authorize access to blob data](https://learn.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-python). | ||||||
``` | ||||||
|
||||||
## Use Hook to read `metadata` from `DataCatalog` | ||||||
We recommend to use the `after_catalog_created` hook if you need to access `metadata` to extend Kedro. | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
||||||
```python | ||||||
class MetadataHook: | ||||||
@hook_impl | ||||||
def after_catalog_created( | ||||||
self, | ||||||
catalog: DataCatalog, | ||||||
conf_catalog: Dict[str, Any], | ||||||
conf_creds: Dict[str, Any], | ||||||
feed_dict: Dict[str, Any], | ||||||
save_version: str, | ||||||
load_versions: Dict[str, str], | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. All these parameters are not used right? Maybe we can remove them from the function. Pluggy allows this There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Fair point. We do this inconsistently, sometimes we do include signature that wasn't used (I was asked to add that before, can't find it exactly which PR it is). For example, There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @stichbury do you think we should keep all our example minimals? We can do a clean up if this is agreed. In this case
or
will both work. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think it's fine to remove them too and keep it minimal. |
||||||
): | ||||||
for dataset_name, dataset in catalog.datasets.__dict__.items(): | ||||||
print(f"{dataset_name} metadata: \n {str(dataset.metadata)}") | ||||||
``` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.