Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DataCatalog2.0]: Move pattern resolution logic to the separate component #4123

Merged
merged 89 commits into from
Sep 12, 2024
Merged
Changes from 1 commit
Commits
Show all changes
89 commits
Select commit Hold shift + click to select a range
a8f4fb3
Added a skeleton for AbstractDataCatalog and KedroDataCatalog
ElenaKhaustova Jul 31, 2024
7d56818
Removed from_config method
ElenaKhaustova Jul 31, 2024
787e121
Merge branch 'main' into refactor-pattern-logic
ElenaKhaustova Aug 2, 2024
0b80f23
Implemented _init_datasets method
ElenaKhaustova Aug 2, 2024
5c727df
Implemented get dataset
ElenaKhaustova Aug 2, 2024
05c9171
Started resolve_patterns implementation
ElenaKhaustova Aug 2, 2024
5c804d6
Implemented resolve_patterns
ElenaKhaustova Aug 5, 2024
e9ba5c4
Merge branch 'main' into refactor-pattern-logic
ElenaKhaustova Aug 5, 2024
530f7d6
Fixed credentials resolving
ElenaKhaustova Aug 5, 2024
64be83c
Updated match pattern
ElenaKhaustova Aug 6, 2024
c29828a
Implemented add from dict method
ElenaKhaustova Aug 6, 2024
957403a
Updated io __init__
ElenaKhaustova Aug 6, 2024
14908ff
Added list method
ElenaKhaustova Aug 6, 2024
c5e925b
Implemented _validate_missing_keys
ElenaKhaustova Aug 6, 2024
b9a92b0
Added datasets access logic
ElenaKhaustova Aug 7, 2024
2cb794f
Merge branch 'main' into refactor-pattern-logic
ElenaKhaustova Aug 7, 2024
2f32593
Added __contains__ and comments on lazy loading
ElenaKhaustova Aug 7, 2024
d1ea64e
Renamed dataset_name to ds_name
ElenaKhaustova Aug 8, 2024
fb89fca
Updated some docstrings
ElenaKhaustova Aug 8, 2024
4486939
Merge branch 'main' into refactor-pattern-logic
ElenaKhaustova Aug 12, 2024
c667645
Fixed _update_ds_configs
ElenaKhaustova Aug 12, 2024
be8e929
Fixed _init_datasets
ElenaKhaustova Aug 12, 2024
ec7ac39
Implemented add_runtime_patterns
ElenaKhaustova Aug 12, 2024
8e23450
Fixed runtime patterns usage
ElenaKhaustova Aug 13, 2024
529e61a
Merge branch 'main' into refactor-pattern-logic
ElenaKhaustova Aug 19, 2024
e4cb21c
Merge branch 'main' into refactor-pattern-logic
ElenaKhaustova Aug 21, 2024
50bc816
Moved pattern logic out of data catalog, implemented KedroDataCatalog
ElenaKhaustova Aug 21, 2024
6dfbcb0
Merge branch 'main' into 4110-move-pattern-resolution-logic
ElenaKhaustova Aug 22, 2024
9346f08
KedroDataCatalog updates
ElenaKhaustova Aug 22, 2024
9568e29
Added property to return config
ElenaKhaustova Aug 28, 2024
86efdfe
Merge branch 'main' into 4110-move-pattern-resolution-logic
ElenaKhaustova Aug 28, 2024
5e27660
Added list patterns method
ElenaKhaustova Aug 28, 2024
72b11d0
Renamed and moved ConfigResolver
ElenaKhaustova Aug 29, 2024
f0a4090
Renamed ConfigResolver
ElenaKhaustova Aug 29, 2024
a4da52a
Merge branch 'main' into 4110-move-pattern-resolution-logic
ElenaKhaustova Aug 29, 2024
7d6227f
Cleaned KedroDataCatalog
ElenaKhaustova Aug 29, 2024
4092291
Cleaned up DataCatalogConfigResolver
ElenaKhaustova Aug 29, 2024
63e47f9
Docs build fix attempt
ElenaKhaustova Aug 30, 2024
68f6527
Removed KedroDataCatalog
ElenaKhaustova Sep 5, 2024
2ac4a2f
Updated from_config method
ElenaKhaustova Sep 5, 2024
cb5879d
Updated constructor and add methods
ElenaKhaustova Sep 5, 2024
9038e96
Updated _get_dataset method
ElenaKhaustova Sep 5, 2024
cc89565
Updated __contains__
ElenaKhaustova Sep 5, 2024
59b6764
Updated __eq__ and shallow_copy
ElenaKhaustova Sep 5, 2024
4f5a3fb
Added __iter__ and __getitem__
ElenaKhaustova Sep 5, 2024
12ed6f2
Removed unused imports
ElenaKhaustova Sep 5, 2024
a106cec
Added TODO
ElenaKhaustova Sep 5, 2024
6df04f7
Updated runner.run()
ElenaKhaustova Sep 5, 2024
8566e27
Updated session
ElenaKhaustova Sep 5, 2024
2dcea33
Added confil_resolver property
ElenaKhaustova Sep 5, 2024
a46597f
Updated catalog list command
ElenaKhaustova Sep 5, 2024
3787545
Updated catalog create command
ElenaKhaustova Sep 5, 2024
68d612d
Updated catalog rank command
ElenaKhaustova Sep 5, 2024
af5bee9
Updated catalog resolve command
ElenaKhaustova Sep 5, 2024
acc4d6e
Merge branch 'main' into 4110-move-pattern-resolution-logic
ElenaKhaustova Sep 5, 2024
e67ff0f
Remove some methods
ElenaKhaustova Sep 5, 2024
7b3afa2
Removed ds configs from catalog
ElenaKhaustova Sep 6, 2024
658a759
Fixed lint
ElenaKhaustova Sep 6, 2024
7be2a8e
Fixed typo
ElenaKhaustova Sep 6, 2024
09f3f26
Merge branch 'main' into 4110-move-pattern-resolution-logic
ElenaKhaustova Sep 6, 2024
9e43a9a
Added module docstring
ElenaKhaustova Sep 6, 2024
25b6501
Removed None from Pattern type
ElenaKhaustova Sep 6, 2024
3a646de
Fixed docs failing to find class reference
ElenaKhaustova Sep 6, 2024
5e5df4a
Fixed docs failing to find class reference
ElenaKhaustova Sep 6, 2024
aa59a35
Updated Patterns type
ElenaKhaustova Sep 6, 2024
c7efa3e
Fix tests (#4149)
ankatiyar Sep 6, 2024
023ffc6
Returned constants to avoid breaking changes
ElenaKhaustova Sep 6, 2024
585b44f
Minor fix
ElenaKhaustova Sep 6, 2024
e447078
Updated test_sorting_order_with_other_dataset_through_extra_pattern
ElenaKhaustova Sep 9, 2024
beb0165
Merge branch 'main' into 4110-move-pattern-resolution-logic
ElenaKhaustova Sep 9, 2024
975e968
Removed odd properties
ElenaKhaustova Sep 9, 2024
11d782c
Updated tests
ElenaKhaustova Sep 9, 2024
e4abd23
Removed None from _fetch_credentials input
ElenaKhaustova Sep 9, 2024
6433dd8
Renamed DataCatalogConfigResolver to CatalogConfigResolver
ElenaKhaustova Sep 10, 2024
355576f
Renamed _init_configs to _resolve_config_credentials
ElenaKhaustova Sep 10, 2024
39d9ff6
Moved functions to the class
ElenaKhaustova Sep 10, 2024
659c9da
Refactored resolve_dataset_pattern
ElenaKhaustova Sep 10, 2024
840b32a
Fixed refactored part
ElenaKhaustova Sep 10, 2024
77f551c
Changed the order of arguments for DataCatalog constructor
ElenaKhaustova Sep 10, 2024
6e079a1
Replaced __getitem__ with .get()
ElenaKhaustova Sep 10, 2024
1f7e5f8
Updated catalog commands
ElenaKhaustova Sep 10, 2024
80f0e3d
Moved warm up block outside of the try block
ElenaKhaustova Sep 10, 2024
017cda3
Fixed linter
ElenaKhaustova Sep 10, 2024
cab6f06
Removed odd copying
ElenaKhaustova Sep 10, 2024
8f604d1
Updated release notes
ElenaKhaustova Sep 11, 2024
9a4db18
Returned DatasetError
ElenaKhaustova Sep 11, 2024
0a6946a
Added _dataset_patterns and _default_pattern to _config_resolver to a…
ElenaKhaustova Sep 11, 2024
fee7bd6
Made resolve_dataset_pattern return just dict
ElenaKhaustova Sep 11, 2024
f5a7992
Fixed linter
ElenaKhaustova Sep 11, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Moved pattern logic out of data catalog, implemented KedroDataCatalog
Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
ElenaKhaustova committed Aug 21, 2024
commit 50bc8165297030890dd05d5c0da73c237fc604b8
2 changes: 2 additions & 0 deletions kedro/config/__init__.py
Original file line number Diff line number Diff line change
@@ -7,11 +7,13 @@
BadConfigException,
MissingConfigException,
)
from .config_resolver import ConfigResolver
from .omegaconf_config import OmegaConfigLoader

__all__ = [
"AbstractConfigLoader",
"BadConfigException",
"ConfigResolver",
"MissingConfigException",
"OmegaConfigLoader",
]
237 changes: 237 additions & 0 deletions kedro/config/config_resolver.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,237 @@
import copy
import logging
import re
from typing import Any

from parse import parse

Patterns = dict[str, dict[str, Any]]

CREDENTIALS_KEY = "credentials"


def _get_credentials(credentials_name: str, credentials: dict[str, Any]) -> Any:
"""Return a set of credentials from the provided credentials dict.

Args:
credentials_name: Credentials name.
credentials: A dictionary with all credentials.

Returns:
The set of requested credentials.

Raises:
KeyError: When a data set with the given name has not yet been
registered.

"""
try:
return credentials[credentials_name]
except KeyError as exc:
raise KeyError(
f"Unable to find credentials '{credentials_name}': check your data "
"catalog and credentials configuration. See "
"https://kedro.readthedocs.io/en/stable/kedro.io.DataCatalog.html "
"for an example."
) from exc


def _resolve_credentials(
config: dict[str, Any], credentials: dict[str, Any] | None
) -> dict[str, Any]:
"""Return the dataset configuration where credentials are resolved using
credentials dictionary provided.

Args:
config: Original dataset config, which may contain unresolved credentials.
credentials: A dictionary with all credentials.

Returns:
The dataset config, where all the credentials are successfully resolved.
"""
config = copy.deepcopy(config)

def _map_value(key: str, value: Any) -> Any:
if key == CREDENTIALS_KEY and isinstance(value, str):
return _get_credentials(value, credentials)
if isinstance(value, dict):
return {k: _map_value(k, v) for k, v in value.items()}
return value

return {k: _map_value(k, v) for k, v in config.items()}


def _resolve_config(
ds_name: str,
matched_pattern: str,
config: dict,
) -> dict[str, Any]:
"""Get resolved AbstractDataset from a factory config"""
result = parse(matched_pattern, ds_name)
# Resolve the factory config for the dataset
if isinstance(config, dict):
for key, value in config.items():
config[key] = _resolve_config(ds_name, matched_pattern, value)
elif isinstance(config, (list, tuple)):
config = [_resolve_config(ds_name, matched_pattern, value) for value in config]
elif isinstance(config, str) and "}" in config:
try:
config = str(config).format_map(result.named)
except KeyError as exc:
raise KeyError(
f"Unable to resolve '{config}' from the pattern '{matched_pattern}'. Keys used in the configuration "
f"should be present in the dataset factory pattern."
) from exc
return config


class ConfigResolver:
def __init__(
self,
config: dict[str, dict[str, Any]],
credentials: dict[str, dict[str, Any]] | None = None,
):
self._runtime_patterns = {}
self._dataset_patterns, self._default_pattern = self._get_patterns(
config, credentials
)

self._ds_configs = self._get_ds_configs(config, credentials)

@property
def _logger(self) -> logging.Logger:
return logging.getLogger(__name__)

@staticmethod
def _is_pattern(pattern: str) -> bool:
"""Check if a given string is a pattern. Assume that any name with '{' is a pattern."""
return "{" in pattern

@staticmethod
def _specificity(pattern: str) -> int:
"""Helper function to check the length of exactly matched characters not inside brackets."""
# Remove all the placeholders from the pattern and count the number of remaining chars
result = re.sub(r"\{.*?\}", "", pattern)
return len(result)

@classmethod
def _sort_patterns(cls, dataset_patterns: Patterns) -> dict[str, dict[str, Any]]:
"""Sort a dictionary of dataset patterns according to parsing rules.

In order:

1. Decreasing specificity (number of characters outside the curly brackets)
2. Decreasing number of placeholders (number of curly bracket pairs)
3. Alphabetically
"""
sorted_keys = sorted(
dataset_patterns,
key=lambda pattern: (
-(cls._specificity(pattern)),
-pattern.count("{"),
pattern,
),
)
catch_all = [
pattern for pattern in sorted_keys if cls._specificity(pattern) == 0
]
if len(catch_all) > 1:
raise ValueError(
f"Multiple catch-all patterns found in the catalog: {', '.join(catch_all)}. Only one catch-all pattern is allowed, remove the extras."
)
return {key: dataset_patterns[key] for key in sorted_keys}

def match_pattern(self, ds_name: str) -> str | None:
"""Match a dataset name against patterns in a dictionary."""
all_patterns = list(self._dataset_patterns.keys())
all_patterns.extend(list(self._default_pattern.keys()))
all_patterns.extend(list(self._runtime_patterns.keys()))
matches = (pattern for pattern in all_patterns if parse(pattern, ds_name))
return next(matches, None)

@classmethod
def _get_patterns(
cls,
config: dict[str, dict[str, Any]] | None,
credentials: dict[str, dict[str, Any]] | None,
) -> tuple[Patterns, Patterns]:
dataset_patterns = {}
config = copy.deepcopy(config) or {}
credentials = copy.deepcopy(credentials) or {}
user_default = {}

for ds_name, ds_config in config.items():
if cls._is_pattern(ds_name):
resolved_ds_config = _resolve_credentials(ds_config, credentials)
dataset_patterns[ds_name] = resolved_ds_config

sorted_patterns = cls._sort_patterns(dataset_patterns)
if sorted_patterns:
# If the last pattern is a catch-all pattern, pop it and set it as the default
if cls._specificity(list(sorted_patterns.keys())[-1]) == 0:
last_pattern = sorted_patterns.popitem()
user_default = {last_pattern[0]: last_pattern[1]}

return sorted_patterns, user_default

def _get_ds_configs(
self,
config: dict[str, dict[str, Any]],
credentials: dict[str, dict[str, Any]] | None,
) -> dict[str, dict[str, Any]]:
config = copy.deepcopy(config) or {}
credentials = copy.deepcopy(credentials) or {}
ds_configs = {}
for ds_name, ds_config in config.items():
if not self._is_pattern(ds_name):
ds_configs[ds_name] = _resolve_credentials(ds_config, credentials)

return ds_configs

def resolve_patterns(
self, datasets: str | list[str]
) -> dict[str, Any] | list[dict[str, Any]]:
if isinstance(datasets, str):
datasets_lst = [datasets]
else:
datasets_lst = datasets

resolved_configs = []

for ds_name in datasets_lst:
matched_pattern = self.match_pattern(ds_name)
if matched_pattern and ds_name not in self._ds_configs:
# If the dataset is a patterned dataset, materialise it and add it to
# the catalog
config_copy = copy.deepcopy(
self._dataset_patterns.get(matched_pattern)
or self._default_pattern.get(matched_pattern)
or self._runtime_patterns.get(matched_pattern)
or {}
)
ds_config = _resolve_config(ds_name, matched_pattern, config_copy)

if (
self._specificity(matched_pattern) == 0
and matched_pattern in self._default_pattern
):
self._logger.warning(
"Config from the dataset factory pattern '%s' in the catalog will be used to "
"override the default dataset creation for '%s'",
matched_pattern,
ds_name,
)
resolved_configs.append(ds_config)
elif ds_name in self._ds_configs:
resolved_configs.append(self._ds_configs.get(ds_name))
else:
resolved_configs.append(None)

if isinstance(datasets, str):
return resolved_configs[0]
else:
return resolved_configs

def add_runtime_patterns(self, dataset_patterns: Patterns) -> None:
self._runtime_patterns = {**self._runtime_patterns, **dataset_patterns}
self._runtime_patterns = self._sort_patterns(self._runtime_patterns)
3 changes: 1 addition & 2 deletions kedro/io/__init__.py
Original file line number Diff line number Diff line change
@@ -14,7 +14,7 @@
Version,
)
from .data_catalog import DataCatalog
from .data_catalog_redesign import AbstractDataCatalog, KedroDataCatalog
from .data_catalog_redesign import KedroDataCatalog
from .lambda_dataset import LambdaDataset
from .memory_dataset import MemoryDataset
from .shared_memory_dataset import SharedMemoryDataset
@@ -31,6 +31,5 @@
"MemoryDataset",
"SharedMemoryDataset",
"Version",
"AbstractDataCatalog",
"KedroDataCatalog",
]
Loading