Keep data in memory after _save
#4001
astrojuanlu
started this conversation in
Idea
Replies: 1 comment
-
Looks like the answer to this is just to use https://docs.kedro.org/en/stable/api/kedro.io.CachedDataset.html test_ds:
type: CachedDataset
dataset:
type: pandas.CSVDataset
filepath: example.csv |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
When declaring a dataset in the catalog, the data will be serialized (good). However, at the moment looks like it's an either/or situation: the user needs to choose between serializing the data and passing it in memory. The problem is that users many times want to do both. Typical reasons:
_load
method implemented, so it requires hacksThis week I was asked about this twice already by different users. It has been mentioned several times but I cannot locate specific issues or Slack conversations.
Beta Was this translation helpful? Give feedback.
All reactions