How do I update parameters from the kedro notebook?

[praneeth5222]

Hello

I wanted to update the parameters with in the parameters.yml from the jupyter notebook every-time beore I do the session.run().
I was able to update the parameters from context as mentioned here but when I do the session.run() the parameters are getting loaded from catalog rather than context. Can anyone please advice me how do I update the parameters with in catalog before I run the pipeline?

Thank you!!

[noklam]

The StackOverflow you linked is 2.5 years old so it's a bit outdated. Assuming you are on Kedro 0.18.3

There are two ways to do it

1. Use the %reload_kedro magic, which creates the session object for you

%load_ext kedro.python   # Not needed if you use kedro ipython / kedro jupyter as it will do it for you
%reload_kedro --env=<env> --params=<params>  # Similar to kedro run CLI 
session.run()

2. Create the session manually

from kedro.framework.session import KedroSession
with KedroSession.create(package_name=None, project_path=None, save_on_close=True, env=None, extra_params=None):
    session.run()

tl;dr You can pass the extra_params when creating the session.

The detailed answer is, a new context is created every time you do session.run, as session only allows 1 run now, you cannot reuse the same session.

[noklam]

Did you recreate the context object? I think you are pointing to an old context that's why it isn't updated.

The ipython extension preload a few variables, you can refresh it with the %reload_kedro magic. If you choose to create a session yourself, you need to get a new context object instead of using the preloaded object.

If you can just print out the parameters within the node function, I think you should see 10.

TLDR I think the parameters is updated but you are printing the old context thus not seeing the new parameters.

[praneeth5222]

Hi @noklam
Even after running the %reload_kedro magic, the parameter is still being seen as 16 to the pipeline instead of 10 when the session runs.
At the moment, I was able to do it by initiating the pipeline from the python script rather than jupyter notebook with the same exact code. Going to give up the jupyter notebook approach

Also on the lines of parameter tuning, I also have an another question. Can you let me know if there is a way to specify multiple data folders in catalog.yml?
At the moment my data storage has a root folder which has one folder corresponding to each date like - 2022-09-30, 2022-10-01... etc. and internally each date folder has many parquet files.
I was able to load the multiple parquet files corresponding to one date folder in this way using the PartitionedDataSet-

type: PartitionedDataSet
path: "/Users/praneeth/Desktop/data/2022-09-30"
dataset: pandas.ParquetDataSet
load_args:
engine: pyarrow
use_nullable_dtypes: True

Can you suggest me if there is a way to load the specified folders instead of a single folder? (for e.g -> to load all the folders between 2022-09-26 till 2022-10-03)
I can create a new discussion for this if needed?

[noklam]

@praneeth5222 Would you be able to share an example by modifying a starter project like the iris or spaceflights one? What you have described is very unusual.

[praneeth5222]

Sure, will do that
Should I raise a new discussion about specifying multiple folders in catalog.yml file?

[noklam]

@praneeth5222 yes please