ploomber-core telemetry breaks jupysql when executing on a read only filesystem

[dancoates]

We use papermill to execute jupyter notebooks on AWS lambda functions. The AWS Lambda filesystem is readonly except for the /tmp directory. When the jupysql extension is loaded the telemetry.py file in ploomber-core attempts to create a folder at ~/.ploomber by default. This throws an error and causes the execution of the notebook to fail.

This can be pretty easily worked around by setting:

import os
os.environ["PLOOMBER_HOME_DIR"] = "/tmp/.ploomber"

But it would be good if the telemetry.py file in ploomber-core either wasn't executed at all, or if it does need to be executed, if it more gracefully handled the situation where it is unable to create the config directory.

[edublancas]

thanks for the report! we'll take a look and fix it

btw, check out ploomber-engine, it's a drop-in replacement for papermill with enhanced capabilities!

[dancoates]

Thanks! and thanks for moving, wasn't sure whether to make this in jupysql or ploomber-core.
Testing out ploomber-engine is definitely on the to do list, great to see improvements in the notebook execution space

[mehtamohit013]

AC Criteria:

• In case of unable to write, use the default config, which would contain the path to tmp and use the file there as telemetry and init the different telemetry each time with different uuid
• test case
• Changelog

[edublancas]

In case of unable to write, use the default config, which would contain the path to tmp and use the file there as telemetry and init the different telemetry each time with different uuid

the default value isn't the /tmp/ directory, it's a different one. but that's fine. the idea is that the Config class should work ok in read-only filesystems. it should just skip the filesystem operations and initialize with the default values every time

[mehtamohit013]

In case of unable to write, use the default config, which would contain the path to tmp and use the file there as telemetry and init the different telemetry each time with different uuid

the default value isn't the /tmp/ directory, it's a different one. but that's fine. the idea is that the Config class should work ok in read-only filesystems. it should just skip the filesystem operations and initialize with the default values every time

Update: I understand it now

Yes, I understand that. But as far as I understood config class, all the functions are written such that, they write to a file. So, I need somewhere to write the file. This is the preliminary understanding, i am still exploring it.

Do you want me to change config class, or make a new class such that we don't have to write to a file, even a tmpfile and all the operations can be done in-memory?

[edublancas]

hey @dancoates, we just released ploomber-core 0.2.13 with this fix. if the filesystem is read-only, it won't attempt to write the file. if you encounter any other problems, feel free to open an issue!

[dancoates]

Fantastic, thank you for the fix @edublancas