Build fails with "Object of type PosixPath is not JSON serializable" #113

michaelweinold · 2024-08-25T07:40:05Z

Description

When adding our bw2io package to the environment.yaml as a dependency, I get an error of this kind:

TypeError: Object of type PosixPath is not JSON serializable

Interestingly, the error does not reference the package, nor any of its dependencies. The error trace goes from the micromamba build library to jupyterlite_core/manager > jupyterlite_xeus/add_on > empack/pack and then to the Pyhon JSON decoder.

Any idea what the cause might be?

Reproduce

Run build workflow at michaelweinold/jl_xeus_test

Expected behavior

No error 😇

Context

JupyterLite version: jupyterlite-core=0.3.0, jupyterlite-xeus=0.1.9
Operating System and version: N/A
Browser and version: N/A

Build Log

Transaction finished
To activate this environment, use:
micromamba activate xeus-python-kernel

Or to execute a single command in this environment, use:
micromamba run -n xeus-python-kernel mycommand

[LiteBuildApp] ERROR | [lite] [post_build] [jupyterlite-xeus] [ERR] Object of type PosixPath is not JSON serializable

Traceback (most recent call last):

File "/home/runner/micromamba/envs/build-env/lib/python3.11/site-packages/doit/doit_cmd.py", line 294, in run

return command.parse_execute(args)

^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/home/runner/micromamba/envs/build-env/lib/python3.11/site-packages/doit/cmd_base.py", line 150, in parse_execute

return self.execute(params, args)

^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/home/runner/micromamba/envs/build-env/lib/python3.11/site-packages/doit/cmd_base.py", line 570, in execute

return self._execute(**exec_params)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/home/runner/micromamba/envs/build-env/lib/python3.11/site-packages/doit/cmd_run.py", line 265, in _execute

return runner.run_all(self.control.task_dispatcher())

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/home/runner/micromamba/envs/build-env/lib/python3.11/site-packages/doit/runner.py", line 254, in run_all

self.run_tasks(task_dispatcher)

File "/home/runner/micromamba/envs/build-env/lib/python3.11/site-packages/doit/runner.py", line 213, in run_tasks

node = task_dispatcher.generator.send(node)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/home/runner/micromamba/envs/build-env/lib/python3.11/site-packages/doit/control.py", line 629, in _dispatcher_generator

next_step = node.step()

^^^^^^^^^^^

File "/home/runner/micromamba/envs/build-env/lib/python3.11/site-packages/doit/control.py", line 336, in step

return next(self.generator)

^^^^^^^^^^^^^^^^^^^^

File "/home/runner/micromamba/envs/build-env/lib/python3.11/site-packages/doit/control.py", line 345, in _func

for value in decorated(*args, **kwargs):

File "/home/runner/micromamba/envs/build-env/lib/python3.11/site-packages/doit/control.py", line 473, in _add_task

new_tasks = generate_tasks(to_load, task_gen, ref.doc)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/home/runner/micromamba/envs/build-env/lib/python3.11/site-packages/doit/loader.py", line 390, in generate_tasks

for task_dict, x_doc in flat_generator(gen_result, gen_doc):

File "/home/runner/micromamba/envs/build-env/lib/python3.11/site-packages/doit/loader.py", line 27, in flat_generator

for item in gen:

File "/home/runner/micromamba/envs/build-env/lib/python3.11/site-packages/jupyterlite_core/manager.py", line 138, in _delayed_gather

yield from _gather()

File "/home/runner/micromamba/envs/build-env/lib/python3.11/site-packages/jupyterlite_core/manager.py", line 131, in _gather

raise error

File "/home/runner/micromamba/envs/build-env/lib/python3.11/site-packages/jupyterlite_core/manager.py", line 123, in _gather

for task in getattr(addon, attr)(self):

File "/home/runner/micromamba/envs/build-env/lib/python3.11/site-packages/jupyterlite_xeus/add_on.py", line 127, in post_build

yield from self.copy_kernels_from_prefix()

File "/home/runner/micromamba/envs/build-env/lib/python3.11/site-packages/jupyterlite_xeus/add_on.py", line 179, in copy_kernels_from_prefix

yield from self.copy_kernel(kernel_dir, kernel_wasm, kernel_js)

File "/home/runner/micromamba/envs/build-env/lib/python3.11/site-packages/jupyterlite_xeus/add_on.py", line 261, in copy_kernel

yield from self.pack_prefix(kernel_dir=kernel_dir)

File "/home/runner/micromamba/envs/build-env/lib/python3.11/site-packages/jupyterlite_xeus/add_on.py", line 288, in pack_prefix

pack_env(

File "/home/runner/micromamba/envs/build-env/lib/python3.11/site-packages/empack/pack.py", line 186, in pack_env

pack_pkg_impl(

File "/home/runner/micromamba/envs/build-env/lib/python3.11/site-packages/empack/pack.py", line 54, in pack_pkg_impl

json.dump(pkg_meta, f)

File "/home/runner/micromamba/envs/build-env/lib/python3.11/json/init.py", line 179, in dump

for chunk in iterable:

File "/home/runner/micromamba/envs/build-env/lib/python3.11/json/encoder.py", line 432, in _iterencode

yield from _iterencode_dict(o, _current_indent_level)

File "/home/runner/micromamba/envs/build-env/lib/python3.11/json/encoder.py", line 406, in _iterencode_dict

yield from chunks

File "/home/runner/micromamba/envs/build-env/lib/python3.11/json/encoder.py", line 326, in _iterencode_list

yield from chunks

File "/home/runner/micromamba/envs/build-env/lib/python3.11/json/encoder.py", line 439, in _iterencode

o = _default(o)

^^^^^^^^^^^

File "/home/runner/micromamba/envs/build-env/lib/python3.11/json/encoder.py", line 180, in default

raise TypeError(f'Object of type {o.class.name} '

TypeError: Object of type PosixPath is not JSON serializable

The text was updated successfully, but these errors were encountered:

michaelweinold · 2024-08-25T07:41:57Z

And @martinRenou, I guess you have some bots to block (see above) 🙈

michaelweinold · 2024-08-25T07:53:03Z

Since the main error seems to originate with this line:

conda_meta_filename = f"{filename_base_from_meta(pkg_meta)}.json"
    with open(filtered_prefix / "conda-meta" / conda_meta_filename, "w") as f:
        json.dump(pkg_meta, f)

Did we (in our Brightway packages) use some non-standard package name?

DerThorsten · 2024-08-25T08:13:19Z

I am a bit surprised by this error. It seems like a missing str(thePath) somewhere. But I would not expect this to happen for certain packages only.

michaelweinold · 2024-08-25T08:27:07Z

from empack import filter_env

path_env = "/opt/homebrew/Caskroom/miniconda/base/envs/xeus-python-kernel-test"

list_pkg_meta = [pkg_meta for pkg_meta in filter_env.iterate_env_pkg_meta(path_env)]

for pkg_meta in list_pkg_meta:
    tempfile = "/tmp/tmp.json"
    with open(tempfile, 'w') as file:
        json.dump(pkg_meta, file)

works fine on my local machine. Weird 🙈

michaelweinold · 2024-08-25T13:13:30Z

I found the bug - the issue originates from the iterate_pip_pkg_record() function in combination with the better_exceptions_fork package.

Running iterate_pip_pkg_record() while passing it the path to the temporary directory which is generated by the jupyter lite build command:

from empack import filter_env
[pip_pkg for pip_pkg in filter_env.iterate_pip_pkg_record("(...)/tmpfwwgtj1q/env/envs/xeus-python-kernel")]

returns this dictionary:

{'name': 'better_exceptions_fork',
 'version': '0.2.1.post6',
 'files': [PosixPath('lib/python3.11/site-packages/better_exceptions/__init__.py'),
  PosixPath('lib/python3.11/site-packages/better_exceptions/__init__.pyc'),
  PosixPath('lib/python3.11/site-packages/better_exceptions/__main__.py'),
  PosixPath('lib/python3.11/site-packages/better_exceptions/__main__.pyc'),
  PosixPath('lib/python3.11/site-packages/better_exceptions/color.py'),
  PosixPath('lib/python3.11/site-packages/better_exceptions/color.pyc'),
  PosixPath('lib/python3.11/site-packages/better_exceptions/formatter.py'),
  PosixPath('lib/python3.11/site-packages/better_exceptions/formatter.pyc'),
  PosixPath('lib/python3.11/site-packages/better_exceptions/highlighter.py'),
  PosixPath('lib/python3.11/site-packages/better_exceptions/highlighter.pyc'),
  PosixPath('lib/python3.11/site-packages/better_exceptions/log.py'),
  PosixPath('lib/python3.11/site-packages/better_exceptions/log.pyc'),
  PosixPath('lib/python3.11/site-packages/better_exceptions/repl.py'),
  PosixPath('lib/python3.11/site-packages/better_exceptions/repl.pyc'),
  PosixPath('lib/python3.11/site-packages/better_exceptions_fork/__init__.py'),
  PosixPath('lib/python3.11/site-packages/better_exceptions_fork/__init__.pyc'),
  PosixPath('lib/python3.11/site-packages/better_exceptions_fork/__main__.py'),
  PosixPath('lib/python3.11/site-packages/better_exceptions_fork/__main__.pyc'),
  PosixPath('lib/python3.11/site-packages/better_exceptions_fork/color.py'),
  PosixPath('lib/python3.11/site-packages/better_exceptions_fork/color.pyc'),
  PosixPath('lib/python3.11/site-packages/better_exceptions_fork/formatter.py'),
  PosixPath('lib/python3.11/site-packages/better_exceptions_fork/formatter.pyc'),
  PosixPath('lib/python3.11/site-packages/better_exceptions_fork/highlighter.py'),
  PosixPath('lib/python3.11/site-packages/better_exceptions_fork/highlighter.pyc'),
  PosixPath('lib/python3.11/site-packages/better_exceptions_fork/log.py'),
  PosixPath('lib/python3.11/site-packages/better_exceptions_fork/log.pyc'),
  PosixPath('lib/python3.11/site-packages/better_exceptions_fork/repl.py'),
  PosixPath('lib/python3.11/site-packages/better_exceptions_fork/repl.pyc'),
  PosixPath('lib/python3.11/site-packages/better_exceptions_hook.pth')],
 'fn': 'better_exceptions_fork-0.2.1.post6',
 'build': 'pip',
 'build_number': 0,
 'depends': []}

...and that cannot be serialized to JSON.

I'll see if I can figure out how to mitigate this.

michaelweinold · 2024-08-27T05:21:14Z

Why is better_exceptions_fork even installed into the environment? It is not a dependency of bw2io or xeus-python - and apparently also not of any of the packages in build-environment.yml. Odd that it would not be there if I just use the standard recipe of the template:

name: xeus-python-kernel
channels:
  - https://repo.mamba.pm/emscripten-forge
  - conda-forge
dependencies:
  - xeus-python
  - numpy
  - matplotlib

...but only appears if I add bw2io.

michaelweinold added the bug Something isn't working label Aug 25, 2024

jupyterlite deleted a comment Aug 25, 2024

michaelweinold mentioned this issue Aug 25, 2024

Change list-of-POSIX-paths to list-of-strings in yield of iterate_pip_pkg_record() function emscripten-forge/empack#104

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build fails with "Object of type PosixPath is not JSON serializable" #113

Build fails with "Object of type PosixPath is not JSON serializable" #113

michaelweinold commented Aug 25, 2024

michaelweinold commented Aug 25, 2024

michaelweinold commented Aug 25, 2024

DerThorsten commented Aug 25, 2024

michaelweinold commented Aug 25, 2024

michaelweinold commented Aug 25, 2024 •

edited

Loading

michaelweinold commented Aug 27, 2024

Build fails with "Object of type PosixPath is not JSON serializable" #113

Build fails with "Object of type PosixPath is not JSON serializable" #113

Comments

michaelweinold commented Aug 25, 2024

Description

Reproduce

Expected behavior

Context

michaelweinold commented Aug 25, 2024

michaelweinold commented Aug 25, 2024

DerThorsten commented Aug 25, 2024

michaelweinold commented Aug 25, 2024

michaelweinold commented Aug 25, 2024 • edited Loading

michaelweinold commented Aug 27, 2024

michaelweinold commented Aug 25, 2024 •

edited

Loading