Error on using laconic-so commands

[nikugogoi]

• On trying to use latest SO release for deploying MobyMask, got error with laconic-so version command

  $ laconic-so version
  Traceback (most recent call last):
    File "/home/dev/bin/laconic-so/_bootstrap/__init__.py", line 76, in import_string
    File "/home/dev/.shiv/laconic-so_1d1696dfb73c0ef92795f7b95b3d6bac18c9edb389eb82952720f9e3760cb229/site-packages/cli.py", line 20, in <module>
      from app import build_containers
    File "/home/dev/.shiv/laconic-so_1d1696dfb73c0ef92795f7b95b3d6bac18c9edb389eb82952720f9e3760cb229/site-packages/app/build_containers.py", line 31, in <module>
      from app.base import get_npm_registry_url
    File "/home/dev/.shiv/laconic-so_1d1696dfb73c0ef92795f7b95b3d6bac18c9edb389eb82952720f9e3760cb229/site-packages/app/base.py", line 18, in <module>
      from app.deploy import get_stack_status
    File "/home/dev/.shiv/laconic-so_1d1696dfb73c0ef92795f7b95b3d6bac18c9edb389eb82952720f9e3760cb229/site-packages/app/deploy.py", line 26, in <module>
      from python_on_whales import DockerClient, DockerException
    File "/home/dev/.shiv/laconic-so_1d1696dfb73c0ef92795f7b95b3d6bac18c9edb389eb82952720f9e3760cb229/site-packages/python_on_whales/__init__.py", line 1, in <module>
      from .client_config import ClientNotFoundError
    File "/home/dev/.shiv/laconic-so_1d1696dfb73c0ef92795f7b95b3d6bac18c9edb389eb82952720f9e3760cb229/site-packages/python_on_whales/client_config.py", line 10, in <module>
      import pydantic
    File "/home/dev/.shiv/laconic-so_1d1696dfb73c0ef92795f7b95b3d6bac18c9edb389eb82952720f9e3760cb229/site-packages/pydantic/__init__.py", line 3, in <module>
      import pydantic_core
    File "/home/dev/.shiv/laconic-so_1d1696dfb73c0ef92795f7b95b3d6bac18c9edb389eb82952720f9e3760cb229/site-packages/pydantic_core/__init__.py", line 6, in <module>
      from ._pydantic_core import (
  ModuleNotFoundError: No module named 'pydantic_core._pydantic_core'
  
  During handling of the above exception, another exception occurred:
  
  Traceback (most recent call last):
    File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
      return _run_code(code, main_globals, None,
    File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
      exec(code, run_globals)
    File "/home/dev/bin/laconic-so/__main__.py", line 3, in <module>
    File "/home/dev/bin/laconic-so/_bootstrap/__init__.py", line 253, in bootstrap
    File "/home/dev/bin/laconic-so/_bootstrap/__init__.py", line 81, in import_string
    File "/home/dev/bin/laconic-so/_bootstrap/__init__.py", line 59, in import_string
    File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
      return _bootstrap._gcd_import(name[level:], package, level)
    File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
    File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
    File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
    File "<frozen importlib._bootstrap_external>", line 883, in exec_module
    File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
    File "/home/dev/.shiv/laconic-so_1d1696dfb73c0ef92795f7b95b3d6bac18c9edb389eb82952720f9e3760cb229/site-packages/cli.py", line 20, in <module>
      from app import build_containers
    File "/home/dev/.shiv/laconic-so_1d1696dfb73c0ef92795f7b95b3d6bac18c9edb389eb82952720f9e3760cb229/site-packages/app/build_containers.py", line 31, in <module>
      from app.base import get_npm_registry_url
    File "/home/dev/.shiv/laconic-so_1d1696dfb73c0ef92795f7b95b3d6bac18c9edb389eb82952720f9e3760cb229/site-packages/app/base.py", line 18, in <module>
      from app.deploy import get_stack_status
    File "/home/dev/.shiv/laconic-so_1d1696dfb73c0ef92795f7b95b3d6bac18c9edb389eb82952720f9e3760cb229/site-packages/app/deploy.py", line 26, in <module>
      from python_on_whales import DockerClient, DockerException
    File "/home/dev/.shiv/laconic-so_1d1696dfb73c0ef92795f7b95b3d6bac18c9edb389eb82952720f9e3760cb229/site-packages/python_on_whales/__init__.py", line 1, in <module>
      from .client_config import ClientNotFoundError
    File "/home/dev/.shiv/laconic-so_1d1696dfb73c0ef92795f7b95b3d6bac18c9edb389eb82952720f9e3760cb229/site-packages/python_on_whales/client_config.py", line 10, in <module>
      import pydantic
    File "/home/dev/.shiv/laconic-so_1d1696dfb73c0ef92795f7b95b3d6bac18c9edb389eb82952720f9e3760cb229/site-packages/pydantic/__init__.py", line 3, in <module>
      import pydantic_core
    File "/home/dev/.shiv/laconic-so_1d1696dfb73c0ef92795f7b95b3d6bac18c9edb389eb82952720f9e3760cb229/site-packages/pydantic_core/__init__.py", line 6, in <module>
      from ._pydantic_core import (
  ModuleNotFoundError: No module named 'pydantic_core._pydantic_core'
  

  Error was thrown on using other laconic-so commands too

• Checked with other releases of laconic-so:

  • The error seems to be introduced in release v1.1.0-8eed5a7-202307310428
  • Release v1.1.0-758ec16-202307300438 prior to that is working and not throwing the error

[dboreham]

This doesn't appear to be related to any recent commit. I'm wondering if it might be some build environment thing?

This may be related: pydantic/pydantic#6557

[dboreham]

btw, the problematic release works for me (although it does introduce a weird warning message from the same library):

$ ./laconic-so version
/home/david/.shiv/laconic-so_adfaa4046412af789761876fd25d439b3a10be5cb12b0acd910a72a86e5ad6f0/site-packages/pydantic/_internal/_config.py:269: UserWarning: Valid config keys have changed in V2:
* 'allow_population_by_field_name' has been renamed to 'populate_by_name'
  warnings.warn(message, UserWarning)
Version: 1.1.0-8eed5a7-202307310428

[dboreham]

This seems to be some error from a mystery meat transitive dependency. The error's manifestation depends on the version of Python used.

[dboreham]

Should be fixed in the latest release, please re-test.

[nikugogoi]

Tested with latest release and getting same error.

The error's manifestation depends on the version of Python used.

What python version should we use?

Current python version

$ python3 --version
Python 3.10.6

[dboreham]

stack orchestrator supports Python 3.8 and later so 3.10.6 is supported.

[dboreham]

It looks like just updating our first order dependencies to the latest wasn't enough to pull in the updated pydantic.
I'll force its version explicitly.

[dboreham]

The underlying cause is : gabrieldemarmiesse/python-on-whales#455

[dboreham]

Ok, hopefully this one #473 should get us going again.

I don't fully understand what happened, but roughly: we use Docker on Whales which is a Python library for docker.
That library uses a library called pydantic, the purpose of which I don't understand fully, but it appears to involve extreme cleverness.
Pydantic people re-wrote their code totally, making pydantic v2.
Yesterday docker on whales people merged a PR that uses pydantic v2 by default.
This appears to produce two problems : spurious warning messages emitted to stdout/stderr from pydantic code; and exceptions thrown if you execute with some version of Python vs the version used to build the shiv package.

The second problem actually sounds pretty ominous because it has the feel that pydantic v2 is not "python-version-neutral". This would be a big problem for us because we are trying to ship a version (minor version) neutral package.

I've been able to force pydantic back to v1 for now, which works around the problem.

[dboreham]

Should finally fix the issue: #473