Pip and conda install pydantic 2.0.2 by default, that causes ImportError #18027

simonreise · 2023-07-08T11:02:39Z

Bug description

I am developing library that requires lightning dependency. I am trying to build it using conda-build, but it crashes on testing with error ImportError: cannot import name 'ModelMetaclass' from 'pydantic.main'. It also crashes with the same error when I tried to test it with tox in appveyor in python 3.8, 3.9, 3.10 and 3.11 on both windows and ubuntu.

I also tried to create new clean venv in pycharm, install lightning via pip and import lightning shows the same error.

Looks like pip and conda install pydantic 2.0.2 by default, but in this version 'ModelMetaclass' is not located in 'pydantic.main' anymore.

Manual installation of pydantic <2.0 or adding requirement pydantic <2.0 to requirements in my setup.py and meta.yaml fixes the problem.

UPD: also I found out that in wheel from pypi in METADATA file pydantic dependency is Requires-Dist: pydantic (<4.0,>=1.7.4), in tar.gz from pypi in requirements/base.txt it is pydantic<3.0,>=1.7.4 and in requirements/app/base.txt it is pydantic >=1.7.4, <2.0.0. conda search lightning --info also shows dependency - pydantic <4.0,>=1.7.4. I think it was supposed to be <2.0, not <4.0?

What version are you seeing the problem on?

v2.0

How to reproduce the bug

1. Create new python venv
2. pip install lightning
3. import lightning

Error messages and logs

ImportError while importing test module 'C:\projects\remote-sensing-processor\tests\test_landsat.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
C:\Python311-x64\Lib\importlib\__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
test_landsat.py:4: in <module>
    import remote_sensing_processor as rsp
..\.tox\python3.11\Lib\site-packages\remote_sensing_processor\__init__.py:21: in <module>
    from remote_sensing_processor import segmentation
..\.tox\python3.11\Lib\site-packages\remote_sensing_processor\segmentation\__init__.py:3: in <module>
    from remote_sensing_processor.segmentation.segmentation import segmentation_train, segmentation_test
..\.tox\python3.11\Lib\site-packages\remote_sensing_processor\segmentation\segmentation.py:12: in <module>
    import lightning as l
..\.tox\python3.11\Lib\site-packages\lightning\__init__.py:18: in <module>
    from lightning.app import storage  # noqa: E402
..\.tox\python3.11\Lib\site-packages\lightning\app\__init__.py:24: in <module>
    from lightning.app import components  # noqa: E402, F401
..\.tox\python3.11\Lib\site-packages\lightning\app\components\__init__.py:1: in <module>
    from lightning.app.components.database.client import DatabaseClient
..\.tox\python3.11\Lib\site-packages\lightning\app\components\database\__init__.py:1: in <module>
    from lightning.app.components.database.client import DatabaseClient
..\.tox\python3.11\Lib\site-packages\lightning\app\components\database\client.py:22: in <module>
    from lightning.app.components.database.utilities import _GeneralModel
..\.tox\python3.11\Lib\site-packages\lightning\app\components\database\utilities.py:23: in <module>
    from pydantic.main import ModelMetaclass
E   ImportError: cannot import name 'ModelMetaclass' from 'pydantic.main' (C:\projects\remote-sensing-processor\.tox\python3.11\Lib\site-packages\pydantic\main.py)

Environment

pip list after pip install lightning in clean venv

Package             Version
------------------- --------
aiohttp             3.8.4
aiosignal           1.3.1
annotated-types     0.5.0
ansicon             1.89.0
anyio               3.7.1
arrow               1.2.3
async-timeout       4.0.2
attrs               23.1.0
beautifulsoup4      4.12.2
blessed             1.20.0
certifi             2023.5.7
charset-normalizer  3.2.0
click               8.1.4
colorama            0.4.6
croniter            1.3.15
dateutils           0.6.12
deepdiff            6.3.1
exceptiongroup      1.1.2
fastapi             0.100.0
filelock            3.12.2
frozenlist          1.3.3
fsspec              2023.6.0
h11                 0.14.0
idna                3.4
inquirer            3.1.3
itsdangerous        2.1.2
Jinja2              3.1.2
jinxed              1.2.0
lightning           2.0.4
lightning-cloud     0.5.37
lightning-utilities 0.9.0
markdown-it-py      3.0.0
MarkupSafe          2.1.3
mdurl               0.1.2
mpmath              1.3.0
multidict           6.0.4
networkx            3.1
numpy               1.25.0
ordered-set         4.1.0
packaging           23.1
pip                 22.1.2
psutil              5.9.5
pydantic            2.0.2
pydantic_core       2.1.2
Pygments            2.15.1
PyJWT               2.7.0
python-dateutil     2.8.2
python-editor       1.0.4
python-multipart    0.0.6
pytorch-lightning   2.0.4
pytz                2023.3
PyYAML              6.0
readchar            4.0.5
requests            2.31.0
rich                13.4.2
setuptools          62.3.3
six                 1.16.0
sniffio             1.3.0
soupsieve           2.4.1
starlette           0.27.0
starsessions        1.3.0
sympy               1.12
torch               2.0.1
torchmetrics        1.0.0
tqdm                4.65.0
traitlets           5.9.0
typing_extensions   4.7.1
urllib3             2.0.3
uvicorn             0.22.0
wcwidth             0.2.6
websocket-client    1.6.1
websockets          11.0.3
wheel               0.37.1
yarl                1.9.2

setup.py with requirements

import setuptools

with open("README.md", "r") as fh:
    long_description = fh.read()

with open("src/remote_sensing_processor/__init__.py") as f:
    for line in f:
        if line.find("__version__") >= 0:
            version = line.split("=")[1].strip()
            version = version.strip('"')
            version = version.strip("'")
            continue

setuptools.setup(
	name = 'remote-sensing-processor',
	version = version,
	author = 'Mikhail Moskovchenko',
	author_email = 'moskovchenkomike@gmail.com',
	description = 'RSP is a tool for geospatial raster data processing',
	long_description = long_description,
	long_description_content_type = 'text/markdown',
	url = 'https://github.com/simonreise/remote-sensing-processor',
	project_urls = {
		'Bug Tracker': 'https://github.com/simonreise/remote-sensing-processor/issues',
        'Source': 'https://github.com/simonreise/remote-sensing-processor',
        'Documentation': 'https://remote-sensing-processor.readthedocs.io'
	},
	classifiers = [
		'Programming Language :: Python :: 3',
		'License :: OSI Approved :: GNU General Public License v3 (GPLv3)',
		'Operating System :: OS Independent',
		'Environment :: GPU :: NVIDIA CUDA',
		'Topic :: Scientific/Engineering :: GIS'
	],
    keywords = 'remote sensing, landsat, sentinel, gdal, rasterio',
	package_dir = {"": "src"},
    packages = setuptools.find_packages(where="src"),
    setup_requires = ['cython', 'numpy>=1.17'],
    install_requires = ['numpy>=1.17', 'h5py', 'torch>=1.10', 'torchvision>=0.10', 'lightning', 'tensorboard', 'transformers', 'timm', 'scikit-learn', 'scikit-image', 'rasterio', 'pyproj', 'geopandas'],
    python_requires = ">=3.8",
    include_package_data = True
)

meta.yaml for conda-build

{% set data = load_setup_py_data() %}

package:
  name: remote-sensing-processor
  version: {{ data.get('version') }}

source:
  path: ./..

requirements:
  build:
    - python {{ python }}
    - cython
    - numpy >=1.17
  host:
    - geopandas
    - h5py
    - lightning
    - numpy >=1.17
    - pip
    - pyproj
    - python {{ python }}
    - pytorch >=1.10
    - rasterio
    - scikit-image
    - scikit-learn
    - tensorboard
    - timm
    - torchvision
    - transformers
  run:
    - geopandas
    - h5py
    - lightning
    - numpy >=1.17
    - pyproj
    - python {{ python }}
    - pytorch >=1.10
    - rasterio
    - scikit-image
    - scikit-learn
    - tensorboard
    - timm
    - torchvision
    - transformers


build:
  noarch: python

test:
  imports:
    - remote_sensing_processor
    - remote_sensing_processor.common
    - remote_sensing_processor.imagery_types
    - remote_sensing_processor.indices
    - remote_sensing_processor.landsat
    - remote_sensing_processor.mosaic
    - remote_sensing_processor.segmentation
    - remote_sensing_processor.sentinel2
    - remote_sensing_processor.sentinel2.sen2cor
    - remote_sensing_processor.sentinel2.sentinel_postprocessing
    - remote_sensing_processor.sentinel2.superres
    - remote_sensing_processor.sentinel2.superres.weights
    - remote_sensing_processor.unzip


about:
  home: "https://github.com/simonreise/remote-sensing-processor"
  license: "GNU General Public v3 (GPLv3)"
  license_family: GPL3
  license_file:
  summary: "RSP is a tool for geospatial raster data processing"
  doc_url: "https://remote-sensing-processor.readthedocs.io"
  dev_url:

appveyor.yml

environment:
  matrix:
    - APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2022
      PYTHON: "3.8"
      PYTHONPATH: "C:\\Python38-x64"
      TOXENV: python3.8
    - APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2022
      PYTHON: "3.9"
      PYTHONPATH: "C:\\Python39-x64"
      TOXENV: python3.9
    - APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2022
      PYTHON: "3.10"
      PYTHONPATH: "C:\\Python310-x64"
      TOXENV: python3.10
    - APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2022
      PYTHON: "3.11"
      PYTHONPATH: "C:\\Python311-x64"
      TOXENV: python3.11
    - APPVEYOR_BUILD_WORKER_IMAGE: Ubuntu
      PYTHON: "3.8"
      TOXENV: python3.8
    - APPVEYOR_BUILD_WORKER_IMAGE: Ubuntu
      PYTHON: "3.9"
      TOXENV: python3.9
    - APPVEYOR_BUILD_WORKER_IMAGE: Ubuntu
      PYTHON: "3.10"
      TOXENV: python3.10
    - APPVEYOR_BUILD_WORKER_IMAGE: Ubuntu
      PYTHON: "3.11"
      TOXENV: python3.11
stack: python %PYTHON%

build: false

for:
-
  matrix:
    only:
      - APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2022
        PYTHON: "3.8"
        PYTHONPATH: "C:\\Python38-x64"
        TOXENV: python3.8 
  install:
    - "SET PATH=%PYTHONPATH%;%PYTHONPATH%\\Scripts;%PATH%"
    - "%PYTHONPATH%\\python.exe -m pip install https://github.com/cgohlke/geospatial-wheels/releases/download/v2023.1.10.1/GDAL-3.6.2-cp38-cp38-win_amd64.whl"
    - "%PYTHONPATH%\\python.exe -m pip install https://github.com/cgohlke/geospatial-wheels/releases/download/v2023.1.10.1/rasterio-1.3.4-cp38-cp38-win_amd64.whl"
    - "%PYTHONPATH%\\python.exe -m pip install https://github.com/cgohlke/geospatial-wheels/releases/download/v2023.1.10.1/Fiona-1.8.22-cp38-cp38-win_amd64.whl"
    - "%PYTHONPATH%\\python.exe -m pip install tox"
  test_script:
    - "%PYTHONPATH%\\python.exe -m tox"    

-
  matrix:
    only:
      - APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2022
        PYTHON: "3.9"
        PYTHONPATH: "C:\\Python39-x64"
        TOXENV: python3.9
    
  install:
    - "SET PATH=%PYTHONPATH%;%PYTHONPATH%\\Scripts;%PATH%"
    - "%PYTHONPATH%\\python.exe -m pip install https://github.com/cgohlke/geospatial-wheels/releases/download/v2023.4.22/GDAL-3.6.4-cp39-cp39-win_amd64.whl"
    - "%PYTHONPATH%\\python.exe -m pip install https://github.com/cgohlke/geospatial-wheels/releases/download/v2023.4.22/rasterio-1.3.6-cp39-cp39-win_amd64.whl"
    - "%PYTHONPATH%\\python.exe -m pip install https://github.com/cgohlke/geospatial-wheels/releases/download/v2023.4.22/Fiona-1.9.3-cp39-cp39-win_amd64.whl"
    - "%PYTHONPATH%\\python.exe -m pip install tox"
  test_script:
    - "%PYTHONPATH%\\python.exe -m tox"
    
-
  matrix:
    only:
      - APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2022
        PYTHON: "3.10"
        PYTHONPATH: "C:\\Python310-x64"
        TOXENV: python3.10
    
  install:
    - "SET PATH=%PYTHONPATH%;%PYTHONPATH%\\Scripts;%PATH%"
    - "%PYTHONPATH%\\python.exe -m pip install https://github.com/cgohlke/geospatial-wheels/releases/download/v2023.4.22/GDAL-3.6.4-cp310-cp310-win_amd64.whl"
    - "%PYTHONPATH%\\python.exe -m pip install https://github.com/cgohlke/geospatial-wheels/releases/download/v2023.4.22/rasterio-1.3.6-cp310-cp310-win_amd64.whl"
    - "%PYTHONPATH%\\python.exe -m pip install https://github.com/cgohlke/geospatial-wheels/releases/download/v2023.4.22/Fiona-1.9.3-cp310-cp310-win_amd64.whl"
    - "%PYTHONPATH%\\python.exe -m pip install tox"
  test_script:
    - "%PYTHONPATH%\\python.exe -m tox"

-
  matrix:
    only:
      - APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2022
        PYTHON: "3.11"
        PYTHONPATH: "C:\\Python311-x64"
        TOXENV: python3.11
    
  install:
    - "SET PATH=%PYTHONPATH%;%PYTHONPATH%\\Scripts;%PATH%"
    - "%PYTHONPATH%\\python.exe -m pip install https://github.com/cgohlke/geospatial-wheels/releases/download/v2023.4.22/GDAL-3.6.4-cp311-cp311-win_amd64.whl"
    - "%PYTHONPATH%\\python.exe -m pip install https://github.com/cgohlke/geospatial-wheels/releases/download/v2023.4.22/rasterio-1.3.6-cp311-cp311-win_amd64.whl"
    - "%PYTHONPATH%\\python.exe -m pip install https://github.com/cgohlke/geospatial-wheels/releases/download/v2023.4.22/Fiona-1.9.3-cp311-cp311-win_amd64.whl"
    - "%PYTHONPATH%\\python.exe -m pip install tox"
  test_script:
    - "%PYTHONPATH%\\python.exe -m tox"

-
  matrix:
    only:
      - APPVEYOR_BUILD_WORKER_IMAGE: Ubuntu
        PYTHON: "3.8"
        TOXENV: python3.8
  install: 
    - sh: sudo add-apt-repository ppa:ubuntugis/ppa
    - sh: sudo apt-get update
    - sh: sudo apt-get -y install python-numpy gdal-bin libgdal-dev
    - sh: sudo apt-get -y install python3-pip
    - pip3 install tox
  test_script:
    - sh: python3 -m tox

-
  matrix:
    only:
      - APPVEYOR_BUILD_WORKER_IMAGE: Ubuntu
        PYTHON: "3.9"
        TOXENV: python3.9
  install: 
    - sh: sudo add-apt-repository ppa:ubuntugis/ppa
    - sh: sudo apt-get update
    - sh: sudo apt-get -y install python-numpy gdal-bin libgdal-dev
    - sh: sudo apt-get -y install python3-pip
    - pip3 install tox
  test_script:
    - sh: python3 -m tox

-
  matrix:
    only:
      - APPVEYOR_BUILD_WORKER_IMAGE: Ubuntu
        PYTHON: "3.10"
        TOXENV: python3.10
  install:
    - sh: sudo add-apt-repository ppa:ubuntugis/ppa
    - sh: sudo apt-get update
    - sh: sudo apt-get -y install python-numpy gdal-bin libgdal-dev
    - sh: sudo apt-get -y install python3-pip
    - pip3 install tox
  test_script:
    - sh: python3 -m tox

-
  matrix:
    only:
      - APPVEYOR_BUILD_WORKER_IMAGE: Ubuntu
        PYTHON: "3.11"
        TOXENV: python3.11
  install:
    - sh: sudo add-apt-repository ppa:ubuntugis/ppa
    - sh: sudo apt-get update
    - sh: sudo apt-get -y install python-numpy gdal-bin libgdal-dev
    - sh: sudo apt-get -y install python3-pip
    - pip3 install tox
  test_script:
    - sh: python3 -m tox

More info

No response

cc @tchaton

The text was updated successfully, but these errors were encountered:

huweiATgithub · 2023-07-08T13:33:48Z

From pydantic/pydantic#6381, it seems that ModelMetaclass has been considered private and removed from public interface in Pydantic v2.

AnezeR · 2023-07-08T21:23:07Z

Seems to be fixed by #18022

RongqinChen · 2023-07-09T09:22:01Z

I solved this issue by degrading pydantic using this command pip install pydantic==1.10.11 --upgrade.

awaelchli · 2023-07-09T18:14:35Z

Thanks everyone. We are aware, the fix is in #18018 and 2.0.5 will be released very shortly. Downgrading the package is the immediate solution.

Fixed; context below: Lightning-AI/pytorch-lightning#18018 Lightning-AI/pytorch-lightning#18027

function2-llx · 2023-07-12T17:31:01Z

@awaelchli Hi, it seems that the conda-forge build of 2.0.5 is failed due to this error: conda-forge/lightning-feedstock#8.

awaelchli · 2023-07-12T22:05:22Z

I believe the correct fix is here: #18030 @Borda This commit was missed in the 2.0.5 release.

@function2-llx The solution is to downgrade pydantic to < 2.0 until Lightning 2.0.6 is released.

weiji14 · 2023-07-17T04:01:26Z

@function2-llx, I've pinned pydantic<2.0 in the lightning=2.0.5 conda-forge package at conda-forge/lightning-feedstock#8, maybe try installing again and see if it works for you.

dinhanhx · 2023-08-11T12:19:39Z

And don't forget to uninstall pydantic_core

I just use pydantic==1.10.9

silverlining21 · 2023-08-17T07:57:05Z

pip install lighting==2.0.7

pydantic==2.1.1 +lighting==2.0.7, while import lightning, got pydantic.errors.PydaydanticUserError
pydantic==1.10.11 +lighting==2.0.7, works well

Borda · 2023-08-17T08:36:04Z

pydantic==2.1.1 +lighting==2.0.7, while import lightning, got pydantic.errors.PydaydanticUserError

strange as we fixed 2626bf6 and released in 2.0.6

UPDATE: can you share your trace as I am not able to reproduce it

$ pip list | grep -e lightning -e pydantic
WARNING: Ignoring invalid distribution -orch (/mnt/69B27B700DDA7D73/venvs/lai/lib/python3.9/site-packages)
lightning                     2.0.7             /home/jirka/Workingspace/lightning
lightning-fabric              1.9.5             /home/jirka/Workingspace/lightning
lightning-utilities           0.9.0
pt-lightning-sphinx-theme     0.0.31
pydantic                      2.1.1
pydantic_core                 2.4.0
pytorch-lightning             2.1.0rc0          /home/jirka/Workingspace/lightning
$ python
Python 3.9.17 (main, Jun  6 2023, 20:11:04) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import lightning
>>> import lightning.app

hannahbrown7 · 2023-08-23T09:54:57Z

I'm also getting a pydantic.errors.PydanticUserError, see attached a traceback when I try to import lightning.

UPDATE: Based on @silverlining21 comment, I install pydantic==1.10.11 and that gets rid of the error.

Borda · 2023-08-23T17:37:15Z

I'm also getting a pydantic.errors.PydanticUserError, see attached a traceback when I try to import lightning.

seems something strange with mlflow as the trace states

lightning -> mlflow -> pydantic

ItamarKanter · 2023-09-11T08:10:32Z

I have similar issue,
import pytorch_lightning works well
However from lightning.pytorch.utilities import rank_zero_only failed with the following error
ImportError: cannot import name 'ModelMetaclass' from 'pydantic.main'
pydantic 2.3.0 pyhd8ed1ab_0 defaults
pydantic-core 2.6.3 py310hcb5633a_0 defaults

pytorch-lightning 2.0.8 pyhd8ed1ab_0 defaults

zareami10 · 2023-10-06T19:36:24Z

Facing the same issue. Maybe it's better to reopen this?

Borda · 2023-10-07T02:27:40Z

Another set of patches was landed on master in #18676 so pls try install from master

simonreise added bug Something isn't working needs triage Waiting to be triaged by maintainers labels Jul 8, 2023

github-actions bot added the ver: 2.0.x label Jul 8, 2023

awaelchli closed this as completed Jul 9, 2023

awaelchli added app (removed) Generic label for Lightning App package and removed needs triage Waiting to be triaged by maintainers labels Jul 9, 2023

luigibonati mentioned this issue Jul 9, 2023

specify dependency version requirement luigibonati/mlcolvar#89

Merged

EricThomson mentioned this issue Jul 10, 2023

Lightning import fails rasbt/scipy2023-deeplearning#2

Closed

salil-gtm added a commit to salil-gtm/emlov3_assignment_7 that referenced this issue Jul 12, 2023

Update requirements.txt

0df792c

Fixed; context below: Lightning-AI/pytorch-lightning#18018 Lightning-AI/pytorch-lightning#18027

awaelchli added this to the 2.0.x milestone Jul 12, 2023

awaelchli added the priority: 0 High priority task label Jul 12, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pip and conda install pydantic 2.0.2 by default, that causes ImportError #18027

Pip and conda install pydantic 2.0.2 by default, that causes ImportError #18027

simonreise commented Jul 8, 2023 •

edited by github-actions bot

Loading

huweiATgithub commented Jul 8, 2023

AnezeR commented Jul 8, 2023

RongqinChen commented Jul 9, 2023

awaelchli commented Jul 9, 2023

function2-llx commented Jul 12, 2023

awaelchli commented Jul 12, 2023

weiji14 commented Jul 17, 2023

dinhanhx commented Aug 11, 2023

silverlining21 commented Aug 17, 2023

Borda commented Aug 17, 2023 •

edited

Loading

hannahbrown7 commented Aug 23, 2023 •

edited

Loading

Borda commented Aug 23, 2023

ItamarKanter commented Sep 11, 2023

zareami10 commented Oct 6, 2023

Borda commented Oct 7, 2023 •

edited

Loading

Pip and conda install pydantic 2.0.2 by default, that causes ImportError #18027

Pip and conda install pydantic 2.0.2 by default, that causes ImportError #18027

Comments

simonreise commented Jul 8, 2023 • edited by github-actions bot Loading

Bug description

What version are you seeing the problem on?

How to reproduce the bug

Error messages and logs

Environment

More info

huweiATgithub commented Jul 8, 2023

AnezeR commented Jul 8, 2023

RongqinChen commented Jul 9, 2023

awaelchli commented Jul 9, 2023

function2-llx commented Jul 12, 2023

awaelchli commented Jul 12, 2023

weiji14 commented Jul 17, 2023

dinhanhx commented Aug 11, 2023

silverlining21 commented Aug 17, 2023

Borda commented Aug 17, 2023 • edited Loading

hannahbrown7 commented Aug 23, 2023 • edited Loading

Borda commented Aug 23, 2023

ItamarKanter commented Sep 11, 2023

zareami10 commented Oct 6, 2023

Borda commented Oct 7, 2023 • edited Loading

simonreise commented Jul 8, 2023 •

edited by github-actions bot

Loading

Borda commented Aug 17, 2023 •

edited

Loading

hannahbrown7 commented Aug 23, 2023 •

edited

Loading

Borda commented Oct 7, 2023 •

edited

Loading