⬆️ Looser 2.0 numpy functionality

[daquinteroflex]

Closes #1912

• Possibly add the trapz logic per numpy version
• Work out how to create the locked resolved environment.
  • Update jax dependency. Had to update jax due to a weird segmentation fault with the older version.
  • Update pandas dependency to greater than 2.2.2 INFO: Please pin your numpy < 2 if installing pandas via pip pandas-dev/pandas#55519 Not required below as not related to the error.
  • Wait for the next autograd release with this PR Fix numpy v2 breaking changes HIPS/autograd#618
  • Increase boto3 version.
  • Identify the right gdstk dependencies (thanks Yannick)
• Get the CI tests to pass.
• Proper backend testing too.
• Finish the backend PR

[daquinteroflex]

I've just rerun the poetry lock cleanly:

Ran into this:

  OverrideNeeded

  {Package('jax', '0.4.25', features=frozenset({'cpu'}), source_type='legacy', source_url='https://storage.googleapis.com/jax-releases/jax_releases.html', source_reference='jaxsource'): {'numpy': <Dependency numpy (>=1.23.2)>, 'scipy': <Dependency scipy (>=1.9)>}, Package('jaxlib', '0.4.25', source_type='legacy', source_url='https://storage.googleapis.com/jax-releases/jax_releases.html', source_reference='jaxsource'): {'scipy': <Dependency scipy (>=1.9)>}, Package('botocore', '1.35.2'): {'urllib3': <Dependency urllib3 (>=1.25.4,!=2.2.0,<3)>}, Package('pylint', '3.2.6'): {'dill': <Dependency dill (>=0.3.6)>}, Package('pandas', '2.2.1'): {'numpy': <Dependency numpy (>=1.23.2,<2)>}, Package('pydantic', '2.8.2'): {'typing-extensions': <Dependency typing-extensions (>=4.6.1)>}, Package('ml-dtypes', '0.4.0'): {'numpy': <Dependency numpy (>=1.23.3)>}, Package('flax', '0.8.5'): {'numpy': <Dependency numpy (>=1.23.2)>}, Package('flax', '0.8.4'): {'numpy': <Dependency numpy (>=1.23.2)>}, Package('jax', '0.4.25', source_type='legacy', source_url='https://storage.googleapis.com/jax-releases/jax_releases.html', source_reference='jaxsource'): {'numpy': <Dependency numpy (>=1.23.2)>, 'scipy': <Dependency scipy (>=1.9)>}, Package('oldest-supported-numpy', '2023.12.21'): {'numpy': <Dependency numpy (==1.23.2)>}, Package('oldest-supported-numpy', '2023.12.12'): {'numpy': <Dependency numpy (==1.23.2)>}, Package('oldest-supported-numpy', '2023.10.25'): {'numpy': <Dependency numpy (==1.23.2)>}, Package('oldest-supported-numpy', '2023.8.3'): {'numpy': <Dependency numpy (==1.23.2)>}}

Hence, we might need to increase the jax dependency to support numpy==2.0

[daquinteroflex]

Note that np.trapezoid and np.trapz have the same API interface so can be used intechageably

[daquinteroflex]

Even if we do increase the jax requirements, we also have to increase botocore

  {Package('botocore', '1.35.2'): {'urllib3': <Dependency urllib3 (>=1.25.4,<1.27)>}, Package('pylint', '3.2.6'): {'dill': <Dependency dill (>=0.2)>}, Package('pandas', '2.2.1'): {'numpy': <Dependency numpy (>=1.22.4,<2)>}, Package('pydantic', '2.8.2'): {'typing-extensions': <Dependency typing-extensions (>=4.6.1)>}, Package('jax', '0.4.31', features=frozenset({'cpu'}), source_type='legacy', source_url='https://storage.googleapis.com/jax-releases/jax_releases.html', source_reference='jaxsource'): {'numpy': <Dependency numpy (>=1.24)>, 'scipy': <Dependency scipy (>=1.10)>}, Package('jaxlib', '0.4.31', source_type='legacy', source_url='https://storage.googleapis.com/jax-releases/jax_releases.html', source_reference='jaxsource'): {'scipy': <Dependency scipy (>=1.10)>}, Package('jax', '0.4.30', features=frozenset({'cpu'}), source_type='legacy', source_url='https://storage.googleapis.com/jax-releases/jax_releases.html', source_reference='jaxsource'): {'numpy': <Dependency numpy (>=1.22)>, 'scipy': <Dependency scipy (>=1.9)>}, Package('ml-dtypes', '0.4.0'): {'numpy': <Dependency numpy (>1.20)>}, Package('flax', '0.8.5'): {'numpy': <Dependency numpy (>=1.22)>}, Package('jaxlib', '0.4.30', source_type='legacy', source_url='https://storage.googleapis.com/jax-releases/jax_releases.html', source_reference='jaxsource'): {'scipy': <Dependency scipy (>=1.9)>}, Package('jax', '0.4.30', source_type='legacy', source_url='https://storage.googleapis.com/jax-releases/jax_releases.html', source_reference='jaxsource'): {'numpy': <Dependency numpy (>=1.22)>, 'scipy': <Dependency scipy (>=1.9)>}, Package('oldest-supported-numpy', '2023.12.21'): {'numpy': <Dependency numpy (==1.22.2)>}, Package('oldest-supported-numpy', '2023.12.12'): {'numpy': <Dependency numpy (==1.22.2)>}, Package('oldest-supported-numpy', '2023.10.25'): {'numpy': <Dependency numpy (==1.22.2)>}, Package('oldest-supported-numpy', '2023.8.3'): {'numpy': <Dependency numpy (==1.22.2)>}}

This also means we need to increase our pandas and boto3->botocore version.

[tylerflex]

Hence, we might need to increase the jax dependency to support numpy==2.0

We can try. What version do we need to support? we can bump it and see if the tests pass.

[agriyakhetarpal]

Hi there, we released a new version of Autograd: https://github.com/HIPS/autograd/releases/tag/v1.7.0, which includes NumPy v2 support. It will be available via PyPI later in the day.

[daquinteroflex]

Thanks @agriyakhetarpal this is great!!!

@tylerflex Been trying to debug that. Jax has a really complex packaging structure, so it's not exactly clear to me what's the exact dependency limitation source within either jax or boto3 (but that doesn't look like it) - and it's hard to see as the lockfile doesn't actually generate the graph since the dependency resolution fails.

I keep getting this error no matter what relevant dependencies I loosen:

The following error occurred when trying to handle this error:


  OverrideNeeded

  {Package('botocore', '1.35.4'): {'urllib3': <Dependency urllib3 (>=1.25.4,<1.27)>}, Package('pylint', '3.2.6'): {'dill': <Dependency dill (>=0.2)>}, Package('pandas', '2.2.2'): {'numpy': <Dependency numpy (>=1.22.4)>}, Package('pydantic', '2.8.2'): {'typing-extensions': <Dependency typing-extensions (>=4.6.1)>}, Package('jax', '0.4.31', source_type='legacy', source_url='https://storage.googleapis.com/jax-releases/jax_releases.html', source_reference='jaxsource'): {'numpy': <Dependency numpy (>=1.24)>, 'scipy': <Dependency scipy (>=1.10)>}, Package('jaxlib', '0.4.31', source_type='legacy', source_url='https://storage.googleapis.com/jax-releases/jax_releases.html', source_reference='jaxsource'): {'scipy': <Dependency scipy (>=1.10)>}, Package('jax', '0.4.30', source_type='legacy', source_url='https://storage.googleapis.com/jax-releases/jax_releases.html', source_reference='jaxsource'): {'numpy': <Dependency numpy (>=1.22)>, 'scipy': <Dependency scipy (>=1.9)>}, Package('ml-dtypes', '0.4.0'): {'numpy': <Dependency numpy (>1.20)>}, Package('flax', '0.8.5'): {'numpy': <Dependency numpy (>=1.22)>}, Package('jaxlib', '0.4.30', source_type='legacy', source_url='https://storage.googleapis.com/jax-releases/jax_releases.html', source_reference='jaxsource'): {'scipy': <Dependency scipy (>=1.9)>}, Package('oldest-supported-numpy', '2023.12.21'): {'numpy': <Dependency numpy (==1.22.2)>}, Package('oldest-supported-numpy', '2023.12.12'): {'numpy': <Dependency numpy (==1.22.2)>}, Package('oldest-supported-numpy', '2023.10.25'): {'numpy': <Dependency numpy (==1.22.2)>}, Package('oldest-supported-numpy', '2023.8.3'): {'numpy': <Dependency numpy (==1.22.2)>}}

  at ~/.local/share/pipx/venvs/poetry/lib/python3.10/site-packages/poetry/puzzle/provider.py:653 in complete_package
      649│                     current_overrides.update({package: package_overrides})
      650│                     overrides.append(current_overrides)
      651│ 
      652│             if overrides:
    → 653│                 raise OverrideNeeded(*overrides)
      654│ 
      655│         # Modifying dependencies as needed
      656│         clean_dependencies = []
      657│         for dep in dependencies:

The following error occurred when trying to handle this error:


  IncompatibleConstraintsError

  Incompatible constraints in requirements of oldest-supported-numpy (2022.11.19):
  numpy (==1.22.2) ; platform_machine == "loongarch64" and python_version < "3.11"
  numpy (==1.23.3) ; python_version == "3.9" and platform_system == "OS400"
  numpy ; python_version >= "3.12" or python_version >= "3.9" and platform_python_implementation == "PyPy"

  at ~/.local/share/pipx/venvs/poetry/lib/python3.10/site-packages/poetry/puzzle/provider.py:925 in _resolve_overlapping_markers
      921│                     specific_source_dependency = dep
      922│                 constraint = constraint.intersect(dep.constraint)
      923│             if constraint.is_empty():
      924│                 # conflict in overlapping area
    → 925│                 raise IncompatibleConstraintsError(package, *used_dependencies)
      926│ 
      927│             if not any(uses):
      928│                 # This is an edge case where the dependency is not required
      929│                 # for the resulting marker. However, we have to consider it anyway

Doesn't look like a scipy support issue, nor pandas, technically it's not jax, nor boto3 has no direct dependencies but could be related to this based on the packages in the error.

Will keep debugging but is a weird issue. I'll continue on this next week I think. Unless, possibly @yaugenst-flex might be interested to give it a shot too earlier?

For reference, in a clean pip environment, it does install. So it's a poetry reproducible issue on broad dependency management issue.

± |dario/loosen_numpy_2_requirements {5} ✓| → pip list
Package                       Version        Editable project location
----------------------------- -------------- ------------------------------------------
absl-py                       2.1.0
accessible-pygments           0.0.5
alabaster                     1.0.0
annotated-types               0.7.0
anyio                         4.4.0
argon2-cffi                   23.1.0
argon2-cffi-bindings          21.2.0
arrow                         1.3.0
astroid                       3.2.4
asttokens                     2.4.1
async-lru                     2.0.4
attrs                         24.2.0
autograd                      1.6.2
babel                         2.16.0
beautifulsoup4                4.12.3
black                         24.8.0
bleach                        6.1.0
boto3                         1.35.4
botocore                      1.35.4
bracex                        2.5
bump-my-version               0.26.0
cachetools                    5.5.0
certifi                       2024.7.4
cffi                          1.17.0
cfgv                          3.4.0
chardet                       5.2.0
charset-normalizer            3.3.2
chex                          0.1.86
click                         8.1.7
cloudpickle                   3.0.0
cma                           3.4.0
colorama                      0.4.6
comm                          0.2.2
contourpy                     1.2.1
coverage                      7.6.1
cycler                        0.12.1
dask                          2024.8.1
debugpy                       1.8.5
decorator                     5.1.1
defusedxml                    0.7.1
devsim                        2.8.2
dill                          0.3.8
distlib                       0.3.8
docutils                      0.21.2
etils                         1.9.2
executing                     2.0.1
fastcore                      1.7.1
fastjsonschema                2.20.0
filelock                      3.15.4
flax                          0.8.5
fonttools                     4.53.1
fqdn                          1.5.1
fsspec                        2024.6.1
future                        1.0.0
gdspy                         1.6.13
gdstk                         0.9.53
gitdb                         4.0.11
GitPython                     3.1.43
grcwa                         0.1.2
h11                           0.14.0
h5netcdf                      1.0.2
h5py                          3.11.0
httpcore                      1.0.5
httpx                         0.27.0
humanize                      4.10.0
identify                      2.6.0
idna                          3.7
imagesize                     1.4.1
importlib_metadata            8.4.0
importlib_resources           6.4.4
iniconfig                     2.0.0
ipykernel                     6.29.5
ipython                       8.26.0
ipywidgets                    8.1.5
isoduration                   20.11.0
isort                         5.13.2
jax                           0.4.31
jaxlib                        0.4.31
jaxtyping                     0.2.33
jedi                          0.19.1
Jinja2                        3.1.4
jmespath                      1.0.1
joblib                        1.4.2
json5                         0.9.25
jsonpointer                   3.0.0
jsonschema                    4.23.0
jsonschema-specifications     2023.12.1
jupyter                       1.0.0
jupyter_client                8.6.2
jupyter-console               6.6.3
jupyter_core                  5.7.2
jupyter-events                0.10.0
jupyter-lsp                   2.2.5
jupyter_server                2.14.2
jupyter-server-mathjax        0.2.6
jupyter_server_terminals      0.5.3
jupyterlab                    4.2.4
jupyterlab_pygments           0.3.0
jupyterlab_server             2.27.3
jupyterlab_widgets            3.0.13
kiwisolver                    1.4.5
klujax                        0.2.5
locket                        1.0.0
markdown-it-py                3.0.0
MarkupSafe                    2.1.5
matplotlib                    3.9.2
matplotlib-inline             0.1.7
mccabe                        0.7.0
mdit-py-plugins               0.4.1
mdurl                         0.1.2
memory-profiler               0.61.0
mistune                       3.0.2
ml-dtypes                     0.4.0
msgpack                       1.0.8
mypy-extensions               1.0.0
myst-parser                   4.0.0
natsort                       8.4.0
nbclient                      0.10.0
nbconvert                     7.16.4
nbdime                        4.0.1
nbformat                      5.10.4
nbsphinx                      0.9.5
nest-asyncio                  1.6.0
networkx                      2.8.8
nodeenv                       1.9.1
notebook                      7.2.1
notebook_shim                 0.2.4
numpy                         2.1.0
oldest-supported-numpy        2022.5.28
opt-einsum                    3.3.0
optax                         0.2.3
orbax-checkpoint              0.6.1
orjson                        3.10.7
overrides                     7.7.0
packaging                     24.1
pandas                        2.2.2
pandocfilters                 1.5.1
parso                         0.8.4
partd                         1.4.2
pathspec                      0.12.1
pexpect                       4.9.0
pillow                        10.4.0
pip                           24.0
platformdirs                  4.2.2
pluggy                        1.5.0
pre-commit                    3.8.0
prometheus_client             0.20.0
prompt_toolkit                3.0.47
protobuf                      5.27.3
psutil                        6.0.0
ptyprocess                    0.7.0
pure_eval                     0.2.3
pybind11                      2.13.5
pycparser                     2.22
pydantic                      2.8.2
pydantic_core                 2.20.1
pydantic-settings             2.4.0
pydata-sphinx-theme           0.15.4
Pygments                      2.18.0
PyJWT                         2.9.0
pylint                        3.2.6
pyparsing                     3.1.2
pyproject-api                 1.7.1
pyroots                       0.5.0
pyswarms                      1.3.0
pytest                        8.3.2
pytest-timeout                2.3.1
python-dateutil               2.9.0.post0
python-dotenv                 1.0.1
python-json-logger            2.0.7
pytz                          2024.1
PyYAML                        6.0.2
pyzmq                         26.2.0
qtconsole                     5.5.2
QtPy                          2.4.1
questionary                   1.10.0
referencing                   0.35.1
requests                      2.32.3
responses                     0.25.3
rfc3339-validator             0.1.4
rfc3986-validator             0.1.1
rich                          13.7.1
rich-click                    1.8.3
rpds-py                       0.20.0
Rtree                         1.2.0
ruff                          0.5.5
s3transfer                    0.10.2
sax                           0.11.4
scikit-rf                     1.2.0
scipy                         1.14.1
Send2Trash                    1.8.3
setuptools                    65.5.0
shapely                       2.0.6
signac                        2.2.0
six                           1.16.0
smmap                         5.0.1
sniffio                       1.3.1
snowballstemmer               2.2.0
soupsieve                     2.6
Sphinx                        8.0.2
sphinx-book-theme             1.1.3
sphinx-copybutton             0.5.2
sphinx-favicon                1.0.1
sphinx-notfound-page          1.0.4
sphinx-sitemap                2.6.0
sphinx-tabs                   3.4.5
sphinxcontrib-applehelp       2.0.0
sphinxcontrib-devhelp         2.0.0
sphinxcontrib-htmlhelp        2.1.0
sphinxcontrib-jsmath          1.0.1
sphinxcontrib-qthelp          2.0.0
sphinxcontrib-serializinghtml 2.0.0
sphinxemoji                   0.3.1
stack-data                    0.6.3
synced-collections            1.0.0
tensorstore                   0.1.64
terminado                     0.18.1
tidy3d                        2.7.2          /home/daquintero/flexcompute/tidy3d_urgent
tinycss2                      1.3.0
tmm                           0.1.8
toml                          0.10.2
tomlkit                       0.13.2
toolz                         0.12.1
tornado                       6.4.1
tox                           4.18.0
tqdm                          4.66.5
traitlets                     5.14.3
trimesh                       4.4.7
typeguard                     2.13.3
types-python-dateutil         2.9.0.20240821
typing_extensions             4.12.2
tzdata                        2024.1
uri-template                  1.3.0
urllib3                       2.2.2
virtualenv                    20.26.3
vtk                           9.3.1
wcmatch                       9.0
wcwidth                       0.2.13
webcolors                     24.8.0
webencodings                  0.5.1
websocket-client              1.8.0
widgetsnbextension            4.0.13
xarray                        2024.7.0
zipp                          3.20.0

[notice] A new release of pip is available: 24.0 -> 24.2
[notice] To update, run: python3.11 -m pip install --upgrade pip

[tylerflex]

ok thanks for looking into it. maybe Yannick has some idea.. or perhaps we need to add numpy < 2 in the "jax" dependency group?

[yaugenst-flex]

Hi all, so after a little bit of digging around I found that the culprit is the latest gdstk version (0.9.53), because that version drops support for numpy<2 entirely, see here.

So, the only changes to the pyproject.toml that are required for numpy2 compatibility are the following:

numpy = ">=1.2"
pandas = "*"
autograd = ">=1.6.2"
gdstk = { version = ">=0.9.49,<0.9.53", optional = true }

boto3 does not depend on numpy, and while jax is a pretty heavy dependency in its own right, it does not depend on many external libraries, see here, so both of these should not be problematic w.r.t. numpy2 support.

[yaugenst-flex]

Ah wait no, regarding gdstk, there is actually a problem. gdstk<0.9.53 technically supports numpy>=2 because it does not explicitly disallow numpy<2 and will likely just break if used with numpy>=2.
So if we specify gdstk>=0.9.49,<0.9.53" }, dependency resolution will work fine but things will probably just break when we are calling into gdstk.
In other words, we cannot really support both numpy <2 and >=2 as long as gdstk allows installation of numpy versions that it is incompatible with. Any ideas how to solve this @lucas-flexcompute?
I think the simplest solution would be if the latest gdstk version could support both numpy versions, then we could depend on gdstk>=0.9.54 (which I'm assuming would be the next version 😉) and call it a day.

[lucas-flexcompute]

Ah wait no, regarding gdstk, there is actually a problem. gdstk<0.9.53 technically supports numpy>=2 because it does not explicitly disallow numpy<2 and will likely just break if used with numpy>=2. So if we specify gdstk>=0.9.49,<0.9.53" }, dependency resolution will work fine but things will probably just break when we are calling into gdstk. In other words, we cannot really support both numpy <2 and >=2 as long as gdstk allows installation of numpy versions that it is incompatible with. Any ideas how to solve this @lucas-flexcompute? I think the simplest solution would be if the latest gdstk version could support both numpy versions, then we can install gdstk>=0.9.54 (which I'm assuming would be the next version 😉) and call it a day.

gdstk 0.9.53 supports installation with numpy < 2. It requires numpy >= 2 only for building (at least that's my understanding of how the 2.0 API works: runtime is compatible with 1.x, but the build step must be done against 2).

[yaugenst-flex]

Thanks @lucas-flexcompute yeah you're right, I was wrong about that. Tbh I don't really understand what the problem is then. Fact is that we get a dependency resolution error when trying trying to use gdstk=0.9.53...

Oooh poetry is probably trying to build it? I'm not sure about how the distribution part works, I would have assumed that poetry just pulls pre-built wheels from pypi? But maybe that's not the case? @daquinteroflex

[lucas-flexcompute]

@yaugenst-flex Maybe pip or poetry force looking into the build system dependencies instead of the runtime dependencies? Could that be it?

[daquinteroflex]

Thanks @yaugenst-flex and @lucas-flexcompute thanks for this! I am looking to understand exactly how the poetry lockfile resolution works now, I just believed it was based on the package metadata rather than build-time resolution, and it does pull the pre-built wheels but we have control over specifically which build files it uses. I was consistently getting resolution errors despite multiple tweaks and now it's resolving with Yannick's suggestion (but I also this weekend rewiped my ubuntu so could also be that...) Am looking into this more as you suggest Lucas.

Generating the lockfile would resolve the reproducible environment locking problem, but then within that we'll still actually have to test that multiple numpys actually work within variations of the numpy used.

[daquinteroflex]

So basically, poetry has not been great giving information on the failed resolution. I've been using uv recently so tried it to debug this and it's turned out to be pretty helpful (look how fast it is!):

This is for:

# gdspy
gdspy = { version = "*", optional = true }

# gdstk
gdstk = { version = ">=0.9.53", optional = true }

± |dario/loosen_numpy_2_requirements U:1 ✗| → uv pip install -e .[dev]
Resolved 242 packages in 4.06s
   Built tidy3d @ file:///home/daquintero/flexcompute/tidy3d_urgent
error: Failed to prepare distributions
  Caused by: Failed to fetch wheel: gdspy==1.6.13
  Caused by: Build backend failed to build wheel through `build_wheel()` with exit status: 1
--- stdout:
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-cpython-311
creating build/lib.linux-x86_64-cpython-311/gdspy
copying gdspy/operation.py -> build/lib.linux-x86_64-cpython-311/gdspy
copying gdspy/path.py -> build/lib.linux-x86_64-cpython-311/gdspy
copying gdspy/polygon.py -> build/lib.linux-x86_64-cpython-311/gdspy
copying gdspy/__init__.py -> build/lib.linux-x86_64-cpython-311/gdspy
copying gdspy/library.py -> build/lib.linux-x86_64-cpython-311/gdspy
copying gdspy/hobby.py -> build/lib.linux-x86_64-cpython-311/gdspy
copying gdspy/viewer.py -> build/lib.linux-x86_64-cpython-311/gdspy
copying gdspy/curve.py -> build/lib.linux-x86_64-cpython-311/gdspy
copying gdspy/gdsiiformat.py -> build/lib.linux-x86_64-cpython-311/gdspy
copying gdspy/label.py -> build/lib.linux-x86_64-cpython-311/gdspy
creating build/lib.linux-x86_64-cpython-311/gdspy/data
copying gdspy/data/07.xbm -> build/lib.linux-x86_64-cpython-311/gdspy/data
copying gdspy/data/00.xbm -> build/lib.linux-x86_64-cpython-311/gdspy/data
copying gdspy/data/down.xbm -> build/lib.linux-x86_64-cpython-311/gdspy/data
copying gdspy/data/04.xbm -> build/lib.linux-x86_64-cpython-311/gdspy/data
copying gdspy/data/02.xbm -> build/lib.linux-x86_64-cpython-311/gdspy/data
copying gdspy/data/up.xbm -> build/lib.linux-x86_64-cpython-311/gdspy/data
copying gdspy/data/06.xbm -> build/lib.linux-x86_64-cpython-311/gdspy/data
copying gdspy/data/09.xbm -> build/lib.linux-x86_64-cpython-311/gdspy/data
copying gdspy/data/05.xbm -> build/lib.linux-x86_64-cpython-311/gdspy/data
copying gdspy/data/01.xbm -> build/lib.linux-x86_64-cpython-311/gdspy/data
copying gdspy/data/outline.xbm -> build/lib.linux-x86_64-cpython-311/gdspy/data
copying gdspy/data/03.xbm -> build/lib.linux-x86_64-cpython-311/gdspy/data
copying gdspy/data/08.xbm -> build/lib.linux-x86_64-cpython-311/gdspy/data
running build_ext
building 'gdspy.clipper' extension
creating build/temp.linux-x86_64-cpython-311
creating build/temp.linux-x86_64-cpython-311/gdspy
clang++ -pthread -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -fPIC -I/home/daquintero/.cache/uv/builds-v0/.tmpJ9wO4E/include -I/home/daquintero/.local/share/uv/python/cpython-3.11.9-linux-x86_64-gnu/include/python3.11 -c gdspy/clipper.cpp -o build/temp.linux-x86_64-cpython-311/gdspy/clipper.o
--- stderr:
/home/daquintero/.cache/uv/builds-v0/.tmpJ9wO4E/lib/python3.11/site-packages/setuptools/_distutils/dist.py:268: UserWarning: Unknown distribution option: 'tests_require'
  warnings.warn(msg)
error: command 'clang++' failed: No such file or directory
---

I think maybe we need to start thinking depreciating gdspy in this case? @lucas-flexcompute

[yaugenst-flex]

@daquinteroflex this just looks like you're missing the compiler?

gdspy was not giving any problems in the poetry lockfile generation...

[daquinteroflex]

Whoops sorry my bad, still recovering from the ubuntu wipe and these minor issues have been popping up. It's weird as poetry compiled it but uv didn't, interesting how the environments get configured differnetly.

[heitzmann]

Gdspy has been in bug fix mode only for quite a long time, and I've recently archived the project, because I have no time to actually maintain it, so I think that's a good option

…

On Mon, Aug 26, 2024, 13:53 Dario Quintero (Flexcompute) < ***@***.***> wrote: So basically, poetry has not been great giving information on the failed resolution. I've been using uv recently so tried it debug this and it's turned out to be pretty helpful: ± |dario/loosen_numpy_2_requirements U:1 ✗| → uv pip install -e .[dev] Resolved 242 packages in 4.06s Built tidy3d @ file:///home/daquintero/flexcompute/tidy3d_urgent error: Failed to prepare distributions Caused by: Failed to fetch wheel: gdspy==1.6.13 Caused by: Build backend failed to build wheel through `build_wheel()` with exit status: 1--- stdout:running bdist_wheelrunning buildrunning build_pycreating buildcreating build/lib.linux-x86_64-cpython-311creating build/lib.linux-x86_64-cpython-311/gdspycopying gdspy/operation.py -> build/lib.linux-x86_64-cpython-311/gdspycopying gdspy/path.py -> build/lib.linux-x86_64-cpython-311/gdspycopying gdspy/polygon.py -> build/lib.linux-x86_64-cpython-311/gdspycopying gdspy/__init__.py -> build/lib.linux-x86_64-cpython-311/gdspycopying gdspy/library.py -> build/lib.linux-x86_64-cpython-311/gdspycopying gdspy/hobby.py -> build/lib.linux-x86_64-cpython-311/gdspycopying gdspy/viewer.py -> build/lib.linux-x86_64-cpython-311/gdspycopying gdspy/curve.py -> build/lib.linux-x86_64-cpython-311/gdspycopying gdspy/gdsiiformat.py -> build/lib.linux-x86_64-cpython-311/gdspycopying gdspy/label.py -> build/lib.linux-x86_64-cpython-311/gdspycreating build/lib.linux-x86_64-cpython-311/gdspy/datacopying gdspy/data/07.xbm -> build/lib.linux-x86_64-cpython-311/gdspy/datacopying gdspy/data/00.xbm -> build/lib.linux-x86_64-cpython-311/gdspy/datacopying gdspy/data/down.xbm -> build/lib.linux-x86_64-cpython-311/gdspy/datacopying gdspy/data/04.xbm -> build/lib.linux-x86_64-cpython-311/gdspy/datacopying gdspy/data/02.xbm -> build/lib.linux-x86_64-cpython-311/gdspy/datacopying gdspy/data/up.xbm -> build/lib.linux-x86_64-cpython-311/gdspy/datacopying gdspy/data/06.xbm -> build/lib.linux-x86_64-cpython-311/gdspy/datacopying gdspy/data/09.xbm -> build/lib.linux-x86_64-cpython-311/gdspy/datacopying gdspy/data/05.xbm -> build/lib.linux-x86_64-cpython-311/gdspy/datacopying gdspy/data/01.xbm -> build/lib.linux-x86_64-cpython-311/gdspy/datacopying gdspy/data/outline.xbm -> build/lib.linux-x86_64-cpython-311/gdspy/datacopying gdspy/data/03.xbm -> build/lib.linux-x86_64-cpython-311/gdspy/datacopying gdspy/data/08.xbm -> build/lib.linux-x86_64-cpython-311/gdspy/datarunning build_extbuilding 'gdspy.clipper' extensioncreating build/temp.linux-x86_64-cpython-311creating build/temp.linux-x86_64-cpython-311/gdspyclang++ -pthread -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -fPIC -I/home/daquintero/.cache/uv/builds-v0/.tmpJ9wO4E/include -I/home/daquintero/.local/share/uv/python/cpython-3.11.9-linux-x86_64-gnu/include/python3.11 -c gdspy/clipper.cpp -o build/temp.linux-x86_64-cpython-311/gdspy/clipper.o--- stderr:/home/daquintero/.cache/uv/builds-v0/.tmpJ9wO4E/lib/python3.11/site-packages/setuptools/_distutils/dist.py:268: UserWarning: Unknown distribution option: 'tests_require' warnings.warn(msg)error: command 'clang++' failed: No such file or directory--- I think maybe we need to start thinking depreciating gdspy in this case? @heitzmann <https://github.com/heitzmann> — Reply to this email directly, view it on GitHub <#1913 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABNJ2E3BSCXHE7QYMOHSS4DZTNMR7AVCNFSM6AAAAABM3SCQNWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMJQGY2DMMBVGM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[yaugenst-flex]

I think it makes sense to keep gdspy around as long as we don't run into actual issues with it, since deprecating support would also involve removing the corresponding functions from tidy3d core.

Regarding gdstk I suspect that poetry might be trying to either build or run using the wrong numpy version (since those two steps require different numpy versions). Poetry should support different build systems, but maybe there is some weird interaction with scikit-build. No idea.

[daquinteroflex]

Yeah agree with Yannick, it's interesting, the uv and poetry have different ways of managing the source packaging installations. gdspy gets installed fine in poetry, but built from source in uv. uv and pip also manage to resolve the environment on our pyproject.toml, but poetry is unable to resolve an environment with gdstk >=0.9.53.

This gives me the impression it could be a poetry bug, but still unclear... It definitely just sounds like a build configuration problem and managing the source installation in the lockfile generation.

[yaugenst-flex]

uv and pip also manage to resolve the environment on our pyproject.toml

I don't know about uv, but I believe pip just resolves the dependencies based on [tool.poetry.extras], which are less constrained than what poetry uses (which has to take into account all version requirements specified in [tool.poetry.dependencies]). Not completely sure on this but extras, when installed via pip, probably does not include the versions specified in [tool.poetry.dependencies]?

[daquinteroflex]

So the backend PR is ready, seems to be working, and it's only awaiting the final go ahead. Realised I had cleaned and opened that PR up but not this one so opening it for review now

[tylerflex]

it looks good to me! but I'd suggest getting someone who's more familiar to take a closer look