Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

python312Packages.blis: 0.7.11 -> 1.0.0 #333221

Merged
merged 4 commits into from
Sep 5, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 25 additions & 14 deletions pkgs/development/python-modules/blis/default.nix
Original file line number Diff line number Diff line change
Expand Up @@ -3,46 +3,55 @@
buildPythonPackage,
fetchFromGitHub,
setuptools,
cython_0,
cython,
hypothesis,
numpy,
pytestCheckHook,
pythonOlder,
blis,
numpy_2,
gitUpdater,
}:

buildPythonPackage rec {
pname = "blis";
version = "0.7.11";
version = "1.0.0";
pyproject = true;

disabled = pythonOlder "3.7";
disabled = pythonOlder "3.9";

src = fetchFromGitHub {
owner = "explosion";
repo = "cython-blis";
rev = "refs/tags/v${version}";
hash = "sha256-p8pzGZc5OiiGTvXULDgzsBC3jIhovTKUq3RtPnQ/+to=";
rev = "refs/tags/release-v${version}";
hash = "sha256-XS6h2c+8BJ9pAvIX8340C4vRZEBRmEZc6/6tH7ooqNU=";
};

postPatch = ''
# The commit pinning numpy to version 2 doesn't have any functional changes:
# https://github.com/explosion/cython-blis/pull/108
# BLIS should thus work with numpy and numpy_2.
substituteInPlace pyproject.toml setup.py \
--replace-fail "numpy>=2.0.0,<3.0.0" numpy

# See https://github.com/numpy/numpy/issues/21079
# has no functional difference as the name is only used in log output
substituteInPlace blis/benchmark.py \
--replace 'numpy.__config__.blas_ilp64_opt_info["libraries"]' '["dummy"]'
--replace-fail 'numpy.__config__.blas_ilp64_opt_info["libraries"]' '["dummy"]'
'';

preCheck = ''
# remove src module, so tests use the installed module instead
rm -rf ./blis
'';

nativeBuildInputs = [
build-system = [
setuptools
cython_0
cython
numpy
];

propagatedBuildInputs = [ numpy ];
dependencies = [ numpy ];
Comment on lines +51 to +54
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't that a bit weird to put it in both places?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's what upstream specifies. IIRC numpy is imported in setup.py.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's what upstream specifies. IIRC numpy is imported in setup.py.

Hmm if so, the only good reason to still put Numpy also in the native inputs is for cross compilation, which will not work on NixOS with the way upstream uses the native Numpy in their setup.py:

https://github.com/explosion/cython-blis/blob/b33c2ed9cf7ed51d999a926c4950dd8dec4d0286/setup.py#L86C39-L86C58

I think that for cross compilation you'd have to patch that line, and for non-cross compilation, you'd get the host's Numpy available anyway, so there is no need for numpy in the build-system list IMO.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We plan to separate build dependencies and runtime dependencies in the future. #272178
So, we need to add numpy to both build-system and dependencies.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We plan to separate build dependencies and runtime dependencies in the future. #272178 So, we need to add numpy to both build-system and dependencies.

I'm not sure I agree with everything said there. Even if I had agreed, #272179 should have been finished before you'd apply that policy upon other packages.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The order is reversed.
Without proper separation, we cannot start #272179.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you'd get the host's Numpy available anyway, so there is no need for numpy in the build-system list IMO.

There is because in the future dependencies might not be available at build time in order to reduce the number of rebuilds cause by updates, overrides, etc.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you'd get the host's Numpy available anyway, so there is no need for numpy in the build-system list IMO.

There is because in the future dependencies might not be available at build time in order to reduce the number of rebuilds cause by updates, overrides, etc.

I support this motivation and this goal, but in this case, due to the goal of upstream's import numpy in setup.py, it is completely incorrect to add numpy to build-system, but rather you should patch (downstream or upstream) the file to support specifying manually the host machine's Numpy headers.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that cross doesn't work as is. But at least this continues to work if dependencies are unavailable.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that cross doesn't work as is. But at least this continues to work if dependencies are unavailable.

OK I understand now. Still it would have been nice if you had put a TODO comment or something. Also the intentions of #272178 should be communicated in the documentation IMO, even though not everything is implemented yet.


nativeCheckInputs = [
hypothesis
Expand All @@ -52,16 +61,18 @@ buildPythonPackage rec {
pythonImportsCheck = [ "blis" ];

passthru = {
# Do not update to BLIS 0.9.x until the following issue is resolved:
# https://github.com/explosion/thinc/issues/771#issuecomment-1255825935
Comment on lines -55 to -56
Copy link
Member Author

@dotlambda dotlambda Aug 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@davidak Can thinc's blis pin be removed?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually upstream removed it in explosion/thinc@efda3b4 so this should be fine.

skipBulkUpdate = true;
tests = {
numpy_2 = blis.overridePythonAttrs (old: {
numpy = numpy_2;
});
};
updateScript = gitUpdater {
rev-prefix = "v";
ignoredVersions = "0\.9\..*";
rev-prefix = "release-v";
};
};

meta = with lib; {
changelog = "https://github.com/explosion/cython-blis/releases/tag/release-v${version}";
description = "BLAS-like linear algebra library";
homepage = "https://github.com/explosion/cython-blis";
license = licenses.bsd3;
Expand Down
34 changes: 16 additions & 18 deletions pkgs/development/python-modules/spacy/default.nix
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,11 @@
fetchPypi,
hypothesis,
jinja2,
jsonschema,
langcodes,
mock,
murmurhash,
numpy,
packaging,
pathy,
preshed,
pydantic,
pytestCheckHook,
Expand All @@ -29,7 +27,6 @@
thinc,
tqdm,
typer,
typing-extensions,
wasabi,
weasel,
writeScript,
Expand All @@ -40,36 +37,39 @@

buildPythonPackage rec {
pname = "spacy";
version = "3.7.5";
version = "3.7.6";
pyproject = true;

disabled = pythonOlder "3.7";

src = fetchPypi {
inherit pname version;
hash = "sha256-pkjGy/Ksx6Vaae6ef6TyK99pqoKKWHobxc//CM88LdM=";
hash = "sha256-9AZcCqxcSLv7L/4ZHVXMszv7AFN2r71MzW1ek0FRTjQ=";
};

pythonRelaxDeps = [
"smart-open"
"typer"
];
postPatch = ''
# thinc version 8.3.0 had no functional changes
# also see https://github.com/explosion/spaCy/issues/13607
substituteInPlace pyproject.toml setup.cfg \
--replace-fail "thinc>=8.2.2,<8.3.0" "thinc>=8.2.2,<8.4.0"
'';

nativeBuildInputs = [
build-system = [
cymem
cython_0
murmurhash
numpy
thinc
];

propagatedBuildInputs = [
blis
dependencies = [
catalogue
cymem
jinja2
jsonschema
langcodes
murmurhash
numpy
packaging
pathy
preshed
pydantic
requests
Expand All @@ -82,16 +82,14 @@ buildPythonPackage rec {
typer
wasabi
weasel
] ++ lib.optionals (pythonOlder "3.8") [ typing-extensions ];
];

nativeCheckInputs = [
pytestCheckHook
hypothesis
mock
];

doCheck = true;

# Fixes ModuleNotFoundError when running tests on Cythonized code. See #255262
preCheck = ''
cd $out
Expand Down Expand Up @@ -132,7 +130,7 @@ buildPythonPackage rec {
description = "Industrial-strength Natural Language Processing (NLP)";
mainProgram = "spacy";
homepage = "https://github.com/explosion/spaCy";
changelog = "https://github.com/explosion/spaCy/releases/tag/v${version}";
changelog = "https://github.com/explosion/spaCy/releases/tag/release-v${version}";
license = licenses.mit;
maintainers = [ ];
};
Expand Down
46 changes: 25 additions & 21 deletions pkgs/development/python-modules/thinc/default.nix
Original file line number Diff line number Diff line change
Expand Up @@ -16,59 +16,66 @@
mock,
murmurhash,
numpy,
plac,
preshed,
pydantic,
pytestCheckHook,
python,
pythonOlder,
setuptools,
srsly,
tqdm,
typing-extensions,
wasabi,
}:

buildPythonPackage rec {
pname = "thinc";
version = "8.2.3";
format = "setuptools";
version = "8.3.0";
pyproject = true;

disabled = pythonOlder "3.7";

src = fetchPypi {
inherit pname version;
hash = "sha256-9a/FIikSqAvai9zslYNiorpTjXAn3I22FUhF0oWdynY=";
hash = "sha256-6zvtVPXADsmt2qogjFHM+gWUg9cxQM1RWqMzc3Fcblk=";
};

postPatch = ''
# As per https://github.com/explosion/thinc/releases/tag/release-v8.3.0 no
# code changes were required for NumPy 2.0. Thus Thinc should be compatible
# with NumPy 1.0 and 2.0.
substituteInPlace pyproject.toml setup.cfg \
--replace-fail "numpy>=2.0.0,<2.1.0" numpy
substituteInPlace setup.cfg \
--replace "preshed>=3.0.2,<3.1.0" "preshed"
--replace-fail "numpy>=2.0.1,<2.1.0" numpy
'';

nativeBuildInputs = [ setuptools ];
build-system = [
blis
cymem
cython_0
murmurhash
numpy
preshed
setuptools
];

buildInputs =
[ cython_0 ]
++ lib.optionals stdenv.isDarwin [
Accelerate
CoreFoundation
CoreGraphics
CoreVideo
];
buildInputs = lib.optionals stdenv.isDarwin [
Accelerate
CoreFoundation
CoreGraphics
CoreVideo
];

propagatedBuildInputs = [
dependencies = [
blis
catalogue
confection
cymem
murmurhash
numpy
plac
preshed
pydantic
srsly
tqdm
wasabi
] ++ lib.optionals (pythonOlder "3.8") [ typing-extensions ];

Expand All @@ -78,10 +85,7 @@ buildPythonPackage rec {
pytestCheckHook
];

# Add native extensions.
preCheck = ''
export PYTHONPATH=$out/${python.sitePackages}:$PYTHONPATH

# avoid local paths, relative imports wont resolve correctly
mv thinc/tests tests
rm -r thinc
Expand Down