Build poetry package with extra or optional dependencies

[JulianRMedina]

• [ x] I have searched the issues of this repo and believe that this is not a duplicate.
• [x ] I have searched the documentation and believe that my question is not covered.

Feature Request

Looking through the documentation I was able to find adding optional/extra dependencies to a project as well as installing, but I could not find information about their inclusion in the build process. I see two scenarios:

1. Poetry already supports this and the documentation should be edited accordingly to accommodate.
2. Poetry does not support this and should provide some sort of API to do so similarly to the current process for installing extra req.s, but for building:

poetry build --extras "mysql pgsql"
poetry build -E mysql -E pgsql

[teichert]

@JulianRMedina Is this stack-overflow answer related?

[JulianRMedina]

@teichert, thank you for the response but this is a different problem than what I was thinking. That stack-overflow post talks about the install of optional or extra dependencies, I'm specifically talking about the inclusion of optional/extra dependencies in a package wheel.

My exact use-case was the following:
I have a package that uses onnxruntime. You can install onnxruntime or onnxruntime-gpu but not both as they're not compatible with each other. Ideally, I'd like to be able to have the following work-flow:

poetry build --E onnxgpu
poetry build --E onnxcpu

or something similar, so that I can create two different wheels, one with support for a CPU, and another with support for a GPU.

After considering this, it may be some type of anti-pattern, and I curse onnxruntime for separating support for CPU and GPU but it's the scenario I am in.

[abn]

1. Wheels themselves do not include the dependencies, but rather only metadata specifying dependencies. An exception to this rule is when the package maintainer bundles or vendors dependencies. This is warranted only in very specific scenarios and is not considered best practice in a lot of cases and requires custom tooling. This also means that when the wheel is built, only the dependencies specified for the build backend (independent of package requirements) are required.
2. Python packaging does not have a standard way to manage build time platform dependencies, for example any compile time libraries (cuda etc.) required by your build script. However, Python package dependencies for package builds can be specified thanks to PEP 518. This does not apply in your case.
3. A wheel binary built by poetry follows the naming convention as specified in PEP 427. While this is convention, downstream tools do tend to rely on this, and likely will make your wheels be installable only via direct reference url installs as opposed to by specifying name and version constraints.
4. You can specify dependencies such that they install only if certain environment markers apply. However, this does not help in your scenario since the decision to install a cpu or gpu runtime needs to be made ahead of time by the end user.

Considering all of the above, it is unlikely that such a feature, as you have described, will be implemented. Especially since the wheels need to adhere PEP 427. Typically, this is usually handled by specifying extras (as mentioned above).

There is a broader feature that might benefit a use case like this described in #2270. But that is still under discussion. As for a way forward for your package, I would recommend something like the following.

[tool.poetry]
name = "awesome"

[tool.poetry.dependencies]
onnxruntime = { version = "^1.3.0", optional = true }
onnxruntime-gpu = { version = "^1.3.0", optional = true }

[tool.poetry.extras]
cpu = ["onnxruntime"]
gpu = ["onnxruntime-gpu"]

While this would mean that the user needs to decide if they want to install the cpu version or the gpu version ahead of time, this will ensure that the user only installs the required dependencies for their environment.

# this will install onnxruntime and not onnxruntime-gpu
pip install awesome[cpu]

# this will install onnxruntime-gpu and not onnxruntime
pip install awesome[gpu]

Even if you had two separate wheels built as you described, you will end up with exactly the same content in both wheels except for one requirement specification in the wheel metadata having a suffix -gpu.

If you absolutely must have a separate package you could, split the project into three. A common package that contains your logic and a -cpu and -gpu version that each depend on your common package and also their corresponding onxruntime. Less than ideal, I know. Note that since you do not have any cpu/gpu specific files in your own project, this is not something I would recommend.

[JulianRMedina]

@abn I appreciate the feedback and feel settled in the matter, and I thought I may have been chasing some sort of anti-pattern to my problem. I'll resolve this now.

[pvardanis]

1. Wheels themselves do not include the dependencies, but rather only metadata specifying dependencies. An exception to this rule is when the package maintainer bundles or vendors dependencies. This is warranted only in very specific scenarios and is not considered best practice in a lot of cases and requires custom tooling. This also means that when the wheel is built, only the dependencies specified for the build backend (independent of package requirements) are required.
2. Python packaging does not have a standard way to manage build time platform dependencies, for example any compile time libraries (cuda etc.) required by your build script. However, Python package dependencies for package builds can be specified thanks to PEP 518. This does not apply in your case.
3. A wheel binary built by poetry follows the naming convention as specified in PEP 427. While this is convention, downstream tools do tend to rely on this, and likely will make your wheels be installable only via direct reference url installs as opposed to by specifying name and version constraints.
4. You can specify dependencies such that they install only if certain environment markers apply. However, this does not help in your scenario since the decision to install a cpu or gpu runtime needs to be made ahead of time by the end user.

Considering all of the above, it is unlikely that such a feature, as you have described, will be implemented. Especially since the wheels need to adhere PEP 427. Typically, this is usually handled by specifying extras (as mentioned above).

There is a broader feature that might benefit a use case like this described in #2270. But that is still under discussion. As for a way forward for your package, I would recommend something like the following.

[tool.poetry]
name = "awesome"

[tool.poetry.dependencies]
onnxruntime = { version = "^1.3.0", optional = true }
onnxruntime-gpu = { version = "^1.3.0", optional = true }

[tool.poetry.extras]
cpu = ["onnxruntime"]
gpu = ["onnxruntime-gpu"]

While this would mean that the user needs to decide if they want to install the cpu version or the gpu version ahead of time, this will ensure that the user only installs the required dependencies for their environment.

# this will install onnxruntime and not onnxruntime-gpu
pip install awesome[cpu]

# this will install onnxruntime-gpu and not onnxruntime
pip install awesome[gpu]

Even if you had two separate wheels built as you described, you will end up with exactly the same content in both wheels except for one requirement specification in the wheel metadata having a suffix -gpu.

If you absolutely must have a separate package you could, split the project into three. A common package that contains your logic and a -cpu and -gpu version that each depend on your common package and also their corresponding onxruntime. Less than ideal, I know. Note that since you do not have any cpu/gpu specific files in your own project, this is not something I would recommend.

I'm confused. Even if we build and publish the package with poetry, what happens when the user wants to install the package with one of the 2 extras?

I published the package and tried to install it in a different environment from PyPI but something like:
pip install awesome[cpu]

didn't work in that case.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.