Provide support for passing build flags down the dependency chain

[marcusva]

Abstract

poudriere creates packages from ports in isolated jails, which feature their own set of environment variables that are usually set on starting the jail. A dependency X of a port Y thus is unaware of any environment settings the ports framework does on Y's behalf.
In some situations, it is necessary that environment settings are passed down the build chain to allow dependencies to build correctly based on whatever the framework decides "to be the best" for port Y.

Problem Space

As of writing, the FreeBSD ports tree allows users and ports to build, install and use different versions of the Python language (lang/python) at the same time. Ports using the Python bits of the ports framework (namely Mk/Uses/python.mk) are enabled to choose a specific Python version or can be marked to support a series of Python versions, giving users (and other ports) the opportunity and flexibility to pick their specific preferred and supported version of the Python language.

The pre-pkg era did not allow ports to install packages for different Python versions, which used the same origin information, which was resolved with the availability of pkg 1.3. The ports and pkg framework are capable of installing e.g. py33-foo and py27-foo originating from devel/py-foo at the same time.

A single poudriere instance however is unable to create such a set of packages. Within a single run, it can only create a package py33-foo or py27-foo, but not both. This in turn limits the possibilities of creating different version-specific packages based on the same origin and won't allow different ports to refer to a shared dependency, which would support those versions.

Previous Workarounds

The limiting situation was resolved in the pre-pkg era by duplicating ports, renaming them and referring to either one of them within the specific consumers.

A port py-onlypy33 may be limited to Python 3.3 and may depend on py-foo, which supports not only Python 3.3, but also Python 2.7. Another port py-onlypy27may also depend on py-foo, but only works with Python 2.7. This lead to the typical following workaround:

• Create port for py-foo, using Python 2.7
• Create port for py33-foo, using Python 3.3
• py-onlypy33 refers to py33-foo in the BUILD_DEPENDS or RUN_DEPENDS
• py-onlypy27 refers to py-foo in the BUILD_DEPENDS or RUN_DEPENDS

Current Situation within the Ports

With pkg 1.3 and newer, a workaround as shown above is not necessary anymore. The Python bits of the ports framework allow ports to pass certain flags (specifically DEPENDS_ARGS=PYTHON_VERSION=<versionnumber>) to their dependencies, which can (re)act according to those. If a user wants to install py-onlypy33, the following will happen:

1. User executes make install for py-onlypy33
2. python.mk will append PYTHON_VERSION=python3.3 to the DEPENDS_ARGS
3. ports framework traverses into the py-foodependency
4. python.mk for py-foo notices PYTHON_VERSION and will instruct it to be built for Python 3.3
5. py33-foo is installed
6. py-onlypy33 detects the proper py33-foo package and will continue to build and install

If the user is going to install py-onlypy27, the following will happen:

1. User executes make install for py-onlypy27
2. python.mk will append PYTHON_VERSION=python2.7 to the DEPENDS_ARGS
3. ports framework traverses into the py-foodependency
4. python.mk for py-foo notices PYTHON_VERSION and will instruct it to be built for Python 2.7
5. py27-foo is installed
6. py-onlypy27 detects the proper py27-foo package and will continue to build and install

Current Situation with Poudriere

Poudriere builds ports and their dependencies in (mostly) isolated jail environments, with the ability to parallelize builds. Information, such as the DEPENDS_ARGS from above are not shared or passed down the build chain. This results in the following behaviour for py-onlypy33:

1. poudriere gathers dependencies for py-onlypy33
2. poudriere runs the build for py-foo using the configured default python version (e.g. 2.7)
3. poudriere registers a package py27-foo
4. poudriere raises an error for py-onlypy33 due to the unsatisfied dependency for py33-foo

Solution

Poudriere needs to be able to pass around specific flags or settings, which can be set by a port or the ports framework. The ports framework may define a specific set of global variables, which should be respected by package builders (poudriere, tinderbox, etc.).

To prevent misbehaviour and to avoid limiting the parallelization features of package builders, those global variables should be available prior to any specific task for a port (make fetch, make patch, etc.), so that the internal package builder workflow may look like:

1. detect port
2. calculate global flags
3. gather dependencies to be built (with the global flags)
4. build dependencies (with the global flags)
5. build port

A single variable PACKAGE_BUILD_FLAGS may be sufficient and should be able to be passed down to dependencies of dependencies of dependencies of ...

[bdrewery]

You've misunderstood something. Ports does recursive builds with chaining down deps. Poudriere does not. It gathers a list of dependencies and then builds them in 1 level separate from each other. There is no recursion or chain to pass environment variables down into.

There's no relation in package building between py-foo and py-dep. There's no way to have py-foo tell py-dep it wants version 33 versus 27.

Yes we could have py-foo have a DEP_FLAGS variable in its Makefile that Poudriere will ensure all deps are built with, but that would still limit you to 1 version. How would poudriere know it needs to build py-foo twice? Building py-dep twice should be fine (py-dep27 and py-dep33) as they have different PKGNAMES can be stored in the queue separately. But I see no way to have the leaf port built twice as Poudriere all origin-based for this piece.

I still think the simplest solution is to either package all versions of python in py-* (as gentoo does) or create slave ports as the rest of the ports framework/tree does. Creating them on-demand would be the most interesting. I would prefer this be in the framework though and not part of Poudriere as then we get into another "Port build tool XYZ is broken" like we are with Tinderbox now and portupgrade/portmaster often.

[marcusva]

This is about building dependencies with the correct information. py-foo tells the world that it (as well as its dependencies) needs version 33. A dependency thus needs to receive a piece of information that it shall built itself with version 33.
I do not see anything in your comment arguing against that (as long as we both have the same understanding of the leaf port being the one "owning" the dependencies).

Or do you talk about non-first-class ports (direct dependency of a port vs. dependency of a dependency)? That also should not be a problem as long as the initial port causing the build will set everything correctly (the same happens right now within a ports tree build)

Slave ports are not a viable option here, since this easily grows out of hand (it already does for the small couple of gnome-related ones, which only target two version (ranges). In the worst case, one needs a slave port for each particular version (with 4-5 python version being supported at the same time in the tree), which easily adds up to hundreds of slave ports.

[bdrewery]

I'm taking another look at this. I noticed that even bsd.ruby.mk has the DEPENDS_ARGS. I misunderstood some the changes that were made leading up to this problem.

[bdrewery]

Caching DEPENDS_ARGS per port and passing it to their dependencies may discover new packages that need to be built.

If we get dependencies B and C from PortA and dependencies C and D from PortB, we need to lookup further dependencies given the tuples B:PortA_DEPENDS_ARGS, C:PortA_DEPENDS_ARGS, C:PortB_DEPENDS_ARGS, and D:PortB_DEPENDS_ARGS. The DEPENDS_ARGS may all match so there would only be 1 actual lookup, but if they don't then we can discover new PKGNAMES from those lookups to queue. It could end up with A B C D, or A B B-II C C-II D.

I think implementing that makes sense and would solve this.

The problem then is that it's not easy with the current code (especially to keep performance acceptable for 24000 port build startups) but I do consider it a bug and in need of fixing.

[bdrewery]

And those DEPENDS_ARGS for the generated package would have to be passed into the actual build of it as well of course.

[bdrewery]

I think to fix this and set us up for the future of sub-packages, everything tracking an origin needs to instead track a "package" object which has stuff like the DEPENDS_ARGS and the origin it came from for the sake of building. The queue is already all pkgname-oriented at least.

[bdrewery]

dns/unbound/Makefile:DEPENDS_ARGS+=     WITH_GOST=yes
dns/unbound/Makefile:DEPENDS_ARGS+=     WITH_ECDSA=yes

These are wrong since we don't generally support multiple packages installed at once but the plan with Poudriere will allow supporting something like this in the future perhaps.

[bdrewery]

@bapt 's comment on this is that the queue should track origin:key=vaue:key2=value which would be an extensible format to support variants and other features in the future.

[bdrewery]

There's 2 implementations that tackle this problem.

https://github.com/bbqsrc/poudriere/commit/73dfbc4a88b79ca2288c0732522e04f38eabdcd8 and #314

[bdrewery]

I have a PoC for this working.

testport net/tiny-network-utilities
Current:
[00:00:08] ====>> Debug: net/tiny-network-utilities |  depends on net/py-netifaces
[00:00:08] ====>> Debug: net/tiny-network-utilities |  depends on textproc/py-hexdump
[00:00:09] ====>> Debug: net/py-netifaces depends on lang/python27
[00:00:09] ====>> Debug: net/py-netifaces depends on devel/py-setuptools27
[00:00:09] ====>> Debug: devel/py-setuptools27 depends on lang/python27
[00:00:09] ====>> Debug: textproc/py-hexdump depends on devel/py-setuptools27
[00:00:09] ====>> Debug: textproc/py-hexdump depends on lang/python27
Fixed:
[00:00:08] ====>> Debug: net/tiny-network-utilities | tiny-network-utilities-0.150519 depends on net/py-netifaces
[00:00:08] ====>> Debug: net/tiny-network-utilities | tiny-network-utilities-0.150519 depends on textproc/py-hexdump
[00:00:10] ====>> Debug: net/py-netifaces | py34-netifaces-0.10.4 depends on lang/python34 | python34-3.4.4_2
[00:00:10] ====>> Debug: net/py-netifaces | py34-netifaces-0.10.4 depends on devel/py-setuptools34 | py34-setuptools34-20.0
[00:00:10] ====>> Debug: textproc/py-hexdump | py34-hexdump-3.3 depends on devel/py-setuptools34 | py34-setuptools34-20.0
[00:00:10] ====>> Debug: textproc/py-hexdump | py34-hexdump-3.3 depends on lang/python34 | python34-3.4.4_2

There's several challenges to fix still. One is that for bulk -a we optimize the queue by assuming that we will visit all origins once when looking up dependencies and thus don't do recursive dependency lookup. That optimization will have to come out to make this work properly. That could seriously hurt the package build times on the cluster. So I need to come up with more optimizations for it.

Another challenge is the TRIM_ORPHANED_BUILD_DEPS feature which only works off of origins. This feature is enabled by default since it is an optimization to avoid building something unneeded.

What I've done is make a 2-tuple of ORIGIN:DEPENDS_ARGS to map to a particular PKGNAME. For the VARIANTS feature I would just need to add in the VARIANT as well. It's not too bad to extend for that, but I suspect that the proper VARIANTS implementation should be using DEPENDS_ARGS anyhow, so there may be nothing that needs to be extended.

[bdrewery]

Actually I will just add in a VARIANT now to the tuple. While a port may need to pass its VARIANT in DEPENDS_ARGS, the port itself relies on a particular VARIANT to determine its PKGNAME. So it does need to go into the set.

[bdrewery]

This may also break some ports.

Example at https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=195342 with fix at https://bz-attachments.freebsd.org/attachment.cgi?id=150142

[bdrewery]

There's an actual patch to resolve this and to take it a step further to support FLAVORS in #483. Note the problem is fixed in that PR even without FLAVORS; FLAVORS is just the next step.

[bdrewery]

6ce2d0b dropped this support. FLAVORS is the replacement.