Incremental build changes

[bdrewery]

We should make bulk -S the default and rely on PORTREVISION bumps. This will greatly reduce package rebuilds.
It depends on freebsd/pkg#1831 which has more details on what needs to change here and how to interact with pkg differently.

Edit: bulk -S isn't exactly same behavior being discussed here. The idea that we don't force rebuild packages because a dependency is updated is what this is. A port version/revision/epoch bump would still do a rebuild.

[bdrewery]

#806 needs to be considered here too.

[karolyi]

I'm +1ing here, with elaborating as to why I need this functionality:

I have 2 jails building with mysql dependencies, which in turn (at the time of writing this) builds llvm90 each time of a base system update (freebsd patch level change, for example). llvm90 builds for ~25 minutes each time, with an AMD Ryzen server with 16 cores, while maxing out load.

I think rebuilding llvm only to build mysql-(client|server) is totally superfluous and unnecessary, since it has no custom build options. As it is only needed for the mysql build, it would be fine just to install it from a package repository, for which I have no option now.

So as you can see, the issue isn't just about 'incremental builds', because on a base system change (i.e. the aforementioned freebsd patch level change) poudriere deletes the entire built package tree and rebuilds everything. Which is fine, but llvm could be seeded into the build environment from a binary package, saving lots of time and resources.

[bdrewery]

I have a prototype for this that I will put into the next poudriere-devel for wider testing.

[heliosyne]

Is there progress on this? I've had to abandon poudriere entirely because I can't get it to not build the entire dependency tree when all I need it to do is build one port with non-default options, and use its testport facility on stuff I maintain.

[bdrewery]

Is there progress on this? I've had to abandon poudriere entirely because I can't get it to not build the entire dependency tree when all I need it to do is build one port with non-default options, and use its testport facility on stuff I maintain.

I'm sorry for the trouble. I feel your pain. The frustration from rebuilds and rust crashing my build system so many times is what keeps me working on this currently.

I have it working locally and have been working over the past month to clean it up before pushing it out. Right now I am focused on adding more tests to capture some of the specific odd cases that the changes ran into. In particular there were no build queue, testport, options, pkgclean, distclean tests committed yet. I have some for each of those now.

It's a big change and as I tested it more I found more pieces which were needing manual testing. So I've been adding more tests around those areas.

 148 files changed, 8783 insertions(+), 2863 deletions(-)

Most of that is just in the test dir:

 108 files changed, 4087 insertions(+), 459 deletions(-)

The new build tests explicitly are checking for what is built to make sure only what is needed to rebuild is rebuilt.

It's coming soon. I want it out within the next week or two.

[karolyi]

Really looking forward to this.

Just another example: today another ca_root_nss arrived, which in turn made curl to rebuild, which in turn made rust to rebuild, which in turn made clamd rebuild. I only really use clamd.

Rust builds for ~35 minutes, burning all 16 cores in my server.

[michael-o]

Really looking forward to this.

Just another example: today another ca_root_nss arrived, which in turn made curl to rebuild, which in turn made rust to rebuild, which in turn made clamd rebuild. I only really use clamd.

Rust builds for ~35 minutes, burning all 16 cores in my server.

I feel with you. ca_root_nss is one of the most useless ports these days, it has been long superseded with the CA certs system in base (though a few fixes need to be done). I am look at you @kevans91 ;-)

[igalic]

this makes me think that there probably are a number of dependencies out there that when bumped don't need packages depending on them to rebuild.

as in: it might need verification / testing, but not a rebuild.

How many people who use poudriere run a package's tests?

[bdrewery]

Poudriere running tests will be handled in #355. Probably a lot don't get ran.

Not exactly responding to what you're asking but it made realize I should brain dump this out somewhere.

In general what I can think of that needs a chased rebuild (PORTREVISION bump) are

• Package provides a shared library with a new name/version
• Package provides a static library that has changed
• Package is a generator tool like a compiler
• Package is a script runtime (Perl, Ruby, Python, PHP)
  • Ruby has bitten me here but I am unsure why.
• Package consumers modules built by other packages with strict API requirements.
  • PHP will bump its API version and fail to load extensions not rebuilt to match the new version.
• Default version changes like freebsd/freebsd-ports@8fd85e6

Through testing this change over the past year I have ran into most of those. Each was very confusing to debug and they were not obviously because of this change. That's a big reason I've been moving slow on releasing this.

Simple changes like a shared library package has behavior changed or docs changed but the shlib version does not change - no PORTREVISION chase is needed.

Some of these don't require a chased rebuild for functionality but for the sake of reproducible builds they do. When we communicate the "new" rebuild to porters we'll want the key point that anything that changes reproducibility of downstream packages will require a PORTREVISION chase.

Committers do forget to do a chase, or are unaware that it is needed, or rely on Poudriere to do the chasing for them. Relying on Poudriere was never the right thing because for reproducibility.

The only case I have special checks for are shared libraries. If a package requires shared libraries it will check that the package that provides the library still provides it. If not then it rebuilds the package assuming that the shared library has a newer version/name to use. This is non-trivial. Ports don't tell what libraries they provide in an easy way. Ports do say which they require but not what they provide. So Poudriere needs to do a lot of work to make these decisions which can lead to a NOP-build spending minutes solving which packages provide which libraries. This special case is a big reason I haven't pushed this out yet as I'm unhappy with how it is implemented. I need to make it smarter but need tests for it first. I didn't add tests initially.

I could see handling some of the other special cases depending on community testing of the change.

[bdrewery]

I have 2 more commits to add tests for. It's almost done.

[grahamperrin]

Does work in progress on this issue explain why ports such as llvm15 are sometimes queued, then not (a good thing)?

[karolyi]

3.4.0 (installed from latest) shipped with a change that reads amongst others:

…
- testport|bulk -b: New feature to fetch remote packages. See PACKAGE_FETCH
  options in poudriere.conf.sample.
…

@bdrewery, is this the change we've been waiting for? I've set this up, couldn't test it as there are no updates for my jails now (in terms of curl/rust or anything that pulls these). I've did a quick run, couple python packages updated, nothing significant that would put a real strain on my server.

Merry Christmas btw.

[karolyi]

Well, if it is, it's not working. Even when I specified PACKAGE_FETCH_WHITELIST="rust", it gets rebuilt no matter what.

[igalic]

sometimes, rust changes underneath, due to transient dependencies, such as curl updating.

if the rust in package repos is recorded with the old version of curl, your local build will, unfortunately correctly, determine that rust needs to be rebuilt.

it would be nice to find ways around this, especially in cases where there was no ABI change to a library.

[ohaucke]

@igalic You could get the lastest commit hash, that was used for the builds, from here https://pkg-status.freebsd.org/api/1/builds?type=package&latest=True and use it for the git checkout to run your build.

That way, i usually don't need to build rust, gcc & llvm

[bdrewery]

I merged in the major test framework changes this depends on today. I plan to merge the feature in "soon". Subpackages became a distraction but I've decided I'll leave subpackages broken and focus on getting this in.

[bdrewery]

It's happening. I had a few hundreds commits behind this work that are now merged. There's only 3 commits left to merge. I'm fixing up some shared library tracking changes and then this will be in.

[markmcb]

@bdrewery, not sure if this is the best place to call it out, but -S is still listed 3 times in the poudriere-bulk(8) synopsis (version poudriere-git-3.4.99.20240424).

SYNOPSIS
     poudriere bulk -a -j name [-CcFIikNnRrSTtvw] [-B name] [-b branch]
                    [-J maxjobs[:prebuildmaxjobs]]
                    [-O overlay [-O overlay2 ...]] [-p tree] [-z set]
     poudriere bulk -f file [-f file2 ...] -j name [-CcFIikNnRrSTtvw]
                    [-B name] [-b branch] [-J maxjobs[:prebuildmaxjobs]]
                    [-O overlay [-O overlay2 ...]] [-p tree] [-z set]
     poudriere bulk -j name [-CcFIikNnRrSTtvw] [-B name] [-b branch]
                    [-J maxjobs[:prebuildmaxjobs]]
                    [-O overlay [-O overlay2 ...]] [-p tree] [-z set]
                    origin ...

[bdrewery]

@markmcb thanks, fixed.

I have merged this into git. Updating the port now. Off-by-default. I will send an email to ports later today explaining how to enable, use, and debug it.

[bdrewery]

See freebsd/freebsd-ports@eef5ff6 for summary of feature

[karolyi]

Hey,

I started using poudriere-devel a while ago, and wanted to wait for this moment, until curl becomes updated and per the poudriere-stable branch, rust would be recompiled because of it.

I was hoping that this devel version and the changes you added within would solve this issue but it didn't. Curl had a new version today, and rust will be recompiled, whereas rust is only needed for the compilation of clamd, which means a binary version of it would be totally fine for the time of the compilation. Same goes for mysql-server in another jail, that is only pulled in by teamspeak-server and isn't used.

Deleting rust-1.81.0.pkg: missing dependency: curl-8.10.0_1

So my question is: do you have information on how I can avoid this with poudriere-devel? Compiling rust is not only unnecessary, it takes huge resources, as it takes 36 minutes with my server sweating blood. Same with mysql-server.

Thanks for your help in advance.

[markmcb]

@karolyi do you have PKG_NO_VERSION_FOR_DEPS=yes set in poudriere.conf?

I had the same update and a rust rebuild was not triggered. It’s been working as expected for me.

[karolyi]

@markmcb thanks for your reply. No I don't, because I wasn't sure what exactly I should have used to not have rust rebuilt.

I'll try next time (I guess I can't avoid building it now), but the cleanest solution would be just to pull a rust binary for the build time, and maybe to the packages. In my case, it isn't even installed when I install the jail, since it's not a requirement.

[karolyi]

Hm. I made a build with -S (no PKG_NO_VERSION_FOR_DEPS=yes added yet), and it skipped building mysql-server for the other jail.

Now I removed -S as it is claimed to be deprecated and added PKG_NO_VERSION_FOR_DEPS=yes. Now it started to rebuild cmake-core and will rebuild mysql-server too I think.

Update: yes it is rebuilding mysql-server too.