RPM 4.20 change in directory structure changes expectations, thus making tests fail

[stratakis]

RPM 4.20 introduced rpm-software-management/rpm#2078, which basically makes the %prep section of the rpmbuild process extract the sources to BUILD/foo-<version>-build/foo-<version>/ instead of BUILD/foo-<version>/, essentially changing the use case we had before with the $TMT_SOURCE_DIR variable.

This breaks expectations, for example our Cython rawhide package testing fails as the dir structure is wrong, while the same works for Fedora 40, where RPM doesn't have this feature.

We can fix it by changing the directories of course, but ideally a conditionalized value of $TMT_SOURCE_DIR depending on the feature existence would be non-disruptive. Currently way too many packages that we have set a CI for will fail due to this change.

[hroncok]

IMHO $TMT_SOURCE_DIR remains the same, but the sources are now unpacked one level deeper.

[stratakis]

IMHO $TMT_SOURCE_DIR remains the same, but the sources are now unpacked one level deeper.

edited accordingly.

[lukaszachy]

I received some tips what to do, lets see.

[lukaszachy]

@pmatilai So I've tried the builddir=$(dirname $(find /foo demo.spec)) but the 'demo.spec' isn't copied into _builddir.

rpm-4.19.91-6.fc41.x86_64
rpm-build-4.19.91-6.fc41.x86_64

$ cat demo.spec 
Name: demo
Version: 1.0
Summary: Trying discover with applied patches
Release: 1
License: MIT
BuildArch: noarch
Source0: no-tmt-2.tgz

%description
Some tests are being added by patches, lets discover them correctly

%prep
%autosetup -n no-tmt-2

%install

%changelog
* Thu Jun 1 2023 Lukas Zachar <lzachar@redhat.com> 0.1-1

And when I do # rpmbuild -bp --define "_sourcedir $(pwd)" demo.spec --define '_builddir /foo' the created '/foo' contains no spec file.

# find /foo
/foo
/foo/demo-1.0-build
/foo/demo-1.0-build/SPECPARTS
/foo/demo-1.0-build/no-tmt-2
/foo/demo-1.0-build/no-tmt-2/all_in_one

Could the feasible workaround be the find where SPECPARTS directory is instead?

[pmatilai]

Sorry, I must've misread/misunderstood the case. It's hard to advice when you don't know the whole picture - I don't even know what TMT is, much less how it's used. The point is, you can compatibly get the directory that way by looking for a known "anchor" with find and take the dirname of that.

cd $(dirname $TMT_SOURCE_DIR/cython-*/tests)

Here, the anchor would be "tests", assuming I understood the context correctly, in the no-tmt-2 case it'd be all_in_one.

The SPECPARTS directory is unlikely to move again so that could sort of be used to detect the new layout, but these hardcoded assumptions is what led to this problem in the first place so it'd be nice to figure out something better. Note that older rpm versions will not have a literal "SPECPARTS" directory, it'll be something else in another directory.

The fundamental issue in this and several similar cases is that rpmbuild does not expect to be used this way at all, all this build directory stuff ultimately only exists for the purpose of building rpms. And so it doesn't export the environment in any way at all, because there has been no point in doing so. The most compatible way of doing things is allowing rpmbuild to run it.

I've now come across a case where a test unpacked an src.rpm, locally built it and ran the test-suite within. It failed with 4.20 because the paths were different, but it might've just well used rpmbuild -bi for running the %check (maybe "--with check" if disabled by default), and, with the realization the test-suite was enabled and that already happens during the build of the package this is supposed to be testing, the solution was to just delete the test. Not saying that's the case here, just an example of how we sometimes fail to see the forest from the trees.

I'd be open to considering some means of exporting that info, minimally output the relevant paths in the build log for example. (thoughts + feedback in rpm-software-management/rpm#3147 welcome)

[AdamWill]

I've now come across a case where a test unpacked an src.rpm, locally built it and ran the test-suite within.

We also have an entire standardized mechanism for doing this, which the change broke in the same way. https://bugzilla.redhat.com/show_bug.cgi?id=2290735

[pmatilai]

Yup, this is an entire pattern whose existence has taken us by surprise. We'll need to look into better supporting the use-case, but for the short-term, I came up with a trick to fish the contentious builddir from rpmbuild in what should be a compatible manner across old and new rpm versions. See rpm-software-management/rpm#3147 (comment)