-
Notifications
You must be signed in to change notification settings - Fork 50
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Flatcar SDK: add experimental prefix builds #1158
Conversation
Cgroupsv1 and Cilium tests failed: https://github.com/flatcar/scripts/actions/runs/6246234944?pr=1158 (and the test results markdown is too large to be added as a comment here, lol) - but this seems unrelated to this PR? All other tests pass. |
Build action triggered: https://github.com/flatcar/scripts/actions/runs/6362248084 |
As per Dongsu's comment the cgroupsv1 issue is unrelated, and all other tests pass. The PR should be ready for review. |
81fb322
to
beb1bc3
Compare
Rebased on latest main to resolve a conflict with glibc ebuilds. |
beb1bc3
to
e8d7d81
Compare
Rebased on the latest main. |
e8d7d81
to
4bc898d
Compare
Rebased on latest main to ingest the kernel update that fixes the cgroups v1 issue. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not entirely sure if another overlay is really needed. I think that ideally everything we need is in coreos-overlay for our stuff and in portage-stable for stuff straight from Gentoo.
Not a fan of having duplicated packages in prefix overlay. I assume this is temporary as they have some fixes for prefix builds that will eventually find their way to either portage-stable or coreos-overlay (I'm looking at app-misc/ca-certificates and sys-apps/baselayout here).
Also, is this really Flatcar specific? I mean - it uses Flatcar tooling and build scripts, but looks like a thing you could pull off using Gentoo repo. I suppose it could be made generic to support building stuff either from Gentoo as primary + extra overlays or from portage-stable as primary + coreos-overlay.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is this eclass being added here instead of using the eclass from portage-stable?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is an import from upstream Gentoo to fix a prefix build issue.
Integrating it with portage-stable is part of the prefix stabilisation process; for now prefix is experimental.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(Also, we carry prefix specific patches on top of upstream)
sdk_container/src/third_party/prefix-overlay/prefix/prefix-final/prefix-final-0.0.1.ebuild
Outdated
Show resolved
Hide resolved
@@ -0,0 +1 @@ | |||
sys-apps/portage ~amd64 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure I like the idea of pulling in unstable portage.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This shouldn't be necessary now that 3.0.51 is stable. At least 3.0.50 was needed for the transition to Meson, which was primarily motivated by this work.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Unfortunately this is necessary for cross-boss' cb-bootstrap to work. It's only used for bootstrapping the "staging" area and explicitly disabled via PREFIX_KEYWORDS when emerging into the prefix.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
gcc-13 is unstable too, right? But if we switch to gcc-13 then we can remove the above.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Stable on amd64, unstable on arm64. We have some accept keywords in coreos-overlay, I suppose that prefix-overlay will need its own.
app-misc/ca-certificates::coreos # coreos ca-certificates doesn't support prefix | ||
sys-apps/baselayout::coreos # coreos baselayout depends on systemd |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah, so this explains why those packages are added to the prefix-overlay. Do you plan to work on adding the missing features to coreos-overlay counterparts?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The packages are fundamentally different to their upstream counterparts, which is a shame. I feel like they don't have to differ quite so much. The coreos baselayout could possibly even be an additional package instead.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It depends on the feasibility - but it's sure worth researching. Baselayout in coreos-overlay in particular is very Flatcar specific; if we can manage to use stock Gentoo baselayout then this can be removed here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Using Gentoo baselayout instead of our own stuff would be nice. Maybe it's even possible, who knows. Just no one had a look at it yet. We have a bunch of our own stuff in our baselayout fork, though.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
But until then, I suppose we would need to do some work on our fork to support eprefix in order to make the prefix overlay come out of experimental status.
@@ -0,0 +1 @@ | |||
>=sys-devel/gcc-13 # FIXME: Cannot cross-compile gcc 13+ using older version |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We are going to have gcc 13 soon in SDK, so this could be dropped, I think.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hopefully! For now it's required to make bootstrapping work. BTW if we switch to gcc-13 then I believe we can also remove the unstable un-mask from bootstrapping.
sdk_container/src/third_party/prefix-overlay/dev-libs/libgpg-error/libgpg-error-1.47.ebuild
Outdated
Show resolved
Hide resolved
Absolutely agree - for the first (experimental) prefix support in the SDK these are a (temporaty) necessity for 2 reasons:
We aim for prefix-overlay to eventually be merged into our regular overlays. Keeping it separate for now allows us to ingest the prefix feature and ship it with the SDK, which will enable more devs to work on this feature.
All packages in prefix-overlay need to be justified - if it's just duplicates then we should not carry them.
This feature is meant to extend the scope of the SDK to also build distro independent packages. It's not Flatcar specific but it uses the portability of our SDK container and the well-defined environment we provide in our SDK releases to make prefix builds approachable and easy for a larger developer audience. |
This change adds experimental prefix builds to the Flatcar SDK. Prefix builds use a custom sys prefix path and emerge all binaries and runtime dependencies into that prefix. This path can then e.g. be shipped as a portable sysext since it includes all dependencies, and has libraries at a custom path so these do not conflict with libraries on target systems. Prefix uses a staging environment (path) featuring a full-blown development environment, and a "final" environment for installing. Staging and final need to be created using setup_prefix first, which will also create an emerge wrapper to emerge ebuilds into staging and subsequently final. The root fs in final may then e.g. be used to create a distro independent, portable sysext. Co-authored-by: James Le Cuirot <chewi@gentoo.org> Co-authored-by: Jeremi Piotrowski <jpiotrowski@microsoft.com> Co-authored-by: Thilo Fromm <thilofromm@microsoft.com>
Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
Thank you @krnowak! Great review and awesome feedback, as always. Co-authored-by: Krzesimir Nowak <knowak@microsoft.com>
- remove explicit "-multilib" from prefix keywordsas it is set in profile - split heredoc for generating emerge wrapper so we don't need to escape - add sys-apps/bubblewrap and virtual/tmpfiles to package update automation list - use prefix build fix for libgpg-error from upstream Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
fe39f9a
to
f846b03
Compare
NOTE that for testing the PR you'll need to use the nightly SDK from 2 nights ago - last night's build failed and there is no SDK. To do this, start the container like this:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd say let's get this in. Ideally there would be some automatic thing going on (like a github action), so it would catch breakage quickly before it bitrots too much.
Agreed @krnowak . I've created flatcar/Flatcar#1195 for tracking and added it to the relevant boards. |
This change adds experimental prefix builds to the Flatcar SDK.
Prefix builds use a custom sys prefix path and emerge all binaries and runtime dependencies into that prefix.
This path can then e.g. be shipped as a portable sysext since it includes all dependencies, and has libraries at a custom path so these do not conflict with libraries on target systems.
Prefix uses a staging environment (path) featuring a full-blown development environment, and a "final" environment for installing. Staging and final need to be created using setup_prefix first, which will also create an emerge wrapper to emerge ebuilds into staging and subsequently final. The root fs in final may then e.g. be used to create a distro independent, portable sysext.
See the readme included with this PR, at https://github.com/flatcar/scripts/blob/t-lo/experimental-prefix-support/PREFIX.md, for more information.
This PR is a vetted, rebased, and cleaned up version of #1151 . In the previous PR @pothos had helpful and interesting feedback regarding library linker paths - happy to continue this discussion here!
That PR in turn was a rebased version of the original Flatcar prefix work by @chewi , @jepio , and @HappyTobi here: https://github.com/chewi/flatcar-distro/tree/hackathon-prefix-builds .
How to use
Prefix requires cross-boss to be available locally.
Tested with the nightly SDK version
3740.0.0+nightly-20230927-2100
(the nightly after that failed so there's no newer SDK available).The latest development is in the
t-lo/experimental-prefix-support
branch.git clone https://github.com/chewi/flatcar-distro.git cd flatcar-distro git checkout t-lo/experimental-prefix-support git clone https://github.com/chewi/cross-boss ./run_sdk_container -t -V 3740.0.0+nightly-20230927-2100
In the SDK container, initialise a prefix, then build (for example) python into it:
Prefixes are architecture aware; the same prefix name can be used for multiple archs
"Final" environments will be in
__prefix__/<board>/prefixtest
after the builds conclude.__prefix__/<board>/prefixtest/root
contain the root FS and can be used to create sysexts.Testing done
Built the above (see "How to use") for amd64 and arm64, created a python sysext, tested on Flatcar x86_64 and arm64 QEmu instances.
Also, added upstream Gentoo as a 4th overlay to a prefix and was able to seamlessly emerge Gentoo packages. Tested a "midnight commander" sysext on both amd64 and arm64 qemu instances.
Advanced usage
Developers can actually add Gentoo as a custom overlay to their prefix once it was set up.
my-testprefix
with path/usr/local/pfx
, simply:/build/prefix-amd64-usr/my-testprefix/root/usr/local/pfx/etc/portage/repos.conf/gentoo.conf
and/home/sdk/trunk/src/scripts/__prefix__/amd64-usr/my-testprefix/root/usr/local/pfx/etc/portage/repos.conf/gentoo.conf
with
emerge-prefix-my-testprefix-amd64-usr app-misc/mc
to emerge midnight commander from Gentoo.changelog/
directory (user-facing change, bug fix, security fix, update)/boot
and/usr
size, packages, list files for any missing binaries, kernel modules, config files, kernel modules, etc.