[RFE] Stabilise SDK Prefix builds

[t-lo]

Current situation

Prefix builds is an experimental feature of the Flatcar SDK.
While the feature works and is usable, it is not well integrated with the SDK's board cross compiler environment.
As a result, it uses its own tools and wrappers, and it has a dependency on the external cross-boss cross compilation toolbox.
Lastly, Prefix builds are not automatically tested and rely on developer usage to detect breakage.

Impact

While prefix apps can be produced manually, they cannot be reliably integrated into automation for risk of random breakage.

Ideal future situation

Prefix builds are fully integrated with the SDK and regularly tested.

Implementation options

Fully integrating and stabilising it requires additional steps

• Integrate cb-bootstrap with the Flatcar SDK.
  Currently, setup_prefix uses cross-boss' cb-bootstrap to set up the prefix environment.
  Bootstrapping must be fully integrated with the Flatcar SDK before prefix builds are considered stable.
• Integrate prefix builds with /build/<board> environment and use board cross toolchain.
  Prefix builds currently use the SDK cross toolchains (/usr/<arch>-gnu/) instead of board toolchains in /build/<board>.
  Prefix builds must be integrated with the board toolchains and stop using cb-emerge before considered stable.
• Add prefix wrappers for all portage tools (similar to board wrappers), not just emerge.
• Add test cases for prefix builds to mantle/kola
  • An example prefix application is built for each architecture
  • The prefix application is converted into a sysext and functionality is tested in a qemu VM.

Stretch goals

• Optionally integrate Gentoo as a fourth overlay (requested via a commandline option to setup_prefix)

[pothos]

• Integrate prefix builds with /build/<board> environment and use board cross toolchain.
  Prefix builds currently use the SDK cross toolchains (/usr/<arch>-gnu/) instead of board toolchains in /build/<board>.
  Prefix builds must be integrated with the board toolchains and stop using cb-emerge before considered stable.

Does it really make sense to have prefix builds part of the board? I think these are separate things: one is for the final image, one is for a prefix build - having both binary packages end up at the same place sounds like it will cause problems because they are not compatible.

[jepio]

Integrate prefix builds with /build/ environment and use board cross toolchain.
Prefix builds currently use the SDK cross toolchains (/usr/-gnu/) instead of board toolchains in /build/.
Prefix builds must be integrated with the board toolchains and stop using cb-emerge before considered stable.

Thinking about this: /build/<board> does not (and can't) contain a cross-compiler and the sysroot part of it (glibc/linux-headers) is not used by the prefix build because it needs to build it's own version of them. Can you elaborate on what you imagine needs to be done here?