Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enhance build times disabling command-not-found during rootfs creation #6616

Merged
merged 14 commits into from
Jun 9, 2024

Conversation

alexl83
Copy link
Contributor

@alexl83 alexl83 commented May 16, 2024

Description

fix issue #6531 by disabling "command-not-found" apt hook during build-time (which takes ages under qemu) and re-enabling after succesfully creating rootfs
No impact for end user as command-not-found will create/update its internal db natively/transparently on SBC during first sudo apt update

This shaves and additional ~2 minutes on a cached build with ORAS and CCACHE

How Has This Been Tested?

  • Built vanilla trixie & bookworm & noble & jammy for opi03
  • Built vanilla trixie & bookworm & noble & jammy for opi5-plus

Checklist:

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code

@alexl83 alexl83 requested review from a team and igorpecovnik as code owners May 16, 2024 20:42
@github-actions github-actions bot added size/small PR with less then 50 lines Framework Framework components labels May 16, 2024
@alexl83 alexl83 changed the title Fix issue#6531 by disabling command-not-found at build time Fix issue #6531 by disabling command-not-found at build time May 16, 2024
extensions/disable-cnf_chroot.sh Outdated Show resolved Hide resolved
lib/functions/configuration/main-config.sh Outdated Show resolved Hide resolved
@igorpecovnik igorpecovnik added 08 Milestone: Third quarter release Needs review Seeking for review labels May 17, 2024
@alexl83 alexl83 force-pushed the fix_slow_image_creation branch 4 times, most recently from 490ba36 to 7725ac2 Compare May 17, 2024 19:20
@alexl83 alexl83 requested a review from rpardini May 17, 2024 19:59
@alexl83 alexl83 requested a review from rpardini May 18, 2024 09:37
@rpardini
Copy link
Member

I agree, c-n-f is useful yes, but adds a lot of overhead.
It's also unclear to me when the c-n-f database is updated: is it 1) when the first apt update happens after cnf is installed, 2) when c-n-f itself is installed or 3) somehow else?
This somehow relates to the synaptic index debacle we had a few years ago, which related to shipping-or-not the apt package lists in the images. Having those in increases the image size, but possibly lowers the time needed for the first apt update when running on the board.
I'd sincerely prefer to have everything in place (possibly at the very latest possible moment, so that it does not run/affect the build process), but also not ship c-n-f/synaptic/apt-lists in the image: leave it to the user doing the first apt-update after booting on real hardware. But not everyone agrees...

@alexl83
Copy link
Contributor Author

alexl83 commented May 18, 2024

I agree, c-n-f is useful yes, but adds a lot of overhead. It's also unclear to me when the c-n-f database is updated: is it 1) when the first apt update happens after cnf is installed, 2) when c-n-f itself is installed or 3) somehow else? This somehow relates to the synaptic index debacle we had a few years ago, which related to shipping-or-not the apt package lists in the images. Having those in increases the image size, but possibly lowers the time needed for the first apt update when running on the board. I'd sincerely prefer to have everything in place (possibly at the very latest possible moment, so that it does not run/affect the build process), but also not ship c-n-f/synaptic/apt-lists in the image: leave it to the user doing the first apt-update after booting on real hardware. But not everyone agrees...

I think it makes sense to have it installed in rootfs

cnf works like this:
/etc/apt/apt.conf.d/50command-not-found updates or re-creates database /var/lib/command-not-found/commands.db{,.metadata} everytime apt update is run (incrementally if db is already present)

stand-alone: root user can always run /usr/sbin/update-command-not-found to create/populate/update db at any given time (it was standard behaviour for first run after install many ubuntus ago) - it's somehow rendered useless by apt hook IMHO

While debootstrap and rootfs creation is non-interactive, cnf is basically useless at build-time: saving less than 20 seconds on native builds might be overkill and not useful for end-user, while saving 25-30 minutes on quemu builds makes all the sense of the world to me, especially since at first apt-update on SBC, cnf (which my changes re-enable before creating final image) will kick in natively and create its first database in less than 20 seconds during apt update

Everything is reasonably transparent to end user in my opinion, ponderating calculated impacts reported above
As an end-user I always appreciated cnf as it saved me countless apt queries to find which package to install in order to get specific binaries

regarding this:

I agree, c-n-f is useful yes, but adds a lot of overhead. It's also unclear to me when the c-n-f database is updated: is it 1) when the first apt update happens after cnf is installed, 2) when c-n-f itself is installed or 3) somehow else?

Correct answer is 1, with a twist - first apt update + every subsequent apt update (which will be incremental since first run already created a db) -- still during build time every subesquent run make build times a lot longer and excruciating by adding more pain

hope it helps :)

@alexl83 alexl83 force-pushed the fix_slow_image_creation branch 4 times, most recently from 71fb331 to 23ba9e2 Compare May 23, 2024 16:36
@alexl83 alexl83 changed the title Fix issue #6531 by disabling command-not-found at build time Enhance build times disabling command-not-found during rootfs creation Jun 9, 2024
@igorpecovnik igorpecovnik added Ready to merge Reviewed, tested and ready for merge and removed Needs review Seeking for review labels Jun 9, 2024
@igorpecovnik igorpecovnik merged commit eb7b39c into armbian:main Jun 9, 2024
@alexl83 alexl83 deleted the fix_slow_image_creation branch June 14, 2024 19:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
08 Milestone: Third quarter release Framework Framework components Ready to merge Reviewed, tested and ready for merge size/small PR with less then 50 lines
Development

Successfully merging this pull request may close these issues.

3 participants