System Update utility #33

probonopd · 2021-01-17T21:05:12Z

Write a System Update utility that installs the update into a new Boot Environment.

Along the lines of

 pkg upgrade --fetch-only -y
 beadm list
 beadm create c572e
 beadm mount c572e /media/c572e
 chroot /media/c572e
  pkg upgrade -y
  pkg upgrade -r FreeBSD -y
  exit
 beadm list
 beadm umount c572e

Thanks @grahamperrin

We could then make the next boot go into the new Boot Environment, and ask the user there whether all subsequent boots should continue to go there. Or something like that.

The text was updated successfully, but these errors were encountered:

grahamperrin · 2021-01-18T02:55:27Z

Thanks, that's from my quick and dirty record of a first attempt https://pastebin.com/ppT7LfVT where (amongst other things) the pkg upgrade -y failed due to poudriere-related files being not found in the chroot(8) environment.

I took my hints from Upgrading from FreeBSD 9.3 to FreeBSD 10.1 using beadm and freebsd-update – Dan Langille's Other Diary (2015-03-13).

I see options such as --chroot and --rootdir in pkg(8), my first experiment with --chroot failed (through me not yet having a full understanding of the manual page). I guess that using one or both of those options in combination with (or in lieu of) chroot(8) will allow things to work with e.g. my poudriere repository, I simply haven't had time to digest the man pages.

For the pkg stuff above, the phrase System Update is potentially misleading in that base (the system) is not yet packaged. So better name the utility Software Update or something like that.

A System Update to the FreeBSD -RELEASE base of helloSystem should use freebsd-update(8).

A System Update to the FreeBSD -CURRENT base of helloSystem might be out of scope (without PkgBase).

`/dev`

~~Installed helloSystem 0.4.0 (0D26) mount point /dev has the required file system.~~

Old files

That reminds me,

root@mowa219-gjp4-8570p:~ # cd /usr/src && make -DBATCH_DELETE_OLD_FILES delete-old
>>> Removing old files (only deletes safe to delete libs)
/usr/sbin/fmtree
/usr/lib/debug/usr/sbin/fmtree.debug
/usr/share/man/man8/fmtree.8.gz
>>> Old files removed
>>> Removing old directories
>>> Old directories removed
To remove old libraries run 'make delete-old-libs'.
root@mowa219-gjp4-8570p:/usr/src #

– and

I do not make delete-old-libs
(off-topic from helloSystem) I can't easily recall what I have installed from ports
(OT) I do update (from source) my jail for poudriere, but I have never given thought to deletion of old files within the jail (does it take care of itself?) … maybe saner to simply create from source.

grahamperrin · 2021-02-16T18:37:37Z

helloSystem/docs#17

probonopd · 2021-02-16T20:23:12Z

We could also take an entirely different approach to "updating", e.g., installing from a new Live ISO while keeping parts of the old system around, similar to what the Mac does... or use an A/B partitioning scheme, similar to what ChromeOS does, or...

grahamperrin · 2021-02-16T20:40:26Z

… installing from a new Live ISO while keeping parts of the old system …

The parts gained from an ISO might as effectively be gained with a PkgBase approach. I think first of @igalic https://alpha.pkgbase.live/

probonopd · 2021-02-16T21:15:19Z

Not really: Integration and end-to-end tests can be run on an ISO, but not on an upgraded system. At least not easily.

grahamperrin · 2021-02-16T21:31:41Z

Integration

Way beyond me but 👍

end-to-end tests

Is an intention, to verify (with checksums) that what's installed is as it should be?

grahamperrin · 2021-02-16T21:38:35Z

@vermaden

#33 (comment) my bad,

Installed helloSystem 0.4.0 (0D26) mount point /dev has the required file system.

That was, not taking into account the chroot step. So, yes, mount -t devfs devfs /dev is a necessary next step.

vermaden · 2021-02-16T22:03:35Z

#33 (comment) thanks @vermaden

`/dev`

Installed helloSystem 0.4.0 (0D26) mount point /dev has the required file system.

Old files

That reminds me,

root@mowa219-gjp4-8570p:~ # cd /usr/src && make -DBATCH_DELETE_OLD_FILES delete-old
>>> Removing old files (only deletes safe to delete libs)
/usr/sbin/fmtree
/usr/lib/debug/usr/sbin/fmtree.debug
/usr/share/man/man8/fmtree.8.gz
>>> Old files removed
>>> Removing old directories
>>> Old directories removed
To remove old libraries run 'make delete-old-libs'.
root@mowa219-gjp4-8570p:/usr/src #

– and

* I **do not** `make delete-old-libs`

* (off-topic from helloSystem) I can't easily recall what I have installed from ports

* (OT) I do _update_ (from source) my jail for poudriere, but I have never given thought to deletion of old files within the jail (does it take care of itself?) … maybe saner to simply _create_ from source.

Yes, this is that step.

igalic · 2021-02-17T12:19:55Z

just one small note re beadm: if you replace it with bectl(8), which is in base, that's one less thing to install

vermaden · 2021-02-17T12:25:26Z

@vermaden FYI https://pastebin.com/crR6WFCG – first observed whilst testing an upgrade with helloSystem, it's reproducible with a clean installation of FreeBSD 12.1-STABLE (without helloSystem as a starting point.

If I (exit the chroot environment etc. and) restart the system before the second run of freebsd-update install then there are no such bad system calls (and /lib/libmd.so.6 is not empty).

If moving from 12.x to 13.x its best (and probably stated in the FreeBSD Handbook) to first upgrade 12.1 to 12.2 (or other latest 12.x version at the moment) and then do the major upgrade to 13.x.

Regards.

probonopd · 2021-02-17T17:49:01Z

The more I think about it, I come to the conclusion that I am really more in favor of a "clean install" at least for major revisions/releases that would work roughly like this:

User downloads the latest ISO
In a running helloSystem, user double-clicks the ISO
User double-clicks "Install FreeBSD", the usual installer opens
The disk from which helloSystem is currently booted is shown
If the user selects that disk, then the following happens:
- A new jail/chroot is created that is also a boot environment
- helloSystem is installed to that jail/chroot
- /home is the same for the old and the new boot environment (as far as I remember this is even the default)
- /Applications may need some special treatment (tbd)
The new boot environment is selected for the next boot (only)
Only if we reach the full graphical desktop in the next boot, then set the new boot environment permanently

Advantages:

"Clean install" = predictable results
ISO can be tested beforehand
More familiar process to Mac users, but using ZFS power

For minor (security) updates, we could still use a more traditional approach.

What do you think?

vermaden · 2021-02-17T18:51:46Z

Agreed.

kettle-7 · 2021-05-25T08:03:34Z

What if the base system was a git repository mounted ro and you mounted a separate zfs subvolume for rw folders (like /Users), and in an update you'd just run

mount -o rw,remount / # is remount even necessary on BSD?
cd /
git pull
cd -
mount -o ro /

probonopd · 2021-05-25T17:32:34Z

I don't think that git is a good match for managing gigabytes of binaries. Plus, git does ask the user questions in the case of merge conflicts.

kettle-7 · 2021-05-25T20:14:10Z

There's always ostree, it's like git but endless os uses it.

probonopd · 2021-05-25T20:21:58Z

Definitely unwelcome technology. Not human-friendly. Cryptic GUID-like names of files. Not "manageable" by hand. Not simple.

kettle-7 · 2021-05-25T20:42:42Z

The other thing I can think of is SystemImageKit and SystemExtensionKit

probonopd · 2021-05-25T21:16:27Z

SystemImageKit is Linux/GRUB only.

probonopd · 2021-10-16T15:53:19Z

tl;dr

This is mainly a summary for my future self:

FreeBSD itself

FreeBSD isn't sufficiently described with versions like "12.2". You can have "12.2" and it is still outdated
You also need to look at the number that follows RELEASE-.... For me it helps to think of 12.2-RELEASE-p7 as 12.2.7
In fact, the kernel and the FreeBSD "userland" can have different versions. freebsd-version -ku can be used to show the kernel version and the userland version
Note that "userland" here does not mean anything installed using Ports or Packages
To update FreeBSD "itself" (not: ports and packages) in one go without asking questions one can use sudo env PAGER=cat freebsd-update fetch install and then reboot

Thanks to @grahamperrin for being patient enough explaining it to me as long as was needed for me to finally get it.

(still to be found out whether we can prevent this from overwriting our carefully crafted custom config files such as modified rc scripts)

Packages

sudo env PAGER=cat pkg.real update
sudo env PAGER=cat pkg.real upgrade --yes

(still to be found out how to prevent them from overwriting our carefully crafted custom config files)

grahamperrin · 2021-10-16T16:05:52Z

… still to be found out how to update those without them asking questions, …

helloSystem/hello#161 (comment) showed:

csh

setenv PAGER cat …

… (Still a bit unclear) sudo env PAGER=cat freebsd-update fetch install may need to be run again after rebooting …

Not required. The advice to run the command was because you did not mention restarting the system.

probonopd · 2021-10-16T16:11:59Z

(moved to helloSystem/hello#161 (comment))

probonopd · 2021-10-16T16:30:22Z

For updating packages, we could probably re-use most of the code from our application installer GUI. It already knows how to display a grapical progress bar for pkg, and allows more advanced users to look at the command line output.

probonopd · 2021-10-17T10:03:37Z

Create a new BE prior to doing anything (TODO: only if FreeBSD and/or package updates are available).

Then, update FreeBSD:

Next, update packages:

Anyone volunteers to test?

probonopd · 2021-10-17T12:20:03Z

Should appear in the "Developer Preview" folder in the next continuous build, 0G25.

probonopd · 2021-10-17T14:41:53Z

Known issues

computer:///...drive items appear (items for unmounted drives). gvfs to blame?
Everything that is contained in FreeBSD packages (but has been modified by helloSystem) gets overwritten. We need to find a way to tell pkg to leave certain files alone. Examples:
- /usr/local/share/slim/themes/default/background.jpg (wallpaper)
- /usr/local/sbin/automount (automounter; we use a much newer version than is in packages
Files that we forcefully deleted (because we absolutely don't want them on the system but many packages depend on them) re-appear. Examples:
- /usr/local/share/fonts/dejavu/
- /usr/local/share/fonts/Liberation/

Essentially much of what script.hello does in terms of customizing the system gets undone when one updates packages.

We'd probably have to rethink (=make more complicated) the way we put the system together if we really want end users to update the system using packages.

In the meantime, sudo pkg lock --yes automount slim gvfs dejavu liberation-fonts-ttf) can somewhat work around this. (There may be more packages that need to be locked as well.)

probonopd · 2022-10-03T12:36:56Z

sudo env PAGER=cat freebsd-update upgrade -r 13.1-RELEASE does not silence the questions; still getting

The following components of FreeBSD seem to be installed:
kernel/generic src/src world/base world/lib32

The following components of FreeBSD do not seem to be installed:
kernel/generic-dbg world/base-dbg world/lib32-dbg

Does this look reasonable (y/n)?

Would env EDITOR=cat work? Trying sudo env EDITOR=nano PAGER=cat freebsd-update upgrade -r 13.1-RELEASE but having no questions asked at all is what we really need.

probonopd · 2022-10-03T13:06:25Z

My real life experience.

Thought it would be nice to run the latest version of PrusaSlicer.

Wouldn't it be awesome if this could be done without having to change half the operating system?

FreeBSD% sudo pkg  install PrusaSlicer
Passwort: 
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
The following 45 package(s) will be affected (of 0 checked):

Installed packages to be REMOVED:
        FreeCAD: 0.19.2_3
        frei0r-plugins: 1.7.0.18
        frei0r-plugins-opencv: 1.7.0.18
        gdal: 3.3.3_2
        opencv: 4.5.3_4
        shotcut: 21.03.21_3

New packages to be INSTALLED:
        abseil: 20211102.0 [FreeBSD]
        gcc11: 11.3.0 [FreeBSD]
        libixion: 0.17.0_1 [FreeBSD]
        py39-boost-libs: 1.79.0_1 [FreeBSD]
        py39-numpy: 1.22.4_1,1 [FreeBSD]
        py39-setuptools: 62.1.0_1 [FreeBSD]

Installed packages to be UPGRADED:
        PrusaSlicer: 2.3.3 -> 2.3.3_4 [FreeBSD]
        boost-libs: 1.72.0_6 -> 1.79.0_1 [FreeBSD]
        botan2: 2.19.1 -> 2.19.2 [FreeBSD]
        codeblocks: 20.03_1 -> 20.03_2 [FreeBSD]
        firebird25-client: 2.5.9_1 -> 2.5.9_2 [FreeBSD]
        firefox: 95.0.2_1,2 -> 105.0.1,2 [FreeBSD]
        gnustep-base: 1.27.0_2 -> 1.27.0_6 [FreeBSD]
        gnustep-gui: 0.28.0_2 -> 0.28.0_3 [FreeBSD]
        harfbuzz-icu: 3.2.0 -> 4.4.1_1 [FreeBSD]
        icu: 70.1_1,1 -> 71.1,1 [FreeBSD]
        libcdr01: 0.1.7_1 -> 0.1.7_2 [FreeBSD]
        libcmis: 0.5.2_2 -> 0.5.2_5 [FreeBSD]
        libe-book: 0.1.3_21 -> 0.1.3_24 [FreeBSD]
        libmspub01: 0.1.4_19 -> 0.1.4_20 [FreeBSD]
        liborcus: 0.16.1_1 -> 0.17.2_1 [FreeBSD]
        libqxp: 0.0.0_19 -> 0.0.0_20 [FreeBSD]
        libreoffice: 7.2.5.2 -> 7.3.5.2 [FreeBSD]
        libvisio01: 0.1.7_5 -> 0.1.7_8 [FreeBSD]
        libzmf: 0.0.2_24 -> 0.0.2_25 [FreeBSD]
        openjfx14: 14.0.2.1+1_4 -> 14.0.2.1+1_7 [FreeBSD]
        openvdb: 8.1.0_5 -> 9.1.0 [FreeBSD]
        poppler: 21.12.0 -> 22.06.0 [FreeBSD]
        poppler-glib: 21.12.0 -> 22.06.0 [FreeBSD]
        poppler-qt5: 21.12.0 -> 22.06.0 [FreeBSD]
        poppler-utils: 21.12.0 -> 22.06.0 [FreeBSD]
        qt5-core: 5.15.2p263 -> 5.15.5p165_2 [FreeBSD]
        qt5-webkit: 5.212.0.a4_7 -> 5.212.0.a4_10 [FreeBSD]
        raptor2: 2.0.15_19 -> 2.0.15_22 [FreeBSD]
        sfcgal: 1.4.0 -> 1.4.1_1 [FreeBSD]
        spidermonkey78: 78.9.0_4 -> 78.15.0_2 [FreeBSD]
        tesseract: 5.1.0 -> 5.1.0_1 [FreeBSD]
        thunderbird: 91.9.1 -> 102.3.1_1 [FreeBSD]
        webkit2-gtk3: 2.34.2 -> 2.34.6_3 [FreeBSD]

Number of packages to be removed: 6
Number of packages to be installed: 6
Number of packages to be upgraded: 33

The process will require 46 MiB more space.
535 MiB to be downloaded.

Proceed with this action? [y/N]:

Chose to sudo pkg upgrade instead. At first it said it would uninstall nothing, so I started to do it.

But then, after having downloaded and installed a bunch of stuff, it said the following. Had I known it would end like this, I would not even have attempted the upgrade.

Checking integrity... done (0 conflicting)
Conflicts with the existing packages have been found.
One more solver iteration is needed to resolve them.
The following 874 package(s) will be affected (of 0 checked):

Installed packages to be REMOVED:
        drm-fbsd13-kmod: 5.4.144.g20211013
        e2fsprogs-libss: 1.46.4
        hdf-szip: 2.1.1
        p7zip: 16.02_3
        py38-cairo: 1.18.1_2,1
        py38-gobject3: 3.38.0
        py38-markdown: 3.3.4
        py38-matplotlib: 3.4.1
        py38-mutagen: 1.42.0_2
        py38-pygments: 2.7.2
        py38-pyliblo: 0.10.0_2
        py38-pyside2: 5.15.2_2
        py38-qt5-core: 5.15.4_2
        py38-qt5-dbus: 5.15.4_2
        py38-qt5-gui: 5.15.4_1
        py38-qt5-multimedia: 5.15.4_3
        py38-qt5-network: 5.15.4_2
        py38-qt5-printsupport: 5.15.4_3
        py38-qt5-qml: 5.15.4_3
        py38-qt5-svg: 5.15.4_3
        py38-qt5-webchannel: 5.15.4_2
        py38-qt5-webengine: 5.15.4_3
        py38-qt5-widgets: 5.15.4_3
        py38-rdflib: 5.0.0
        py38-shiboken2: 5.15.2_2
        py38-sip: 5.5.0_1,1

Grudgingly accepted. Will probably have to install the py39 equivalents by hand. Mere mortals would be totally lost here. This is why I think package managers are tools for developers, but not for non-technical end users.

sudo pkg install e2fsprogs-libss hdf-szip p7zip py39-cairo py39-gobject3 py39-markdown py39-matplotlib py39-mutagen py39-pygments py39-pyliblo py39-pyside2 py39-qt5-core py39-qt5-dbus py39-qt5-gui py39-qt5-multimedia py39-qt5-network py39-qt5-printsupport py39-qt5-qml py39-qt5-svg py39-qt5-webchannel py39-qt5-webengine py39-qt5-widgets py39-rdflib py39-shiboken2 py39-sip

pkg: No packages available to install matching 'e2fsprogs-libss' have been found in the repositories
pkg: No packages available to install matching 'hdf-szip' have been found in the repositories
pkg: No packages available to install matching 'p7zip' have been found in the repositories
pkg: No packages available to install matching 'py39-qt5-core' have been found in the repositories
pkg: No packages available to install matching 'py39-qt5-dbus' have been found in the repositories
pkg: No packages available to install matching 'py39-qt5-gui' have been found in the repositories
pkg: No packages available to install matching 'py39-qt5-multimedia' have been found in the repositories
pkg: No packages available to install matching 'py39-qt5-network' have been found in the repositories
pkg: No packages available to install matching 'py39-qt5-printsupport' have been found in the repositories
pkg: No packages available to install matching 'py39-qt5-qml' have been found in the repositories
pkg: No packages available to install matching 'py39-qt5-svg' have been found in the repositories
pkg: No packages available to install matching 'py39-qt5-webchannel' have been found in the repositories
pkg: No packages available to install matching 'py39-qt5-widgets' have been found in the repositories

Bummer!

Total breakage. Now what?

sudo pkg install devel/py-qt5

After a reboot, I cannot open many applications anymore, getting

This application requires at least version FBSD_1.7 of libc.so.7 to run.

Please update it and try again.

So far I've been running FreeBSD 13.0-RELEASE. So doing

sudo freebsd-update upgrade -r 13.1-RELEASE

takes forever, hanging on Preparing to download files... for a looong time with no progress indication.
Then got Fetching 18519 patches.

The following file could not be merged automatically: /etc/group
Press Enter to edit this file in /usr/bin/vi and resolve the conflicts
manually...

Absolute nightmare!
I don't even know how to get out of vi, much less how to edit anything using it.
Please just leave my files alone!

Typing the cryptic :q! a couple of times gets me out of vi.

Stuff like

+<<<<<<< current version
 video:*:44:user
+=======
+video:*:44:
+realtime:*:47:
+idletime:*:48:
+>>>>>>> 13.1-RELEASE

absolutely makes me fear that Xorg will be broken.

So Does this look reasonable (y/n)? n, retry using sudo env PAGER=cat freebsd-update fetch install.

Does sudo env PAGER=cat freebsd-update upgrade -r 13.1-RELEASE similarly work? No.

Trying sudo env EDITOR=nano PAGER=cat freebsd-update upgrade -r 13.1-RELEASE next.

Had to know a lot about the system to manually edit the files in nano.

Then:

To install the downloaded upgrades, run "/usr/sbin/freebsd-update install".
FreeBSD% sudo /usr/sbin/freebsd-update install
Passwort: 
Installing updates...
Kernel updates have been installed.  Please reboot and run
"/usr/sbin/freebsd-update install" again to finish installing updates.

After having done that, I now seem to have successfully moved from 13.0 to 13.1, but it is not something a non-technical user could have done.

probonopd · 2022-10-03T20:21:19Z

EDITOR=edit is even better because it comes with FreeBSD base.txz.

grahamperrin · 2022-10-03T22:58:29Z

… nightmare! … vi, …

⚙ D36786 freebsd-update.sh: EDITOR: if not set, prefer ee

grahamperrin · 2022-10-03T23:08:49Z

… Would env EDITOR=cat work? …

No. cat(1) is not an editor.

grahamperrin · 2022-10-03T23:21:08Z

… PrusaSlicer …

Not part of the operating system (not integral to FreeBSD).

… Total breakage. Now what? …

Boot the boot environment that preceded the breakage.

Questions about pkg(8), and other things that are not part of the operating system, will be easier to answer if asked as questions in Q&A: https://github.com/helloSystem/hello/discussions/categories/q-a

grahamperrin · 2022-10-03T23:30:18Z

… Stuff like

+<<<<<<< current version
 video:*:44:user
+=======
+video:*:44:
+realtime:*:47:
+idletime:*:48:
+>>>>>>> 13.1-RELEASE

…

Think of that as you would think of a merge with Git. From Git - Advanced Merging:

probonopd · 2022-10-04T09:30:48Z

The purpose of this thread is to show which challenges we have to overcome in oder to arrive at a no-questions-asked upgrade process that we could integrate in the System Update utility.

I fear this would have to be solved by FreeBSD proper.
In the meantime, package-based system upgrades are not suitable for non-technical users.

grahamperrin · 2022-10-06T07:00:05Z

~~NB, my earlier comment:~~

~~… I don't recall suggesting freebsd-update(8) as a basis for a System Update/Upgrade utility. …~~

freebsd-update(8) is, essentially, the norm for binary updates and binary upgrades to supported versions of RELEASE.

I thought that you already decided, long ago, to not use freebsd-update in the context of an update utility for helloSystem.

… package-based system upgrades …

Do you mean PkgBase (for the operating system, FreeBSD)? Or pkg-upgrade(8) (for packages of ports)?

grahamperrin · 2022-10-09T07:47:26Z

#33 (comment)

… package-based system upgrades are not suitable for non-technical users.

Maybe true from the helloSystem perspective; not true as a sweeping statement.

From https://www.ghostbsd.org/about for GhostBSD, which takes pkg-based approaches:

… a graphical application to install software and update your system. …

… We try to simplify FreeBSD to lower the entry-level of using FreeBSD on a desktop or laptop. We provide all the benefits of the FreeBSD operating system benefits combined with our in-house GUI tool. …

From #33 (comment) (2021-01-18, the first response to the opening post):

… GhostBSD's Update Station, …

Pictured:

The kernel, and other essentials, amongst items to be upgraded:

Limitations, maintenance

From helloSystem/hello#161 (comment):

… helloSystem does not want to operate a package repository because it would require infrastructure to be maintained. …

GhostBSD maintains the type of thing that helloSystem does not want to maintain:

a repository – FreeBSD ports with GhostBSD ports.

Via ghostbsd/ghostbsd-ports@b70c442?diff=split#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5R44:

https://github.com/ghostbsd/ghostbsd-ports/discussions/26#discussion-55695 (2020-12-09)

grahamperrin · 2022-10-15T18:10:37Z

Is pkg_replace(1) of interest?

I never used it, but it caught my eye.

https://github.com/kdeguchi/pkg_replace#readme | ports-mgmt/pkg_replace