Auto Updater + Per Platform Installer RFC

[johnwparent]

Auto Updater + Per Platform Installer RFC
Authors:
@johnwparent

1 Summary

Gpt4All auto updater responsible for detecting when an upgrade to Gpt4All (or any subcomponent if and when there are any) on a given system is available, and in the event that an upgrade is desired, for facilitating the process of that upgrade on the local system.

Gpt4All per platform installers provide binary installers for Gpt4All from a platform specific installer framework, taking advantage of the aforementioned auto updater to facilitate upgrades.
To facilitate the above, an bespoke installer manifest scheme and update flow provides the mechanisms required for an auto updater to drive the custom installers to drive their own upgrades using platform specific behavior.

2 Motivation

Currently Gpt4All uses the Qt IFW installers + associated CPack generator to construct binary installers for distribution. Alternatively, Gpt4All could be taking advantage of CPack’s other generators to create more customized, platform specific installers. Platform specific installers would allow for more customization with respect to their given platform’s consideration (such as the Windows registry or DragNDrop on MacOS) and the features of the specific installer frameworks. Further still per platform installers would allow for more optimization and better user experiences as each installer is specialized for its given context. Signing too, becomes a simpler process when using non IFW installers.
At the moment, usage of the IFW installers is fixed due to a reliance on the Qt maintenance tool to facilitate the update/repair process as well as the upgrade process on the advent of a new Gpt4All release. The maintenance tool is generated as a part of, and shipped inside of the IFW installers Gpt4All currently generates and relies on. It’s primarily responsible for detecting updates per installation component, and allowing the optional re-install or upgrade of these components, chief of which is Gpt4All itself. In order to function properly, detect updates, and update/install/repair components, the maintenance tool is reliant on a hosted repository of a highly specific structure, which requires the use of another Qt tool to generate, adding further complexity to the build and release process. In order to move away from the IFW installers, Gpt4All needs to move away from the utilization of the maintenance tool, as other installers are incompatible with the repository layout the maintenance tool requires, and in general, are not compatible with being upgraded by any tool other than themselves. Gpt4All then, needs an in house auto-updater to replace the Qt maintenance tool, and once that reliance is severed, it can progress to implementing custom installers, providing the benefits mentioned above.

3 Proposed Implementation

3.1.

First it is required to replace the maintenance tool so our installers can receive updates. This requires creating our own auto updater. The autoupdater has three primary responsibilities:

1. Determining when an upgrade is available
2. Determining how to obtain that upgrade
3. Driving said upgrade

In order to do all of the above, it requires an understanding of the current version of Gpt4All, as well as an understanding of other versions available, and important metadata about those versions. To do this, we need a new hosted repository layout, more generic than the maintenancetool repositories, with compatibility for each of the platform installers.
At the root of this repository resides a manifest, denoting available versions, providing information about those versions, and other metadata detailed below.

Metadata scheme

<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" elementFormDefault="unqualified" version="1.0">
    <xs:element name="pkg-description" type="root"/>
    <xs:complexType name="pkg-description">
        <xs:sequence>
            <xs:element name="name" type="xs:string"/>
            <xs:element name="version" type="xs:decimal"/>
            <xs:element name="notes" type="xs:string"/>
            <xs:element name="authors" type="xs:string"/>
            <xs:element name="date" type="xs:date"/>
            <xs:element name="release-type" type="xs:string"/>
            <xs:element name="last-compatible-version" type="xs:string"/>
            <xs:element name="entity" type="xs:string"/>
            <xs:element name="component-list" type="xs:string"/>
           <xs:element name="pkg-manifest" type="child"/>
        </xs:sequence>
    </xs:complexType>
    <xs:complexType name="pkg-manifest">
        <xs:sequence>
		<xs:element name="installer-uri" type="xs:string"/>
             <xs:element name="sha256" type="xs:integer"/>
             <xs:element name="args" type="xs:string"/>
             <xs:element name="signed" type="xs:boolean"/>
		<xs:element name="components" type="xs:string"/>
             <xs:element name="sbom-manifest" type="xs:string"/>
        </xs:sequence>
    </xs:complexType>    
</xs:schema>

This would replace the existing release.json file gpt4all currently uses to provide context about upgrades + availability

At the root of the repository, a single manifest file named gpt4all.xml will be present. This will be a symlink to the most recent version’s manifest file. Otherwise the repository consists of a manifest file per release and the structure required by a given installer (installer dependent)

The auto updater fetches this root, symlinked manifest file via web request, and parses the information within.
At this point the behavior is much the same as what exists in gpt4all at present, a version is returned, and if more recent than the current version, Gpt4all prompts the user to upgrade.

In the event of an upgrade operation, because we use our own updater, and powerful, platform specific installers, the best actors to actually perform the upgrade are the installers themselves, which are capable of doing so. To that end, they are not capable of detecting upgrades or finding new versions of themselves, instead they can update themselves when they’re already on the system. To that end the auto updater reads the manifest file, which per the scheme provides the url to download the latest installer from. The auto updater then downloads that installer, kills the gpt4all process, initiates the installer with the platform appropriate arguments and any arguments denoted in the manifest.

.
├── manifest.xml --> manifest-xx.x.xml
├── version-xx.x/
│   ├── manifest-xx.x.xml
│   ├── linux/
│   │   └── installer.rpm/deb/tgz
│   ├── macos/
│   │   └── installer.dmg
│   └── windows/
│       └── installer.exe
└── version-xx.y/
    ├── manifest-xx.y.xml
    ├── linux/
    │   └── installer.rpm/deb/tgz
    ├── macos/
    │   └── installer.dmg
    └── windows/
        └── installer.exe

3.2 installers

The installers themselves are a vital part of the upgrade, platform specific implementations are as follows:

Windows: InnoSetup

MacOs: DragNDrop (DMG’s)

Linux: rpm, deb, archives
(linux is driven entirely by maintenance tool and system package managers)

Actual implementations are effected via Cpack's packaging generators and whatever customization is determined to be required.

[johnwparent]

cc @manyoso

[manyoso]

Thanks for the RFC. I've added this to our project page and marked it as high priority. Some questions:

1. Will the updater allow to go backwards to a previous version?
2. Let's say an update produced a crashing program. Can the user launch the updater manually to back track?
3. Will the installers/updaters have headless versions?

[johnwparent]

1. Will the updater allow to go backwards to a previous version?

If the installer supports it. MacOS, it will to be sure, for Windows, I know installers generally complain about downgrades, but I'm fairly sure with some registry tweaks that's viable.

2. Let's say an update produced a crashing program. Can the user launch the updater manually to back track?

Yes, the new updater would be a standalone exe (it needs to to actually update Gpt4All in the first place), so users can invoke that to restart the upgrade process.
The upgrade tool would ostensibly also be able to update itself, so it's processes can be independent of Gpt4All.

3. Will the installers/updaters have headless versions?

Yes, all of the above on Windows. I'm less familiar with MacOS, but I'm pretty sure you can mount, extract, and install a DMG from the command line, so also yes.

[manyoso]

Okay, please proceed.

[johnwparent]

Status and timeline for feature implementation, testing, and deployment:

Current Status (10/9): Headless version for offline implemented, being tested by @johnwparent

Timeline:

• 10/18 : Headless version for offline + online installers with support for in-place upgrading/downgrading/replacement distributed to @manyoso for testing
• 10/25: GUI version deployed for testing
• 11/1: Overhauled, platform specific installers available for testing
• 11/4 - ?: Iterating on any issues raised with previous sprints and implementing feature requests/iterating on feedback

Once final sprint is complete, general deployment + larger scale testing.

cc @manyoso