Switching to Sparkle 2.x as the main version

[kornelski]

Work required to make Sparkle 2.x official:

• Port EdDSA keys Add ed25519 support to XPC branch #1423
• Create documentation for installation and configuration of 2.x
• Create migration guide for 1.x users
• Add banners/labels to every 1.x documentation page that it is outdated
• Add banners/labels to 2.x documentation pages that they apply only to new Sparkle
• Add issue templates for 2.x
• Add "which version am I using?" help page to the website that helps select the right documentation
• Add version to logs (perhaps even call it Sparkle2?)

[DivineDominion]

I'm watching this issue now. I'm the author of the blog posts @michelf mentioned:

• https://christiantietze.de/posts/2019/06/sparkle-xpc-branch/
• https://christiantietze.de/posts/2019/06/sparkle-xpc-or-no/
• https://christiantietze.de/posts/2019/06/sparkle-xpc-setup/

I can condense and rewrite them and then change the docs. I don't know anything about the EdDSA stuff that's still open: does it make a difference regarding the documentation? If not, I'll go ahead with the documentation.

[kornelski]

Switching to new codesigning can be handled separately.

[zorgiepoo]

I'm jotting (mostly internal) areas here that I believe have changed, been documented, or may want to be looked over by someone curious since 1.x.

• INSTALL.markdown (ignore bit saying to use SparkleInstallerLauncher if not sandboxed), Documentation/Security.md for initial motivation of branch (ignore outdated info encouraging user driver to be in separate process from updater). Also see repo README. [edit]: also this thread from a while back.
• SPUUserDriver and SPUUserDriverCoreComponent as exposed APIs (see 2.x UserDriver API Redesign #1551 for some discussion), other SU_EXPORT APIs like SPUStandardUserDriverDelegate, SPUUpdaterSettings, SPUUpdatePermissionRequest, SPUUpdaterDelegate. And essential ones (SPUUpdater, SPUUpdaterDelegate, SPUStandardUpdaterController)
• Autoupdate's daemon/agent (AppInstaller), IPC communication, installer stages (see Documentation/Installation.md I believe)
• Installer Progress agent (install progress, sending app termination signal, app relaunching).
• Whether or not SUUpdater, SUUpdaterDelegate shim should be "deprecated"; compatibility goals (API compatibility, binary compatibility, SPU prefix usage?). [Back in the day I injected Sparkle 2 in several apps using Sparkle 1 for testing and it worked "as-is" in fair number of cases]
• The XPC services including InstallerLauncher that submits daemon/agent(s)
• Re-architectured or newer update control drivers / update drivers (SPUUpdateDriver, automatic, probing, scheduled, user initiated, basic, core, download driver, appcast driver, ..), based on 1.x but more composed instead of subclassing
• SparkleCore framework for all not-linked-AppKit / non-UI pieces
• sparkle-cli: arbitrary bundle updating + command line UI + exposed "API/UI"
• Workflows: using with/without XPC services / sandboxing, making new UIs, making updaters to update other bundles (note host vs app bundle distinction), CLI like tool for users or plug-ins that can't invasively load Sparkle into a process it doesn't own, non-admin or updates requiring authorization (including authorization before quitting app, automatic downloading + later resuming with authorization prompt)

I gave a sandboxed + hardened runtime + notarized test app with the xpc services a quick go and it seemed to work.

One note is that because codesign_embedded_executable signs with the hardened runtime, that prevents debuggers from debugging the prebuilt XPC services (Xcode normally inserts an entitlement exception before signing a 'hardened runtime' binary for debugging workflows). That interestingly results in Xcode being unable to debug them causing Sparkle not work with those services when ran from Xcode. The debugging XPC services option in the scheme can be disabled to work around that, or one can choose not to sign with with hardened runtime for debugging purposes. Normally XPC services probably aren't pre-built, so there's an interesting integration dilemma here. Not a critical issue though.

[zorgiepoo]

The public facing documentation part the original post describes here is done now, but what really is essential if this is to ever take off is a transitional phase of re-targetting primary development branch and freezing 1.x only additions. Secondary is re-evaluating changes from 1.x that make sense to bring back (I have a collected list, but not most productive use of time without the former guarantee).

[zorgiepoo]

Preliminary documentation as far as I see it is done, and is more of an iterative process during development, so I marked the original post points as done.

Next actionable steps are:

• Guarantee future changes are made in 2.x to prevent maintenance burden (Add pull request template and update issue templates #1727)
• Change development branch from master to 2.x (TBD when)
• Integrate / Sync changes from 1.x

I have a bunch of easily adaptable changes from #1722

[zorgiepoo]

2.x is sufficiently in sync with 1.x now in order to switch the main development branch and feature freeze 1.x. There are some changes not in 2.x for various (and sometimes intentional) reasons but none are blockers.

There is one small localization change in generate_appcast (not my expertise) I didn't quite know how to port over because both branches diverged (@adamtharani @gwynne). Might need to look at it more later. [edit]: Actually, looking at code in 2.x again, this already looks OK.

Other things not ported:

• Exposing Code Signing Verifier. Small thing, but I'd rather not consider this until someone pushes for it again, doesn't really seem to be Sparkle's duty here. Could be exposed as Private API headers though.
• Ignoring permissions in creating binary delta files - this really seems like a bug since hash verification looks at permissions. Format needs to be changed properly if a change is to be made here.
• installUpdatesIfAvailable API which has been deprecated in 1.x and caused issues.
• updaterDidRelaunchApplication: API - I need to know a concrete use case before adding such functionality (this approach is often misused), and the current way it's implemented by propagating a temporary default is not good (attempted in Add/port update relaunched API #1891 but rejected).
• Delegate methods to suppress new alert: conflicts with customizable user driver API, needs contribution from someone that still needs such functionality (if still relevant). [edit]: equivalent functionality has been implemented in 2.2/2.3 now.
• Using pubDate for sorting appcast items to support situation where different versions of an app have same bundle version, or an invalid formatted bundle version. Dash separators were implemented to be ignored however (Port #1504 (dash separators in version strings) to 2.x #1845).
• Lockfile for 3rd party updaters. This doesn't work with 2.x's architecture, or essentially the lock file already is the installer service that is submitted and running for a specific bundle. The updater queries the service before checking for new updates, maybe an API could be added to query the service but not adding new API for an anticipated use case.

[kornelski]

Yeah, I generally agree about the changes. It makes sense to clean up the API and quirky behaviors.

However, I would prefer to keep these two:

• Supporting predictable order when versions are equal, preferably using pubDate. This allows auto-generating feeds from git and nightly releases, which can't rely on versions being bumped every build.

• Lock file in /tmp. Can't we just drop any file in there? I'd like to support compatibility with external update tools, as that's one of Sparkle's unique features.

[zorgiepoo]

Yeah, I generally agree about the changes. It makes sense to clean up the API and quirky behaviors.

However, I would prefer to keep these two:

• Supporting predictable order when versions are equal, preferably using pubDate. This allows auto-generating feeds from git and nightly releases, which can't rely on versions being bumped every build.

This one is a matter of correctness and policy rather than technical reason. If we support this, then we also support or recommend applications using incorrectly formatted / invalid CFBundleVersion values. Developers that submit nightly builds need to auto-increment their bundle versions, manually bumping it themselves or automating it (Apple even provides such a tool for that). If you support valid bundles versions, then how can Sparkle be able to compare different identical versions to tell the user they're out of date? The only reasonable answer is, you need to use another field stored in the app for comparison, and that's why there is also a short version string for display. I'd also like to minimize using or parsing/comparing the pubDate; this feature had caused a major regression in 1.x a while back due to more complex comparison logic.

• Lock file in /tmp. Can't we just drop any file in there? I'd like to support compatibility with external update tools, as that's one of Sparkle's unique features.

The current approach is to probe if there's an existing installer running (in a separate process) for a bundle ID; that's how 2.x picks up resumable updates and it supports external updaters doing the same. There are some architectural implications about writing to /tmp anywhere due to sandboxing and I don't particularly like dropping or cleaning up a global file that is only respected by choice (based on how the logic works in 1.x and adapting it to 2.x, another external updater could just easily remove the lock file on startup!) It's likely, though I cannot confirm for certainty, the external updater in question here is based on sparkle-cli or 2.x architecture which enabled this, because it added support for updating external bundles. This may require a deeper discussion elsewhere for best integration. (Also there are already applications using 2.x, so this approach is already not working or being used there rather than being a regression ;)).

[digitalmoksha]

does this mean we'll soon have tagged builds of 2.x?

[zorgiepoo]

A pre-release tagged build would come first. Things I want to review over:

• Story for pref panes / plug-ins (things here have largely changed since 2016)
• SPARKLE_NORMALIZE_INSTALLED_APPLICATION_NAME reviewal
• XPC Service Integration (XPC Service Integration Changes #1814)
• User Driver API
• Push a pre-release
• Story for package management

.. and some miscellaneous code base review and testing.

The main development branch has since been changed, and build artifacts are being uploaded by our CI in the meantime here.

[zorgiepoo]

The current status is

• Wait for Sparkle 1.27.0 release to ease adoption of some major DSA deprecation / EdDSA changes
• Wait to ship 2.0 after official pre-releases have been out for "some time" for testing purposes.

I don't have anything else on 2.0 milestone at this moment.

The website / API documentation has been updated, Swift Package Manager and CocoaPods support has been added, and pre-release distributions are on GitHub's releases page.

[zorgiepoo]

I finally pulled the trigger here and pushed the release button. Thanks to everyone involved contributing and using the beta versions, and especially @kornelski for maintaining the project and reviewing a whole lot of pull requests.

[DivineDominion]

That's great post-Christmas news! 👏 Thanks everyone