Build a working distributable macOS application (signed and notarized)

[simonw]

Follows #25.

[nikvdp]

Hi @simonw I've been following along with this as we recently faced a similar problem on an electron project and after seeing your recent update on HN thought some of our learnings might be helpful for you.

I've been working on porting our setup into the datasette repo over at nikdvp/datasette.app and saw from #27 that we seem to have converged on pretty similar solutions! I also started with pyinstaller but quickly switched over to bundling a full venv (using conda-pack). Still in flux and not quite ready for a PR yet, but some of https://github.com/nikvdp/datasette might be useful.

Some bits that might be of interest:

• An alternative way to build a single-file binary without relying on pyinstaller. This uses dgiagio/warp, which has some advantages over pyinstaller. In particular it doesn't rely on python's zipfile machinery so many bugs that show up in pyinstaller produced bins don't happen with warp since it runs it the same way python normally does. I've commented it out for now because packing it into a single binary breaks the codesigning process, but may be helpful as an alternative to Package as standalone binary datasette#93 (comment). (code).

• Mac Codesigning: once you've gone through the signing certificate export dance you can set the CSC_LINK and CSC_KEY_PASSWORD env vars in the 'Build Electron app' step as documented here, and electron-builder will then take care of making sure everything is signed correctly.

• the entitlements.plist file and associated config in electron-builder.yml allow the bundled python interpreter to inherit the codesigning config of the parent electron app. This ended up being super important for us cause it fixed the main issue we ran into in our project, where macos' security prevented the electron app from launching our bundled interpreter. With these entitlements the Electron app's own signature can be inherited by the bundled python interpreter allowing the electron app to launch it succesfully.

[simonw]

@nikvdp wow that's really useful, thanks!

My development version right now actually does the following:

1. Bundles standalone Python inside the application - so /Applications/Datasette.app/Contents/Resources/python/bin/python3.9 is a valid Python interpreter.
2. When the app first starts up, runs mkdir ~/.datasette-app and then /Applications/Datasette.app/Contents/Resources/python/bin/python3.9 -m venv ~/.datasette-app/venv to create a new virtual environment for installing Datasette and plugins
3. Runs ~/.datasette-app/venv/bin/pip install datasette==0.59a2 datasette-app-support to install Datasette in that new virtual environment
4. Runs ~/.datasette-app/venv/bin/datasette to start the server process running

This is all highly experimental but it seems to work - at least on my machine, and with the .dmg installer I generated which I then installed using the macOS security pane "open this thing anyway" option.

What I'm now wondering is if the signing of the binaries will survive this - I need to be able to execute ~/.datasette-app/venv/bin/datasette which was created by the python3.9 binary that lives inside the /Application/Datasette.app package!

[simonw]

@nikvdp the reason I'm currently leaning towards this slightly terrifying idea of shipping an entire Python environment (from https://github.com/indygreg/python-build-standalone) and then using it to create a virtual environment is that I really want users of this to be able to install Datasette plugins - see #5 - and I don't think the PyInstaller route will let them do that.

Do you have any idea if the combo of conda-pack and warp might enable Python plugin installation?

[simonw]

Without signing, the user needs to open the app, then go to Apple -> System Preferences -> General and click this:

[iansinnott]

Hey @simonw, friend of Nik here (thus how I found this issue) but also a fan of datasette. Thought I'd chime in as I've run into these issues in the past as well.

Without signing, the user needs to

It doesn't help much, but a user can also right click the app and then click "Open" via the context menu. Perhaps slightly less friction than having to go through the settings pane.

signing of the binaries will survive this

In my experience (mostly with Alfred plugins) if the executable is signed (such as python3) then any code run through the executable will work. Issues crop up when you ship binaries to end users. This has been an on-going pain point in one of my projects.

Point being, I think plugins will work fine if they are just python code that is run through your binary which the user has already given a security exception to.

[nikvdp]

What I'm now wondering is if the signing of the binaries will survive this - I need to be able to execute ~/.datasette-app/venv/bin/datasette which was created by the python3.9 binary that lives inside the /Application/Datasette.app package!

The problem we ran into initially was that although the electron app itself was signed correctly, without the correct entitlement settings (what worked for us was this and this) the signed-ness didn't extend to the bundled interpreter, only to the electron app's own binaries, ymmv

I'm not familiar with python-build-standalone, but from the docs I think what what it's doing is pretty similar to what conda-pack does, except that with conda-pack the venv (with datasette already installed into it) is built during CI and baked into the mac .app bundle rather than spun up on the user's machine. Saves a few moving parts, but probably includes files you don't actually need, and the user might need admin permissions to install new packages or modify the venv on mac/win if they installed to /Applications or Program Files.

Do you have any idea if the combo of conda-pack and warp might enable Python plugin installation?

Warp might be helpful here because under the hood it just extracts the bundle into a cache folder it manages in the user's home folder and then execs it. From then on it's basically a normal python venv that just so happens to be installed at eg ~/Library/Application Support/warp/packages/datasette/bin/python on mac. As @iansinnott was saying, so long as that python bin is signed correctly before warp copies it to its cache folder then it should be as free to install and run plugins as any other python env.

I don't think conda-pack would be necessary for this either, bundling up a python-build-standalone env into a single executable with warp should be a similar process to doing it for a conda-packed env.

[simonw]

As of #50 this now works! Every commit gets a built, signed, notarized, zipped Datasette.dmg attached to it as an artifact.

[simonw]

Turned this all into a TIL: https://til.simonwillison.net/electron/sign-notarize-electron-macos