Apple Notarization of migration assets

[lidel]

Friends from Brave noted that upgrading go-ipfs 0.7.0 to 0.9.0 seems to cause security error on macOS now:

Initializing daemon...
go-ipfs version: 0.9.0
Repo version: 11
System version: amd64/darwin
Golang version: go1.16.5
Found outdated fs-repo, migrations need to be run.
Looking for suitable migration binaries.
Need 1 migrations, downloading.
Downloading migration: fs-repo-10-to-11...
Fetching with HTTP: "https://ipfs.io/ipfs/QmVxxcTSuryJYdQJGcS8SyhzN7NBNLTqVPAxpu6gp2ZcrR/fs-repo-10-to-11/versions"
Fetching with HTTP: "https://ipfs.io/ipfs/QmVxxcTSuryJYdQJGcS8SyhzN7NBNLTqVPAxpu6gp2ZcrR/fs-repo-10-to-11/v1.0.0/fs-repo-10-to-11_v1.0.0_darwin-amd64.tar.gz"
Downloaded and unpacked migration: /var/folders/44/9r6j8dks44xdv4947lc_fkmh0000gn/T/migrations669767337/fs-repo-10-to-11 (v1.0.0)
Running migration fs-repo-10-to-11 ...
  => Running: /var/folders/44/9r6j8dks44xdv4947lc_fkmh0000gn/T/migrations669767337/fs-repo-10-to-11 -path=/Users/sdonner/Library/Application Support/BraveSoftware/Brave-Browser-Nightly/brave_ipfs -verbose=true
The migrations of fs-repo failed:
  migration fs-repo-10-to-11 failed: signal: killed
If you think this is a bug, please file an issue and include this whole log output.
  https://github.com/ipfs/fs-repo-migrations
Error: migration fs-repo-10-to-11 failed: signal: killed

Perhaps Apple hardened their policies since we shipped 0.7.0 → 0.8.0 to our Desktop users?
First step is to confirm this is real issue.

Step 1: confirm this reproducible

Need help from macOS user (~15min):

1. Get the official 0.7.0 for macOS from https://dist.ipfs.io/go-ipfs/v0.7.0/go-ipfs_v0.7.0_darwin-amd64.tar.gz
2. Use it to create an empty repo: execute export IPFS_PATH=/tmp/test-7-to-9 to point at a custom directory, then run ipfs init and ipfs daemon and confirm that it works ok.
3. Stop 0.7.0 daemon
4. Get official 0.9.0 for macOS from https://dist.ipfs.io/go-ipfs/v0.9.0/go-ipfs_v0.9.0_darwin-amd64.tar.gz
5. Start ipfs daemon
6. Comment below if migration was successful, or if the above error was displayed instead.

Step 2: fix the issue

• (ad-hoc fix for 7–9 migration) manually notarize related migration binary
• (real fix) add notarization workflow to dist.ipfs.io
  • mvp would be workflow_dispatch job that does notarization of a binary in specified *_darwin-*.tar.gz archive
  • harder: include notarization stapling as part of the build (tbd if worth it, one can assume that most of IPFS users on macOS is online at the time of migration)

[mburns]

0.7.0 -> 0.9.0 migration Worked For Me. No security pop-up about unknown developer...

Initializing daemon...
go-ipfs version: 0.9.0
Repo version: 11
System version: amd64/darwin
Golang version: go1.16.5
Found outdated fs-repo, migrations need to be run.
Run migrations now? [y/N] y
Looking for suitable migration binaries.
Need 1 migrations, downloading.
Downloading migration: fs-repo-10-to-11...
Fetching with HTTP: "https://ipfs.io/ipfs/QmVxxcTSuryJYdQJGcS8SyhzN7NBNLTqVPAxpu6gp2ZcrR/fs-repo-10-to-11/versions"
Fetching with HTTP: "https://ipfs.io/ipfs/QmVxxcTSuryJYdQJGcS8SyhzN7NBNLTqVPAxpu6gp2ZcrR/fs-repo-10-to-11/v1.0.0/fs-repo-10-to-11_v1.0.0_darwin-amd64.tar.gz"
Downloaded and unpacked migration: /var/folders/ql/lq9gwpgx3nggmjy13syp055m0000gn/T/migrations511631181/fs-repo-10-to-11 (v1.0.0)
Running migration fs-repo-10-to-11 ...
  => Running: /var/folders/ql/lq9gwpgx3nggmjy13syp055m0000gn/T/migrations511631181/fs-repo-10-to-11 -path=/tmp/test-7-to-9 -verbose=true
applying 10-to-11 repo migration
opening datastore at "/tmp/test-7-to-9"
upgrading pinning to use datastore
converted 2 pins from ipld storage into datastore
updated version file
Migration 10 to 11 succeeded
Success: fs-repo migrated to version 11.
Swarm listening on /ip4/127.0.0.1/tcp/4001
Swarm listening on /ip4/127.0.0.1/udp/4001/quic
Swarm listening on /ip4/192.168.1.245/tcp/4001
Swarm listening on /ip4/192.168.1.245/udp/4001/quic
Swarm listening on /ip6/::1/tcp/4001
Swarm listening on /ip6/::1/udp/4001/quic
Swarm listening on /p2p-circuit
Swarm announcing /ip4/127.0.0.1/tcp/4001
Swarm announcing /ip4/127.0.0.1/udp/4001/quic
Swarm announcing /ip4/192.168.1.245/tcp/4001
Swarm announcing /ip4/192.168.1.245/udp/4001/quic
Swarm announcing /ip4/50.39.131.247/udp/4001/quic
Swarm announcing /ip6/::1/tcp/4001
Swarm announcing /ip6/::1/udp/4001/quic
API server listening on /ip4/127.0.0.1/tcp/5001
WebUI: http://127.0.0.1:5001/webui
Added migration file "fs-repo-10-to-11_v1.0.0_darwin-amd64.tar.gz": /ipfs/QmaCqq6KLw7p9DBYtVgegMmkDEuRXkS395LVweYqdY7vcq
Could not add migration to IPFS: nothing downloaded by ipfs fetcher
Gateway (readonly) server listening on /ip4/127.0.0.1/tcp/8080
Daemon is ready

[lidel]

I believe @mburns was able to replicate since his last comment was posted.

@stephendonner @mburns what version of macOS are you running when experiencing this?

[stephendonner]

@lidel sorry for the delay; Monterey (public beta), version 12.0 Beta (21A5268h)

[lidel]

Quick update – successful replication, but this is limited to GUI apps, it seems.

I was able to replicate this error while simulating 0.7.0 → 0.9.0
with the latest Brave Nightly and macOS Big Sur 11.2:

It is a bit ridiculous that macOS does not complain when I run migration using the same go-ipfs binary via Terminal app instead of being spawned via Brave:

[lidel]

ipfs/distributions#367 (macos signing and notarization of darwin binaries at dist.ipfs.io via github actions) is ready for final review

[lidel]

Signed migrations for Intel and Apple Silicon are ready: ipfs/distributions#381 → ipfs/distributions#381
Remaining work here:

• retest 0.7.0 → 0.9.x migration in Brave using IPFS_DIST_PATH from feat: fs-repo-migrations v2.0.2 (macos signed) distributions#381 (@lidel)
  • wip, waiting for new CID with the fix feat: fs-repo-migrations v2.0.2 (macos signed) distributions#381 (comment)
• merge feat: fs-repo-migrations v2.0.2 (macos signed) distributions#381 (@lidel)
• (if migration works as expected) open a PR against go-ipfs to switch to signed migrations by default (and mark it to close this issue) (@lidel)

[lidel]

Reopening as this needs #8333 to be merged.