Automate copying of signed binaries from dist.ipfs.io/go-ipfs

[lidel]

Current state

• macOS signing is being added in feat: macos signing and notarization  distributions#367 which means dist.ipfs.io is the source of the truth when it comes to official binaries
• We want binaries attached to releases at https://github.com/ipfs/go-ipfs/releases to be signed as well, so they need to come from /ipns/dist.ipfs.io/go-ipfs/

Proposed change

(iiuc) binaries attached to GitHub release are copied from /ipns/dist.ipfs.io/go-ipfs/ manually by @aschmahmann or @Stebalien – we should automate this.

Solution design

Create a GitHub Action workflow (triggered manually via workflow_dispatch and maybe daily via cron to avoid human error) that iterates over every vA.B.C release and attaches any missing files from /ipns/dist.ipfs.io/go-ipfs/VA.B.C (including manifest and .cid and .sha512 files) if not present yet.

This ensures that going forward https://github.com/ipfs/go-ipfs/releases is in sync with dist.ipfs.io.

[BigLep]

@schomatis : this is a good candidate that we could use your help for.

For testing/development, you can use your own repo for testing GitHub actions.

[schomatis]

@BigLep I'm not familiar with GitHub actions nor with our distribution system so I'm not sure I'm the right person for this one. However, if you think it's worthwhile I can spend some time familiarizing myself with this and give it a try.

[BigLep]

@schomatis : we talked about it more internally. The other go-ipfs Stewards will handle this.

[petar]

@lidel Could you provide a bit more detail on what needs to be done here?

[lidel]

For v0.10.0 we had to manually copy all files:

• from https://dist.ipfs.io/go-ipfs/v0.10.0
• to "Assets" section at the bottom of https://github.com/ipfs/go-ipfs/releases/tag/v0.10.0

@petar I think MVP we want here is a Github Action workflow that utilizes https://github.com/actions/github-script to automate the copying if release files. We could run it once a day or trigger it manually.

---

Some notes / suggestions around implementation, hopefully you find them useful:

• Use Github API to get the list of GitHub Releases (https://github.com/ipfs/go-ipfs/releases)
  • github-script action has octokit library and access token built-in, so you can call github.rest.repos.listReleases + find an example of the response output here
  • prior art for how we use github-script in a workflow at ipfs/distributions/../nightly.yml
• For every release version found via listReleases
  • calculate the size of assets array (number of files attached to Github Release)
  • count how many files got published at dist.ipfs.io
    • for example, ipfs ls /ipns/dist.ipfs.io/go-ipfs/v0.10.0 | wc -l returns 48 files
  • if release on GitHub has less than the directory on dist.ipfs, download and attach missing ones to it
    • ipfs get /ipns/dist.ipfs.io/go-ipfs/v0.10.0 will download everything to a directory named v0.10.0
    • then you can attach each file to Github Release via github.rest.repos.uploadReleaseAsset
      • sidenote: this is tricky to test without introducing garbage release. I think we could test this with 0.10.0 with minimal disruption: manually remove results file from the 0.10.0 release, and then observe is automation restores it back

lmk if you get blocked on anything, or think there is a better way for removing the need for manual copying of files on each release

[Stebalien]

We could run it once a day or trigger it manually.

As long as it runs when sent a custom "event", we can trigger it from the https://github.com/ipfs/distributions actions.