feat: macos signing and notarization #367
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -7,35 +7,130 @@ on: | |
| pull_request: | ||
| branches: | ||
| - master | ||
| workflow_dispatch: | ||
| inputs: | ||
| dist_root: | ||
| description: 'DIST_ROOT' | ||
| required: true | ||
| default: '/ipns/dist.ipfs.io' | ||
|
|
||
| env: | ||
| DIST_ROOT: ${{ github.event.inputs.custom_dist_root || '/ipns/dist.ipfs.io' }} # content root used for calculating diff to build | ||
| GO_IPFS_VER: 'v0.9.1' # go-ipfs daemon used for chunking and applying diff | ||
| CLUSTER_CTL_VER: 'v0.14.0' # ipfs-cluster-ctl used for pinning | ||
|
|
||
| jobs: | ||
| build: | ||
| runs-on: "ubuntu-latest" | ||
| steps: | ||
| - uses: actions/checkout@v2 | ||
| - uses: actions/setup-node@v2 | ||
| with: | ||
| node-version: '14' | ||
| - name: Setup IPFS | ||
| run: ./scripts/ci/setup-ipfs.sh | ||
| env: | ||
| CLUSTER_USER: ${{ secrets.CLUSTER_USER }} | ||
| CLUSTER_PASSWORD: ${{ secrets.CLUSTER_PASSWORD }} | ||
| timeout-minutes: 5 | ||
| - name: Build any new ./releases | ||
| run: ./dockerized make all_dists | ||
| - name: Inspect git status and contents of ./releases | ||
| run: git status && ls -Rhl ./releases | ||
| - name: Temporarily save ./releases artifacts | ||
| uses: actions/upload-artifact@v2 | ||
| with: | ||
| name: releases-unsigned-diff | ||
| path: releases | ||
| retention-days: 1 | ||
|
|
||
| lint: | ||
| runs-on: "ubuntu-latest" | ||
| steps: | ||
| - uses: actions/checkout@v2 | ||
| - uses: actions/setup-node@v2 | ||
| with: | ||
| node-version: '14' | ||
| - run: npm ci --no-audit --progress=false | ||
| - run: npm run lint | ||
|
|
||
| sign-macos: | ||
| runs-on: "macos-latest" | ||
| needs: build | ||
| steps: | ||
| - uses: actions/checkout@v2 | ||
| - name: Retrieve unsigned artifacts | ||
| uses: actions/download-artifact@v2 | ||
| with: | ||
| name: releases-unsigned-diff | ||
| path: releases | ||
| continue-on-error: true # skip if no releases | ||
| - name: List ./releases before | ||
| run: ls -Rhl ./releases || echo "No ./releases" | ||
| - name: Install gon via HomeBrew for code signing and app notarization | ||
| run: | | ||
| brew tap mitchellh/gon | ||
| brew install ipfs coreutils gawk gnu-sed jq mitchellh/gon/gon | ||
|
Comment on lines
+72
to
+73
There was a problem hiding this comment. ℹ️ gon package is used by Terraform project, so multiple eyes are looking at this |
||
| ipfs init --profile test # needed for calculating NEW_CID later | ||
| - name: Import Keychain Certs | ||
| uses: apple-actions/import-codesign-certs@253ddeeac23f2bdad1646faac5c8c2832e800071 # v1@2020-02-03 | ||
|
There was a problem hiding this comment. ℹ️ 👮♂️ I hardcoded specific revision because this is a third-party action and in theory someone could swap-out git tag and steal our signing keys. |
||
| with: | ||
| p12-file-base64: ${{ secrets.APPLE_CERTS_P12 }} | ||
| p12-password: ${{ secrets.APPLE_CERTS_PASS }} | ||
| - name: Verify identity used for signing | ||
| run: security find-identity -v | ||
| - name: Sign any new releases | ||
| run: ./scripts/ci/sign-new-macos-releases.sh | ||
| env: | ||
| WORK_DIR: ${{ github.workspace }} | ||
| AC_USERNAME: ${{ secrets.APPLE_AC_USERNAME }} # implicitly read from env by gon | ||
| AC_PASSWORD: ${{ secrets.APPLE_AC_PASSWORD }} | ||
| - name: List ./releases after | ||
| run: ls -Rhl ./releases || echo "No ./releases" | ||
| - name: Temporarily save notarized artifacts | ||
| uses: actions/upload-artifact@v2 | ||
| with: | ||
| name: releases-signed-macos-diff | ||
| path: releases | ||
| retention-days: 1 | ||
| continue-on-error: true # skip if no releases | ||
|
|
||
| persist: | ||
| runs-on: "ubuntu-latest" | ||
| needs: sign-macos | ||
| steps: | ||
| - uses: actions/checkout@v2 | ||
| - name: Retrieve signed artifacts | ||
| uses: actions/download-artifact@v2 | ||
| continue-on-error: true # skip if no releases | ||
| with: | ||
| name: releases-signed-macos-diff | ||
| path: releases | ||
| - name: List ./releases | ||
| run: ls -Rhl ./releases || echo "No ./releases" | ||
| - name: Setup IPFS | ||
| run: ./scripts/ci/setup-ipfs.sh | ||
| env: | ||
| CLUSTER_USER: ${{ secrets.CLUSTER_USER }} | ||
| CLUSTER_PASSWORD: ${{ secrets.CLUSTER_PASSWORD }} | ||
| timeout-minutes: 5 | ||
| - run: ./dockerized make publish | ||
| - run: git status | ||
| - name: Read CID of updated DAG | ||
| id: cid-reader | ||
| run: echo "::set-output name=CID::$(tail -1 ./versions)" | ||
| - name: Pin new website to ipfs-websites.collab.ipfscluster.io | ||
| run: ./scripts/ci/pin-to-cluster.sh | ||
| env: | ||
| PIN_CID: ${{ steps.cid-reader.outputs.CID }} | ||
| PIN_NAME: "https://github.com/ipfs/distributions/commits/${{ github.sha }}" | ||
| PIN_ADD_EXTRA_ARGS: "" | ||
| CLUSTER_USER: ${{ secrets.CLUSTER_USER }} | ||
| CLUSTER_PASSWORD: ${{ secrets.CLUSTER_PASSWORD }} | ||
| timeout-minutes: 60 | ||
| - name: Update PR status with preview link | ||
| run: ./scripts/ci/github-preview-link.sh | ||
| env: | ||
| CONTENT_PATH: "/ipfs/${{ steps.cid-reader.outputs.CID }}/" | ||
| GIT_REVISION: ${{ github.sha }} | ||
| GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -25,4 +25,4 @@ if [ "$failed" = true ]; then | |
| fi | ||
|
|
||
| echo "npm install" | ||
| exec npm ci --prefer-offline --no-audit --progress=false | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| #!/usr/bin/env bash | ||
| set -e | ||
|
|
||
| PREVIEW_URL="https://dweb.link$CONTENT_PATH" | ||
| API_PARAMS=$(jq --monochrome-output --null-input \ | ||
| --arg state "success" \ | ||
| --arg target_url "$PREVIEW_URL" \ | ||
| --arg description "Preview updated website on IPFS" \ | ||
| --arg context "Preview is ready" \ | ||
| '{ state: $state, target_url: $target_url, description: $description, context: $context }' ) | ||
| curl --output /dev/null --silent --show-error \ | ||
| -X POST -H "Authorization: Bearer $GITHUB_TOKEN" -H 'Content-Type: application/json' \ | ||
| --data "$API_PARAMS" 'https://api.github.com/repos/ipfs/distributions/statuses/${GIT_REVISION}' | ||
|
There was a problem hiding this comment. ℹ️ we could replace this script with JS call and |
||
| echo "Pinned to IPFS - $PREVIEW_URL" | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,28 @@ | ||
| #!/usr/bin/env bash | ||
| set -e | ||
|
|
||
| echo "::group::pin add" | ||
| ipfs-cluster-ctl --enc=json \ | ||
| --host "/dnsaddr/ipfs-websites.collab.ipfscluster.io" \ | ||
| --basic-auth "${CLUSTER_USER}:${CLUSTER_PASSWORD}" \ | ||
| pin add \ | ||
| --name "${PIN_NAME}" \ | ||
| --no-status $PIN_ADD_EXTRA_ARGS \ | ||
| "$PIN_CID" | ||
| echo "::endgroup::" | ||
|
|
||
| echo "::group::waiting until pinned" | ||
| while true; do | ||
| ipfs-cluster-ctl --enc=json \ | ||
| --host "/dnsaddr/ipfs-websites.collab.ipfscluster.io" \ | ||
| --basic-auth "${CLUSTER_USER}:${CLUSTER_PASSWORD}" \ | ||
| status "$PIN_CID" | tee cluster-pin-status | ||
| if [[ $(jq '.peer_map[].status' cluster-pin-status | grep '"pinned"' | wc -l) -ge 2 ]]; then | ||
| echo "Got 2 pin confirmations, finishing the workflow" | ||
|
Comment on lines
+20
to
+21
There was a problem hiding this comment. ℹ️ to make CI faster and avoid hanging forever if parts of cluster are out of sync, cluster pinning is considered "done" after we have 2 copies. Here, I do it manually in userland, but filled ipfs-cluster/ipfs-cluster#1427 to add this sort of thing to ipfs-cluster-ctl CLI tool. There was a problem hiding this comment. Similar to above: when do we decide to put this in the PR vs. the code itself? Do folks tend to do a git blame and look for comments to find more context? There was a problem hiding this comment. I often use git blame, but in this specific case those comments are provided mostly to make PR review easier. There was a problem hiding this comment. Ok. I'll defer to the team for the standard. I was bringing this up because it was hard for me to imagine a case where a comment is useful during PR where it isn't also useful 6 months in the future when looking at the code with fresh eyes. Anyways, sounds good. |
||
| break | ||
| else | ||
| echo "(sleeping for 15 seconds)" | ||
| sleep 15 | ||
| fi | ||
| done | ||
| echo "::endgroup::" | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,40 @@ | ||
| #!/usr/bin/env bash | ||
| set -e | ||
|
|
||
| echo "::group::Install go-ipfs and ipfs-cluster-ctl" | ||
| curl -s https://dist.ipfs.io/go-ipfs/${GO_IPFS_VER}/go-ipfs_${GO_IPFS_VER}_linux-amd64.tar.gz | sudo tar vzx -C /usr/local/bin/ go-ipfs/ipfs --strip-components=1 | ||
| curl -s https://dist.ipfs.io/ipfs-cluster-ctl/${CLUSTER_CTL_VER}/ipfs-cluster-ctl_${CLUSTER_CTL_VER}_linux-amd64.tar.gz | sudo tar vzx -C /usr/local/bin/ ipfs-cluster-ctl/ipfs-cluster-ctl --strip-components=1 | ||
| echo "::endgroup::" | ||
|
|
||
| # fix resolv - DNS provided by Github is unreliable for DNSLik/dnsaddr | ||
| sudo sed -i -e 's/nameserver 127.0.0.*/nameserver 1.1.1.1/g' /etc/resolv.conf | ||
|
Comment on lines
+9
to
+10
There was a problem hiding this comment. ℹ️ This was pretty annoying. Entire build failed because Githubs DNS proxying cache at |
||
|
|
||
| # QUIC perf: https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size | ||
| sudo sysctl -w net.core.rmem_max=2500000 | ||
|
|
||
| # init ipfs | ||
| echo "::group::Set up IPFS daemon" | ||
| ipfs init --profile flatfs,server,test,lowpower | ||
| # make flatfs async for faster ci | ||
| new_config=$( jq '.Datastore.Spec.mounts[0].child.sync = false' ~/.ipfs/config) && echo "${new_config}" > ~/.ipfs/config | ||
| # restore deterministic port (changed by test profile) | ||
| ipfs config Addresses.API "/ip4/127.0.0.1/tcp/5001" | ||
| # wait for ipfs daemon | ||
| ipfs daemon --enable-gc=false & while (! ipfs id --api "/ip4/127.0.0.1/tcp/5001"); do sleep 1; done | ||
| echo "::endgroup::" | ||
|
|
||
|
|
||
| echo "::group::Preconnect to cluster peers" | ||
| echo '-> preconnect to cluster peers' | ||
| ipfs-cluster-ctl --enc=json \ | ||
| --host "/dnsaddr/ipfs-websites.collab.ipfscluster.io" \ | ||
| --basic-auth "${CLUSTER_USER}:${CLUSTER_PASSWORD}" \ | ||
| peers ls > cluster-peers-ls | ||
| for maddr in $(jq -r '.[].ipfs.addresses[]?' cluster-peers-ls); do | ||
| ipfs swarm connect "$maddr" || continue | ||
| done | ||
| echo '-> manual connect to cluster.ipfs.io' | ||
| ipfs swarm connect /dnsaddr/cluster.ipfs.io | ||
| echo '-> list swarm peers' | ||
| ipfs swarm peers | ||
| echo "::endgroup::" | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What's the process for updating these? Do we rely on someone manually updating it? What's their trigger for doing so?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think those are low importance: could be kept as-is until someone cares to update or there is a breaking change that forces version bump.
Having static versions of build tools makes the builds more robust and reproducible, but we could also switch to always run on the latest version for dogfooding – don't feel strongly either way, but leaning towards more reproducibility.