bug: pinning fails during CI

[SgtPooki]

Describe the bug
Pinning is failing during CI actions: https://github.com/ipfs/ipfs-webui/actions/runs/6165506297/job/16733763437. We currently use both estuary and our own ipfs-cluster; both are failing.

We should investigate using fleek.co (since most of our applications are using fleek.co now) or web3.storage for pinning webui files instead.

Things that could reduce the complexity of CI by moving to...

Fleek

• creating CID with contents of build/ -

  ipfs-webui/.github/workflows/ci.yml

  Lines 69 to 74 in 346c367

  	- uses: ipfs/start-ipfs-daemon-action@v1
  	- name: Import build/ to IPFS
  	id: ipfs
  	run: |
  	root_cid=$(ipfs add --cid-version 1 -Q -r ./build)
  	echo "cid=$root_cid" >> $GITHUB_OUTPUT

• ? create car file -

  ipfs-webui/.github/workflows/ci.yml

  Lines 78 to 80 in 346c367

  	- name: Create ipfs-webui.car file
  	run: |
  	ipfs dag export ${{ steps.ipfs.outputs.cid }} > ipfs-webui_${{ github.sha }}.car

• pinning -

  ipfs-webui/.github/workflows/ci.yml

  Lines 88 to 125 in 346c367

  	- name: Pin to estuary.tech
  	id: pin-estuary
  	continue-on-error: true
  	run: |
  	curl -X POST https://api.estuary.tech/pinning/pins -d '{ "name": "'$PIN_NAME'", "cid": "'$PIN_CID'" }' -H "Content-Type: application/json" -H "Authorization: Bearer $ESTUARY_API_TOKEN"
  	curl -X GET https://api.estuary.tech/pinning/pins -H "Content-Type: application/json" -H "Authorization: Bearer $ESTUARY_API_TOKEN"
  	env:
  	ESTUARY_API_TOKEN: ${{ secrets.ESTUARY_API_TOKEN }}
  	PIN_CID: ${{ steps.ipfs.outputs.cid }}
  	PIN_NAME: "ipfs-webui@${{ github.sha }}"
  	- name: Pin to ipfs-websites.collab.ipfscluster.io
  	id: pin-cluster
  	continue-on-error: true
  	run: |
  	ipfs-cluster-ctl --enc=json \
  	--host "${CLUSTER_HOST}" \
  	--basic-auth "$CLUSTER_USER:$CLUSTER_PASSWORD" \
  	peers ls > cluster-peers-ls
  	for maddr in $(jq -r '.ipfs.addresses[]?' cluster-peers-ls); do
  	ipfs swarm peering add "$maddr" &
  	ipfs swarm connect "$maddr" &
  	done
  	ipfs-cluster-ctl --enc=json \
  	--host "${CLUSTER_HOST}" \
  	--basic-auth "${CLUSTER_USER}:${CLUSTER_PASSWORD}" \
  	pin add \
  	--name "${PIN_NAME}" \
  	--replication-min 1 \
  	--replication-max 3 \
  	--wait \
  	$PIN_CID
  	env:
  	CLUSTER_HOST: "/dnsaddr/ipfs-websites.collab.ipfscluster.io"
  	CLUSTER_USER: ${{ secrets.CLUSTER_USER }}
  	CLUSTER_PASSWORD: ${{ secrets.CLUSTER_PASSWORD }}
  	PIN_CID: ${{ steps.ipfs.outputs.cid }}
  	PIN_NAME: "ipfs-webui@${{ github.sha }}"

• dnslink -

  ipfs-webui/.github/workflows/ci.yml

  Lines 134 to 144 in 346c367

  	# dev dnslink is updated on each main branch update
  	- run: npx dnslink-dnsimple --domain dev.webui.ipfs.io --link /ipfs/${{ steps.ipfs.outputs.cid }}
  	if: github.ref == 'refs/heads/main'
  	env:
  	DNSIMPLE_TOKEN: ${{ secrets.DNSIMPLE_TOKEN }}
  	# production dnslink is updated on release (during tag build)
  	- run: npx dnslink-dnsimple --domain webui.ipfs.io --link /ipfs/${{ steps.ipfs.outputs.cid }}
  	if: github.ref == 'refs/heads/main' && github.event_name == 'workflow_dispatch'
  	env:
  	DNSIMPLE_TOKEN: ${{ secrets.DNSIMPLE_TOKEN }}

web3.storage

• pinning -

  ipfs-webui/.github/workflows/ci.yml

  Lines 88 to 125 in 346c367

  	- name: Pin to estuary.tech
  	id: pin-estuary
  	continue-on-error: true
  	run: |
  	curl -X POST https://api.estuary.tech/pinning/pins -d '{ "name": "'$PIN_NAME'", "cid": "'$PIN_CID'" }' -H "Content-Type: application/json" -H "Authorization: Bearer $ESTUARY_API_TOKEN"
  	curl -X GET https://api.estuary.tech/pinning/pins -H "Content-Type: application/json" -H "Authorization: Bearer $ESTUARY_API_TOKEN"
  	env:
  	ESTUARY_API_TOKEN: ${{ secrets.ESTUARY_API_TOKEN }}
  	PIN_CID: ${{ steps.ipfs.outputs.cid }}
  	PIN_NAME: "ipfs-webui@${{ github.sha }}"
  	- name: Pin to ipfs-websites.collab.ipfscluster.io
  	id: pin-cluster
  	continue-on-error: true
  	run: |
  	ipfs-cluster-ctl --enc=json \
  	--host "${CLUSTER_HOST}" \
  	--basic-auth "$CLUSTER_USER:$CLUSTER_PASSWORD" \
  	peers ls > cluster-peers-ls
  	for maddr in $(jq -r '.ipfs.addresses[]?' cluster-peers-ls); do
  	ipfs swarm peering add "$maddr" &
  	ipfs swarm connect "$maddr" &
  	done
  	ipfs-cluster-ctl --enc=json \
  	--host "${CLUSTER_HOST}" \
  	--basic-auth "${CLUSTER_USER}:${CLUSTER_PASSWORD}" \
  	pin add \
  	--name "${PIN_NAME}" \
  	--replication-min 1 \
  	--replication-max 3 \
  	--wait \
  	$PIN_CID
  	env:
  	CLUSTER_HOST: "/dnsaddr/ipfs-websites.collab.ipfscluster.io"
  	CLUSTER_USER: ${{ secrets.CLUSTER_USER }}
  	CLUSTER_PASSWORD: ${{ secrets.CLUSTER_PASSWORD }}
  	PIN_CID: ${{ steps.ipfs.outputs.cid }}
  	PIN_NAME: "ipfs-webui@${{ github.sha }}"

[SgtPooki]

i'm investigating using w3cli for this, but running into some issues.. will need some more debugging and testing: https://filecoinproject.slack.com/archives/C027XP5BTGB/p1694813264220989

[SgtPooki]

we can potentially use fleek (even though pinning is currently behind closed alpha) via their site:deploy command and then getting the CID via dig or other: dig +noall +answer TXT \_dnslink.ipfs-webui.on.fleek.co

[SgtPooki]

pinning seems to be succeeding on our ipfs cluster now: https://github.com/ipfs/ipfs-webui/actions/runs/6150397438/job/16688431637

[SgtPooki]

testing out w3cli at https://github.com/SgtPooki/ipfs-webui/actions/workflows/ci.yml

[SgtPooki]

Process for pinning with w3cli

The below steps are finalized and working in this repo now

Given the following env vars:

export W3CLI_SPACE_NAME="foobar123"
export W3CLI_SPACE_DELEGATION_PROOF_FILENAME="ipfs-webui-ci-space-delegation.proof"
export W3CLI_SPACE_DELEGATION_PROOF_BASE64_FILENAME="ipfs-webui-ci-space-delegation.proof.base64"
export W3_STORE_NAME="ipfs-webui-store"
export UCAN_PRINCIPAL_FOR_CI_FILENAME="w3cli-ci-principal.json"

on your computer

# install w3cli, exposes command `w3`
npm install -g @web3-storage/w3cli 

# verify email if you don't already have an account
w3 login <email> 

# creates a space and returns a DID we store into `W3CLI_SPACE_DID`
W3CLI_SPACE_DID=$(w3 space create $W3CLI_SPACE_NAME) 

# Create a private key and DID our CI agent will use. 
# See https://github.com/web3-storage/w3cli#w3_principal & the `const principal = Signer.parse(process.env.KEY)` line at https://github.com/web3-storage/w3up/tree/main/packages/w3up-client#bringing-your-own-agent-and-delegation for more information
npx -y ucan-key ed --json > $UCAN_PRINCIPAL_FOR_CI_FILENAME

# this is a private key for your CI "agent". This needs uploaded to github as a SECRET.
W3_AGENT_PRINCIPAL=$(cat $UCAN_PRINCIPAL_FOR_CI_FILENAME | jq -r .key)

# this is the DID for your CI Agent
W3_AGENT_DID=$(cat $UCAN_PRINCIPAL_FOR_CI_FILENAME | jq -r .did)


# create a delegation that has permissions for the CI runner, and save delegation to a file
w3 delegation create $W3_AGENT_DID -n 'ipfs-webui-CI' --can 'store/*' --can 'upload/*' --output $W3CLI_SPACE_DELEGATION_PROOF_FILENAME 

# convert space delegation proof car to a base64 encoded string and save in a file with the name $W3CLI_SPACE_DELEGATION_PROOF_BASE64_FILENAME
# Note that on my machine, base64 is BSD and gbase64 is from GNU-tools, so I use gbase64
cat $W3CLI_SPACE_DELEGATION_PROOF_FILENAME | gbase64 -w0 > $W3CLI_SPACE_DELEGATION_PROOF_BASE64_FILENAME 

on CI

Make sure you set your GitHub repo secrets:

Set CI SECRETS:

• W3_AGENT_DID = the content of $W3_AGENT_DID from above
• W3_AGENT_PRINCIPAL = the content of $W3_AGENT_PRINCIPAL from above
• W3CLI_SPACE_DELEGATION_PROOF_BASE64_STRING = the result of $(cat $W3CLI_SPACE_DELEGATION_PROOF_BASE64_FILENAME)

When running your action, you will need to ensure you do:

    env:
      W3_STORE_NAME: ipfs-webui-ci
      W3_AGENT_DID: ${{ secrets.W3_AGENT_DID }}
      # set w3cli principal from https://github.com/web3-storage/w3cli#w3_principal
      W3_PRINCIPAL: ${{ secrets.W3_AGENT_PRINCIPAL }}
      W3CLI_SPACE_DELEGATION_PROOF_BASE64_STRING: ${{ secrets.W3CLI_SPACE_DELEGATION_PROOF_BASE64_STRING }}

### Non w3cli stuff:

# create car file with contents you want to upload, stored in `my-car.car`

### web3cli stuff

# auth/delegation with w3cli still unclear.. 
# some instructions at https://github.com/web3-storage/w3up/tree/main/packages/w3up-client#bringing-your-own-agent-and-delegation

# ensure whoami shows we are who we think we are. This should be the same as $W3_AGENT_DID
npx -y --package=@web3-storage/w3cli@latest -- w3 whoami

# convert W3CLI_SPACE_DELEGATION_PROOF_BASE64_STRING to a file:
echo $W3CLI_SPACE_DELEGATION_PROOF_BASE64_STRING | base64 -d > $W3CLI_SPACE_DELEGATION_PROOF_FILENAME


# I'm not sure if I need this, or I just need to use `space use`
npx -y --package=@web3-storage/w3cli@latest -- w3 proof add $W3CLI_SPACE_DELEGATION_PROOF_FILENAME


# add that space
W3CLI_SPACE_DID=$(npx -y --package=@web3-storage/w3cli@latest -- w3 space add $W3CLI_SPACE_DELEGATION_PROOF_FILENAME)

# now ensure the w3cli is using that space
npx -y --package=@web3-storage/w3cli@latest -- w3 space use $W3CLI_SPACE_DID

# upload the car 
npx -y --package=@web3-storage/w3cli@latest -- w3 up --no-wrap -c my-car.car 

[SgtPooki]

I think I finally got the w3cli stuff down: https://github.com/SgtPooki/ipfs-webui/actions/runs/7066688311/job/19239006867