Superfluid GitHub Actions Deep Dive

Current Monorepo Layout

• Currently we use lerna to manage the monorepo packages.

About package.json

Even for projects that are not npm projects, a package.json is still provided in order to:

1. Be part of the lerna workspace.
2. Support repo-wide standard build pipeline tasks, often used via yarn workspace or yarn workspaces:

• yarn workspace @superfluid-finance/$i build
• yarn workspace @superfluid-finance/$i clean
• yarn workspaces run lint
• yarn workspaces run pre-commit

3. "tasks" folders are for build pipeline. 4 "scripts" are scripts to be used by local development or for distribution.

Note: targets for supply chain security examination:

• prepare
• postinstall

Common Pattersn

• Using ':' to separate npm sub scripts, use run-s|run-p to in series or in parallel.
  • use run-s to enforce pipeline sequence.
  • use run-p if parallel is desired.
• Common npm scripts as standard "verbs":
  • to be used in CI pipeline:
    • install
    • lint
    • build
    • test
    • clean
  • to be used in local development:
    • git-submodule:*, git submodule workflows
    • check-updates
    • pre-commit # husky pre-commit hook (.husky/pre-commit)
    • shell # nix shell
  • for monorepo workflow:
    • manage-versions
• Required workspace package common "verbs":
  • pre-commit
  • lint
  • build
  • test
  • clean
  • check-updates
  • cloc
• Common Workflow Patterns:
  • Use nix shell to deterministically setup CI environment.
  • "Small and lean": use as much common verbs as possible.
  • Complex scripts to put into tasks/ of the workspace package.

Purpose and General Philosophy

The GitHub Actions in our monorepo leverage automations to enable:

• actions to run tests to ensure a baseline level of security
• actions to run tests to minimize unexpected behavior
• actions to automate tasks we might forget
• verifiability (deployments, governance actions, etc.)

Some principles we strive to follow:

• DRY (Don't Repeat Yourself)
• Don't trust, verify
• Speed
• Internal developer experience

File Categories

call.*.yml

call.*.yml files are the reusable workflows we use throughout our workflow files. The purpose of this is to reduce duplication of jobs that may need to be run in different workflow files and steps. This improves maintainability and generally reduces complexity when making changes/reading code.

ci.*.yml

CI = Continuous Integration

ci.*.yml files are continuous integration workflows we use to run tests at different stages to ensure regression does not occur and that obvious breaking changes are not pushed to prod.

cd.*.yml

CD = Continuous Deployment

cd.*.yml files are continuous deployment workflows we use to handle a lot of tasks surrounding deployments. Using GitHub Actions to deploy contracts allows anyone to verify that a specific contract deployment is tied to a specific commit.

handler.*.yml

handler.*.yml files are intermediary files used manually to execute multi-step operations OR to execute multi-step operations upon completion of another workflow.

daily-*.yml

daily-*.yml files are used to run daily checks or to create daily messages.

File-by-file Breakdown

How to use

I would recommend that you just read the one liner description of each of the files just to have an idea of the purpose of each file. If you are making changes to the file, then it makes a bit more sense to read further: e.g. where the file is used and the satellite files (files surrounding the specified workflow file which might need special attention/modification).

call.check-query-schema-against-subgraph.yml

A reusable workflow to check that the SDK-Core GraphQL schema is consistent with the deployed subgraph's schema.

Expand

Used in:

• ci.canary
• ci.feature
• daily-query-check

Satellite Files:

• packages/sdk-core/tasks/package.json
• packages/sdk-core/tasks/startHardhatNode.sh
• packages/sdk-core/tasks/setupTestEnvironment.sh
• packages/sdk-core/tasks/testSchemasAndQueries.sh
• packages/sdk-core/previous-versions-testing/runQueryTests.ts

call.deploy-subgraph.yml

A reusable workflow to deploy the subgraph.

Expand

Used in:

• ci.canary
• handler.deploy-subgraph

Secrets:

• SATSUMA_DEPLOY_KEY
• THE_GRAPH_ACCESS_TOKEN

Satellite Files:

• packages/subgraph/package.json
• packages/subgraph/scripts/getAbi.js
• packages/subgraph/subgraph.template.yaml
• packages/subgraph/tasks/getSFMeta.sh
• packages/subgraph/tasks/deploy.sh
• packages/subgraph/tasks/deploy-all-networks.sh
• packages/subgraph/tasks/deploy-to-network.sh
• packages/subgraph/tasks/deploy-to-satsuma.sh

call.setup-deploy-and-test-local-subgraph.yml

A reusable workflow to run unit and integration tests on the subgraph, it also tests the new subgraph schema against the local

Expand

Used in:

• ci.canary
• ci.feature
• ci.pre-release-subgraph

Satellite Files:

• packages/ethereum-contracts/dev-scripts/runDeployContractsAndToken.js
• packages/sdk-core/tasks/startHardhatNode.sh
• packages/subgraph/src/*: mapping files
• packages/subgraph/test/*
• packages/subgraph/tests/*
• packages/subgraph/tasks/setup-graph-node.sh
• packages/subgraph/tasks/testenv-ctl.sh
• packages/subgraph/docker-compose.yml
• packages/subgraph/scripts/getAbi.js
• packages/subgraph/subgraph.template.yaml
• packages/subgraph/src/addresses.template.ts

call.test-automation-contracts.yml

A reusable workflow that builds and tests the automation contracts.

Expand

Used in:

• ci.feature
• ci.canary

Satellite Files:

• packages/automation-contracts/autowrap/*
• packages/automation-contracts/scheduler/*

call.test-sdk-core.yml

A reusable workflow that builds and tests SDK-Core.

Expand

Used in:

• ci.feature
• ci.pre-release-sdk-core

Satellite Files:

• packages/sdk-core/tasks/testenv-ctl.sh
• packages/sdk-core/tasks/startHardhatNode.sh
• packages/sdk-core/tasks/setupTestEnvironment.sh
• packages/ethereum-contracts/dev-scripts/runDeployContractsAndToken.js

call.test-spec-haskell.yml

A reusable workflow that builds and tests the Haskell spec.

Expand

Used in:

• ci.canary
• ci.feature

Satellite Files:

• packages/spec-haskell/*

call.test-subgraph-on-previous-sdk-core-versions.yml

A reusable workflow that tests the deployed subgraph against previous SDK-Core versions.

Expand

Used in:

• ci.canary
• ci.feature
• ci.pre-release-subgraph

cd.ethereum-contracts-stable.create-release-drafts.yml

A continuous deployment workflow that creates a release draft to be filled out and published for ethereum contracts.

Expand

Satellite Files:

• packages/ethereum-contracts/package.json

cd.feature.create-pr-artifact.yml

A continuous deployment workflow that creates a PR artifact when a pull request is opened.

Expand

Satellite Files:

• tasks/check-changeset.sh

cd.sdk-core-stable.create-release-drafts.yml

A continuous deployment workflow that creates a release draft to be filled out and published for sdk-core.

Expand

Satellite Files:

• packages/sdk-core/package.json

cd.sdk-redux-stable.create-release-drafts.yml

A continuous deployment workflow that creates a release draft to be filled out and published for sdk-redux.

Expand

Satellite Files:

• packages/sdk-redux/package.json

cd.subgraph-stable.create-release-drafts.yml

A continuous deployment workflow that creates a release draft to be filled out and published for the subgraph.

Expand

Satellite Files:

• packages/subgraph/package.json

ci.canary.yml

A continuous integration workflow that builds, tests and publishes coverage reports for all packages in the monorepo upon a PR being merged to dev. It also publishes a dev build for ethereum-contracts, sdk-core and sdk-redux and deploys the subgraph if there were changes made to subgraph files.

Expand

Satellite Files:

• tasks/check-changeset.sh: Only used for determining if we need to deploy the subgraph
• ./.github/workflows/call.setup-deploy-and-test-local-subgraph.yml@dev
• ./.github/workflows/call.test-subgraph-on-previous-sdk-core-versions.yml
• ./.github/workflows/call.check-query-schema-against-subgraph.yml
• ./.github/workflows/call.deploy-subgraph.yml
• ./.github/workflows/call.test-spec-haskell.yml

ci.feature.yml

A continuous integration workflow that builds, tests and publishes coverage reports for all packages in the monorepo upon a PR being opened. It uses tasks/check-changeset.sh to check what files were changed compared to dev branch and will run specific jobs based on the output of the check job.

Expand

Satellite Files:

• tasks/check-changeset.sh

NOTE: It is important that you modify this file if you are adding a new package to the monorepo if you plan on adding tests for it to ci.feature.yml.

• ./.github/workflows/call.setup-deploy-and-test-local-subgraph.yml
• ./.github/workflows/call.test-sdk-core.yml
• ./.github/workflows/call.test-subgraph-on-previous-sdk-core-versions.yml
• ./.github/workflows/call.check-query-schema-against-subgraph.yml
• ./.github/workflows/call.test-spec-haskell.yml

ci.pre-release-sdk-core.yml

A continuous integration workflow that builds, tests and publishes coverage reports for all packages in the monorepo. It also publishes a dev build for ethereum-contracts, sdk-core and sdk-redux.

Expand

Satellite Files:

• ./.github/workflows/call.check-query-schema-against-subgraph.yml
• ./.github/workflows/call.test-sdk-core.yml

ci.pre-release-subgraph.yml

A continuous integration workflow that runs a series of tests upon PRs to release-subgraph-v1 branch to ensure that a subgraph deployment to V1 will not break the subgraph, previous versions and the current version of the sdk-core.

Expand

Satellite Files:

• ./.github/workflows/call.setup-deploy-and-test-local-subgraph.yml
• ./.github/workflows/call.test-subgraph-on-previous-sdk-core-versions.yml

daily-query-check.yml

A cron job run daily to check that the sdk-core query GraphQL schema representation is synced with the deployed v1 and dev endpoint schema representations.

Expand

Satellite Files:

• ./.github/workflows/call.check-query-schema-against-subgraph.yml

daily-slack-message.yml

A cron job run daily to report open PRs and the status of the latest canary (dev) build.

Expand

Satellite Files:

• tasks/daily-slack-bot.js

handler.changelog-reminder.yml

A handler workflow which creates a reminder to update the changelogs whenever a PR is created.

handler.deploy-subgraph.yml

A handler workflow used to deploy the subgraph manually using the deploy subgraph reusable workflow.

Expand

Satellite Files:

• ./.github/workflows/call.deploy-subgraph.yml

handler.deploy-to-mainnet.yml

A handler workflow used to deploy Superfluid contracts to mainnet.

Expand

Satellite Files:

• packages/ethereum-contracts/ops-scripts/deploy-framework.js

handler.deploy-to-testnets.yml

A handler workflow used to deploy Superfluid contracts to testnets.

Expand

Satellite Files:

• packages/ethereum-contracts/ops-scripts/deploy-test-environment.js
• packages/ethereum-contracts/ops-scripts/info-print-contract-addresses.js
• packages/ethereum-contracts/tasks/etherscan-verify-framework.sh

handler.list-super-token.yml

A handler workflow used to list super tokens on our resolver.

Expand

Satellite Files:

• packages/ethereum-contracts/ops-scripts/resolver-list-super-token.js

handler.placeholder-for-testing.yml

A handler workflow that can be used for testing and reverted before merging.

handler.publish-pr-packages.yml

A handler workflow which publishes PR builds to Github upon a PR successfully passing all checks and our continuous deployment workflow file creating a PR artifact.

Expand

Satellite Files:

• tasks/npmrc-use-github.sh

handler.publish-release-packages.yml

A handler workflow which publishes release packages to NPM upon a release draft being published.

Expand

Satellite Files:

• tasks/npm-publish.sh
• packages/sdk-core/src/*
• packages/sdk-core/tsconfig.json
• packages/sdk-core/tsconfig.module.json
• packages/sdk-core/typedoc.js
• packages/sdk-redux/src/*
• packages/sdk-redux/tsconfig.json
• packages/sdk-redux/tsconfig.module.json

handler.run-ethereum-contracts-script.yml

A handler workflow which can execute any arbitrary script in our ethereum-contracts/scripts folder.

Expand

Satellite Files:

• packages/ethereum-contracts/ops-scripts/*.js

handler.update-evm-contracts-docs.yml

A handler workflow which uses docgen to generate docs for our solidity contracts and handles deployment to our documentation site.

handler.verify-contracts.yml

A handler workflow which is called to verify contracts on etherscan.

Expand

Satellite Files:

• packages/ethereum-contracts/ops-scripts/info-print-contract-addresses.js
• packages/ethereum-contracts/tasks/etherscan-verify-framework.sh

handler.xkcd.yml

A handler workflow which creates a comment upon a PR being merged to dev which gets the corresponding xkcd comic based on the merged PR number.

Further reading:

• https://docs.github.com/en/actions (use as reference)
• https://forum.openzeppelin.com/t/a-case-for-secure-web3-operations/32141