Test: Find a solution to running tests that require GitHub secrets from forked PRs

[erezrokah]

Context #1768

We have several tests that require Netlify API access. We enable API access in CI by configuring a GitHub secret

cli/.github/workflows/main.yml

Line 38 in 2110739

NETLIFY_AUTH_TOKEN: ${{ secrets.NETLIFY_AUTH_TOKEN }}

which is a Netlify API token.

The problem is that GitHub secrets are not available for workflows running from forked PRs (for security reasons).
We should figure out a way to still run those tests on such PRs.

One possible solution is to mock API calls for those tests.
Another would be to manually run the tests on the reviewer's machine as a part of the PR review process.

Thoughts

[eduardoboucas]

What do you think of trying https://github.com/imjohnbo/ok-to-test?

[erezrokah]

That flow looks cool 🚀
However I'm not sure how they enforce this:

As someone with write access, comment /ok-to-test sha= on an incoming pull request

I'm not sure but this https://github.com/imjohnbo/ok-to-test/blob/1b4bf926f1c3ed5b945803742a455d6b18529b53/.github/workflows/ok-to-test.yml#L5 should run on any comment?

[eduardoboucas]

However I'm not sure how they enforce this:

I believe this will tell the slash-command-dispatch action that only comments made by someone with write access will be valid (see https://github.com/peter-evans/slash-command-dispatch#action-inputs).

I could set this up in a repo on my personal account and take it for a spin. If we're happy with how it works, we can adopt it. What do you think?

[erezrokah]

I believe this will tell the slash-command-dispatch action that only comments made by someone with write access will be valid (see https://github.com/peter-evans/slash-command-dispatch#action-inputs).

You're correct. That should work.

I could set this up in a repo on my personal account and take it for a spin. If we're happy with how it works, we can adopt it. What do you think?

Sounds good to me. Using a GitHub app at an org level to create elevated tokens would be great too (at the moment we're using Personal Access Tokens from a bot account for each repo that needs it).