Refactor e2e tests with wiremock scenarios

[thomashoneyman]

This adds a wiremock state machine in config.nix so that we can transition package states in mock environments. Specifically, we can do something like publish effect-4.0.0, at which point its tarball will be available in storage, the available-versions returned by Pursuit include the version, and so on.

The reason to do this is so we can try workflows like: publish effect-4.0.0, then publish console-6.0.0 (which depends on effect), and then try to unpublish effect-4.0.0 (should not be possible, because console depends on it).

I've implemented this, and I've also added a number of new e2e tests. Specifically:

• each endpoint gets its own test file focused on testing the primary use of that endpoint, success or error
• there is a GitHubIssue test for testing workflows triggered by github issues
• there is a Workflows test which tries edge cases like the publish-publish-unpublish above, or which checks on job priority like publishing effect, then prelude, and making sure prelude finishes before all the matrix jobs do.

Basic idea is to merge this into the concurrent-jobs branch to be our test suite as we continue iterating on things.

[thomashoneyman]

The server logs have some messages about not being able to write to scratch/.cache which I'll look into tomorrow, but otherwise this seems like a pretty good set of tests for core registry functionality we can use as we iterate — sorry it turned out a bit bigger than expected 😬

The one bummer here is that we now reset state between test runs, which means modifying the database, checked out repos, and so on. The upside is we have a clean state on each run and can truly inspect the results of each test. The downside is that you can no longer just start test-env in a terminal and do spago run in the other. Instead, you need to source the test-env.sh produced by test-env first, then run spago run.

Small additional hassle, but I'm not sure of a better way to get around it...if you're on linux you can just do nix build .#checks.x86_64-linux.integration, but that's not available on macOS.

E2E Tests » Endpoints » Jobs » Status endpoint
  ✓︎ can reach the status endpoint
E2E Tests » Endpoints » Jobs » Jobs list
  ✓︎ lists jobs and respects include_completed filter
E2E Tests » Endpoints » Jobs » Jobs API error handling
  ✓︎ returns HTTP 404 for non-existent job ID
E2E Tests » Endpoints » Publish » Publish workflow
  ✓︎ can publish effect@4.0.0 and filter logs
E2E Tests » Endpoints » Publish » Publish state machine
  ✓︎ returns same jobId for duplicate publish requests
E2E Tests » Endpoints » Unpublish » Publish-Unpublish workflow
  ✓︎ can publish effect@4.0.0 then unpublish it
E2E Tests » Endpoints » Transfer » Transfer workflow
  ✓︎ can transfer effect to a new location
E2E Tests » Workflows » GitHubIssue » GitHubIssue end-to-end
  ✓︎ handles publish via GitHub issue, posts comments, and closes issue on success
  ✓︎ posts failure comment and leaves issue open when job fails
  ✓︎ calls Teams API to verify trustee membership for authenticated operation
  ✓︎ posts error comment when issue body contains invalid JSON
E2E Tests » Workflows » Multi-operation » Concurrent git operations
  ✓︎ git repos remain clean after all matrix jobs complete
E2E Tests » Workflows » Multi-operation » Dependency and unpublish interactions
  ✓︎ publishing a package fails when its dependency was unpublished
  ✓︎ unpublishing a package fails when dependents exist in manifest index
E2E Tests » Workflows » Multi-operation » Job priority
  ✓︎ second publish job completes before first package's matrix jobs finish

[thomashoneyman]

This is potentially an artifact of WIP code? But the server always starts off with matrix jobs to build for existing published packages.

[f-f]

Instead, you need to source the test-env.sh produced by test-env first, then run spago run.

that's fine and possibly better if we can bundle that in a single invocation.

how does one terminate wiremock? is it now started by the tests, or how does that work?

[thomashoneyman]

how does one terminate wiremock? is it now started by the tests, or how does that work?

It's the same as before — .#test-env starts process-compose with the wiremock servers, and then you can Ctrl+C or process-compose down to shut them down.

[thomashoneyman]

The new way to run the env is:

# to start the wiremock servers, git mocks, etc.
nix run .#test-env -- --tui

# to execute the tests
spago-test-e2e

They're intentionally separate because this lets you iterate on tests and keep running spago-test-e2e without having to redo the nix build for the env. Alternately, on Linux, if you just want to do it in one command you can run the Nix integration check.