Run "craft prepare" as part of CI

[rhcarvalho]

Publishing a release in getsentry/publish#629 failed because, after preparing a release, tests failed in the release branch (and have passed for 25+ days in master, since #4040).

Build logs

Copying some logs for posteriority

/usr/bin/docker run --name getsentrycraftlatest_e455e7 --label f88420 --workdir /github/workspace --rm -e CRAFT_LOG_LEVEL -e CRAFT_DRY_RUN -e GIT_COMMITTER_NAME -e GIT_AUTHOR_NAME -e EMAIL -e GITHUB_API_TOKEN -e COCOAPODS_TRUNK_TOKEN -e CRAFT_GCS_TARGET_CREDS_JSON -e CRAFT_GCS_STORE_CREDS_JSON -e CRATES_IO_TOKEN -e DOCKER_USERNAME -e DOCKER_PASSWORD -e TWINE_USERNAME -e TWINE_PASSWORD -e NPM_TOKEN -e GEM_HOST_API_KEY -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY -e NUGET_API_TOKEN -e INPUT_ENTRYPOINT -e INPUT_ARGS -e HOME -e GITHUB_JOB -e GITHUB_REF -e GITHUB_SHA -e GITHUB_REPOSITORY -e GITHUB_REPOSITORY_OWNER -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RETENTION_DAYS -e GITHUB_RUN_ATTEMPT -e GITHUB_ACTOR -e GITHUB_WORKFLOW -e GITHUB_HEAD_REF -e GITHUB_BASE_REF -e GITHUB_EVENT_NAME -e GITHUB_SERVER_URL -e GITHUB_API_URL -e GITHUB_GRAPHQL_URL -e GITHUB_WORKSPACE -e GITHUB_ACTION -e GITHUB_EVENT_PATH -e GITHUB_ACTION_REPOSITORY -e GITHUB_ACTION_REF -e GITHUB_PATH -e GITHUB_ENV -e RUNNER_OS -e RUNNER_NAME -e RUNNER_TOOL_CACHE -e RUNNER_TEMP -e RUNNER_WORKSPACE -e ACTIONS_RUNTIME_URL -e ACTIONS_RUNTIME_TOKEN -e ACTIONS_CACHE_URL -e GITHUB_ACTIONS=true -e CI=true --entrypoint "/bin/bash" -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/home/runner/work/_temp/_github_home":"/github/home" -v "/home/runner/work/_temp/_github_workflow":"/github/workflow" -v "/home/runner/work/_temp/_runner_file_commands":"/github/file_commands" -v "/home/runner/work/publish/publish":"/github/workspace" getsentry/craft:latest -e -c " cd __repo__/.; exec craft publish 6.14.0; "
[debug] craft 0.28.0-dev.0 (e4f9962a309557aa975d559dc0ce497ca8dca1dc)
[debug] No environment file found in the home directory: /github/home/.craft.env
[debug] No environment file found in the configuration directory: /github/workspace/__repo__/.craft.env
[debug] Not initializing Sentry SDK - no valid DSN found in environment or config files
[debug] Configuration file found:  /github/workspace/__repo__/.craft.yml
[debug] Parsing and validating the configuration file...
[debug] "craft" version is compatible with the minimal version from the configuration file.
[debug] Catching Ctrl-C is disabled by default. See https://github.com/getsentry/craft/issues/21
[info] Publishing version: "6.14.0"
[debug] Working directory: /github/workspace/__repo__
[debug] Checking out release branch release/6.14.0
[debug] Revision to publish:  7e0badbd937281b05138f4b0e7ea54461d28f5f5
[debug] Using "github" for status checks
[debug] Using "github" for artifacts
[debug] Fetching repository information...
[debug] GET /repos/getsentry/sentry-javascript - 200 in 457ms
[debug] Repository info received
[debug] Fetching combined revision status...
[debug] Fetching Checks API status...
[debug] Fetching Checks API status...
[debug] GET /repos/getsentry/sentry-javascript/commits/7e0badbd937281b05138f4b0e7ea54461d28f5f5/status - 200 in 182ms
[debug] Combined revision status received.
[debug] GET /repos/getsentry/sentry-javascript/commits/7e0badbd937281b05138f4b0e7ea54461d28f5f5/check-runs - 200 in 325ms
[debug] Revision checks received.
[debug] GET /repos/getsentry/sentry-javascript/commits/7e0badbd937281b05138f4b0e7ea54461d28f5f5/check-suites - 200 in 343ms
[debug] Revision check suites received.
[debug] No config provided for Github status provider, calculating the combined status...
[debug] No legacy checks detected, checking for runs...
[debug] Check runs exist, continuing...
[debug] Commit API status result: success
[debug] Check runs API result: failure
[debug] [[status-provider/github]] Got status "failure" for revision 7e0badbd937281b05138f4b0e7ea54461d28f5f5
Error:  Build(s) for revision 7e0badbd937281b05138f4b0e7ea54461d28f5f5 have not succeeded. Please check the revision's status.
  at pr (/usr/local/bin/craft:623:319056)
  at RH.waitForTheBuildToSucceed (/usr/local/bin/craft:632:1802)
  at async iYt (/usr/local/bin/craft:693:1243)
  at async vUe (/usr/local/bin/craft:693:3075)
  at async Object.AJ [as handler] (/usr/local/bin/craft:696:73)

We could ensure master is ready for a release by automating and testing release preparation more often, either in every PR, or after every push to master or as a recurring "cron job".

[rhcarvalho]

Maybe "craft prepare" is not exactly what we actually need, but rather to apply the version bump script create branch as a release process would (but with a different branch name) and then see the status of the tests, as close as the release process would do, but without publishing anything to NPM/etc.

[lobsterkatie]

apply the version bump script create branch as a release process would (but with a different branch name) and then see the status of the tests, as close as the release process would do, but without publishing anything to NPM/etc.

I suppose there's no harm in doing this, but TBH it feels like overkill to me. The only thing that's different about the tests when run by craft prepare vs the tests when run on a PR or on master is the version bump, and that's not something we should ever be testing against a literal value (a problem which ideally would be caught on code review, but also not something we're likely to be doing a ton of, making this a rare problem to run into).

[lobsterkatie]

(repeating what I said on Slack so the whole convo is centralized)

It's also worth noting that the reason we're testing against a literal version value in the first place is because we're using an inline snapshot test. I'll admit this is a matter of opinion, but I actually really dislike inline snapshots in general, because a) as seen here, you can only test against literal values, which is often not what you want, b) not so much here but often they take up a whole bunch of room and make your test harder to read, and c) relatedly, often a lot of the data in the snapshot is irrelevant and so it makes it much less clear what you’re actually testing. If it were up to me, I’d add this to our eslint rules, in fact - no-inline-snapshot or something.