test: run the development environment steps in CI

[miketheman]

Exercise some of our Makefile, docker-compose.yml and shell script wrapper utilities.

Originally conceived in #10052, simplified with current commands.

Closes #10052

[miketheman]

I don't know if this is a great idea yet.

It seems like a good idea, but it's also another ~12 minutes CI step.
Doesn't have to be required, but also maybe should run post-merge on main? Or some other time? Up for debate.

[di]

Doesn't have to be required, but also maybe should run post-merge on main?

Given that deploys are blocked on status checks passing, this might be a worse time to run it, unless we made it not required, which I'm not sure we'd ever see.

Instead, maybe we can run it regularly (daily?) via cron and alert us if it fails?

[miketheman]

Given that deploys are blocked on status checks passing, this might be a worse time to run it, unless we made it not required, which I'm not sure we'd ever see.

Excellent point - definitely should not be in the critical path of anything, will remove those triggers.

Instead, maybe we can run it regularly (daily?) via cron and alert us if it fails?

I like this better. Have we established a notification pattern elsewhere? I have used Datadog CI before - I don't think we have that set up yet, but once it is, we can set a CI Monitor to tell us things we want to know.

[di]

Have we established a notification pattern elsewhere?

Sort of, we do issue some Sentry messages like this:

warehouse/warehouse/ip_addresses/models.py

Line 89 in fec156f

sentry_sdk.capture_message(f"Attempted to store invalid ip_address: {e}")

I think that works fairly well? But definitely is made easier by the fact that we already have a Sentry integration present there.

[miketheman]

... the fact that we already have a Sentry integration present there.

Yeah, that's during runtime, and bootstrapped the Sentry config already. I think with the schedule changes we can ship this, and figure out notification out of band.