canary-cloud: Use Redpanda Cloud

[def-]

Fixes: https://github.com/MaterializeInc/cloud/issues/9721

Nightly run: https://buildkite.com/materialize/nightly/builds/9274

Checklist

• This PR has adequate test coverage / QA involvement has been duly considered. (trigger-ci for additional test/nightly runs)
• This PR has an associated up-to-date design doc, is a design doc (template), or is sufficiently small to not require a design.
• If this PR evolves an existing $T ⇔ Proto$T mapping (possibly in a backwards-incompatible way), then it is tagged with a T-proto label.
• If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label (example).
• If this PR includes major user-facing behavior changes, I have pinged the relevant PM to schedule a changelog post.

[benesch]

This looks fantastic, @def-—thank you!

We can only run one cloud canary test with Redpanda Cloud at a time, right? Should we create a new pipeline ("Expensive nightly"? "External services nightly"?) so that folks using trigger-ci don't run this test when validating a PR?

[def-]

This is already only executed on main and release branches, and also locked by using a concurrency group. So no problem for PRs.

[benesch]

Ah, beautiful! Well this basically LGTM. Are you still working to get it to pass nightlies? Let me know when you're ready for a full review (and how thorough of a review you feel you need).

[def-]

Ah, beautiful! Well this basically LGTM. Are you still working to get it to pass nightlies?

It worked before, I broke it again by some refactoring. Hopefully the new run is green now: https://buildkite.com/materialize/nightly/builds/9274
If not, should only be some minor changes required. So this should be ready for review.
Edit: Succeeded now.

[benesch]

This is fantastic work, Dennis—thank you so much!