Launch ncov builds using the up-and-coming new nextstrain/base image

[tsibley]

This helps us test the new image internally before making it the default.¹

I launched a quick test with:

./bin/launch-build --exec uname /var/empty

to make sure that builds can still start. Waiting on success as Batch hasn't scheduled the job onto a node yet. Update: it started fine and the job definition uses the expected image.

¹ nextstrain/docker-base#21

[tsibley]

@eharkins @emmahodcroft @MoiraZuber This PR is to test nextstrain/docker-base#21 (comment) a little more broadly before releasing it widely. Is it ok with you if we merge this later today, aiming to see how ncov builds go on Monday and Tuesday? I expect they'll run fine, but it may introduce some snafus in the build.

If y'all don't like the idea of the potential disruption/being the test subjects, then what I'd suggest instead is running each launched ncov build in parallel: one as things are today, and one with the new base image (configured not to update staging/production). Then we can hopefully spot problems without impacting your work. This sounds better and better as I type it out. Should we just do this instead? It's slightly more work to arrange, but probably not too much. Y'all have the best sense of how costly disruption is.

[emmahodcroft]

Sorry Tom, it's late here and my brain is already at half-mast. I think running them both in parallel sounds good as presumably if something goes wrong we'd still have one to work with without much delay. But I'm not really parsing how we do that, and how we ensure that they both don't end up trying to write the same thing at the write time/that we can access the files for the one that works... if you or Eli wouldn't mind outlining this before you log off today, I'd appreciate it, and we can give it a go on Monday!

[tsibley]

@emmahodcroft No worries! Thanks for chiming in.

The second trial build in parallel to the primary build would be an automatic thing kicked off anytime a primary build was, and as part of that would be configured automatically to put results elsewhere. I think this makes sense, but admit I don't have a great sense of your process anymore.

Here's my proposed implementation: 3d8ab2f

It's no big deal if enabling this slips until next week. Maybe for the best anyway. @eharkins Let's coordinate when you have a chance? No rush.

[eharkins]

This plan sounds good, 3d8ab2f makes sense to me at a glance. I will be running my next build on Thursday and am available to coordinate and problem solve if necessary then @tsibley. Sorry I didn't see this sooner!

[tsibley]

@eharkins Great, thanks! I'm going to merge this now and test launching some builds (which I'll immediately cancel). Scratch that, I'll wait until no live builds are running.