CI: deploy script goes off the rails if external DNS doesn't start

[jgallagher]

This test failed on a CI run on 5198b40 (pre-PR):

https://github.com/oxidecomputer/omicron/runs/31089252234

Log showing the specific test failure:

https://buildomat.eng.oxide.computer/wg/0/details/01J9C20P89V4YQSG2E0ARCWCHT/EYt2W7eh7IIwcossCxtFwwpEOJA28LwZLLIegY835dykgGNr/01J9C216DWS3S4ZRPN1HW8FD0J

Excerpt from the log showing the failure:

1071	2024-10-04T16:21:16.536Z	+ curl -sSf /v1/ping --resolve '' --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
1072	2024-10-04T16:21:16.559Z	curl: (49) Couldn't parse CURLOPT_RESOLVE entry ''
1073	2024-10-04T16:21:16.559Z	+ [[ 6 -gt 60 ]]

[iliana]

There's definitely a bug in the script here:

2024-10-04 16:20:54.292981389 UTC: attempting to log into API
2024-10-04 16:21:09.326121866 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local)
Error: logging in

Caused by:
    timed out after 609.344661693s

Stack backtrace:
    [...]
+ eval ''
+ export OXIDE_HOST OXIDE_TOKEN

But the real problem is that the Nexus zone came up but didn't respond to a login request within 10 minutes. Unfortunately the panic backtrace is useless, but given the error message we can start with looking at external DNS and Nexus logs. And as it turns out, neither external DNS zone has started:

https://buildomat.eng.oxide.computer/wg/0/artefact/01J9C20P89V4YQSG2E0ARCWCHT/EYt2W7eh7IIwcossCxtFwwpEOJA28LwZLLIegY835dykgGNr/01J9C216DWS3S4ZRPN1HW8FD0J/01J9C5FJR6X9937YQ74Q2006NZ/oxide-external_dns:default.log.0?format=x-bunyan
https://buildomat.eng.oxide.computer/wg/0/artefact/01J9C20P89V4YQSG2E0ARCWCHT/EYt2W7eh7IIwcossCxtFwwpEOJA28LwZLLIegY835dykgGNr/01J9C216DWS3S4ZRPN1HW8FD0J/01J9C5HN0EXBZ9A47JMJCB6WZK/oxide-external_dns:default.log.0?format=x-bunyan

[ Oct  4 16:11:21 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/dns-server/bin/dns-server --config-file /var/svc/manifest/site/external_dns/config.toml --http-address [fd00:1122:3344:101::9]:5353 --dns-address 10.151.1.165:53 &"). ]
[ Oct  4 16:11:21 Method "start" exited with status 0. ]
Config { log: File { level: Info, path: "/dev/stdout", if_exists: Append }, dropshot: ConfigDropshot { bind_address: [fd00:1122:3344:101::9]:5353, request_body_max_bytes: 104857600, default_handler_task_mode: Detached, log_headers: [] }, storage: Config { storage_path: "/data/dns", keep_old_generations: 3 } }
note: configured to log to "/dev/stdout"
dns-server: config
    config = Config { log: File { level: Info, path: "/dev/stdout", if_exists: Append }, dropshot: ConfigDropshot { bind_address: [fd00:1122:3344:101::9]:5353, request_body_max_bytes: 104857600, default_handler_task_mode: Detached, log_headers: [] }, storage: Config { storage_path: "/data/dns", keep_old_generations: 3 } }
    dns_config = Config { bind_address: 10.151.1.165:53 }
dns-server (store): opening sled database
    path = /data/dns
dns-server (store): pruning trees for generations newer than 0
dns-server (store): pruning trees for generations older than 0
    keep = 3
Error: starting DNS server
Caused by:
0: DNS server start: UDP bind to 10.151.1.165:53
1: Cannot assign requested address (os error 126)
[ Oct  4 16:11:21 Stopping because all processes in service exited. ]
[ Oct  4 16:11:21 Executing stop method (:kill). ]
[ Oct  4 16:11:21 Restarting too quickly, changing state to maintenance. ]

[iliana]

I think this might be a problem in the underlying commit and not a flake since the branch name is called remove-omicron-zone-config-ip, but I will keep this issue open to make the script fail in a more obvious way when external DNS is clearly unavailable.

[jgallagher]

I think this might be a problem in the underlying commit and not a flake since the branch name is called remove-omicron-zone-config-ip

Definitely! Thanks for finding the underlying cause.

[andrewjstone]

I believe I just hit this same thing.

https://buildomat.eng.oxide.computer/wg/0/details/01JDJF154WD63YW4HBBV3FG28W/G2BygwaJa4V4G4Lra5OPyY8HobQV0RuAFMyeSrPT17vE8oDb/01JDJF1HBBHF9V56HJETXNYG8G

[iliana]

Different cause but same symptom, where the bootstrap binary's failing exit code isn't handled.

bootstrap prints out environment variables that we need, so we run it like this:

omicron/.github/buildomat/jobs/deploy.sh

Lines 400 to 403 in e87ac05

	export RUST_BACKTRACE=1
	export E2E_TLS_CERT IPPOOL_START IPPOOL_END
	eval "$(./target/debug/bootstrap)"
	export OXIDE_HOST OXIDE_TOKEN

but this means when it fails catastrophically we just continue carrying on.

This timing out may be a result of your PR:

2024-11-25 20:57:49.620186854 UTC: login succeeded
waiting for nexus to come up...
creating IP pool... 10.151.2.175 - 10.151.2.227
setting up quotas...
ensuring datasets are ready...
Error: timed out after 121.15667209s

[andrewjstone]

Thanks @iliana!

This totally looks like an error on my end. I just discovered I typo'd a new field in dbinit.sql and up1.sql, but not in the code.