Surface hook failure logs in CI

[72636c]

There are a lot of systems at play in a CodeDeploy-based deployment. It may involve the application itself, its logs, a load balancer, CodeDeploy, a hook, the hook's logs, etc. In practice, this means that it can take a while for an engineer to first figure out why a deployment failed, and this starts with figuring out where to find further details about the failure.

The CodeDeploy API does not seem to allow a hook to enrich the deployment with any details beyond the status enum in PutLifecycleEventHookExecutionStatus. Some alternatives we could explore include:

• A Buildkite plugin that echos CloudWatch to Buildkite log output as a post-command hook. This may be desirable as we use varying deployment tools (AWS CDK, Gantry, Serverless Framework) and it may be difficult to build equivalent functionality into each tool independently yet consistently.

  However, it may be difficult to implement this without passing any information about the deployment from command to post-command. We could try to expose correlating references (deployment ID?) as environment variables. Otherwise, we can see how clever we can get with inference by introspecting the step definition, build logs and timestamps, plugin configuration options, etc.

• Investigate per-tool extensions to can perform this action as part of deployment.

  It should be feasible to directly extend Gantry in this manner. On the AWS CDK end we may need to create a custom accompanying CLI; their triggers do not seem intended to affect cdk deploy CLI output, and I'm not sure if much is possible with other commands like cdk metadata.