No space left on device blocks Provider upgrades

[t0yv0]

Some recent changes in Go SDK generation pushed the builds over the limit of disk space.

• Apply larger runners to all workflows pulumi-aws-native#1083
• Upgrade terraform-provider-aws to v5.15.0 pulumi-aws#2792
• Upgrade terraform-provider-azurerm to v3.72.0 pulumi-azure#1350
• Dependencies: pulumi 3.81.0 pulumi-kubernetes#2556
• Automated Pulumi/Pulumi upgrade pulumi-google-native#925

build_sdk (go)](https://github.com/pulumi/pulumi-azure/actions/runs/6116569557/job/16603257286)
Unhandled exception. System.IO.IOException: No space left on device : '/home/runner/runners/2.308.0/_diag/Worker_20230908-024953-utc.log'
   at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan`1 buffer, Int64 fileOffset)
   at System.IO.Strategies.BufferedFileStreamStrategy.FlushWrite()
   at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
   at System.Diagnostics.TextWriterTraceListener.Flush()
   at System.Diagnostics.TraceSource.Flush()
   at GitHub.Runner.Common.TraceManager.Dispose(Boolean disposing)
   at GitHub.Runner.Common.TraceManager.Dispose()
   at GitHub.Runner.Common.HostContext.Dispose(Boolean disposing)
   at GitHub.Runner.Common.HostContext.Dispose()
   at GitHub.Runner.Worker.Program.Main(String[] args)
System.IO.IOException: No space left on device : '/home/runner/runners/2.308.0/_diag/Worker_20230908-024953-utc.log'
   at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan`1 buffer, Int64 fileOffset)
   at System.IO.Strategies.BufferedFileStreamStrategy.FlushWrite()
   at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
   at System.Diagnostics.TextWriterTraceListener.Flush()
   at GitHub.Runner.Common.HostTraceListener.WriteHeader(String source, TraceEventType eventType, Int32 id)
   at GitHub.Runner.Common.HostTraceListener.TraceEvent(TraceEventCache eventCache, String source, TraceEventType eventType, Int32 id, String message)
   at System.Diagnostics.TraceSource.TraceEvent(TraceEventType eventType, Int32 id, String message)
   at GitHub.Runner.Worker.Worker.RunAsync(String pipeIn, String pipeOut)
   at GitHub.Runner.Worker.Program.MainAsync(IHostContext context, String[] args)
System.IO.IOException: No space left on device : '/home/runner/runners/2.308.0/_diag/Worker_20230908-024953-utc.log'
   at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan`1 buffer, Int64 fileOffset)
   at System.IO.Strategies.BufferedFileStreamStrategy.FlushWrite()
   at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
   at System.Diagnostics.TextWriterTraceListener.Flush()
   at GitHub.Runner.Common.HostTraceListener.WriteHeader(String source, TraceEventType eventType, Int32 id)
   at GitHub.Runner.Common.HostTraceListener.TraceEvent(TraceEventCache eventCache, String source, TraceEventType eventType, Int32 id, String message)
   at System.Diagnostics.TraceSource.TraceEvent(TraceEventType eventType, Int32 id, String message)
   at GitHub.Runner.Common.Tracing.Error(Exception exception)
   at GitHub.Runner.Worker.Program.MainAsync(IHostContext context, String[] args)

I've not investigated deeply but this can be also related to Go build dependency caching. If changes in dependencies invalidate the cache but we don't track cache key accurately, it can download the previous cache, and then download new packages again anyway during go build, this can psh things over the line. Workaround to try here is tinkering with cache keys to force a miss.

Or it can be excessively chatty logging we might need to compress here.

[iwahbe]

pulumi/pulumi-aws@0301de4 fixed the build error. This is a bandaid more then a cure, but it will get us going again.

[t0yv0]

Yeah this is what @aq17 and I landed on trying based on @thomas11 idea earlier in the day. More power.

[t0yv0]

Unfortunately K8S repo now fails in test(go) target with the same OOD. This is in on the way to my P1s fixes so I'd like to take this and chase it down a bit deeper.

[t0yv0]

We've leaned heavily into scheduling workloads on the pulumi-ubuntu-8core runner. This solution seems ok for now but may cause problems if the custom runner is out of capacity. In that case recommendation is to use the GitHub runners with more disk space. I was not able to full root cause for lack of time but I was able to measure K8s disk draw in the Go test job:

free: 231G
$ install stuff
free: 226G (-5G)
$ go test -run COMPILEONLY
free: 218G (-8G)
$ go test
free: 213G (-5G)

This runs out of 14G available on stock runners.

There are multiple reasons Go is very resource hungry here but I don't have exact data. 1. Azure SDK and 2 AWS SDKs and GCP SDKs are pulled into the compilation unit via program test pulumi/pkg spurious dependencies on Pulumi state backends; 2. when tests are run, more disk space is used by ProgramTest creating project copies; there might be some cleanup opportunity that's being missed.

[t0yv0]

I'll close for now as I'm not sure it affects us anymore atm with the workarounds in place.

[pulumi-bot]

Cannot close issue:

• does not have required labels: resolution/

Please fix these problems and try again.

[iwahbe]

It's worth noting that pulumi-aws fails when building the complete go SDK, before it reaches go integration tests.

[t0yv0]

Yes most providers just fail to build but K8S also fails to test. In both cases the compilation burden of either SDK or the tests with all the transitive dependencies is what I think sinks the runner.