Docker driver: ContainerStats with stream=true returns empty PreCPUStats causing incorrect CPU metrics

[ynl]

Nomad version

Nomad v1.9.4-dev

Operating system and Environment details

• OS: Linux

Issue

When using Docker's ContainerStats API with stream=true in the Docker driver, the PreCPUStats field in the returned stats is always zero/empty, causing incorrect CPU usage calculations.

The issue occurs in drivers/docker/stats.go when calling:

statsReader, err := h.dockerClient.ContainerStats(ctx, h.containerID, true)

This results in inaccurate CPU usage metrics and resource utilization reporting, as the previous CPU statistics are required for calculating the correct CPU usage percentages.

Reproduction steps

1. Configure a Nomad client with the Docker driver
2. Run any Docker-based Nomad job
3. Examine the stats collection code path in the Docker driver
4. Observe that when stream=true is used, the PreCPUStats values are consistently zero/empty

Expected Result

The Docker stats API should return valid PreCPUStats data regardless of whether streaming mode is enabled or not, allowing for accurate CPU usage calculations.

Actual Result

When stream=true is used:

• PreCPUStats fields are zero/empty
• CPU usage calculations are incorrect due to missing previous CPU statistics
• Resource utilization reporting is inaccurate

Job file (if appropriate)

This issue is reproducible with any Docker-based job, as it's related to the Docker driver's stats collection mechanism rather than specific job configurations.

Additional Context

Currently, the only workaround is to use stream=false, which returns correct PreCPUStats values but may not be optimal for continuous monitoring. This appears to be related to how Docker's stats API handles the streaming mode, where it omits the previous CPU stats in the streaming response.

The issue impacts:

• CPU usage monitoring accuracy
• Resource utilization reporting
• Potential CPU quota enforcement

[jrasell]

Hi @ynl and thanks for raising this issue. I'll mark this for internal investigation and get this roadmapped.