Agent reports if there is insufficient disk space during upgrade, cleans up, and prevents upgrade

[ycombinator]

When Agent tries to perform an upgrade, the local disk may not have sufficient space to a) download the new Agent artifact or b) to extract the downloaded Agent artifact. When this happens, Agent should:

• reports this situation to the user (by reporting it to Fleet for Fleet-managed Agents),
• cleans up any partially downloaded or extracted files, and
• prevents the upgrade from proceeding further

Acceptance critieria includes an E2E test proving the above steps are being taken.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[nimarezainia]

@cmacknz Is this complicated to achieve? I think everything is there we just need the message sent when this situation arises.
I believe the download clean up is also happening today.

cc: @ebeahan

[cmacknz]

It should not be complicated no. We will need to make sure we have a cross platform way to detect disk space on the correct drive (and permissions to do so) which will take some experimenting, but the upgrade status reporting to mark the upgrade as failed already exists.

[michalpristas]

my take is that we should still treat runtime failures as primary source of truth and doing some pre-emptive checks (optionally in one take or 2 PRs) based on disk size and content-length.
there are few issues with pre-checks. first free disk space can change, we can have process poluting disk space that would make upgrade fail. we can check content-length if repository reports it and fail if we don't have sufficient space for download, but we actually don't know what is the size of archive + extracted agent upfront.
we need to figure out on which volume target path is located and support detection of free space across different OSes and filesystems (or use some 3rd party library). Statfs used in go library is using statfs syscall that was deprecated by LSB

[kaanyalti]

my take is that we should still treat runtime failures as primary source of truth and doing some pre-emptive checks (optionally in one take or 2 PRs) based on disk size and content-length. there are few issues with pre-checks. first free disk space can change, we can have process poluting disk space that would make upgrade fail. we can check content-length if repository reports it and fail if we don't have sufficient space for download, but we actually don't know what is the size of archive + extracted agent upfront. we need to figure out on which volume target path is located and support detection of free space across different OSes and filesystems (or use some 3rd party library). Statfs used in go library is using statfs syscall that was deprecated by LSB

I agree with @michal’s comment and wanted to share some thoughts and clarification questions.

There are a few challenges with pre-checks:
• Disk space can change at any moment, so even if we check it beforehand, a concurrent process could consume space before or during the upgrade, still resulting in failure.
• Content-Length may not always be available. The downloader code already handles this case by falling back to runtime errors.
• We don’t know the final size on disk after download and extraction. So even with a known Content-Length, we’d still be guessing unless we can accurately estimate.

Disk space calculation across OSes

On Unix systems, I explored using Statfs golang.org/x/sys/unix. While it does indicate which volume a path is on, it uses the statfs syscall, which is deprecated by LSB https://cs.opensource.google/go/go/+/refs/tags/go1.24.5:src/syscall/zsyscall_linux_amd64.go;l=1350

For a basic test, I wrote this script: https://gist.github.com/kaanyalti/82877273e6b76639a120a4f0fa1e3b96. The results were insightful:

=== Report: Disk full write comparison (reserved=0%) ===
Block size: 4096 bytes
User                 ExpectedBlocks  BlocksWritten  DiffBlocks  BytesExpected  BytesWritten  DiffBytes
-------------------- -------------- ------------- ---------- -------------- ------------- ----------
root                          22954         22442        512       94019584      91922432    2097152
unprivileged-user             22442         22441          1       91922432      91918336       4096

=== Report: Disk full write comparison (reserved=10%) ===
Block size: 4096 bytes
User                 ExpectedBlocks  BlocksWritten  DiffBlocks  BytesExpected  BytesWritten  DiffBytes
-------------------- -------------- ------------- ---------- -------------- ------------- ----------
root                          22954         22441        513       94019584      91918336    2101248
unprivileged-user             19882         19881          1       81436672      81432576       4096

=== Final Comparison Report ===
Test Case           User                 ExpectedBlocks  BlocksWritten  DiffBlocks  BytesExpected  BytesWritten  DiffBytes
------------------- -------------------- -------------- ------------- ---------- -------------- ------------- ----------
reserved=0%         root                          22954         22442        512       94019584      91922432    2097152
reserved=0%         unprivileged-user             22442         22441          1       91922432      91918336       4096
reserved=10%        root                          22954         22441        513       94019584      91918336    2101248
reserved=10%        unprivileged-user             19882         19881          1       81436672      81432576       4096

When no space is reserved for root (reserved=0%), both root and non-root users can access Bavail. When space is reserved (reserved=10%), root can access more than Bavail, but still less than Bfree. So, it’s unclear how we can accurately determine available space for the root user, other than by making a conservative assumption.

I haven’t tested this on Windows yet, but the API there seems more straightforward, it returns the available space for the calling user directly.

Clarification question:

Looking at the downloader logic, the agent already appears to handle disk space issues at runtime by checking errors returned from io.Copy. I haven’t dug into the extraction step yet, but if it’s handled similarly, is the goal of this issue to improve the user-facing error message rather than introduce full-blown pre-checks?

In my opinion, as @michalpristas also suggested, the agent should rely on runtime failures.

cc: @ycombinator @cmacknz @nimarezainia @ebeahan

[cmacknz]

reports this situation to the user (by reporting it to Fleet for Fleet-managed Agents),

cleans up any partially downloaded or extracted files, and

prevents the upgrade from proceeding further

The steps in the description don't require a pre-check and would still be satisfied if we attempted the upgrade and set the upgrade state to failed with a clear and standardized "out of disk space" message and tidied up after the failure as we should already be doing.

We already have an integration that can report disk space across OSes (system), if we wanted to go a bit further we could turn it on as part of self-monitoring and show a disk full indicator in agent details along side agent cpu + memory for the volume agent is installed on. This would be a separate scope of work.

[cmacknz]

There are three points where agent can fail due to disk space errors I think, not just two:

1. When downloading the compressed archive
2. When extracting the compressed archive
3. When starting the agent for the first time at which point it will use additional disk by writing to its log files and local state.

The third one would probably result in an upgrade rollback and would be harder to get a standardized error from.

[michalpristas]

the third one is also a use case not specific to upgrade, we could think about reporting out of disk as an error state to fleet for visibility (degraded + message)

[cmacknz]

I think the first two cases will be by far the most common where you need a few hundred MB of space for the download and ~1.5 GB to extract on Linux. The third failure is what you'd hit if you had a couple of megabytes or less of free disk space.

What I'd suggest is focusing on the first two in #5235 (comment) and reporting via upgrade details on failure and then we fork the 3rd case out into a separate problem. That this third case isn't specific to upgrade is a great point.

[kaanyalti]

I ran a quick test to see how the agent behaves when it tries to download the agent artifacts while upgrading. Reading the description of this issue, I was under the impression that the agent is not surfacing any disk space error in any way when it runs into space problems. This contradicts what I see in the code, specifically in the http downloader code. As I had suspected, the agent indeed reports insufficient disk space. However, after reporting "no space left on device", the agent keeps retrying and eventually the context times out, at which point the disk space error gets masked by the context deadline error. As a result the disk space error does not show up on fleet ui. It is possible to see the insufficient disk space error in the status report though as seen below

┌─ fleet
│  └─ status: (HEALTHY) Connected
├─ elastic-agent
│  └─ status: (HEALTHY) Running
└─ upgrade_details
   ├─ target_version: 9.0.4
   ├─ state: UPG_FAILED
   ├─ action_id: dad9e942-1812-4fe3-8c57-a29e87a094a8
   └─ metadata
      ├─ failed_state: UPG_DOWNLOADING
      ├─ error_msg: failed download of agent binary: context deadline exceeded
      ├─ retry_until: -19m6.933408225s
      └─ retry_error_msg: unable to download package: package '/opt/Elastic/Agent/data/elastic-agent-8.10.0-SNAPSHOT-58886e/downloads/elastic-agent-9.0.4-linux-arm64.tar.gz' not found: open /opt/Elastic/Agent/data/el
astic-agent-8.10.0-SNAPSHOT-58886e/downloads/elastic-agent-9.0.4-linux-arm64.tar.gz: no such file or directory
         copying fetched package failed: write /opt/Elastic/Agent/data/elastic-agent-8.10.0-SNAPSHOT-58886e/downloads/elastic-agent-9.0.4-linux-arm64.tar.gz: no space left on device

I don't see a reason why the agent should keep trying if there isn't enough space on disk. Available disk space can change as the agent is working on the upgrade download; however, aborting the upgrade with a failed status would be simpler. The other option would be to let the agent retry downloading and remember the disk space error so that it does not get covered up by the context deadline error.

@cmacknz @michalpristas