Skip to content
This repository has been archived by the owner on Nov 1, 2023. It is now read-only.

Limit azcopy to just half a gig of RAM #3293

Merged
merged 5 commits into from
Jul 24, 2023

Conversation

DrChat
Copy link
Member

@DrChat DrChat commented Jul 14, 2023

Summary of the Pull Request

By default, azcopy can consume an unbounded amount of RAM.
In conjunction with any other jobs or processes running that also consume a huge amount of RAM, this can cause the system to reach out-of-memory conditions and cause random processes to fail.

However, azcopy will query the system for an environment variable named AZCOPY_BUFFER_GB.
If found, it will limit its internal memory consumption to whatever is specified by that environment variable.
So, we can simply declare that environment variable and limit memory consumption to something reasonable.

From local testing, I have found that half a gigabyte to be a reasonable limit (with the option to decrease the buffer size to something even smaller if necessary).

PR Checklist

  • Applies to work item: #xxx
  • CLA signed. If not, go over here and sign the CLI.
  • Tests added/passed
  • Requires documentation to be updated
  • I've discussed this with core contributors already. If not checked, I'm ready to accept this work might be rejected in favor of a different grand plan. Issue number where discussion took place: #xxx

References

https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-optimize#optimize-memory-use
Azure/azure-storage-azcopy#802
Azure/azure-storage-azcopy#1215

@codecov-commenter
Copy link

codecov-commenter commented Jul 14, 2023

Codecov Report

Merging #3293 (3eec6a2) into main (e68fc45) will decrease coverage by 0.01%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##             main    #3293      +/-   ##
==========================================
- Coverage   31.80%   31.80%   -0.01%     
==========================================
  Files         307      307              
  Lines       37334    37335       +1     
==========================================
  Hits        11874    11874              
- Misses      25460    25461       +1     
Files Changed Coverage Δ
src/agent/onefuzz/src/az_copy.rs 0.00% <0.00%> (ø)

@Porges Porges enabled auto-merge (squash) July 24, 2023 23:25
@Porges Porges merged commit e5d7a60 into microsoft:main Jul 24, 2023
@AdamL-Microsoft AdamL-Microsoft mentioned this pull request Jul 26, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants