Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion lib/analyze-action-post.js

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion lib/analyze-action.js

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion lib/autobuild-action.js

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion lib/init-action-post.js

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion lib/init-action.js

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion lib/resolve-environment-action.js

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion lib/start-proxy-action-post.js

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion lib/upload-lib.js

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion lib/upload-sarif-action-post.js

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion lib/upload-sarif-action.js

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

15 changes: 10 additions & 5 deletions src/overlay-database-utils.ts
Original file line number Diff line number Diff line change
Expand Up @@ -22,18 +22,23 @@ export const CODEQL_OVERLAY_MINIMUM_VERSION = "2.22.3";

/**
* The maximum (uncompressed) size of the overlay base database that we will
* upload. Actions Cache has an overall capacity of 10 GB, and the Actions Cache
* client library uses zstd compression.
* upload. By default, the Actions Cache has an overall capacity of 10 GB, and
* the Actions Cache client library uses zstd compression.
*
* Ideally we would apply a size limit to the compressed overlay-base database,
* but we cannot do so because compression is handled transparently by the
* Actions Cache client library. Instead we place a limit on the uncompressed
* size of the overlay-base database.
*
* Assuming 2.5:1 compression ratio, the 6 GB limit on uncompressed data would
* translate to a limit of around 2.4 GB after compression.
* Assuming 2.5:1 compression ratio, the 15 GB limit on uncompressed data would
* translate to a limit of around 6 GB after compression. This is a high limit
* compared to the default 10GB Actions Cache capacity, but enforcement of Actions
* Cache quotas is not immediate.
Comment on lines +35 to +36
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the addition of a comment acknowledging that this is a large limit, but I don't think that justifying it based on the speed of the enforcement makes sense.

Suggested change
* compared to the default 10GB Actions Cache capacity, but enforcement of Actions
* Cache quotas is not immediate.
* compared to the default 10GB Actions Cache capacity.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do think the speed of enforcement makes quite a big difference in practice with a limit that is over 50% of the Actions Cache quota. If each compressed overlay base database was +5GB and existing cache entries were immediately evicted from the cache to make room before a new cache entry could be uploaded, then the cache wouldn't be able to fit two overlay base databases and other concurrent CodeQL action runs on PRs would effectively fall back to normal analysis while a new overlay base databases was being uploaded. That will not happen because enforcement of quotas is not immediate.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that the cache quota is not specific to CodeQL and is shared among all Actions caches for a repo. It may be that the speed of enforcement makes a difference for the overlay databases if those are the only caches for a repo and they are this large, but it would still hurt customers if we rapidly take up most of their quota and the rate of cache churn resulting from overlay caches competing with customer caches within the standard 10GB quota would plausibly also evict overlay databases before they can be useful.

*
* TODO: revisit this limit before removing the restriction for overlay analysis
* to the `github` and `dsp-testing` orgs.
*/
const OVERLAY_BASE_DATABASE_MAX_UPLOAD_SIZE_MB = 6000;
const OVERLAY_BASE_DATABASE_MAX_UPLOAD_SIZE_MB = 15000;
const OVERLAY_BASE_DATABASE_MAX_UPLOAD_SIZE_BYTES =
OVERLAY_BASE_DATABASE_MAX_UPLOAD_SIZE_MB * 1_000_000;

Expand Down
Loading