Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Chromium: We have exhausted all available IDs in the disk pool #17332

Open
Manouchehri opened this issue Aug 29, 2024 · 12 comments
Open

Chromium: We have exhausted all available IDs in the disk pool #17332

Manouchehri opened this issue Aug 29, 2024 · 12 comments
Labels
bug Something isn't working

Comments

@Manouchehri
Copy link

When building a large application, I got the following error:

7b012e813b8c62de4e17eb1e.trap (maglev-graph-builder.cc.adcdb985_0.trap.tar.zst), 22: com.semmle.util.exception.CatastrophicError: We have exhausted all available IDs in the disk pool /databases/v1/demo/db-cpp/default/idPool

Unsure of what it means, any ideas?

@Manouchehri Manouchehri added the question Further information is requested label Aug 29, 2024
@jketema
Copy link
Contributor

jketema commented Aug 29, 2024

Hi @Manouchehri ,

This means you've hit some internal limit of our database finalization process, due to the size of your application. Which version of CodeQL are you using? We have recently made some improvements in this area, so it would be good to know if you're benefitting from these, or not.

If your application is in fact a large monorepo, the general guidance would be to create a database per application, and not one that includes everything. I cannot tell from your question whether this is the case or not.

@Manouchehri
Copy link
Author

Which version of CodeQL are you using?

2.18.3+202408191541

If your application is in fact a large monorepo, the general guidance would be to create a database per application

It's a single application, Chromium.

@jketema
Copy link
Contributor

jketema commented Aug 29, 2024

2.18.3+202408191541

Thanks. Note that we do not recommend using nightly builds, but that will of course have all the relevant fixes for you.

It's a single application, Chromium.

Can I assume this was just a standard build of a recently checked out copy of Chromium?

@Manouchehri
Copy link
Author

Can I assume this was just a standard build of a recently checked out copy of Chromium?

Correct. No patches or anything made to it.

@jketema
Copy link
Contributor

jketema commented Aug 29, 2024

Thanks. I'll need to discuss this further internally.

My hypothesis is that although we tried to limit the use of the ID pool, the fixes that we've applied for the issues filed by @flowerhack, which I know you've seen, have caused substantially more code to be extracted, pushing us over the ID pool limit.

My expectation is that this is likely going to take some time to fix.

@jketema jketema added bug Something isn't working and removed question Further information is requested labels Aug 29, 2024
@Manouchehri
Copy link
Author

Is there another limiting factor, like reaching the max size of a unit32?

The reason I'm asking, is if it's a just a constant, I might be able to just monkey patch the CodeQL binary myself in the meantime.

@jketema
Copy link
Contributor

jketema commented Aug 29, 2024

The limiting factor here is indeed some 32-bit integer. However, 32-bit integers are pretty fundamental to both the finalization code and the query evaluator, so patching the binary is likely going to be extremely difficult.

@jketema jketema changed the title We have exhausted all available IDs in the disk pool? Chromium: We have exhausted all available IDs in the disk pool? Aug 30, 2024
@jketema jketema changed the title Chromium: We have exhausted all available IDs in the disk pool? Chromium: We have exhausted all available IDs in the disk pool Aug 30, 2024
@jketema
Copy link
Contributor

jketema commented Aug 30, 2024

Would you be able to share the maglev-graph-builder.cc.adcdb985_0.trap.tar.zst file mentioned in the error message. The file should contain a number of text files (that have a .trap extension). The only sensitive information that is potentially in there are the path names of files that were parsed by the tool.

@Manouchehri
Copy link
Author

What folder should that be in? I oddly don't seem to have any *.tar.zst files.

@jketema
Copy link
Contributor

jketema commented Sep 2, 2024

It should be in a subdirectory of /databases/v1/demo/trap/cpp/tarballs, where /databases/v1/demo/ seems to be your database directory.

@jketema
Copy link
Contributor

jketema commented Sep 13, 2024

Hi @Manouchehri,

The latest nightly v2.18.4+202409122320 has a fix that addresses what seemed to be one of the major causes of the ID pool exhaustion, so you might want to see if this fixes the issue you.

@rvermeulen
Copy link
Contributor

Hi @Manouchehri,

Were you able to create a database with a newer CLI version?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants