several source files are included in src.zip, but not in the database

[KeuntaeShin]

Version: CodeQL command-line toolchain release 2.7.3.
Target language: cpp
Target source: Linux kernel

While checking the query result finding function call sites, i notice that statements of several files were not included in the result.

After checking the database directory:

1. The src.zip includes the several files

2. In database-create-xxx.log, i can't see traps of several files, build messages ("[build-stdout] CC [M] drivers/xxx.o") are shown though.

3. In database-create-xxx.log, there are error messages:
   java.io.IOException: Brotli stream decoding failed
   ...
   org.brotli.dec.BrotliRuntimeException: Unused space
   ...
   java.io.IOException: Brotli stream decoding failed
   ...
   org.brotli.dec.BrotliRuntimeException: Corrupted padding bits

I feel that some source files are not treated as targets of database. Is there any way to figure out what files are handled for creating database, and are not? And why the files are not targeted?

[MathiasVP]

Hi @KeuntaeShin,

Thanks for the issue. We'll look into this, but first, we need some additional information:

• What operating system are you on?
• What SHA of the Linux kernel are you trying to build?
• What command line are you using to create the database?

Note that there should be a database of torvalds/linux available for you to download via https://lgtm.com/projects/g/torvalds/linux, but we recently had a technical issue which meant that we couldn't provide that database. I expect the database to be available from that link very soon again, though!

Additionally, it would be super helpful if you upload your logs somewhere so that we can inspect them.

[KeuntaeShin]

Hello @MathiasVP,

1. The codeql is running on Linux xxx 5.4.0-92-generic # 103-Ubuntu SMP Fri Nov 26 16:13:00 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

2. The several files are drivers and under development, so it is hard to answer. Some of driver under development are targeted as database, some of not.

3. codeql database create linux_database --language=cpp --command="[a custom build script]"

4. Due to the circumstance mentioned in second answer, it is hard to share the logs.

Please advise me to clear this problem.

Thanks,
Keuntae Shin

[KeuntaeShin]

A clue that I checked:

AST VIEWER in Visual Code shows nothing corresponding to the several files, while others are shown well in there.

Would it relate to conditional compilation? Such as #if, #ifdef, #ifndef, #else, #elif, #endif, and defined?

Thanks,
Keuntae Shin

[KeuntaeShin]

The error messages have been disappeared after crafting cpp/tools/linux64/extractor forcibly using #7557 (comment)

However the several files still are not treated as targets for database.

@MathiasVP , Please feel free to share anything if you have a clue.

[MathiasVP]

Hello @MathiasVP,

1. The codeql is running on Linux xxx 5.4.0-92-generic # 103-Ubuntu SMP Fri Nov 26 16:13:00 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
2. The several files are drivers and under development, so it is hard to answer. Some of driver under development are targeted as database, some of not.
3. codeql database create samsung_galaxy_s22_linux_database --language=cpp --command="[a custom build script]"
4. Due to the circumstance mentioned in second answer, it is hard to share the logs.

Please advise me to clear this problem.

Thanks, Keuntae Shin

Thanks for the additional information. If you feel comfortable sharing the logs with GitHub Support, you can open a support ticket here and upload the logs. They will then get routed to the correct engineering team.

A clue that I checked:

AST VIEWER in Visual Code shows nothing corresponding to the several files, while others are shown well in there.

Would it relate to conditional compilation? Such as #if, #ifdef, #ifndef, #else, #elif, #endif, and defined?

Thanks,
Keuntae Shin

Conditional compilation shouldn't be an issue, no. If the code compiles without CodeQL it should also work when generating a database.

The error messages have been disappeared after crafting cpp/tools/linux64/extractor forcibly using #7557 (comment)

However the several files still are not treated as targets for database.

@MathiasVP , Please feel free to share anything if you have a clue.

I really appreciate your investigation into this! As far as we know, the issue regarding Brotli stream decoding errors in your original post (and in the issue described in #7557) is really just an unfortunate side effect of the real issue, which is that we fail to correctly compile the project. This causes us to emit some partial brotli compressed files, which we then later fail to decompress.

So the main issue is still present: we fail to build parts of the project. Having access to the logs would help us identify why we fail to build the project.

[KeuntaeShin]

@MathiasVP, Finally i found the corner case CodeQL can't handle. Would you inform proper engineers of a below example for fixing it?

Case A - Code

struct a;
typedef int (*func)(struct a);
struct a {
    func f;
};

int main() {
    
}

Case A - Codeql, Failed

Case B - Code

struct a;
typedef int (*func)(struct a *);
struct a {
    func f;
};

int main() {
    
}

Case B - Codeql, Success

Case A&B - Compile

[MathiasVP]

Thank you for this snippet, @KeuntaeShin. I've added these tests to our internal issue, and I'll keep you updated on the progress here :)

[sashabu]

@KeuntaeShin Thanks for the minimal example. We've tracked down the issue and are expecting to have the fix in 2.8.0.

[KeuntaeShin]

@MathiasVP and @sashabu ,

With 2.8.1, the issue i reported has not been observable. Thank you for your efforts.