Response file parsing for GCC and Clang

[Alexei-Barnes]

This fixes #1780

This moves the response file parsing code implemented by @temportalflux out of the MSVC compiler and shares it for all compilers. This enables GCC and Clang to use response files with quoted arguments.

I've added some lib tests to the response file module, so that it can be reasoned about, separately from the compilers using it.

[codecov-commenter]

Codecov Report

Patch coverage: 66.19% and project coverage change: -0.08% ⚠️

Comparison is base (7074753) 29.46% compared to head (3b2f394) 29.38%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1781      +/-   ##
==========================================
- Coverage   29.46%   29.38%   -0.08%     
==========================================
  Files          49       50       +1     
  Lines       17844    17872      +28     
  Branches     8631     8625       -6     
==========================================
- Hits         5257     5252       -5     
- Misses       7347     7356       +9     
- Partials     5240     5264      +24     

Files Changed	Coverage Δ
src/compiler/gcc.rs	53.88% <0.00%> (+0.14%)	⬆️
src/compiler/msvc.rs	44.98% <0.00%> (-0.13%)	⬇️
src/compiler/response_file.rs	70.14% <70.14%> (ø)

... and 12 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[sylvestre]

rustfmt isn't happy, could you please fix it? thanks

[Alexei-Barnes]

Ran cargo fmt and force pushed.

[sylvestre]

i would prefer the move to be done into a single commit
and then, modifications into a second commit

[Alexei-Barnes]

Sure, I'll rewrite the commit into two commits in that order.

[sylvestre]

@Alexei-Barnes ping ? :)

[Alexei-Barnes]

Sorry for the slow action here - I've been pulled into a work project that's eating my free time. I should get back to this soon.

[sylvestre]

are you still going to work on this? thanks

[Alexei-Barnes]

Right, sorry for the mega wait - I'm working on this again now.

[Alexei-Barnes]

OK, this is split into two separate commits now. The first commit just moves the code, with minimal changes necessary to do so. The second change modifies it, adds tests, and uses it in the GCC module. The final code is unchanged from when you last looked at it.

[Alexei-Barnes]

I've now also merged main into this branch, to keep it up to date. No merge conflicts.

[sylvestre]

sorry but it will need to be rebase :(

[robertmaynard]

What is the value of parsing an response file instead of just hashing it? Is this ensures we are invariant to order of flags in the response file?

[glandium]

What is the value of parsing an response file instead of just hashing it? Is this ensures we are invariant to order of flags in the response file?

For the same reasons we don't just hash command line arguments:

• some flags need to be handled in a special way
• some flags need to disable caching because we can't handle them
• we need to distinguish preprocessor flags from other flags so that we can invoke the preprocessor with the right flags
• the input file may be in the response file, and we need to know what it is
• etc.

[robertmaynard]

What is the value of parsing an response file instead of just hashing it? Is this ensures we are invariant to order of flags in the response file?

For the same reasons we don't just hash command line arguments:

• some flags need to be handled in a special way
• some flags need to disable caching because we can't handle them
• we need to distinguish preprocessor flags from other flags so that we can invoke the preprocessor with the right flags
• the input file may be in the response file, and we need to know what it is
• etc.

Makes sense. My feedback would be we should extend ArgData in gcc.rs to have a explicit type for response files.
This would allow compilers that extend gcc such as nvcc to leverage this work even when they have a different command line syntax to represent response files.

[sylvestre]

no activity for a while, closing