Cannot properly set Include and Defines with third-party compilers

[gentooise]

Type: LanguageService

Describe the bug

• OS and Version: Windows 10 (Windows_NT x64 10.0.17763)
• VS Code Version: 1.38.1
• C/C++ Extension Version: 0.25.1
• It does not seem possible to use compileCommands plus other global includes/defines taken from includePath and defines properties respectively.

I'm using a third-party compiler (ARM) which has its own includes and defines, but I also need to use project-related include/defines taken from compile_commands.json (generated via CMake).
I'm trying to set compiler-specific includes/defines manually, and then use compileCommands property for project-specific includes/defines.

This is a minimal version of my c_cpp_properties.json:

{
    "configurations": [
        {
            "name": "ARM",
            "compilerPath": "",
            "cStandard": "c99",
            "cppStandard": "c++03",
            "intelliSenseMode": "gcc-x86",
            "compileCommands": "${workspaceFolder:Project}/.vscode/compile_commands.json",
            "defines": [
                "__ARM__"
            ],
            "includePath": [
                "\"C:\\Program Files (x86)\\ARM_Compiler_5.06u4\\include\""
            ]
        }
    ],
    "version": 4
}

Also I would like to set something like "intelliSenseMode": null, because neither MSVC, GCC nor CLang are correct choices for my case. Why does IntelliSense need this info?

This is the full Log Diagnostics output (except sensitive paths):

-------- Diagnostics - 18/9/2019, 17:30:29
Version: 0.25.1
Current Configuration:
{
    "name": "ARM",
    "compilerPath": "",
    "cStandard": "c99",
    "cppStandard": "c++03",
    "intelliSenseMode": "gcc-x86",
    "compileCommands": "d:\\path/.vscode/compile_commands.json",
    "defines": [
        "__ARM__"
    ],
    "includePath": [
        "\"C:\\Program Files (x86)\\ARM_Compiler_5.06u4\\include\""
    ],
    "compilerArgs": [],
    "browse": {
        "path": [
            "\"C:\\Program Files (x86)\\ARM_Compiler_5.06u4\\include\"",
            "${workspaceFolder}"
        ],
        "limitSymbolsToIncludedHeaders": true
    }
}
Translation Unit Mappings:
[ D:\path\myfile.cpp ]:
    D:\PATH\MYFILE.CPP
Translation Unit Configurations:
[ D:\path\myfile.cpp ]:
    Process ID: 3332
    Memory Usage: 21 MB
    Includes:
        D:\PATH\INCLUDE1
        D:\PATH\INCLUDE2
        D:\...
        ...
    Defines:
        __ARM__
    Standard Version: c++03
    IntelliSense Mode: gcc-x64
    Other Flags:
        --g++
        --gnu_version=80100
Total Memory Usage: 21 MB

This is a screenshot to show the problems encountered:

I see 3 main problems:

• Include paths: the include path list (inside Log Diagnostics) does not contain the includePath specified in c_cpp_properties.json, but only the paths found in compile_commands.json; Since my includePath is not present, the compiler headers are not recognized (e.g. string.h is not found as you can see from the screen);
• Defines: viceversa, the defines list contains only the ones specified manually into c_cpp_properties.json but does not contain the ones from compile_commands.json; moreover, even if the macro is present in the list (e.g. __ARM__), the editor does not recognizes it and the wrong preprocessor branch is greyed out!
• IntelliSense Mode: it seems that there is no suitable intelliSenseMode for my use case; I put gcc-x86, but CPPTools seems to ignore it anyway (you can see gcc-x64 in the Log Diagnostics); I would also like to avoid the section "Other Flags" which seems to contain gcc-related stuff.

Expected behavior
In the Log Diagnostics I expect to see the union of includes/defines from both c_cpp_properties.json and compile_commands.json, and I expect them to work in the editor.

[CfirTsabari]

hi,
try to change this:
"includePath": [
""C:\Program Files (x86)\ARM_Compiler_5.06u4\include""
]
to this:
"includePath": [
""C:\Program Files (x86)\ARM_Compiler_5.06u4\include\**""
]

[sean-mcmanus]

@CfirTsabari Using "**" with system includes is not recommended because the ordering may not be correct. If subfolders are required, it's better to just list them in the correct order.

@gentooise Our compiler only supports emulation of cl.exe, gcc, and clang, so it needs to be set to one of these modes.

It looks like our C/C++: Log Diagnostics command has a bug in the compile_commands.json scenario where it is showing the c_cpp_properties.json instead of the defines obtained from the compile_commands.json -- I could file a new bug for that: #4270 .

Our compile_commands.json and configurationProvider were designed to ignore the includePath and defines in c_cpp_properties.json, unless a source file is encountered that doesn't have configuration info. One workaround is to set your compilerPath to a .bat (Windows) or .sh (Linux/Mac) script that outputs the system includes and defines you want, such as

@ECHO OFF
echo #include "..." search starts here:
echo #include ^<...^> search starts here:
echo End of search list.
echo #define MYDEF1=1

Let us know if that workaround doesn't work for you. We have a fix pending for the lack of support for "${workspaceFolder}" in the compilerPath and compilerArgs: #3440 .

[sean-mcmanus]

Actually, it looks like the arm compiler should be "the same" as gcc. Can you check what error occurs when you specific that compiler as the compilerPath? Is our mechanism for querying the compiler for system includes and defines failing? enabling debug logging might give better failure info.

[sean-mcmanus]

Is the ARM compiler you're using publicly available for us to try out?

[sean-mcmanus]

Also, we don't have the proper flags/settings enabled to give 100% correct results for ARM -- specifically we need to add intelliSenseMode's for gcc-arm, gcc-arm64, clang-arm, clang-arm64, windows-arm, windows-arm64 support. I should file a new issue to track that: #4271

[gentooise]

It looks like our C/C++: Log Diagnostics command has a bug in the compile_commands.json scenario where it is showing the c_cpp_properties.json instead of the defines obtained from the compile_commands.json -- I could file a new bug for that: #4270 .

Ok, is it only an aesthetic problem then?

Our compile_commands.json and configurationProvider were designed to ignore the includePath and defines in c_cpp_properties.json, unless a source file is encountered that doesn't have configuration info. One workaround is to set your compilerPath to a .bat (Windows) or .sh (Linux/Mac) script that outputs the system includes and defines you want, such as

@ECHO OFF
echo #include "..." search starts here:
echo #include ^<...^> search starts here:
echo End of search list.
echo #define MYDEF1=1

Let us know if that workaround doesn't work for you. We have a fix pending for the lack of support for "${workspaceFolder}" in the compilerPath and compilerArgs: #3440 .

I understand. I will try ASAP and let you know, but I do not understand some details of the expected output format (search starts here:, ^<...^>?). Could you directly give me an output example for the script?

Actually, it looks like the arm compiler should be "the same" as gcc. Can you check what error occurs when you specific that compiler as the compilerPath? Is our mechanism for querying the compiler for system includes and defines failing? enabling debug logging might give better failure info.

Unfortunately not, ARM Compiler is not the same as GCC, it is a commercial compiler that has its own includes, macros, and command line arguments. I can put it in compilerPath anyway and let you know ASAP, but I already know how this compiler works:

• it has no specific arg to get built-in include paths (they are fixed and documented here);
• it only has an argument to get builtin macros (--list_macros), but the output format does not seem equivalent to the gcc one.

My point is that it could be useful to have a generic way to handle compilers that are not specifically supported: your hint about a script file could be one way to do it. Another way could be to have the possibility to (manually) put other includes/defines in addition to the ones found in compile_commands.json.

Is the ARM compiler you're using publicly available for us to try out?

Unfortunately not, it can be only used by purchasing a license.

Also, we don't have the proper flags/settings enabled to give 100% correct results for ARM -- specifically we need to add intelliSenseMode's for gcc-arm, gcc-arm64, clang-arm, clang-arm64, windows-arm, windows-arm64 support. I should file a new issue to track that: #4271

Please note that the compiler is not gcc-arm or clang-arm. The correct names are armcc and armclang, and they have nothing to do with gcc. I already know it because I recently encountered similar issues for an eclipse plugin (here).

Finally, I have another question: which is the purpose of the configurationProvider? What does it add with respect to compileCommands option? Because in my case I'm using CMake with CMake Tools VSCode extension, and this extension is made to be a configuration provider. But to me this seems redundant, because the information provided by the CMake Tools is always taken from compile_commands.json, which is the unique source of information that comes out from CMake. So, which is the difference between using compileCommands only, configurationProvider only or both of them?

[gentooise]

Actually, it looks like the arm compiler should be "the same" as gcc. Can you check what error occurs when you specific that compiler as the compilerPath? Is our mechanism for querying the compiler for system includes and defines failing? enabling debug logging might give better failure info.

Yes it is failing because it seems to switch to MinGW GCC. This is the output of Log Diagnostics:

-------- Diagnostics - 19/9/2019, 17:40:14
Version: 0.25.1
Current Configuration:
{
    "name": "ARM",
    "compilerPath": "C:\\Program Files (x86)\\ARM_Compiler_5.06u4\\bin\\armcc.exe",
    "cStandard": "c99",
    "cppStandard": "c++03",
    "intelliSenseMode": "gcc-x86",
    "defines": [
        "__ARM__"
    ],
    "includePath": [
        "\"C:\\Program Files (x86)\\ARM_Compiler_5.06u4\\include\""
    ],
    "compileCommands": "d:\\path/.vscode/compile_commands.json",
    "compilerArgs": [],
    "browse": {
        "path": [
            "\"C:\\Program Files (x86)\\ARM_Compiler_5.06u4\\include\"",
            "${workspaceFolder}"
        ],
        "limitSymbolsToIncludedHeaders": true
    }
}
Translation Unit Mappings:
[ D:\path\myfile.cpp ]:
    D:\PATH\MYFILE.CPP
    C:\MINGW\INCLUDE\STRING.H
Translation Unit Configurations:
[ D:\path\myfile.cpp ]:
    Process ID: 14584
    Memory Usage: 33 MB
    Compiler Path: C:\MinGW\bin\gcc.exe
    Includes:
        D:\PATH\INCLUDE1
        D:\PATH\INCLUDE2
        D:\...
        ...
        C:\MINGW\LIB\GCC\MINGW32\8.2.0\INCLUDE\C++
        C:\MINGW\LIB\GCC\MINGW32\8.2.0\INCLUDE\C++\MINGW32
        C:\MINGW\LIB\GCC\MINGW32\8.2.0\INCLUDE\C++\BACKWARD
        C:\MINGW\LIB\GCC\MINGW32\8.2.0\INCLUDE
        C:\MINGW\INCLUDE
        C:\MINGW\LIB\GCC\MINGW32\8.2.0\INCLUDE-FIXED
    Defines:
        __ARM__
    Standard Version: c++03
    IntelliSense Mode: gcc-x64
    Other Flags:
        --g++
        --gnu_version=80200
Total Memory Usage: 33 MB

As you can see, it assumes that the compiler is MinGW GCC, and takes in all its include paths.

Anyway, I enabled debug logging but I don't see any output difference, please tell me where to find debug logs (if any).

[gentooise]

Our compile_commands.json and configurationProvider were designed to ignore the includePath and defines in c_cpp_properties.json, unless a source file is encountered that doesn't have configuration info. One workaround is to set your compilerPath to a .bat (Windows) or .sh (Linux/Mac) script that outputs the system includes and defines you want, such as

@ECHO OFF
echo #include "..." search starts here:
echo #include ^<...^> search starts here:
echo End of search list.
echo #define MYDEF1=1

Let us know if that workaround doesn't work for you. We have a fix pending for the lack of support for "${workspaceFolder}" in the compilerPath and compilerArgs: #3440 .

I tried also this, and I got the same result as above (same Log Diagnostics), except for the line:
"compilerPath": "${workspaceRoot}/.vscode/compile.bat",

where compile.bat contains the following:

@ECHO OFF
echo #include "..." search starts here:
echo #include ^<...^> search starts here:
echo "C:\\Program Files (x86)\\ARM_Compiler_5.06u4\\include"
echo End of search list.
echo #define __ARM__=1

[sean-mcmanus]

Yeah, the defines being incorrect is a bug in the logging output.

The ^< is just to escape the < so the output should not contain the ^ but otherwise should appear the same -- you may need to escape other characters though too.

If you're using CMake and the CMake Tools extension, you would probably get better results if you set the configurationProvider setting and not the compileCommands setting. Is the CMake Tools extension able to get the correct system includes/defines?

The .bat file probably doesn't work because haven't shipped the fix for ${workspaceFolder} not working in compilerPath: #3440 . However, even after using the full path, I'm getting "Compiler include path not found"...so we need to investigate why that isn't working -- it might only work with Linux-style paths currently.