Feature Request: Export compile_commands.json

[sslupsky]

A useful enhancement to the arduino cli would be the option to generate a compile_commands.json file. This can be used by Microsoft's Visual Studio Code (vscode) application to configure the its' IntelliSense code completion, parameter info, quick info, and member lists capabilities. I find using arduino-cli and vscode is particularly productive for embedded development.

[sudobash1]

I think this is a cool idea, and I have wanted this feature. I think that it would be at least slightly complicated by how Arduino preprocesses the .ino file. It doesn't just feed it directly to the compiler. You can learn about that here.

Without this preprocessing, IntelliSense, clangd and ccls might think they are looking at invalid C++ (which they would be).

cheers

[matthijskooijman]

I'm also interested in this feature, since my current manual compiler-flags configuration for code completion with clangd is a bit fragile (and does not have a complete list of files in the project, preventing refactoring from working).

As for the .ino file, it will indeed be pretty much impossible to let clangd see the preprocessed version, especially when you're editing it (unless the Arduino IDE talks to clangd and does the preprocessing on the fly, but that's not the subject of this issue). But I think it would actually be fine if all .ino files would end up in the compilation database with just the flags used to compile the preprocessed version. This means that if you have just a single .ino file (or multiple that do not depend on each other without forward declarations), and you do not depend on the autogenerated forward declarations (by defining functions in order and/or adding forward declarations manually), clangd should be able to figure things out most of the time.

One additional thing is that preprocessing adds an #include <Arduino.h> line, but I think this could be emulated by adding -include Arduino.h to the compiler flags as well (this could even have been used for the actual compilation, except that would break compatibility now).

So, outputting a compilation database would simply be a matter of dumping all compilation commands ran, and some special casing for the .ino files, which should be fairly easy to implement.

Where should this file be generated? I suspect that the sketch directory is the obvious place, since that's also where editors / clangd will look for it, right?

One thing to think about is partial compilation: When compilation fails, should a database be written with just the commands that were ran? Probably yes if no database existed yet, but what if a database was already present? Then it might be counterproductive to remove entries for files after the failure, since very likely the flags haven't changed. Maybe an existing database should be loaded, updated in memory and written out at the end of the compilation? If compilation failed, it should be written out as-is, if compilation succeeded, any files in the database that were no longer in the compilation should probably be pruned.

[matthijskooijman]

I did a rough implementation of this feature, see the PR linked above. It's still far from finished, but it might serve to trigger some additional discussion. See the PR for a list of things I think need to be addressed.

[ubidefeo]

thank you @matthijskooijman
I saw the PR come in but it's not my turf, so I'll get the team to check it out on monday :)

enjoy the weekend
u.

[matthijskooijman]

Initial support of this has been merged in #1081. Since that merge, all compilations will generate a compile_commands.json file in the build directory. To generate a complete file without actually compiling everything (i.e. when you still have errors in your code), you can use compile --only-compilation-database.

Some open points (notes copied from my earlier attempt in #944):

• Where to write the file: I now write a compile_commands.json to the sketch directory, since that's where editors would look for it. I later realized that it would probably be good to also write a version to the temp build directory (note: I mean the /tmp/arduino-sketch-XYZ directory, not the build subdirectory of the sketch dir where the compilation result is exported), for any tooling that wants to inspect the copied and preprocessed sketch files.
• Updating sketch paths: Currently, the paths are written as they passed to the compiler, so pointing at the copy of the sketch in the temp build dir. This is probably good for the compile_commands.json to be written to the temp build dir, but the sketch dir version should really point to the original sketch files. This means that the file element in the json should be updated, but maybe also any compiler arguments that resolve to sketch files should have their paths updated. This should probably be applied generically (i.e. search replacing back to the original sketch dir) to all arguments, since we can't be sure what a platform.txt does exactly. Alternatively, updating just the file field might be sufficient and can be done by passing the original filename around (does require significant refactoring, I think).
• Handling .ino files: Currently, only one entry is generated for .cpp file resulting from the combined and preprocessed .ino file(s). This is again fine for the temp build version, but for the sketch dir version, I think this should have a separate entry for each .ino file, with an -include=Arduino.h flag added (as suggested in Feature Request: Export compile_commands.json #849 (comment)). This is imperfect, but probably as close as we can get.
• Making this optional: I believe that writing a compile_commands.json to the temp build dir can always be done, but writing to the sketch directory should almost certainly be opt-in (through commandline or config), to prevent user surprise and because the IDE and arduino-cli seem to try hard to prevent touching the sketch dir now. For arduino-cli, it might make sense to always generate the file in the build subdirectory, along with the compiled program, and provide an option to also, or instead of, generate in the sketch directory itself (where it will be mostly automatically picked up by editors or clangd). For the IDE, some extra care should probably be taken to not generate this file in the sketch directory for unmodified examples, even if the user enabled it in the config.

[Falven]

@matthijskooijman
This has a bug/is missing some commands.
If you are trying to use the compilation database as intellisense configuration as is done in pretty much any IDE, for example, clangd under VSCode, the generated compile_commands.json MUST explicity include the system header include directories otherwise clangd cannot find them. This is because Arduino uses different compilers in non-standard locations, with different system includes per compiler, as such the intellisense engines will not know where to find the system includes.

As an example, look at the compilation database, c_cpp_properties.json, that the Arduino vscode extension generates for it's intellisense configuration. They add on the system include directories.

[matthijskooijman]

I see you opened #1286 for this, let's continue discussion there.

[pokeymud]

I have an issue with this - if I compile using either of these options:

--output-dir
--export-binaries

the associated build files are made and put in the local folder/whichever folder I tell it to build to, but compile_commands.json is not generated with them.

To be clear: If I build using no additional flags like this (in the folder with *.ino):

arduino-cli compile --fqbn arduino:avr:mega

then the build files can be found in the system's temp folder and does include compile_commands.json. However it's a pain to parse my way and copy it to where it needs to be. It would be easier if I could use above options to put it in a definite place.

Here's the full command examples I've tried, but get no compile_commands.json:

arduino-cli compile --export-binaries -b arduino:avr:mega
arduino-cli compile --output-dir %cd%\build -b arduino:avr:mega
arduino-cli compile --output-dir %cd%\build --only-compilation-database -b arduino:avr:mega

Does this need it's own bug report or feature request?

[zfields]

I'm having the same issue as @pokeymud. It's presenting for me in the official Microsoft Arduino VSCode Extension.

[umbynos]

Hi @pokeymud with --build-path you can specify the Path where to save compiled files. Hope this solves your problem.

$ arduino-cli compile --build-path ./build -b arduino:avr:uno
Sketch uses 1874 bytes (5%) of program storage space. Maximum is 32256 bytes.
Global variables use 59 bytes (2%) of dynamic memory, leaving 1989 bytes for local variables. Maximum is 2048 bytes.

Used library  Version Path                                         
LiquidCrystal 1.0.7   /home/umberto/Arduino/libraries/LiquidCrystal

Used platform Version Path                                                         
arduino:avr   1.8.99  /home/umberto/.arduino15/packages/arduino/hardware/avr/1.8.99

$ tree build/
build/
├── Blink.ino.eep
├── Blink.ino.elf
├── Blink.ino.hex
├── Blink.ino.with_bootloader.bin
├── Blink.ino.with_bootloader.hex
├── build.options.json
├── compile_commands.json
├── core
├── includes.cache
├── libraries
│   └── LiquidCrystal
│       ├── LiquidCrystal.cpp.d
│       └── LiquidCrystal.cpp.o
├── libraries.cache
├── preproc
│   └── ctags_target_for_gcc_minus_e.cpp
└── sketch
    ├── Blink.ino.cpp
    ├── Blink.ino.cpp.d
    └── Blink.ino.cpp.o

5 directories, 15 files

[bkietz]

I had this issue and the following worked for me:

$ arduino-cli compile --only-compilation-database --build-path ./build -b arduino:avr:uno

So thanks @umbynos!