Share profiles and build/build_profile<profilename> directory

[ssteinerx]

Correct me if I've missed something, but it seems that profiles, ./.vscode configs, and build products (core, libraries, etc.) are currently stored per sketch.

Specifically, the profiles are entirely within the sketch.yaml, and the entire environment is compiled into the ./build directory: ./build contains the build products for a complete compilation of the entire core, and libraries used by the profile. They are not shared, but are recreated for each sketch.

I would have thought that profiles would/could be global or, at least, optionally shared among "sketches" and would have expected that, if build products are to be kept, they'd also be shared among projects that use identical build configurations.

Also, keeping the build products like this seems like it's giving you "reproducible builds" but, given the fact that the Arduino upgrade process destroys the previous release's entire source code tree, your "reproducible build" becomes an orphan and cannot be rebuilt without tediously re-installing the exact version with which it was built. To make them actually "reproducible", you'd have to snapshot the source tree as well, or override Arduino's default behaviour when upgrading.

I understand the issues vis-à-vis reproducible builds, compiler switches, etc. but snapshotting the source tree (with a SHA, or with the git SHA if available), then appending the SHA of the complete config to the profile name i.e. build_profile-<profile name>-<SOURCE-TREE-SHA>-<CONFIG-SHA> handles that issue. This assumes that any issues that impede making a shareable SHA from the config e.g. embedded project paths or other unique-but-not-really-different info, in the config are rectified.

As for libraries, just symlink the ones used in the tagged build dir, adding and/or building as needed.

This would also go a long way to making "Convert Project" work (or exist, rather) since it'd be a matter of just hooking up already built profiles, libraries etc. rather than having to create the whole environment from scratch.

[thelastoutpostworkshop]

Profiles live inside each sketch’s sketch.yaml because we follow the Arduino CLI sketch-project spec; That keeps a sketch self-contained and lets the same profile work even if the project is copied outside VS Code or shared with someone running pure arduino-cli.

The build products contain absolute include paths and hashes produced by Arduino CLI; reusing them across sketches or after changing the toolchain is unsafe. Instead I rely on the CLI’s own global caches (cores/libraries under the CLI data directory) for reuse, so you’re not recompiling everything from scratch even though each sketch has its own build tree.

The reproducibility story is the YAML: the profile records the FQBN plus the platform/library versions you pin in the Profiles Manager. Re-running arduino-cli compile --profile with that file rebuilds the binary from source. Keeping old binaries around without the corresponding toolchain doesn’t help once the underlying Arduino platform is upgraded, which is why I prefer to regenerate from the captured config rather than treat build/ as an archive.

If you want a profile available in another sketch today, copy the relevant section of sketch.yaml (or the whole file) into the other project; the format is identical to what the CLI expects.

[ssteinerx]

Thanks for the quick reply!

I kind of figured it was an artifact of the way the arduino-cli records these things.

I hadn't noticed any particular caching in the Arduino universe in general -- it looks like it recompiles the world every time but maybe it's just that the paths and command lines are long so it's hard to keep track of what it's actually compiling vs. pulling out of cache.

The good news is that sketch.yaml doesn't contain any sketch specific paths etc., so you can symlink sketch.yaml to a shared file containing any number of profiles.

By careful naming like:
<BOARD><PLATFORM><LIBGROUP><PORT>
you can keep a bunch around, fully configured, without having to modify them for e.g. using a different comm port.

There are few enough parameters in the sketch.yaml that spelling out what they contain in the name isn't too unweildy.

Obviously, the caveat is that any configuration changes you make from within the sketch are going back to the global file, but, with a little care, it's a better solution (for me) than copying sketch.yaml around which sets off my DRY alarm.

Also, which wasn't obvious (to me) until I did it, Arduino Maker also creates .vscode/{c_cpp_properties.json,arduino.json} from the sketch.yaml when it compiles so that pretty much takes care of itself, too!

Thanks for a great tool! The MS/forked extension was not working well so I was keeping the Arduino IDE open in the background to compile/upload. I even wasted half a day trying to get PlatformIO to cooperate with RP2040 and the Arduino SDK but, even when I did, it was not very helpful...and everything I ended up using was a non-official fork of some unmaintained bit of PlatformIO. Not great, at best.