Installing a board package via Board Manager completely screwed up another unrelated one

[teo1978]

I have Arduino 1.8.7 and I had installed the esp8266 boards from https://github.com/esp8266/Arduino
I had installed them from the GIT repositiry directly into [arduino folder]/hardware, NOT the Board manager.
Everything was working fine and I had a sketch that I could compile for an ESP8266 board and upload to the board without any issues.

Then I installed the ESP32 cores from Espressif (https://github.com/espressif/arduino-esp32) using the Board Manager.
I didn't even try to use them. I intended to use them later.

So, I opened my sketch for my ESP8266 board, and I kept the ESP8266 board selected. I tried to compile and upload it to the board as I had done hundred of times (I hadn't modified the sketch, I hadn't updated the ESP8266 cores, nothing had changed besides installing the completely unrelated ESP32 boards that I was not using in any way).

When trying to compile, I would get a nonsense error message related to the ESP32 boards, mentioning a file in the ESP32 package, despite the fact that I was trying to compile a sketch for a ESP8266 board:

Error compiling for board SparkFun ESP8266 Thing Dev.
fork/exec /home/teo/.arduino15/packages/esp32/tools/esptool/2.3.1/esptool: no such file or directory
Error compiling for board SparkFun ESP8266 Thing Dev.

This is absolute nonsense.

I reported this to Espressif (espressif/arduino-esp32#2137), but their response was:

the esp32 board package installs nothing outside of the .arduino15/packages/esp32 directory. Any interference with another board's core files is entirely an arduino IDE issue.

So, this seems to be a bug in the Arduino IDE itself.

Additionally, when I removed the ESP32 boards via the Board Manager, I got an error:

though the boards do appear to have been removed and the sketch for the unrelated ESP8266 board now does compile as expected.

[facchinm]

Hi @teo1978 ,
the board manager is a bit stupid (always trusts json as backing storage, does not handle strange configurations perfectly). We hope to resolve this soon by integrating arduino-cli capabilities in the IDE and using the Board Manager simply as a frontend.

Then, the technical part; when compiling/uploading for a core, some related programs (so called tools) are referenced as {runtime.tools.toolname.path} and this variable is expanded. The board manager takes over cores "installed" in the sketchbook, so if any core declares toolname in its json it gets picked as the right one.
Probably it's an undesirable situation in the esp32/esp8266 case, since the tools is called in the same way but the directory structure is different.

To make 100% sure to target the right tool, cores creators should reference it with a unique version and change the references in platform.txt to {runtime.tools.toolname-toolversion.path}

@cmaglie should we have a master issue about cli integration and reference all the issue like this there?

[me-no-dev]

Hey guys :) So what you propose is to rename esptool to let's say esptool-esp32 so they do not mix. Does that need to be done on both sides, or one will be sufficient? Asking because we still reference the tool as esptool in many places in platform.txt for both boards

[teo1978]

By both sides you mean esp32 and esp8266, right?
I guess if one alone does it, it would fix the issue in practice; however there's no reason why one "side" should have more "responsibility" than the other to rename its tool, or more "right" than the other to call it just "esptool", not to mention other variants that may not exist now but appear in the future, so I guess the logical thing to do would be to do it on both sides.
Just my two cents.

However I have very little understanding of how this stuff works.

[facchinm]

@me-no-dev the ideal behaviour is that every platform should uniquely specify what they need to use (via {runtime.tools.toolname-toolversion.path} syntax or via json).
The biggest problem with cores installed outside board manager is that the tool situation is unknown (they could also not be installed).
Then, there's a third problem: cores in sketchbook pick an fqbn based on the folder structure (eg a core in sketchbook/hardware/myarch/mycore will create entries like -DARDUINO_ARCH_MYCORE which may not fulfill architecture specific ifdefs ).
Said that, there's plenty of space for improvement 😄 If you want to contribute in a single place, https://github.com/arduino/arduino-cli is the right place to go (since we are planning to replace the IDE board manager backend with it soon).

[me-no-dev]

@facchinm thanks for the explanation! And thanks for pointing to the cli :) will definitely have a look and see if any ideas pop up :)

@teo1978 the reason why I ask if one side will be enough is that 8266 Arduino has already many releases and is less or more mature platform, while on esp32 we have had one "stable" release so far and would not hurt much if I mess with tool names and jsons. It has nothing to do with responsibility :) I have write access to both repos.

[matthijskooijman]

I came across this issue when writing #9237 and it seems partly related. Reading this issue, I wonder: How did it work in the first place? AFAICS esptool is a tool that needs to be installed somehow. Before you had the esp32 core installed, you installed the esp8266 core manually into your sketchbook, but that does not install any compilers or tools, right? How did you get esptool then? Did you also have the esp8266 core installed through the board manager?

@facchinm

To make 100% sure to target the right tool, cores creators should reference it with a unique version and change the references in platform.txt to {runtime.tools.toolname-toolversion.path}

I'm not sure this actually makes sense, because:

• A core currently does not reference any version numbers. It is the .json file that ties together a core version and the tools it needs, but this file is somewhat out of scope of the core itself. It would seem weird to point the platform.txt to one particular version of the tool (what if the core is installed manually, or through a json file that has different tool versions, etc.?).
• This would not actually fix the problem. If the esp8266 core and esp32 core use the same tool name (esptool), there is nothing to stop them from also using the same version numbers, and the conflict would still exist.

Requiring the use of different tool names could fix this, but that means that every third-party core should somehow prevent using the same names for tools which would require some global registry for tool names perhaps?

Alternatively, you could include the name of the core that installs a tool in the identity of the tool (and only make available tools defined by a core's .json file to that core), but that would prevent sharing a tool (e.g. avr-gcc) between multiple cores (which I think is currently possible), and that would not really help for manually installed cores (which have no .json file).