Library manager: dependency resolver

[cmaglie]

This is an attempt to solve #5795.

The dependency tree is read from the library_index.json, downloaded from arduino.cc, this json is build from the information contained in the library.properties of each library so, to complete this PR, we should add a field in the library.properties to explicitly specificy dependencies.
In this PR the dependent libraries are handled without versions constraints, this will be added in a future development.

The latest commit has been added for testing purposes and it will be removed before merging this PR: it disables the automatic download of library_index.json and reads the index from library_index_test.json, this way we can test this PR by providing a fake library_index_test.json.

This is the index that I used for testing: library_index_test.zip that contains a modified fake entry:

    {
      "name": "ArduinoCloud",
      "version": "1.0.0",
      "author": "Arduino",
      "maintainer": "Arduino \u003cinfo@arduino.cc\u003e",
      "sentence": "Easly connect your Arduino/Genuino board to the Arduino Cloud",
      "paragraph": "Easly connect your Arduino/Genuino board to the Arduino Cloud",
      "website": "https://github.com/arduino-libraries/ArduinoCloud",
      "category": "Communication",
      "architectures": [
        "*"
      ],
      "types": [
        "Arduino"
      ],
      "requires": [
        { "name": "WiFi101" },
        { "name": "ArduinoSound" }
      ],
      "url": "http://downloads.arduino.cc/libraries/github.com/arduino-libraries/ArduinoCloud-1.0.0.zip",
      "archiveFileName": "ArduinoCloud-1.0.0.zip",
      "size": 45097,
      "checksum": "SHA-256:679a73db1aa4825652882ff71d7a3fb4992f4f79b41cb4a0a6c5c912adbfa848"
    },

and this is the dialog that appears when we try to install "ArduinoCloud":

(of course ArduinoCloud doesn't need ArduinoSound, this is just for testing multiple dependencies)

This PR also lead me to think about a possible simplification of the ContributedLibrary object model, but this is something for another PR to build on top of this one.

[ArduinoBot]

✅ Build completed.

Please test this code using one of the following:

⬇️ http://downloads.arduino.cc/javaide/pull_requests/arduino-PR-6004-BUILD-655-linux32.tar.xz
⬇️ http://downloads.arduino.cc/javaide/pull_requests/arduino-PR-6004-BUILD-655-linux64.tar.xz
⬇️ http://downloads.arduino.cc/javaide/pull_requests/arduino-PR-6004-BUILD-655-windows.zip
⬇️ http://downloads.arduino.cc/javaide/pull_requests/arduino-PR-6004-BUILD-655-macosx.zip
⬇️ http://downloads.arduino.cc/javaide/pull_requests/arduino-PR-6004-BUILD-655-linuxarm.tar.xz

ℹ️ The linuxarm build is still experimental and may not be always available.

[matthijskooijman]

I haven't looked at the code (yet), but my first question is: What will we use to identify a library? It looks like a string name is used, which I assume is the same string stored inside the library.properties of the library depended upon? Or is this the identifier used inside the library manager json? Questions that spring to mind:

• How to identify dependencies that are not registered in the library manager (obviously the can't be installed, but how to specify them?)
• What happens when a library name changes? Is this common?

[cmaglie]

It looks like a string name is used, which I assume is the same string stored inside the library.properties of the library depended upon?

yes, it's the name field in library.properties (there is an exception to this, about the libraries in the old pre-1.5 format in that case the folder name containing the library is used).

How to identify dependencies that are not registered in the library manager (obviously the can't be installed, but how to specify them?)

You can't, but the real question is: even if we can specify external dependency in some way how is the user supposed to install them? he needs to do anyway some actions outside the library manager (download zip, unpack etc.).
At the moment if an unavailable dependency is specified it is ignored during install.

What happens when a library name changes? Is this common?

Renames are denied (except when we are forced for admin needs).

[mbanzi]

what's missing for us to be able to apply this patch?

[cmaglie]

Currently:

• add a new depends field to the library.properties specification
• implement into the indexer (so the field is parsed and published in the library_index.json)
• extensive testing

[tuxedo0801]

This sounds like a first step. but I think this is not enough.

If I download a sketch/project from somewhere on the web and want to build the sketch, I first need to manually check the dependencies. I have to check the "includes" and maybe also the documentation. Then I need to search for the right library in the library manager and finally install it. THEN I can try to build it. And if I found the correct library with the correct version it will hopefully compile successfully.

What about having a kind of "dependency-config" for a sketch or a kind of meta-information IN the sketch that lets the Arduino IDE detect which version the build would require and finally download the correct version.
But then of course different sketches might use different version of same library. So arduino IDE would need to use a dedicated lib-folder for each sketch/project.

Of course this would add another level of complexity to a sketch. But it should be possible to have this kind of backward compatible: f dependency-information is available, use it. If it's not available, proceed as before: let the user find the dependencies on his own.

Have there been any thoughts about this kind of sketch-to-lib-dependency-management?

[tuxedo0801]

As I did not get feedback here, I tried to post the same idea/qustions to arduino developer mailinglist. NOw waiting for message approval ...

[Avamander]

I think sketch-to-lib-dependency-management would be too much for AIDE to handle. The original idea by OP would be enough for starters.

[cmaglie]

superseded by #8600