Add library.json for PlatformIO

[cleveHEX]

So dependecies on PIO will be installed automatically.

[ladyada]

hiya this is handled for us by platform.io so we don't have to maintain two json files!

[cleveHEX]

@ladyada You are wrong, they just use library.properties, instead of this one, but that file does not set dependecies. If you install your library now, you will get compilation error about missing Unified library, this does set dependency, so that one gets installed with DHT library automatically.

[ladyada]

i know that this was somehow managed directly at platform.io as none of our libraries have library.json, almost all have dependenceis, but people use the libraries all the time :)

[cleveHEX]

@ladyada That is because they install those libraries manually, so dependecies don't work.

[cleveHEX]

@ivankravets What is your opinion of this?

[ivankravets]

@CornyjK Thank you so much for the PR. I also did a few 4 years ago and they were rejected by @ladyada. This is her decision. Adafruit forces people to Arduino IDE and we can't help here. Sorry.

The alternative solution is to use PlaformIO Project Dependencies and manually declare all dependencies. Also, try to find alternative library, we support over 6,000.

P.S: Meanwhile, https://marketplace.visualstudio.com/items?itemName=platformio.platformio-ide#review-details

Over 700K downloads for the last year and one/two 5-star reviews each day.

[ladyada]

hiya, @ivankravets thanks for comin' by - have you had any progress in adding/merging library.json with library.properties? they are very close in format - and we can only maintain one file!

[ivankravets]

Hi @ladyada,

Arduino uses a proprietary build system which does not allow library developers to control build process, extended it or modify build flags on-the-fly. library.properties is a primitive manifest which just carries a base information (name, author, description).

Today, library.json supports over 15 different frameworks including Arduino and ARM mbed. See full specification https://docs.platformio.org/en/latest/librarymanager/config.html

[ladyada]

hiya, you say library.json supports Arduino framework but not supported by Arduino IDE right? so we'd still have to have two files - library.json and library.properties? can you use the version number data from library.json or, better yet, look at git commit tags?

[ivankravets]

can you use the version number data from library.json

Yes, we use it. So, you change version field in the manifest in the same approach as you do for library.properties. If we see a tag with this version in your repo (1.2.3 or v1.2.3), we will use it as the main source.

[ladyada]

sorry i got them confused! such similar names :)

i meant, can you use the version tag from the Arduino IDE library.properties instead of having to change it in library.json? or use github release tags (we do this in circuitpython so there is no file version number to change)

[ladyada]

Likewise, @per1234 @cmaglie @facchinm @agdl @mbanzi can you use library.json if library.properties is not available? and if not, is there any information that is missing from library.json that @ivankravets can add to make it possible to have only one metadata file that is maintained?

thanks :)

[ivankravets]

can you use the version tag from the Arduino IDE library.properties instead of having to change it in library.json

A library can be linked only with the one manifest at the same time.

[ladyada]

@ivankravets who wrote the manifest parser?

[per1234]

can you use library.json if library.properties is not available?

I can only tell you that use of library.json is not currently supported. I'll need to leave it to others to decide whether that feature should be added. If it doesn't end up getting the desired attention here, you can open a dedicated issue report for the feature request to ensure it isn't lost.

is there any information that is missing from library.json

Here's the library.properties specification:
https://github.com/arduino/Arduino/wiki/Arduino-IDE-1.5:-Library-specification#libraryproperties-file-format

[ladyada]

thanks @per1234 !
@ivankravets - have you submitted an issue or PR to Arduino to add support for library.json? if so, please link it here :)

[ivankravets]

@ladyada Thanks! Done in arduino/Arduino#8443

Actually, I proposed library.json to Arduino core developers (google group, mailing list) in the 2014 year. A few respected people from open source community, such as @PaulStoffregen, supported the idea with the unified library manifest.

Arduino IDE didn't have library manager or library.properties that time. If I'm not mistaken, it was added in 2016.

I'm sure that it's a good idea to have one manifest for different frameworks and maintain specification together.

[ladyada]

@ivankravets do you know who wrote the manifest parser for platformio? that way it can be updated if Arduino does not accept your PR

[ivankravets]

@ladyada - me. What do you propose?

[ladyada]

awesome @ivankravets, what you can do for now (until Arduino accepts your PR) is read the version number from library.platform if library.json does not contain it, or better yet, read the github release tag (which must match anyways). then the version number is only updated in one spot.

#118 (comment)

[ivankravets]

@ladyada we have been doing that since library.properties was "invented". See this library in our registry https://platformio.org/lib/show/19/DHT%20sensor%20library/manifest

Actually, there is no problem with version field. library.properties does not support dependencies. The same with Arduino IDE. This library depends on Adafruit Unified Sensor.

Please take a look at this PR by @CornyjK. The core idea is to specify all dependencies in library manfiest. Later, library manager will install them automatically. See https://docs.platformio.org/en/latest/librarymanager/config.html#dependencies

[ladyada]

ok great @ivankravets - so you are saying we can have library.platform as we do now
https://github.com/adafruit/DHT-sensor-library/blob/master/library.properties
and then add library.json like this PR here but remove the line "version": "1.3.1", and it will all work?

@CornyjK please update this PR to remove the "version": "1.3.1", entry from json!

[ivankravets]

@ladyada Yes, if you remove version field from manifest and specify repository field than Case 3 will work here. I'm not sure that this is a good idea. I think we should propose users the latest stable version instead of the latest build/commit.

Is it difficult for Adafruit team to change version field in both manifests on the next release?

P.S: I personally don't like the current situation where our crawler scans these manifests and checks for new versions/releases. The right solution is when library maintainers manually publish each new release of a library. Normally, we do that for Python's PyPi, Node.JS's NPM, etc.

Our crawler is a temporary workaround. We plan to introduce soon a new Library Registry, where everyone will maintain personal libraries/packages manually.

[PaulStoffregen]

I also usually forget or neglect to update both on the libs I maintain.

[ladyada]

hiya @PaulStoffregen thanks for the input!

@ivankravets library maintainers usually forget or neglect to update version numbers on libraries they maintain.

@CornyjK @ivankravets sounds like the best next step is to remove the version # from this PR. or should we wait for the new Library Registry, where everyone will maintain personal libraries/packages manually? note that adafruit will not be manually updating another site, so it will be up to someone else.

ideally we would have one file only to manage! @per1234 @cmaglie @facchinm @agdl @mbanzi any updates on whether arduino ide can use library.json if library.properties is not available?

[ladyada]

ping @CornyjK @ivankravets @per1234 @cmaglie @facchinm @agdl @mbanzi

[ivankravets]

Hi @ladyada,

Sorry for the delay, I've hoped to see any comments from the Arduino team... They are busy with own products.

There is no difference for us if community uses library.properties or library.json. In the case with the current PR, @CornyjK would like to have dependencies declaration. We know that Arduino IDE does not allow us to manage projects or dependencies per project. All libraries are shared per sketches which leads to a lot of issues. People keep these libraries in the different folders and "rename" folder before compilation to avoid collisions.

PlatformIO resolves these issues and provides a professional ecosystem for embedded development. It supports over 15 different frameworks including Arduino (Wiring). We have a huge community which uses Arduino framework and our main goal is to provide maximum compatibility with Arduino IDE sketch builder and easy migration from Arduino IDE to PlatformIO ecosystem.

How about if we add dependencies field to library.properties even if Arduino IDE does not support it? We can parse this information by our Library Manager. So, you will maintain just only the one manfiest. I hope it will simplify a life for other library maintainers.

---

We had a bad signal in 2017 with a conflict between Arduino.cc and Arduino.Org. The final solution of this conflict (ARM via Softbank occurred Arduino.cc) made a big threat for the Arduino community. All new Arduino.cc boards are ARM-based, new software products are source closed and proprietary, new tools support only official Arduino.cc boards.

We respect a huge work which @dbarragan did with Wiring and how its simple for using API helped to build new companies and communities (Arduino, Particle, and others).

We are so happy that PlatformIO is a fully independent and community-driven project. We are opened under the same rights for all architectures, silicon vendors, IDEs, and operating systems.

"User should have a choice!", - the main credo of PlatformIO. We trust in the power of an open source in the all aspects!

Regards, Ivan.

[ivankravets]

I will not comment on the "I'm more open source than you" claims.

@cmaglie, It's not a problem with open-source. I talk about the freedom of the Arduino ecosystem. No freedom today.

---

• if library.properties doesn't have a `version' field the library indexer will fetch the release version from the git tag if it is a valid semver.

Issue 1

No need to reinvent a wheel. There are a lot of languages, package managers and manifests for them. 99% of them strictly require version field and there is a reason for that. Packages/libraries could be installed without VCS. Very often companies don't share their repository but provide direct ZIP/TAR URL to their package/library. The only version field in a manifiest explains a revision of source code.

See how do we resolve this case when library is published via self-hosted archive.

Issue 2

We have a lot of repositories which contain libraries/packages located in the one repo. No need to go so far, classic ESP8266 core for Arduino.

Issue 3

Developers should have a clear picture about which libraries and versions are used per project. If library was cloned from VCS, than hash of commmit shold be shown. If developer has an issue with a library, it's very easy to report it to library maintaners with all details including a version. Please take a look how PlatformIO LIbrary Dependency Finder does it

Conslusion

I understand very well that library maintainers sometimes forget to update version field. Nevertheless, having it in manfiest will help to avoid further issues desibed above.

[ivankravets]

@PaulStoffregen @cmaglie

support dependency info. PlatformIO's library.json does.
perhaps compiler errors about missing header files would be augmented with a clickable button or hyperlink to install the missing library
Ok, maybe I'm dreaming too much....

Your dreams were realized in the 2015 year thanks to PlatformIO :)

and when you click on "blue" link, you will see

---

Maybe, it's a time to switch to PlatformIO Build System and its Library Manager? End developers will use the same Arduino IDE and have the same UX as before.

Actually, we can add direct supprot for Arduino's platform.txt manifest and its remote Board JSON files.

What do you think?

[cmaglie]

Issue 1 [...]

Very often companies don't share their repository but provide direct ZIP/TAR URL to their package/library. The only version field in a manifiest explains a revision of source code.

In this particular case I would say that the author must specify the version otherwise there is no other way to determine it. Since he's self-hosting the archive he should also take care to update the version field by hand and not delegate it to the indexer (since there is no git-tag or similar to look at).

My proposal is not about denying the possibility to specify a version in library.properties but making it optional if the library is VCS hosted.

Issue 2

We have a lot of repositories which contain libraries/packages located in the one repo. No need to go so far, classic ESP8266 core for Arduino.

We have a strict rule: 1 library - 1 repo.

The repository you linked is a platform (esp8266) that bundles a set of libraries that works with that particular version of the platform. Again, no need to remove the version field on those libs.

Moreover each version of the platform will deliver a fixed set of platform-libraries (and it's not said that upgrading one of the library in the set will still work without upgrading the whole platform).

Issue 3

Developers should have a clear picture about which libraries and versions are used per project. If library was cloned from VCS, than hash of commmit shold be shown. If developer has an issue with a library, it's very easy to report it to library maintaners with all details including a version.

I don't understand what this has to do with the topic? In this case it seems that not having a bogus version written in library.properties is still better...

[PaulStoffregen]

Agreed, too much off-topic stuff. Issue arduino/Arduino#6004 is where the Arduino IDE is deficient. That's where effort should be focused.

But to be realistic, that pull request has been open for 2 years. Back in May 2017 @mbanzi asking what more needed to be done! It was an Arduino 1.8.3 milestone in July 2017. Every appearance is that work has been abandoned. What are the odds it will ever be updated and merged and release (non-beta) to the maker community?

[cmaglie]

Both fix are coming mainline.

[kleini]

Can somebody please summarize the outcome of this discussion. I don't have the time to read multiple pages and find the summarization between the written lines.

[ivankravets]

@kleini Just add the next lines to your platformio.ini

[env:myenv]
platform = ...
lib_deps =
  Adafruit Unified Sensor
  DHT sensor library@^1.3.5

[kleini]

I didn't want to read exactly this solution. But thanks @ivankravets for your efforts. This makes maintaining upgrades of used libraries for a longer list of sketches a hard task now. This forces me to use forks of all these libraries to have easier dependency management.

[ladyada]

we're hoping that arduino ide adds dependency management soon and then platform.io can parse that file. we're happy to add the dependency graph for all ~200 libraries we manage (we need it for travis anyhow :)

[ivankravets]

@ladyada can you work on this with arduino team? It is enough just to have extended specification for library.properties. Maybe, dependencies = libName1, libName2@SemVer. For example, dependencies = OneWire@^2.3.4, DHT sensor library@~1.3.5, etc. Any format is OK for us.

If we wait when arduino ide and their builder supports this, it will take the years... Arduino IDE does not provide project isolated build environments.

[PaulStoffregen]

Looks like Arduino worked on this months ago with arduino/Arduino#8600. But with no further activity for the last 3 months, hard to feel optimistic.

[ladyada]

@ivankravets (and @PaulStoffregen ) thanks for looking in to this, do you have any suggestions who together we could all work with at arduino to assist with this?

[per1234]

For this situation, it's not absolutely necessary to wait for Arduino to merge arduino/Arduino#8600. All that's required is to know for sure how Arduino's implementation of the feature will use library.properties.

It appears this will be a comma separated list in the depends property. That system is already in place in multiple of Arduino's official libraries:

• https://github.com/arduino-libraries/Arduino_OAuth/blob/master/library.properties#L11
• https://github.com/arduino-libraries/Arduino_MKRRGB/blob/master/library.properties#L11
• https://github.com/arduino-libraries/ArduinoBearSSL/blob/master/library.properties#L11
• https://github.com/arduino-libraries/ArduinoLowPower/blob/master/library.properties#L10

These fields are already being stored in the library_index.json file used by Library Manager:
http://downloads.arduino.cc/libraries/library_index.json
And the system is working when you use the test builds proved in arduino/Arduino#8600.

I think the fact that Arduino has already worked to implement this in multiple of their libraries is a good indication they do plan to eventually merge arduino/Arduino#8600 (or something very similar).

If you can get a final decision from Arduino on the library.properties side of things, PlatformIO can proceed to add support for that system and Adafruit can start defining dependencies for all their libraries, without waiting on Arduino to add the feature to the IDE.

I think cmaglie would be the best person to talk to about getting that decision and the pull request thread would be the place to do it.

[ivankravets]

@cmaglie Can we start using depends? Do you have extended format for this field when the specific version of a dependency is provided?

[PaulStoffregen]

.... it's not absolutely necessary to wait for Arduino ....

I would advise waiting. Arduino has a long history of features such as this languishing for years (this one is already code sitting unused for over 2 years), then much later being implemented differently, usually better. Jumping on this before Arduino solidifies it with a non-beta release may work out well, but it may end up being rather painful when/if they eventually implement it differently.

[cmaglie]

arduino/Arduino#8600 is scheduled for the next IDE release.

The format is a comma separated list of libraries:

depends=FirstLibrary, SecondLibrary

since spaces are allowed in the name of a library, but not commas, you can refer to libraries containing spaces in the name without ambiguity:

depends=Very long library name, Another library with long-name

Do you have extended format for this field when the specific version of a dependency is provided?

there is the possibility to add a version constraint in this way:

depends=First Library (=2.0.0), Second Library (>=1.2.0), Third (^1.2.3)

BTW these constraints are implemented on the indexer (so they are reported on the library_index.json) but not yet in the IDE (it will always install the latest version). The constraints supported are the classic > >= = etc. and the special ^ (that means "compatible with").

We are going to release a version of the arduino-cli that can perform a "lint" of the library and validate the library.properties file.

BTW, as suggested by @per1234, you can use the depends field on the library.properties right now, it is already supported by the indexer and is not going to be changed (unless there is a valid reason to do so).

[ladyada]

@ivankravets @PaulStoffregen @cmaglie hiya folks, looks like there's now a build of the IDE with the new library dep manager? arduino/Arduino#8600 (comment)

[cmaglie]

Yes (still making some minor adjustments)

[ptorrone]

been following along, thanks @cmaglie

@ivankravets @PaulStoffregen post is here, we're tryin this out now:
https://blog.adafruit.com/2019/07/18/library-dependency-manager-coming-to-arduino-ide-arduino-platformio-arduino/

[ivankravets]

@ptorrone Thanks for the blog post! Is it possible to have a blog post on how to use PlatformIO IDE with some Adafruit boards? I'm sure your community will be impressed. We have advanced library manager since the 2015 year. Also, there is free debugger with peripheral registers, memory explorer and disassembly. So, this post can help people migrate from Arduino IDE and explorer totally new instruments. Do you have any board with on-board debug probe/adapter?

There are some reviews from users https://marketplace.visualstudio.com/items?itemName=platformio.platformio-ide&ssr=false#review-details

[ptorrone]

hi hi @ivankravets send me an email pt@adafruit.com

[cleveHEX]

Good idea, I like PIO and I think it deserves wider audience.

[ptorrone]

OK! we got this workin' ... trying out the new library manager dependency addition to the Arduino IDE
https://blog.adafruit.com/2019/07/20/trying-out-the-new-library-manager-dependancy-addition-to-the-arduino-ide-arduino-arduino/

[ladyada]

@CornyjK @ivankravets ok! this library has been updated to use the depends line for dependencies. can you update platform.io to use this file, thus solving the initial problem of needing two files for every library repo

[ivankravets]

hi hi @ivankravets send me an email pt@adafruit.com

@ptorrone , done!

can you update platform.io to use this file

@ladyada Yes, we can. See an opened feature request platformio/platformio-core#2781.