Upcoming release of the llir/llvm project

[mewmew]

This notice is intended to give a heads up for those using the llir/llvm library. The next release will include complete support for all intrinsics of the LLVM IR language. The work is currently in a flux, and to experiment with different API designs and simplify the parser logic and reduce the code duplication in the project, a new repo has been created during the experimental phase.

https://github.com/mewmew/l

At the current stage, the grammar is capable of parsing the entirety of the LLVM IR language, including specialized metadata nodes (#26).

While working on this we will also try to take into consideration previous issues that have been identified with the parser (such as the handling of quoted strings #24).

The llir/llvm/ir package will be extended to support the entire LLVM IR language; thus resolving #23 as linkage information will be present in the in-memory intermediate representation form.

With the upcoming release, read support for all of the LLVM IR language concepts will have been implemented; thus resolving #15.

Similarly; we will now have a grammar covering the entire LLVM IR language; thus resolving #2.

With the addition of support for specialized metadata nodes, the second requirement of llgo will also be fully supported (#3); llgo uses the DIBuilder API for generating debug metadata (DWARF, et al.). This could be built outside of the core (it's just a matter of creating metadata nodes in a particular format), just be aware that it's pretty finicky and easy to break..

For IR construction, a similar approach will be used as has been done before. Personally, we feel this approach has worked out well and has been quite pleasant to use. If anyone has input on their own experience using the API of the llir/llvm/ir package to construct LLVM IR, please let us know as that could help shape the upcoming release. As for llgo, the first requirement in terms of using the LLVM API for generating code, it's mostly write-only via the builder API. Bitcode and IR reading is not important (at the moment?), but writing is; one or the other is required, but preferably both. is satisfied by this API, and has been for a while. Although, now the llir/llvm/ir package will contain the support for the entire LLVM IR language, and now just a subset; thus the requirement should be satisfied in full.

Module top-level information such as target triple and data layout has been and will continue to be recorded and maintained by the IR API, thus supporting the third requirement of llgo; llgo needs to be able to look up target data (arch word size, alignment, etc.) from triples.

Generating C-shared library bindings compatible with the official C library of the LLVM project is an ambitious goal that is left for a future release (#12). Anyone specifically interested in this topic, feel free to get in touch with us or continue the discussion in the dedicated issue.

Similarly, interaction with the Go runtime is targeted for a future release, and those with knowledge in this domain are happily invited to the discussion on what is needed and how to bring this about (#18).

As for use-tracking and data analysis support (#19), more thought will be required to get a clean API. This is therefore targeted for a future release.

So, to summarize, the upcoming release of the llir/llvm project will include read and write support for the entire LLVM IR language. In other words, it will be possible to parse arbitrary LLVM IR assembly files into an in-memory representation, aka the one defined in package llir/llvm/ir. And the in-memory IR representation will have support for the entire LLVM IR language, and can be converted back to LLVM IR assembly for interaction with other tools, such as the LLVM optimizer.

Any feedback is welcome, so we know we're heading in the right direction.

Cheerful regards,
/u & i

[pwaller]

Hi! This is a fantastic project and I've really enjoyed using it.

Please could you consider using semantic version numbers for release tags? Just something of the form v[0-9].[0-9].[0-9]

If you're not aware, vgo requires them, and it is likely to become the officially supported way to depend on specific versions of go code in the next release of Go in the near future. I've already started using it, and on this llvm package it currently resorts to git-sha pinning rather than using a version number.

Some references:

• https://research.swtch.com/vgo (tons to read there)
• https://www.youtube.com/watch?v=F8nrpe0XWRg (Russ Cox, one of the earliest Go authors discussing it).

Looking forward to the next release of llir/llvm, keep up the good work! 🚀

[mewmew]

Hi! This is a fantastic project and I've really enjoyed using it.

Happy to hear you enjoy the project! We too have had a lot of fun playing with various aspects of LLVM IR from Go.

Please could you consider using semantic version numbers for release tags? Just something of the form v[0-9].[0-9].[0-9]

Definitely. we'll make sure to tag the upcoming release with vgo compatible semantic versions (v0.3.0).

Looking forward to the next release of llir/llvm, keep up the good work! 🚀

The upcoming release can be expected to arrive in a couple of months. Right now, I'm out enjoying vacation :)

Have a wonderful summer!

Cheers,
/u

[pwaller]

Boom. Have a great holiday. Ciao! :)

[mewmew]

Boom. Have a great holiday. Ciao! :)

Haha, speedy reply. Thanks! You too!

[pwaller]

Definitely. we'll make sure to tag the upcoming release with vgo compatible semantic versions (v0.3.0).

By the way, if you're making a big backwards incompatible change, you should bump the major version number. I can elaborate further on why if it's not obvious, or I can recommend Russ Cox's writings on the matter: https://research.swtch.com/vgo-import

The upshot is that you can do big breaking changes without breaking existing users (even those just using old-style Go get).

[mewmew]

By the way, if you're making a big backwards incompatible change, you should bump the major version number. I can elaborate further on why if it's not obvious, or I can recommend Russ Cox's writings on the matter: https://research.swtch.com/vgo-import

This will definitely be the case after the v1.0.0 release. Breaking changes will cause major version bumps.

While on version 0.x however, API breakage is expected and even encouraged so that the API of 1.0 will be clean and consistent. If I recall correctly, this is allowed by semantic versioning?

Edit: to expand on the above, the readme has had a work in progress notice, which will be removed as soon as we hit version 1.0.

"This project is a work in progress. The implementation is incomplete and subject to change. The documentation may be inaccurate."

[pwaller]

You are correct.

Major version zero (0.y.z) is for initial development. Anything may change at any time. The public API should not be considered stable.

It would be good to try and reach a stable API. A stable public API doesn't mean you can't add to it though. You do already have quite a bit of useful functionality in there already! :)

(Also, feel free to hide all of these comments).

[mewmew]

Just as a notice, version v0.2.1 has been tagged. This version should work with Go modules.

As the upcoming release will introduce API changes (and provide full support for the entire LLVM IR language), users are expected to pin the the version of llir/llvm to v0.2.1, until upgrading to the new API.

[mewmew]

A quick status update.

@pwaller and I have been running a few experiments to improve the performance of the parser. These experiments have been using a different parser generator (Textmapper instead of Gocc), and the initial results look promising.

The approach taken in the https://github.com/mewmew/l-tm seem to pass the desirable speed requirements of mewspring/mewmew-l#6. There is still quite a lot of work to do to finish all instructions, etc. But already from the current work we can estimate to arrive at somewhere between 1x and 2x runtime as compared to the official LLVM distribution for parsing LLVM IR assembly. Compared to the 30x we had before switching from Gocc to Textmapper to generate the LLVM IR parsers, this seems noteworthy enough to start integrating back into the main llir/llvm repo.

This will be done in steps.

1. merge https://github.com/mewmew/l-tm into a development branch.
2. merge https://github.com/llir/l into a development branch.
3. work on the development branch until the code base matures.
4. prepare for the release of v0.3 and merge back into master.

Edit: the generated parser will be split into a dedicated repository (llir/ll) to keep the size of the main repository down. This allows us to force push to the llir/ll repo whenever we re-generate the LLVM IR parser from the Textmapper grammar, thus keeping download size under control.

[pwaller]

@mewmew thanks for your efforts on this!

Any objection if I make a v0.3.0-pre1 tag so that I may reference it from another module?

[mewmew]

Any objection if I make a v0.3.0-pre1 tag so that I may reference it from another module?

None at all :) v0.3.0-pre1 is now tagged, https://github.com/llir/llvm/tree/v0.3.0-pre1

The primary aim is now to stress test the implementation, find and fix bugs, start to profile and further improve the performance, and also to find inconsistencies in the implementation and the API, that we can improve before the v0.3.0 release.

Do your best to break it! :)

[mewmew]

We are getting closer to release :)

v0.3.0-pre2 has just been tagged. https://github.com/llir/llvm/tree/v0.3.0-pre2

Anyone using the llir/llvm packages is encouraged to upgrade to the latest pre-release version (v0.3.0-pre2) and report back anything weird you encounter. This will help us finalize the API and hopefully do a release proper in early December.

Cheers!
/u

[mewmew]

This will help us finalize the API and hopefully do a release proper in early December.

Haha, it was conspicuously cleaver to leave out the year in above statement. It does look like we will have an early/mid December release of v0.3 after all!

[mewmew]

With #111 and #112 now resolved, the only remaining issue to fix before tagging the v0.3 release is #31. 🎉

[mewmew]

v0.3.0 of llir/llvm was released today.

Thanks to everyone who helped get this release in shape by reporting bugs, implementing features, improving test coverage, discussing design ideas and just being friendly and kind.

Closing this meta issue and welcoming the adventures of the new year :) 🎆

[pwaller]

🎉