Timeline for integrating into Rust upstream?

[davystrong]

Hi,

I'm wondering if there's any timeline for this project to be integrated into the upstream Rust compiler? As I understand it, the Rust compiler needs to be modified somewhat to emit the information this project needs. Are these changes too specific to ever get merged? It would be great if this could be installed as a package simply using cargo, instead of having to compile the whole compiler.

Sorry if there is information about this somewhere, I couldn't find it!

[ZuseZ4]

Hi, the plan is to directly merge this work into nightly rustc, it also got an accepted Major Change Proposal in the past, even though rustc dev still need to vote on the final PR.
There are some implementation changes I need to finish, but I got this fork down to only add 2.5k extra lines of logic and the requested changes are comparably small, so I hope to get to it in December.

[ZuseZ4]

No worries for asking and thanks for your interest, most of the discussions happen on the rust-lang zulip, either in the autodiff stream, or in a help stream linked in the autodiff discussion.

[davystrong]

Hi @ZuseZ4, thanks for that, that's really helpful!

[OwenTrokeBillard]

I, too, am looking forward to Enzyme being merged into Rust. Thanks for all your excellent work so far @ZuseZ4!

[OwenTrokeBillard]

@ZuseZ4 Is there a checklist of issues blocking a merge with upstream Rust? Any issues you could use help with?

[ZuseZ4]

Hi @OwenTrokeBillard, thanks for getting back to this. I was on vacation around end of year and previously got stuck for a while on some re-design, but recently managed to re-implement a core part of my autodiff macro in a way that is more liked by other rustc devs. I now just need to port the remaining code of rust-enzyme to the latest branch, since rustc is somewhat fast moving, so there is a little bit more refactoring required. I think it's the easiest if I just finish this on my own.

One place where I would appreciate help though is the website and documentation, hosted at https://enzyme.mit.edu/rust.
It is developed from https://github.com/EnzymeAD/rustbook
In case you know any other AD tools whose documentation you like (e.g. jaxbook, enzyme.mit.edu/julia, Zygote, ...), it would be great if you could try to add missing pieces to our documentation. Jed is also interested, although slightly blocked since he can't test any examples on the latest branch, and needs to use a slightly dated version. In general, also just adding testcases (and finding bugs) would be great.

As an alternative, it would be great to have rust-enzyme in our compiler explorer: https://enzyme.mit.edu/explorer
I am not really great when it comes to webdev / CI, and the rust side is slightly harder than the other language frontends, since in our case we need to compile all of rustc for it to work.

[jedbrown]

It looks like explorer will just need a build script similar to this and then a flag to activate it. https://github.com/EnzymeAD/enzyme-explorer/blob/main/builder/build-enzyme.sh
A good task once the rebase is ready, in no small part since it makes it easier to report issues.

[OwenTrokeBillard]

Documentation and compiler explorer support are important. I'll gladly help where I can. However, those don't appear to block upstreaming. Do they?

It would be great to have a checklist of issues preventing an upstream specifically. Based on other PRs in Rust, it looks like the process can take many months. Better to get the ball rolling early.

[OwenTrokeBillard]

For example, I imagine Windows support is required (#54). I will give that another try soon.

[ZuseZ4]

Windows support should only be fixing the path issues, I'll try to unblock you here after lunch.

Other than that it's the new macro built-in design instead of our old approach which was requested by other rustc devs, since we would otherwise break cross-compilation.

Some rustc devs already gave their ok from their point of expertise, the final approval will be from t-lang though and they have quite limited time. Someone will bring up the autodiff macro at their next meeting once we are ready, but we should have some good docs by then so they can easily grasp what we would introduce on the language level (that is our autodiff macro frontend). The middle and backend are less relevant here, since we can change those without affecting users.

[fknfilewalker]

Is enzyme now part of nightly? I saw the issue was closed but can not get it to work.

[ZuseZ4]

Are you referring to this one rust-lang/compiler-team#611 @fknfilewalker?
That Major Change Proposal (MCP) was accepted, but there is still some implementation work left.
Mainly this fork needs to build and tested under more targets (MSVC under Windows especially),
and in some more configurations (e.g. downloaded LLVM from CI instead of self-build LLVM, in Docker, ...).
Happy to accept help if you have some time, but otherwise can't give a precise timeline on when it will be ready.

[fknfilewalker]

I am not really a rust user, but I thought about switching when I read that EnzymeAD should be in upstream. I was just confused weather it is or not. So rust is not directly one of my priorities, sorry.

[jmelo11]

Hi, where can i find information about the status/installation/usage of this integration?

Thanks in advance

[ZuseZ4]

The installation instructions are in our readme: https://github.com/EnzymeAD/rust
Usage examples are on our website: https://enzyme.mit.edu/index.fcgi/rust/usage/usage.html
The current status is that I'm waiting for llvm/llvm-project#83052 to get merged, afterwards for rustc to move to an llvm with this patch integrated, and finally I will propose to upstream this fork.

[jmelo11]

Thank you so much!!

[ZuseZ4]

@ all. The Rust language Team met this morning and decided to approve this as an experiment within rustc!
rust-lang#124509

So we can now actually upstream this. I will start the refactoring in the next days. There is still the unmerged llvm PR which we want to have in, but in the best case I can hopefully push my upstream PR at the end of this month.

[jmelo11]

@ all. The Rust language Team met this morning and decided to approve this as an experiment within rustc! rust-lang#124509

So we can now actually upstream this. I will start the refactoring in the next days. There is still the unmerged llvm PR which we want to have in, but in the best case I can hopefully push my upstream PR at the end of this month.

Hi @ZuseZ4, does this mean that it can be installed via nightly or is this still in progress?

[ZuseZ4]

The first 13 files were upstreamed last night: rust-lang#129176
The remaining ~70 files will probably take a few more weeks to be reviewed and cleaned up. Once everything is upstreamed, I'll flip the build flag to ship them on nightly without people having to build the official rustc themself.