Support for generated sources, when building out of tree with non-cargo build systems

[dcbaker]

For build systems other than cargo, it is generally considered best practice to build out of tree (ie, generated files go into a directory that is not tracked by version control). With rustc this works great, except when you need to generate rust sources. In that case a build system (meson in this case, but cmake and gnu autotools would be affected too) will place the generated .rs file in the build tree, while the hard written sources are in the source tree. As far as I can find there is no way to tell rustc about this and get it to compile successfully, without resorting to ugly hacks like "copy all of the files from the source tree to the build tree." While this can work, it's rather ugly and error prone.

[sfackler]

What's wrong with the approach Cargo uses of setting the OUT_DIR environment variable at compilation time?

[dcbaker]

For one, I can't set env variables at compile time, ninja doesn't support them, and never will. I can't change that. But that wouldn't solve the problem that my hand written files live in a source tree (which in many cases is read only) and my generated files are in my build tree, which is the only writeable tree I have

[sfackler]

But that wouldn't solve the problem that my hand written files live in a source tree (which in many cases is read only) and my generated files are in my build tree, which is the only writeable tree I have

Why is that a problem?

[karolherbst]

But that wouldn't solve the problem that my hand written files live in a source tree (which in many cases is read only) and my generated files are in my build tree, which is the only writeable tree I have

Why is that a problem?

I've discussed this with @dcbaker and there was just a small missunderstanding on how all the include! stuff works out. In the end we see two use cases

1. include generated files via include!. This needs support for specifying arbitrary env variables which Ninja does not support. So we can't use this
2. Include generated files directly into the source code tree via normal mod declarations. Doesn't work as rustc doesn't really support split source code trees.

For 1. we could write a wrapper to inject arbitrary env variables, but this solution kind of sucks
For 2. we could merge all source code trees into one big tree within the build directory and build from there.

What rustc could help with is to have a stable interface to inject arbitrary data (something alike -D from C/C++ compilers) or allow build systems to have split source code trees.

[dcbaker]

for 1 what would be really useful is an option like --cfg, but for env, so you could do --env SOURCE_TO_INCLUDE="<some value calculated by the build system>". I can workaround that with a wrapper, but it would be much nicer to not have to do that if possible.

for 2 what I have in mind is that we have in our tree a lot of code generated from xml descriptions, but that code additionally needs a lot of handwritten code to be useful. so you'd have something like:

src/mything/interface.rs
build/src/mything/generated_from_xml_backend1.rs
build/src/mything/generated_from_xml_backend2.rs 
...

I'd really like to be able to have an option like (I don't really care what it looks like, but as a strawman):
rustc src/mything/interface.rs --build-tree=../build/src/mything, and have rustc effectively merge the two trees when doing lookups.

[jpakkane]

For 1. we could write a wrapper to inject arbitrary env variables, but this solution kind of sucks

Please don't. Environment variables are an awful way of configuring anything. The compiler switch way is better in every way.

[dcbaker]

I don't think this issue is really specific for what we need, I'm going to close this and open a different one.