Can't use include! macro on generated files

[a1ien]

Last cargo change rust-lang/cargo#5584 deny modify src directory.
Guide lines suggest use OUT_DIR https://doc.rust-lang.org/cargo/reference/build-scripts.html#case-study-code-generation
And include! file in src.
But generated files contain many attributes like #![allow(unknown_lints)] and this file can't be include. rust-lang/rfcs#752
May be add option to remove all attributes in generated files?

[stepancheg]

I'm OK with having an option to skip generating top-level file attributes.

However, I think the issue should probably be fixed differently.

rust-protobuf should generate not just a set of proto files, but also mod.rs which lists all protobuf files, and that mod.rs file should be included.

And for now, you may generate that mod.rs in OUT_DIR file from build.rs, and include generated mod.rs from the src directory.

[SirVer]

@stepancheg Do you have an example implementation ready of what you are suggesting above, i.e. generating mod.rs and including that?

Related, but maybe different: can I make rust protobuf output a hierarchy of protos, ideally following their package names as directory layout? I need to compile ~100 protos that are scattered all over my repo.

[stepancheg]

Do you have an example implementation ready of what you are suggesting above, i.e. generating mod.rs and including that?

Nope sorry.

can I make rust protobuf output a hierarchy of protos

That is on my very long todo list.

ideally following their package names as directory layout

I think it probably should use filesystem hierarchy for layout instead.

Looking at Google's implementation (protoc):

% cat bbb/aa.proto
syntax = "proto3";

package ppppp;

message FooBar {}

% cat gen.sh
#!/bin/sh -e

for gen in cpp csharp java js objc php python ruby; do
    mkdir ${gen}_out
    protoc --${gen}_out=${gen}_out bbb/aa.proto
done

cpp_out/bbb/aa.pb.cc
cpp_out/bbb/aa.pb.h
csharp_out/Aa.cs
java_out/ppppp/Aa.java
js_out/foobar.js
objc_out/bbb/Aa.pbobjc.h
objc_out/bbb/Aa.pbobjc.m
php_out/Ppppp/FooBar.php
php_out/GPBMetadata/Bbb/Aa.php
python_out/bbb/aa_pb2.py
ruby_out/bbb/aa_pb.rb

Which means that PHP and Java generators use packages for layout, and remaining generators use filesystem layout (and Java uses both package and proto file name). JS and C# use no namespaces. All other generators use filesystem layout.

[SirVer]

I think it probably should use filesystem hierarchy for layout instead.

I agree that the choice is not perfectly clear here. For C++ and others that support namespacing, the layout is not really important - the access pattern in code is dictated by the namespaces. For Python, google ignores the package directive and only uses the directory structure.

Rust is kinda in the middle here. It supports namespacing, but also defaults to using the directory layout for namespaces through mod filename. The namespacing can be overwritten with the include! directive and manual mod statements. I'd suggest either prescribing that all protos are following the directory structure and then ignore the package directive (like Python), or creating a new directory structure that follows the package directives, so that namespaces and directory layouts are the same.

I have a slight preference for the ladder, i.e. not following the example of Python. But I do not feel strongly.

[TeXitoi]

A workaround inspired by #324 (comment) : https://github.com/TeXitoi/osmpbfreader-rs/blob/master/build.rs

[ancwrd1]

I ended up with the following solution which works great. Assumes .proto files are under proto directory. It will compile all proto files into respective Rust modules and include them automatically for compilation.

build.rs:

use std::{
    env, fs,
    ops::Deref,
    path::{Path, PathBuf},
};

fn out_dir() -> PathBuf {
    Path::new(&env::var("OUT_DIR").expect("env")).join("proto")
}

fn cleanup() {
    let _ = fs::remove_dir_all(&out_dir());
}

fn compile() {
    let proto_dir = Path::new(&env::var("CARGO_MANIFEST_DIR").expect("env")).join("proto");

    let files = glob::glob(&proto_dir.join("**/*.proto").to_string_lossy())
        .expect("glob")
        .filter_map(|p| p.ok().map(|p| p.to_string_lossy().into_owned()))
        .collect::<Vec<_>>();

    let slices = files.iter().map(Deref::deref).collect::<Vec<_>>();

    let out_dir = out_dir();
    fs::create_dir(&out_dir).expect("create_dir");

    protobuf_codegen_pure::run(protobuf_codegen_pure::Args {
        out_dir: &out_dir.to_string_lossy(),
        input: &slices,
        includes: &[&proto_dir.to_string_lossy()],
        customize: protobuf_codegen_pure::Customize {
            carllerche_bytes_for_bytes: Some(true),
            ..Default::default()
        },
    })
    .expect("protoc");
}

fn generate_mod_rs() {
    let out_dir = out_dir();

    let mods = glob::glob(&out_dir.join("*.rs").to_string_lossy())
        .expect("glob")
        .filter_map(|p| {
            p.ok()
                .map(|p| format!("pub mod {};", p.file_stem().unwrap().to_string_lossy()))
        })
        .collect::<Vec<_>>()
        .join("\n");

    let mod_rs = out_dir.join("mod.rs");
    fs::write(&mod_rs, format!("// @generated\n{}\n", mods)).expect("write");

    println!("cargo:rustc-env=PROTO_MOD_RS={}", mod_rs.to_string_lossy());
}

fn main() {
    cleanup();
    compile();
    generate_mod_rs();
}

src/proto/mod.rs:

// generated protobuf files will be included here. See build.rs for details
#![allow(renamed_and_removed_lints)]

include!(env!("PROTO_MOD_RS"));

[mverleg]

I''m also generating a listing file and including that like @Dremon , and I'm satisfied with that approach. Would be nice if something like that were included in the library so it can be reused.

[ianthetechie]

Thanks so much @ancwrd1! Great workaround! I was quite confused by the docs myself, as this is a big red flag to cargo.

[stepancheg]

New rust-protobuf has an option gen_mod_rs, which is on by default in rust-protobuf=3.