upgrade protobuf to v2.0 and gen binding file in the build script #171

flier · 2018-05-11T03:24:04Z

The pre-generated binding file doesn't work for protobuf v1.6, we should generate it in the build script.

error[E0433]: failed to resolve. Could not find `MessageStatic` in `protobuf`
   --> /Users/fl0120/.cargo/registry/src/github.com-1ecc6299db9ec823/prometheus-0.4.1/src/../proto/metrics.rs:213:21
    |
213 |         ::protobuf::MessageStatic::descriptor_static(None::<Self>)
    |                     ^^^^^^^^^^^^^ Could not find `MessageStatic` in `protobuf`

breezewish · 2018-05-12T08:38:45Z

Thanks! but CI failed. Could you make a fix for that?

siddontang · 2018-05-14T01:53:41Z

any update @flier

flier · 2018-05-14T03:33:09Z

It seems the protoc-rust depend on the protoc program which need be installed before build the crate.

breezewish · 2018-05-14T08:43:22Z

Cargo.toml

@@ -27,13 +27,14 @@ dev = ["clippy"]
 nightly = ["libc", "spin/unstable"]
 push = ["hyper", "libc"]
 process = ["libc", "procinfo"]
+gen = []


I would like to know why do we need to add a new feature? What will happen if user enables / disables this feature?

It seems the protoc-rust depend on the protoc program which need be installed before build the crate.

The feature provide to the library developer who need use the latest protobuf version and installed the protoc program, and it should be disabled to make more easier with pre-generated binding file.

Can you please add this to the README or a comment in the Cargo.toml so that people will know about it?

Note

The crate has a pre-generated protobuf binding file for protobuf v1.6, if you need use the latest version of protobuf, you can generate the binding file on building with the gen feature after you download and install the protocol buffer compiler.

[dependencies.prometheus] git = "https://github.com/pingcap/rust-prometheus.git" features = ["gen"]

breezewish · 2018-05-14T08:43:46Z

build.rs

+            customize: protoc_rust::Customize {
+                ..Default::default()
+            },
+        }).expect("protoc");


can be simplified to .unwrap()

breezewish · 2018-05-14T08:43:59Z

build.rs

+            out_dir: "proto",
+            input: &["proto/metrics.proto"],
+            includes: &["proto"],
+            customize: protoc_rust::Customize {


can we use protoc_rust::Customize::default() here?

sure, fixed, just copy from rust-protobuf example code

This will allow builds until tikv/rust-prometheus#171 lands

siddontang · 2018-05-17T08:29:57Z

LGTM

PTAL @overvenus

breezewish · 2018-05-22T05:13:39Z

LGTM!

breezewish · 2018-05-22T05:14:50Z

CI failed. Seems that protobuf 1.7 comes, and introduced new breaking changes..

flier · 2018-05-22T05:26:48Z

@breeswish That's why we need a build script to generate the binding file on building :)

breezewish · 2018-05-22T05:31:19Z

@flier Agree.. Honestly I prefer to opt-in dynamic generation. However it requires protobuf to be installed, which is not very ideal..

breezewish · 2018-05-22T06:07:32Z

I took a look at protobuf's changelog, and it seems that they republished the breaking change into 2.0 version, and 1.7 should be the same as 1.5. I think it would be better to target at protobuf 2.0 for us? @flier

breezewish · 2018-05-22T06:09:32Z

Also I noticed that there is a pure rust codegen now. Maybe we can utilize that so that no more C++ protobuf dependency is required and we can safely generate proto/metrics.rs on the fly for all users?

breezewish · 2018-05-22T07:32:19Z

LGTM! What will happen if we don't provide a pre-generated file, instead we always generate one at build?

flier · 2018-05-22T08:27:52Z

sure, after we changed to use pure Rust protoc, it will be fine; on the other hand, the pre-generated binding file will make building a little faster and reduce the troubleshooting cost.

overvenus

Thank you for the PR!

overvenus · 2018-05-22T09:30:00Z

build.rs

+fn main() {
+    if cfg!(feature = "gen") {
+        protobuf_codegen_pure::run(protobuf_codegen_pure::Args {
+            out_dir: "proto",


Does not use the "proto", because it will modify the metrics.rs in the local registry, also if we build multiple projects that deps on the same version prometheus, the file may be corrupted due to multiple writers. Instead you can try OUT_DIR[0]. Here is an example [1].

[0] https://doc.rust-lang.org/cargo/reference/build-scripts.html#case-study-code-generation
[1] tikv/tikv#952

The problem is the generated binding file contains a lot of module attributes, that means we can't use the include! macro to include the automatic generated binding file on the building time. The path attribute could do it, but it doesn't accept a environment variable or concat! macro, it only accept the static relative path.

#[path = "../proto/metrics.rs"] pub mod proto;

Try include!(concat!(env!("OUT_DIR"), "/metrics.rs")); + cfg_if! {}. 😉

#[cfg(feature = "gen")] pub mod proto { include!(concat!(env!("OUT_DIR"), "/metrics.rs")); }

that's will cause a compile error, include! macro fails if included file has top-level inner attributes

error: an inner attribute is not permitted following an outer attribute
--> /Users/fl0120/github/rust-prometheus/target/debug/build/prometheus-52580fdca4c6c951/out/metrics.rs:12:3
|
12 | #![allow(missing_docs)]
| ^
|
= note: inner attributes, like #![no_std], annotate the item enclosing them, and are usually found at the beginning of source files. Outer attributes, like #[test], annotate the item following them.

overvenus · 2018-05-22T09:42:12Z

Cargo.toml

@@ -56,5 +57,8 @@ optional = true
 getopts = "0.2"
 hyper = {version = "0.9", default-features = false}

+[build-dependencies]
+protobuf-codegen-pure = "2.0"


Could you make it an optional dependency?

overvenus · 2018-05-23T02:32:59Z

@flier Thank you!

flier · 2018-05-23T02:34:29Z

@overvenus @breeswish Thanks

…kv#171)

upgrade protobuf to v1.6 and gen binding file in the build script

72003b9

auto generate protobuf binding file when set gen feature

b91c711

Merge branch 'master' into protobuf-1.6

bf1f741

breezewish reviewed May 14, 2018

View reviewed changes

fix code review issues

1de43c4

acdha added a commit to acdha/mountstatus that referenced this pull request May 16, 2018

Pin compatible version of protobuf / Prometheus

83f6707

This will allow builds until tikv/rust-prometheus#171 lands

siddontang mentioned this pull request May 17, 2018

Incompatibility with protobuf 1.6 #174

Closed

flier added 3 commits May 22, 2018 11:17

add usage for the gen feature

99acbb3

fix a typo

c8e5001

fix typo again

40357d0

upgrade protobuf to 1.7

ea2c495

flier changed the title ~~upgrade protobuf to v1.6 and gen binding file in the build script~~ upgrade protobuf to v1.7 and gen binding file in the build script May 22, 2018

upgrade protobuf to v2.0 and use pure rust generator

65a01b7

flier changed the title ~~upgrade protobuf to v1.7 and gen binding file in the build script~~ upgrade protobuf to v2.0 and gen binding file in the build script May 22, 2018

overvenus reviewed May 22, 2018

View reviewed changes

make protobuf-codegen-pure optional

c2cd353

overvenus approved these changes May 23, 2018

View reviewed changes

overvenus merged commit 8724e5a into tikv:master May 23, 2018

azdlowry pushed a commit to azdlowry/rust-prometheus that referenced this pull request Sep 10, 2018

upgrade protobuf to v2.0 and gen binding file in the build script (ti…

7d60772

…kv#171)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

upgrade protobuf to v2.0 and gen binding file in the build script #171

upgrade protobuf to v2.0 and gen binding file in the build script #171

flier commented May 11, 2018

breezewish commented May 12, 2018

siddontang commented May 14, 2018

flier commented May 14, 2018

breezewish May 14, 2018

flier May 14, 2018

Hoverbear May 17, 2018

flier May 22, 2018 •

edited

Loading

breezewish May 14, 2018

breezewish May 14, 2018

flier May 14, 2018

siddontang commented May 17, 2018

breezewish commented May 22, 2018

breezewish commented May 22, 2018

flier commented May 22, 2018

breezewish commented May 22, 2018

breezewish commented May 22, 2018

breezewish commented May 22, 2018 •

edited

Loading

breezewish commented May 22, 2018

flier commented May 22, 2018

overvenus left a comment

overvenus May 22, 2018

flier May 22, 2018

overvenus May 22, 2018

flier May 23, 2018 •

edited

Loading

overvenus May 22, 2018

flier May 22, 2018

overvenus commented May 23, 2018

flier commented May 23, 2018

upgrade protobuf to v2.0 and gen binding file in the build script #171

upgrade protobuf to v2.0 and gen binding file in the build script #171

Conversation

flier commented May 11, 2018

breezewish commented May 12, 2018

siddontang commented May 14, 2018

flier commented May 14, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

flier May 22, 2018 • edited Loading

Choose a reason for hiding this comment

Note

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

siddontang commented May 17, 2018

breezewish commented May 22, 2018

breezewish commented May 22, 2018

flier commented May 22, 2018

breezewish commented May 22, 2018

breezewish commented May 22, 2018

breezewish commented May 22, 2018 • edited Loading

breezewish commented May 22, 2018

flier commented May 22, 2018

overvenus left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

flier May 23, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

overvenus commented May 23, 2018

flier commented May 23, 2018

flier May 22, 2018 •

edited

Loading

breezewish commented May 22, 2018 •

edited

Loading

flier May 23, 2018 •

edited

Loading