Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add a key to ctx.action dict to prevent protoc losing the default env #2508

Merged
merged 1 commit into from
Apr 19, 2017

Conversation

yliu120
Copy link
Contributor

@yliu120 yliu120 commented Dec 16, 2016

This is a small change on the protobuf.bzl but very reasonable and useful.

In practice, a few protobuf client (like tensorflow, see below and etc) would call _proto_gen_impl(ctx) indirectly to generate proto implementations. However by the current definition of bazel's ctx.action rule (http://www.bazel.io/versions/master/docs/skylark/lib/ctx.html#action), the default user's custom shell env (PATH/LD_LIBRARY_PATH) is not passed to the compiled protoc binary because the user_default_shell_env is set to be False as default. This will invoke link error cuz users may link protoc with a customized library. There are a lot of reports on this issue.

In principle, when calling _proto_gen_impl(ctx) to generate proto impls, we should pass user's env to the protoc binary.

Another small change is to delete an extra space. (a bonus created by my vim editor).

Please see some issues related to this small CL:
tensorflow/tensorflow#3261

This change will not affect any other reasonable implementations.

@grpc-kokoro
Copy link

Thanks for your pull request. The automated tests will run as soon as one of the admins verifies this change is ok for us to run on our infrastructure.

1 similar comment
@grpc-kokoro
Copy link

Thanks for your pull request. The automated tests will run as soon as one of the admins verifies this change is ok for us to run on our infrastructure.

@bazel-io
Copy link

Can one of the admins verify this patch?

@googlebot
Copy link

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed, please reply here (e.g. I signed it!) and we'll verify. Thanks.


  • If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
  • If you signed the CLA as a corporation, please let us know the company's name.

@yliu120
Copy link
Contributor Author

yliu120 commented Dec 16, 2016

I signed it!

@googlebot
Copy link

CLAs look good, thanks!

jtriley added a commit to fasrc/helmod that referenced this pull request Jan 4, 2017
@mitar
Copy link

mitar commented Jan 24, 2017

This really solves a problem compiling protobuf with tensorflow with environment variables.

@htuch
Copy link
Contributor

htuch commented Mar 26, 2017

+1. Can we get a merge on this? This is very useful when building with non-default toolchains and setting LD_LIBRARY_PATH.

@xfxyjwf
Copy link
Contributor

xfxyjwf commented Apr 6, 2017

@cgrushko Can you help review this PR?

@xfxyjwf
Copy link
Contributor

xfxyjwf commented Apr 6, 2017

@cgrushko The change seems good to me, but given that default value of that flag is False, I'm wondering if there are concerns about making it True. Maybe bazel team has other recommended solutions for overriding CC/LD_LIBRARY_PATH?

@yliu120
Copy link
Contributor Author

yliu120 commented Apr 6, 2017

@xfxyjwf @cgrushko Thanks for LGTM and the comment. I guess Bazel team made that flag False as default just because Bazel usually will let the user define a customized compiler toolchain in a crosstool file, so it means Bazel could setup an internal version of customized environment and always pass them to the binary command right before it (it doesn't export it to create global side effects). When this is true, then it will certainly not use the default shell 's environment.

However, the case here is:
Some package depends on protobuf. Like tensorflow or other protobuf clients, when users compile their code from source, Bazel will pull protobuf down first and compile protobuf using a user-customized environment. It will create a protoc binary and then use that binary to compile .proto source files. Bazel will use the gen_rule defined here to compile those proto files. Bazel may have no responsibility for passing its internal customized envs to proto gen rules since they have already been defined in this file. (I may not be correct). So for not dropping those envs, we really should add the customized env back.

@htuch
Copy link
Contributor

htuch commented Apr 14, 2017

@yliu120 is correctly interpreting the use of this attribute - it's intended to allow the user to override. Bazel accepts CC/LD_LIBRARY_PATH and autogenerates a toolchain when doing so, see https://www.bazel.build/blog/2016/03/31/autoconfiguration.html. This avoids the extremely heavyweight notion of writing a CROSSTOOL everytime you want to point the compiler at a non-standard location.

@xfxyjwf Can we merge this? It's something we'd like for the Envoy Bazel build, to allow it to be used internally in Google where we point CC at crosstools when using the OSS version from https://github.com/lyft/envoy/.

@xfxyjwf xfxyjwf merged commit 357afc3 into protocolbuffers:master Apr 19, 2017
@yliu120 yliu120 deleted the pass_default_env_to_protoc branch April 19, 2017 19:52
PiotrSikora added a commit to PiotrSikora/proxy that referenced this pull request May 17, 2017
Update to master branch, which includes a fix for cc_proto_library() to
make it work in custom build environment (CC/CXX/LD_LIBRARY_PATH), see:
protocolbuffers/protobuf#2508

Signed-off-by: Piotr Sikora <piotrsikora@google.com>
htuch added a commit to htuch/envoy that referenced this pull request Jul 20, 2017
* Use protobuf from HEAD to ensure we have the
  protocolbuffers/protobuf#3327 fix for CI and local
  builds. Users can opt to use a specific release tag with the
  ENVOY_PROTOBUF_COMMIT env var.

* Remove the workaround for protocolbuffers/protobuf#3327
  in do_ci.sh.

* Remove the multiple protobuf versions that existed because
  protocolbuffers/protobuf#2508 wasn't merged.

* Add some evil symlink stuff to get @protobuf_bzl inferred from
  wherever WORKSPACE points the envoy_dependencies path at.

As a bonus, enabled more verbose build of external deps so we don't sit
around for minutes on initial checkout with no activity indicator. This
can be done safely now as everyone should be at Bazel 0.5.2.
htuch added a commit to htuch/envoy that referenced this pull request Jul 21, 2017
* Remove the workaround for protocolbuffers/protobuf#3327
  in do_ci.sh.

* Remove the multiple protobuf versions that existed because
  protocolbuffers/protobuf#2508 wasn't merged.

* Add some evil symlink stuff to get @protobuf_bzl inferred from
  wherever WORKSPACE points the envoy_dependencies path at.

As a bonus, enabled more verbose build of external deps so we don't sit
around for minutes on initial checkout with no activity indicator. This
can be done safely now as everyone should be at Bazel 0.5.2.
htuch added a commit to envoyproxy/envoy that referenced this pull request Jul 23, 2017
* build: undo some protobuf hacks, put some new ones in.

* Remove the workaround for protocolbuffers/protobuf#3327
  in do_ci.sh.

* Remove the multiple protobuf versions that existed because
  protocolbuffers/protobuf#2508 wasn't merged.

* Add some evil symlink stuff to get @protobuf_bzl inferred from
  wherever WORKSPACE points the envoy_dependencies path at.

As a bonus, enabled more verbose build of external deps so we don't sit
around for minutes on initial checkout with no activity indicator. This
can be done safely now as everyone should be at Bazel 0.5.2.

* Don't put verbose Bazel spew into query targets for coverage.

* Always start from a clean test/coverage/BUILD, don't include it in the query.
htuch added a commit to htuch/rules_go that referenced this pull request Aug 31, 2017
Similar to protocolbuffers/protobuf#2508, when using
protoc with non-standard LD_LIBRARY_PATH, we need to be able to have it
pick up from the environment the correct LD_LIBRARY_PATH.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants