Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Windows RBE Tracking Issue #10619

Closed
sunjayBhatia opened this issue Apr 1, 2020 · 21 comments · Fixed by #11107
Closed

Windows RBE Tracking Issue #10619

sunjayBhatia opened this issue Apr 1, 2020 · 21 comments · Fixed by #11107
Assignees
@sunjayBhatia
Labels
area/build enhancement Feature requests. Not bugs or questions.

Comments

@sunjayBhatia
Copy link
Member

sunjayBhatia commented Apr 1, 2020 •
edited
Loading

Description:

We have started extending and enabling what is run in the Windows CI pipeline and found the Azure Pipelines build environment is severely underpowered to do a fully local build in a timely manner. While we may look in to using a user-managed Azure Pipelines worker, we should also explore the idea of using bazel RBE as the Linux CI does to cut down build times etc. Other projects (namely gRPC and tensorflow) seem to have success with this in their CI.

Questions:

  • Are there any restrictions or limitations to the Google hosted RBE product that would prevent us from using it for the Envoy project CI builds?
    • Since Windows security is key (and the execution environment may not be via Hyper-V isolated containers), is it restricted to Google owned projects only or can we set up Envoy to use it?
  • What is the correct entrypoint to start setting up RBE for Envoy CI? (the envoy-build-tools repo + setting up a toolchain? etc.)

Links:
@htuch htuch added area/build enhancement Feature requests. Not bugs or questions. help wanted Needs help! and removed help wanted Needs help! labels Apr 1, 2020
@htuch
Copy link
Member

htuch commented Apr 1, 2020

@nicolasnoble can you share the gRPC experience with RBE here?

@nicolasnoble
Copy link

cc @jtattermusch - we've been able to run Windows RBE after a lot of tweaks in our environment: https://github.com/grpc/grpc/blob/cc43fd64ab97e45fcdd38759dd8d9eb273823b9e/third_party/toolchains/BUILD#L21

@sunjayBhatia
Copy link
Member Author

Ah cool, this dockerfile should be a good reference: https://github.com/grpc/grpc/blob/master/third_party/toolchains/bazel_0.26.0_rbe_windows/Dockerfile

@sunjayBhatia
Copy link
Member Author

Can you speak to what sorts of tweaks you had to make and if the RBE product is available for us to use with Windows?

@jtattermusch
Copy link

Also see https://github.com/grpc/grpc/pull/22473/files which migrates our stack to Win2019 workers (which is what you should use if you wanna go ahead with windows RBE).

What we've done for RBE windows:

  • we haven't really done any custom tweaks, we basically just followed the user guide for RBE windows. Unfortunately the documentation for RBE windows is internal, so I'm not sure if you are able to access it.
  • the process of setting up Windows RBE is more complicated than linux, because you need to create and build your own Dockerfile and you also need to autogenerate your toolchain (basically what's in int bazel_0.26.0_rbe_windows directory. But these steps are basically just mechanic and they are relatively well documented step by step (if you have access to the docs).

@sunjayBhatia sunjayBhatia mentioned this issue Apr 2, 2020
Merged
@sunjayBhatia
Copy link
Member Author

Unfortunately the documentation for RBE windows is internal, so I'm not sure if you are able to access it

I joined a google group and got access to some docs, not seeing any mention of Windows yet but that is a start, I'll keep looking for more docs and see if I can bother someone to get access

+1 to Windows 2019, we're working on a Dockerfile etc. to get that going and will make a PR to https://github.com/envoyproxy/envoy-build-tools

Thanks for the resources/tips, we will hopefully make some progress soon and maybe have more questions

@sunjayBhatia
Copy link
Member Author

PR for docker image: envoyproxy/envoy-build-tools#40

Confirmed that it locally has the dependencies to build envoy-static.exe and tests

@sunjayBhatia
Copy link
Member Author

so far I don't see any flags to be able to choose the OS of the worker pool VMs in the docs: https://cloud.google.com/remote-build-execution/docs/set-up/remote-environment#create_a_worker_pool, does that sound like something that is available in some further hidden docs?

@sunjayBhatia
Copy link
Member Author

sunjayBhatia commented Apr 3, 2020 •
edited
Loading

@jtattermusch could you shed some light on this comment/link? https://github.com/jtattermusch/grpc/blob/f654f8159778d8b6807ddbd8b20bdeb686a5a819/third_party/toolchains/rbe_win2019/Dockerfile#L62-L63

I believe we're running into the same issue on Azure Pipelines which has a recent/latest version of Docker (19.03.5). We have a local version of Docker thats on an older version (18.09.0) and pacman -Suu works just fine.

@jtattermusch
Copy link

The way a windows pool is setup internally is to create a regular linux pool and the file a ticket with the RBE team to switch your pool to windows OS (it's not self-service). So I guess you'll need to ask the RBE team if this is possible for external users.

@sunjayBhatia
Copy link
Member Author

sunjayBhatia commented Apr 8, 2020 •
edited
Loading

a relevant bazel issue for this work: bazelbuild/bazel#11101

@sunjayBhatia
Copy link
Member Author

sunjayBhatia commented Apr 8, 2020 •
edited
Loading

We've hacked together some changes to bazel-toolchains and envoy-build-tools that let us generate a MSVC toolchain:

@sunjayBhatia
Copy link
Member Author

Our next step is to get a test GCP environment with RBE + Windows enabled to test out our generated toolchain

@sunjayBhatia sunjayBhatia mentioned this issue Apr 9, 2020
Closed
@sunjayBhatia
Copy link
Member Author

bazel-toolchains issue to see if they will take our changes: bazelbuild/bazel-toolchains#855

@sunjayBhatia
Copy link
Member Author

PR to enable remote cache in CI: #10831

@sunjayBhatia sunjayBhatia mentioned this issue Apr 24, 2020
Closed
@sunjayBhatia
Copy link
Member Author

We're testing out our generated toolchain config and toolchain image and with a few tweaks are so far successfully building Envoy with RBE, forthcoming PRs to upstream our work coming soon

@sunjayBhatia
Copy link
Member Author

We've started a Draft PR here to demonstrate a working RBE build on Windows workers: #11107 this should get turned into a full PR once we have upstreamed all our changes to the bazel-toolchains and envoy-build-tools dependencies

@sunjayBhatia
Copy link
Member Author

This build log demonstrates a working build in CI that builds all sources and compiles tests not tagged with skip_on_windows: https://dev.azure.com/cncf/envoy/_build/results?buildId=39115&view=logs&j=2d2b3007-3c5c-5840-9bb0-2b1ea49925f3&t=168f295b-0553-5364-35f7-923225ecd8b3

@sunjayBhatia
Copy link
Member Author

See envoyproxy/envoy-build-tools#64 for changes to envoy-build-tools that will enable generating Windows toolchain configs. All changes required for bazel-toolchains have been merged in: bazelbuild/bazel-toolchains#873

@sunjayBhatia sunjayBhatia mentioned this issue May 28, 2020
Merged
@stale
Copy link

stale bot commented Jun 20, 2020

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or other activity occurs. Thank you for your contributions.

@stale stale bot added the stale stalebot believes this issue/PR has not been touched recently label Jun 20, 2020
@sunjayBhatia
Copy link
Member Author

Our toolchain config changes have all been merged, we're just waiting on fixing some Windows specific issues caused by the merge of recent PRs

@stale stale bot removed the stale stalebot believes this issue/PR has not been touched recently label Jun 22, 2020
lizan pushed a commit that referenced this issue Jun 27, 2020
@sunjayBhatia @wrowe
Windows build: Enable RBE build in Windows CI (#11107)
8614e83 
Commit Message:
Enable RBE build in Windows CI

- Bumps envoy-build-tools and bazel-toolchains
- Bumps envoy-build-ubuntu image tag
  - Running Windows CI via docker uses the same mechanism as Linux to find the image tag, the tags are the same for Windows and Linux images
- Temporarily adds a patch to fix rules_go for Windows RBE issue executing batch scripts, see #11657
- Tags/fixes some additional tests that fail in CI

Additional Description: N/A
Risk Level: Low, but we may observe differences in performance of Windows CI running remotely, we may need to adjust the worker pool accordingly
Testing: N/A
Docs Changes: N/A
Release Notes: N/A
Fixes #10619

Signed-off-by: Sunjay Bhatia <sbhatia@pivotal.io>
Co-authored-by: William A Rowe Jr <wrowe@vmware.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sunjayBhatia sunjayBhatia

Labels
area/build enhancement Feature requests. Not bugs or questions.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Windows build: Enable RBE build in Windows CI
4 participants
@sunjayBhatia @nicolasnoble @jtattermusch @htuch