Skip to content

Commit 8626a79

Browse files
HoloRinHoloRin
committed
Add a partial Bazel build (#2938)
Adds WORKSPACE.bazel, BUILD.bazel & *.bzl files for partial build & test with Bazel. Introduces a build-time dependency on https://github.com/rabbitmq/bazel-erlang
1 parent 08b286c commit 8626a79

File tree

119 files changed

+4088
-166
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

119 files changed

+4088
-166
lines changed

‎.github/workflows/base-images.yaml‎

Lines changed: 0 additions & 52 deletions
This file was deleted.

‎.github/workflows/test.yaml‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
name: Test
2+
on: push
3+
jobs:
4+
test:
5+
name: test
6+
runs-on: ubuntu-18.04
7+
strategy:
8+
matrix:
9+
erlang_version:
10+
- "23"
11+
timeout-minutes: 60
12+
steps:
13+
- name: CHECKOUT REPOSITORY
14+
uses: actions/checkout@v2
15+
- name: CONFIGURE BAZEL
16+
run: |
17+
echo "${{ secrets.BUILDBUDDY_CERT }}" > buildbuddy-cert.pem
18+
echo "${{ secrets.BUILDBUDDY_KEY }}" > buildbuddy-key.pem
19+
cat << EOF >> .bazelrc
20+
build --remote_executor=grpcs://cloud.buildbuddy.io
21+
build --host_platform=//:erlang_${{ matrix.erlang_version }}_platform
22+
build --platforms=//:erlang_${{ matrix.erlang_version }}_platform
23+
build --extra_execution_platforms=//:erlang_${{ matrix.erlang_version }}_platform
24+
25+
build --crosstool_top=@buildbuddy_toolchain//:toolchain
26+
build --extra_toolchains=@buildbuddy_toolchain//:cc_toolchain
27+
build --javabase=@buildbuddy_toolchain//:javabase_jdk8
28+
build --host_javabase=@buildbuddy_toolchain//:javabase_jdk8
29+
build --java_toolchain=@buildbuddy_toolchain//:toolchain_jdk8
30+
build --host_java_toolchain=@buildbuddy_toolchain//:toolchain_jdk8
31+
32+
build --@bazel-erlang//:erlang_version=${{ matrix.erlang_version }}
33+
build --@bazel-erlang//:erlang_home=/usr/lib/erlang
34+
build --//:elixir_home=/usr/local
35+
36+
build --bes_results_url=https://app.buildbuddy.io/invocation/
37+
build --bes_backend=grpcs://cloud.buildbuddy.io
38+
build --remote_cache=grpcs://cloud.buildbuddy.io
39+
build --tls_client_certificate=buildbuddy-cert.pem
40+
build --tls_client_key=buildbuddy-key.pem
41+
build --remote_timeout=1200
42+
43+
build --build_metadata=ROLE=CI
44+
build --remote_instance_name=buildbuddy-io/buildbuddy/ci
45+
build --spawn_strategy=remote
46+
build --jobs=50
47+
EOF
48+
#! - name: Setup tmate session
49+
#! uses: mxschmitt/action-tmate@v3
50+
- name: RUN TESTS
51+
run: |
52+
bazelisk test //... \
53+
--test_tag_filters=-exclusive \
54+
--verbose_failures

‎.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -68,3 +68,6 @@ rabbitmq-server-*.zip
6868

6969
traces*
7070
callgrand*
71+
72+
/.bazelrc
73+
/bazel-*

‎BAZEL.md‎

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
# Experimental [Bazel](https://www.bazel.build/) build
2+
3+
From https://docs.bazel.build/versions/master/bazel-overview.html
4+
> Bazel is an open-source build and test tool similar to Make, Maven, and Gradle. It uses a human-readable, high-level build language. Bazel supports projects in multiple languages and builds outputs for multiple platforms. Bazel supports large codebases across multiple repositories, and large numbers of users.
5+
6+
## Why RabbitMQ + Bazel?
7+
8+
RabbitMQ, Tier1 plugins included, is a large codebase. The developer experience benefits from fast incremental compilation.
9+
10+
More importantly, RabbitMQ's test suite is large and takes hours if run on a single machine. Bazel allows tests to be run in parallel on a large number of remote workers if needed, and furthermore uses cached test results when branches of the codebase remain unchanged.
11+
12+
To the best of our knowledge Bazel does not provide built in Erlang or Elixir support, nor is there an available library of bazel rules. Therefore, we have defined our own rules in https://github.com/rabbitmq/bazel-erlang. Elixir compilation is handled as a special case within this repository. To use these rules, the location of your Erlang and Elixir installations must be indicated to the build (see below).
13+
14+
While most of work for running tests happens in Bazel, the suite still makes use of some external tools for commands, notably gnu `make` and `openssl`. Ideally we could bring all of these tools under bazel, so that the only tool needed would be `bazel` or `bazelisk`, but that will take some time.
15+
16+
## Install Bazelisk
17+
18+
On **macOS**:
19+
20+
`brew install bazelisk`
21+
22+
Otherwise:
23+
24+
https://docs.bazel.build/versions/master/install-bazelisk.html
25+
26+
Additionally, create a `.bazelrc` file with at least:
27+
28+
```
29+
build --@bazel-erlang//:erlang_home=/path/to/erlang/installation
30+
build --@bazel-erlang//:erlang_version=23.1
31+
build --@bazel-erlang//:elixir_home=/path/to/elixir/installation
32+
build --test_strategy=exclusive
33+
```
34+
35+
Additionally, on **macOS**, you will likely need to add
36+
37+
```
38+
build --spawn_strategy=local
39+
```
40+
41+
for certain `rabbitmq_cli` tests to pass. This is because `rabbitmqctl wait` shells out to 'ps', which is broken in the bazel macOS (https://github.com/bazelbuild/bazel/issues/7448).
42+
43+
## Run the broker
44+
45+
`bazel run broker`
46+
47+
## Running tests
48+
49+
Many rabbit tests spawn single or clustered rabbit nodes, and therefore it's best to run test suites sequentially on a single machine. Hence the `--test_strategy=exclusive` flag used in `.bazelrc` above. Naturally that restriction does not hold if utilizing remote execution (as is the case for RabbitMQ's CI pipelines).
50+
51+
Erlang Common Test logs will not be placed in the logs directory when run with bazel. They can be found under `bazel-testlogs`. For instance, those of the rabbit application's backing_queue suite will be under `bazel-testlogs/deps/rabbit/backing_queue_SUITE/test.outputs/`.
52+
53+
### Run all tests
54+
55+
Note: This takes quite some time on a single machine.
56+
57+
`bazel test //...`
58+
59+
### Run tests in a 'package' and its 'subpackages'
60+
61+
**rabbit** is an appropriate example because it encloses the **rabbitmq_prelaunch** application.
62+
63+
`bazel test deps/rabbit/...`
64+
65+
### Run tests for a specific 'package'
66+
67+
`bazel test deps/rabbit_common:all`
68+
69+
### Run an individual common test suite
70+
71+
`bazel test //deps/rabbit:lazy_queue_SUITE`

‎BUILD.bazel‎

Lines changed: 124 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,124 @@
1+
load("elixir_home.bzl", "elixir_home")
2+
load(":rabbitmq_home.bzl", "rabbitmq_home")
3+
load(":rabbitmq_run.bzl", "rabbitmq_run", "rabbitmq_run_command")
4+
load(":rabbitmqctl.bzl", "rabbitmqctl")
5+
load(":rabbitmq.bzl", "management_plugins", "required_plugins")
6+
7+
exports_files([
8+
"scripts/bazel/rabbitmq-run.sh",
9+
])
10+
11+
# This allows us to
12+
# `bazel build //my/target \
13+
# --//:elixir_home=/path/to/elixir/installation`
14+
elixir_home(
15+
name = "elixir_home",
16+
build_setting_default = "~/.kiex/elixirs/elixir-1.10.4",
17+
visibility = ["//visibility:public"],
18+
)
19+
20+
platform(
21+
name = "erlang_22_platform",
22+
constraint_values = [
23+
"@platforms//os:linux",
24+
"@platforms//cpu:x86_64",
25+
"@bazel_tools//tools/cpp:clang",
26+
],
27+
exec_properties = {
28+
"OSFamily": "Linux",
29+
"container-image": "docker://pivotalrabbitmq/rabbitmq-server-buildenv:linux-erlang-22.3",
30+
},
31+
)
32+
33+
platform(
34+
name = "erlang_23_platform",
35+
constraint_values = [
36+
"@platforms//os:linux",
37+
"@platforms//cpu:x86_64",
38+
"@bazel_tools//tools/cpp:clang",
39+
],
40+
exec_properties = {
41+
"OSFamily": "Linux",
42+
"container-image": "docker://pivotalrabbitmq/rabbitmq-server-buildenv:linux-erlang-23.2",
43+
},
44+
)
45+
46+
rabbitmq_home(
47+
name = "broker-home",
48+
plugins = required_plugins(rabbitmq_workspace = "@"),
49+
)
50+
51+
rabbitmq_home(
52+
name = "broker-management-home",
53+
plugins = required_plugins(rabbitmq_workspace = "@") + management_plugins(rabbitmq_workspace = "@"),
54+
)
55+
56+
rabbitmq_home(
57+
name = "broker-for-tests-home",
58+
testonly = True,
59+
plugins = [
60+
"//deps/rabbit:test_bazel_erlang_lib", # <- compiled with test erlc_opts
61+
"@rabbitmq_ct_client_helpers//:bazel_erlang_lib",
62+
"@inet_tcp_proxy//:bazel_erlang_lib",
63+
"@meck//:bazel_erlang_lib",
64+
],
65+
)
66+
67+
rabbitmq_home(
68+
name = "broker-for-cli-tests-home",
69+
plugins = required_plugins(rabbitmq_workspace = "@") + [
70+
"//deps/rabbitmq_federation:bazel_erlang_lib",
71+
"//deps/rabbitmq_stomp:bazel_erlang_lib",
72+
"//deps/amqp_client:bazel_erlang_lib",
73+
],
74+
)
75+
76+
rabbitmq_run(
77+
name = "rabbitmq-run",
78+
home = ":broker-home",
79+
visibility = ["//visibility:public"],
80+
)
81+
82+
rabbitmq_run(
83+
name = "rabbitmq-management-run",
84+
home = ":broker-management-home",
85+
visibility = ["//visibility:public"],
86+
)
87+
88+
rabbitmq_run(
89+
name = "rabbitmq-for-tests-run",
90+
testonly = True,
91+
home = ":broker-for-tests-home",
92+
visibility = ["//visibility:public"],
93+
)
94+
95+
rabbitmq_run(
96+
name = "rabbitmq-for-cli-tests-run",
97+
testonly = True,
98+
home = ":broker-for-cli-tests-home",
99+
visibility = ["//visibility:public"],
100+
)
101+
102+
# Allow us to `bazel run broker`
103+
# for the equivalent of `make run-broker`
104+
# (though it as of yet includes no plugins)
105+
rabbitmq_run_command(
106+
name = "broker",
107+
rabbitmq_run = ":rabbitmq-run",
108+
subcommand = "run-broker",
109+
)
110+
111+
# `bazel run broker-management` for the broker with just the
112+
# management plugin
113+
rabbitmq_run_command(
114+
name = "broker-management",
115+
rabbitmq_run = ":rabbitmq-management-run",
116+
subcommand = "run-broker",
117+
)
118+
119+
# `bazel run rabbitmqctl`
120+
rabbitmqctl(
121+
name = "rabbitmqctl",
122+
home = ":broker-home",
123+
visibility = ["//visibility:public"],
124+
)

‎BUILD.inet_tcp_proxy‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
load("@bazel-erlang//:bazel_erlang_lib.bzl", "erlang_lib")
2+
3+
erlang_lib(
4+
app_name = "inet_tcp_proxy_dist",
5+
app_version = "0.1.0",
6+
app_description = "Erlang distribution proxy to simulate network failures",
7+
app_module = "inet_tcp_proxy_dist_app",
8+
)

‎BUILD.osiris‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
load("@bazel-erlang//:bazel_erlang_lib.bzl", "erlang_lib")
2+
3+
APP_ENV = """[
4+
{data_dir, "/tmp/osiris"},
5+
{port_range, {6000, 6500}}
6+
]"""
7+
8+
DEPS = [
9+
"@gen_batch_server//:bazel_erlang_lib",
10+
]
11+
12+
erlang_lib(
13+
app_name = "osiris",
14+
app_version = "master",
15+
app_description = "New project",
16+
app_module = "osiris_app",
17+
app_env = APP_ENV,
18+
extra_apps = [
19+
"sasl",
20+
"crypto",
21+
],
22+
deps = DEPS,
23+
)

‎BUILD.ra‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
load("@bazel-erlang//:bazel_erlang_lib.bzl", "erlang_lib", "test_erlang_lib")
2+
3+
DEPS = [
4+
"@gen_batch_server//:bazel_erlang_lib",
5+
]
6+
7+
RUNTIME_DEPS = [
8+
"@aten//:bazel_erlang_lib",
9+
]
10+
11+
NAME = "ra"
12+
13+
FIRST_SRCS = [
14+
"src/ra_machine.erl",
15+
"src/ra_snapshot.erl",
16+
]
17+
18+
erlang_lib(
19+
app_name = NAME,
20+
first_srcs = FIRST_SRCS,
21+
deps = DEPS,
22+
runtime_deps = RUNTIME_DEPS,
23+
)
24+
25+
test_erlang_lib(
26+
app_name = NAME,
27+
first_srcs = FIRST_SRCS,
28+
erlc_opts = [
29+
"+debug_info",
30+
],
31+
deps = DEPS,
32+
runtime_deps = RUNTIME_DEPS,
33+
)

0 commit comments

Comments
 (0)