The rules justbuild uses for itself also support protocol buffers. This tutorial shows how to use those rules and the targets associated with them. It is not a tutorial on protocol buffers itself; rather, it is assumed that the reader has some knowledge on protocol buffers.
The protobuf repository conveniently contains an
example,
so we can use this and just add our own target files. We create
file repos.json as follows.
{ "repositories":
{ "":
{ "repository": "protobuf"
, "target_root": "tutorial"
, "bindings": {"rules": "rules"}
}
, "tutorial": {"repository": {"type": "file", "path": "."}}
, ...
}
}The remaining entries are just the ones from etc/repos.json from
the justbuild repository; here we keep in mind that we’re now working
in a different location, and hence have to replace the relative paths
by appropriate absolute ones.
To build the example with just, the only task is to write a targets
file at examples/TARGETS. As that contains a couple of new concepts,
we will do this step by step.
First, we have to declare the proto library. In this case, it only contains one file and has no dependencies.
{ "address":
{ "type": ["@", "rules", "proto", "library"]
, "name": ["addressbook"]
, "srcs": ["addressbook.proto"]
}
...In general, proto libraries could also depend on other proto libraries;
those would be added to the "deps" field.
When building the library, there’s very little to do.
$ just-mr build examples address
INFO: Requested target is [["@","","examples","address"],{}]
INFO: Analysed target [["@","","examples","address"],{}]
INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching
INFO: Discovered 0 actions, 0 trees, 0 blobs
INFO: Building [["@","","examples","address"],{}].
INFO: Processed 0 actions, 0 cache hits.
INFO: Artifacts built, logical paths are:
$On the other hand, what did we expect? A proto library is an abstract description of a protocol, so, as long as we don’t specify for which language we want to have bindings, there is nothing to generate.
Nevertheless, a proto library target is not empty. In fact, it can’t be empty,
as other targets can only access the values of a target and have no
insights into its definitions. We already relied on this design principle
implicitly, when we exploited target-level caching for our external dependencies
and did not even construct the dependency graph for that target. A proto
library simply provides the dependency structure of the .proto files.
$ just-mr analyse --dump-nodes - examples address
INFO: Requested target is [["@","","examples","address"],{}]
INFO: Result of target [["@","","examples","address"],{}]: {
"artifacts": {
},
"provides": {
"proto": [
{"id":"2a483a2de7f25c1bc066e47245f55ec9a2d4a719","type":"NODE"}
]
},
"runfiles": {
}
}
INFO: Target nodes of target [["@","","examples","address"],{}]:
{
"089f6cae7ca77bb786578d3e0138b6ff445c5c92": {
"result": {
"artifact_stage": {
"addressbook.proto": {
"data": {
"file_type": "f",
"id": "b4b33b4c658924f0321ab4e7a9dc9cf8da1acec3",
"size": 1234
},
"type": "KNOWN"
}
},
"provides": {
},
"runfiles": {
}
},
"type": "VALUE_NODE"
},
"2a483a2de7f25c1bc066e47245f55ec9a2d4a719": {
"node_type": "library",
"string_fields": {
"name": ["addressbook"],
"stage": [""]
},
"target_fields": {
"deps": [],
"srcs": [{"id":"089f6cae7ca77bb786578d3e0138b6ff445c5c92","type":"NODE"}]
},
"type": "ABSTRACT_NODE"
}
}
$The target has one provider "proto", which is a node. Nodes are
an abstract representation of a target graph. More precisely, there
are two kind of nodes, and our example contains one of each.
The simple kind of nodes are the value nodes; they represent a
target that has a fixed value, and hence are given by artifacts,
runfiles, and provided data. In our case, we have one value node,
the one for the .proto file.
The other kind of nodes are the abstract nodes. They describe the
arguments for a target, but only have an abstract name (i.e., a
string) for the rule. Combining such an abstract target with a
binding for the abstract rule names gives a concrete “anonymous”
target that, in our case, will generate the library with the bindings
for the concrete language. In this example, the abstract name is
"library". The alternative in our proto rules would have been
"service library", for proto libraries that also contain rpc
definitions (which is used by gRPC).
Using proto libraries requires, as discussed, bindings for the
abstract names. Fortunately, our CC rules are aware of proto
libraries, so we can simply use them. Our target file hence
continues as follows.
...
, "add_person":
{ "type": ["@", "rules", "CC", "binary"]
, "name": ["add_person"]
, "srcs": ["add_person.cc"]
, "proto": ["address"]
}
, "list_people":
{ "type": ["@", "rules", "CC", "binary"]
, "name": ["list_people"]
, "srcs": ["list_people.cc"]
, "proto": ["address"]
}
...The first time, we build a target that requires the proto compiler (in that particular version, built in that particular way), it takes a bit of time, as the proto compiler has to be built. But in follow-up builds, also in different projects, the target-level cache is filled already.
$ just-mr build examples add_person
...
$ just-mr build examples add_person
INFO: Requested target is [["@","","examples","add_person"],{}]
INFO: Analysed target [["@","","examples","add_person"],{}]
INFO: Export targets found: 3 cached, 0 uncached, 0 not eligible for caching
INFO: Discovered 5 actions, 2 trees, 0 blobs
INFO: Building [["@","","examples","add_person"],{}].
INFO: Processed 5 actions, 5 cache hits.
INFO: Artifacts built, logical paths are:
add_person [7210834b05139defe783811d77087aa7c256405c:1980320:x]
$If we look at the actions associated with the binary, we find that those are still the two actions we expect: a compile action and a link action.
$ just-mr analyse examples add_person --dump-actions -
INFO: Requested target is [["@","","examples","add_person"],{}]
INFO: Result of target [["@","","examples","add_person"],{}]: {
"artifacts": {
"add_person": {"data":{"id":"51f7e29f0669608f9e0a0d8c8f4946c239a3ed09","path":"add_person"},"type":"ACTION"}
},
"provides": {
},
"runfiles": {
}
}
INFO: Actions for target [["@","","examples","add_person"],{}]:
[
{
"command": ["clang++","-std=c++20","-O2","-Wall","-Wextra","-Wpedantic","-Wsign-conversion","-I","work","-isystem","include","-c","work/add_person.cc","-o","work/add_person.o"],
"env": {
"PATH": "/bin:/sbin:/usr/bin:/usr/sbin"
},
"input": {
...
}
},
"output": ["work/add_person.o"]
},
{
"command": ["clang++","-o","add_person","add_person.o","libaddressbook.a","libprotobuf.a","libprotobuf_lite.a","libzlib.a"],
"env": {
"PATH": "/bin:/sbin:/usr/bin:/usr/sbin"
},
"input": {
...
},
"output": ["add_person"]
}
]
$As discussed, the libaddressbook.a that is conveniently available
during the linking of the binary (as well as the addressbook.pb.h
available in the include tree for the compile action) are generated
by an anonymous target. Using that during the build we already
filled the target-level cache, we can have a look at all targets
still analysed. In the one anonymous target, we find again the
abstract node we discussed earlier.
$ just-mr analyse examples add_person --dump-targets -
INFO: Requested target is [["@","","examples","add_person"],{}]
INFO: Result of target [["@","","examples","add_person"],{}]: {
"artifacts": {
"add_person": {"data":{"id":"51f7e29f0669608f9e0a0d8c8f4946c239a3ed09","path":"add_person"},"type":"ACTION"}
},
"provides": {
},
"runfiles": {
}
}
INFO: List of analysed targets:
{
"#": {
"acde278315be59c6bdf436efa9dc9782a6c59f36": {
"2a483a2de7f25c1bc066e47245f55ec9a2d4a719": [{"AR":null,"ARCH":null,"CC":null,"CFLAGS":null,"CXX":null,"CXXFLAGS":null,"ENV":null,"HOST_ARCH":null,"OS":null,"TARGET_ARCH":null}]
}
},
"@": {
"": {
"examples": {
"add_person": [{"AR":null,"ARCH":null,"CC":null,"CFLAGS":null,"CXX":null,"CXXFLAGS":null,"ENV":null,"HOST_ARCH":null,"OS":null,"TARGET_ARCH":null}],
"address": [{}]
}
},
"protobuf": {
"": {
"C++ runtime": [{"AR":null,"ARCH":null,"CXX":null,"ENV":null,"HOST_ARCH":null,"OS":null,"TARGET_ARCH":null}],
"protoc": [{"AR":null,"ARCH":null,"CXX":null,"ENV":null,"HOST_ARCH":null,"OS":null,"TARGET_ARCH":null}],
"well_known_protos": [{}]
}
},
"rules": {
"CC": {
"defaults": [{}]
}
}
}
}
$It should be noted, however, that this tight integration of proto
into our C++ rules is just convenience of our code base. If had
to cooperate with rules not aware of proto, we could have created
a separate rule delegating the library creation to the anonymous
target and then simply reflecting the values of that target.
Finally, let’s add a test. As we use the protobuf repository as
workspace root, we add the test script ad hoc into the targets file,
using the "file_gen" rule. For debugging a potentially failing
test, we also keep the intermediate files the test generates.
...
, "test.sh":
{ "type": "file_gen"
, "name": "test.sh"
, "data":
{ "type": "join"
, "separator": "\n"
, "$1":
[ "set -e"
, "(echo 12345; echo 'John Doe'; echo 'jdoe@example.org'; echo) | ./add_person addressbook.data"
, "./list_people addressbook.data > out.txt"
, "grep Doe out.txt"
]
}
}
, "test":
{ "type": ["@", "rules", "shell/test", "script"]
, "name": ["read-write-test"]
, "test": ["test.sh"]
, "deps": ["add_person", "list_people"]
, "keep": ["addressbook.data", "out.txt"]
}
}That example also shows why it is important that the generation
of the language bindings is delegated to an anonymous target: we
want to analyse only once how the C++ bindings are generated.
Nevertheless, many targets can depend (directly or indirectly) on
the same proto library. And, indeed, analysing the test, we get
the expected additional targets and the one anonymous target is
reused by both binaries.
$ just-mr analyse examples test --dump-targets -
INFO: Requested target is [["@","","examples","test"],{}]
INFO: Result of target [["@","","examples","test"],{}]: {
"artifacts": {
"result": {"data":{"id":"dd5983ceb5ffbe6bee6da1664485d1948a5e952b","path":"result"},"type":"ACTION"},
"stderr": {"data":{"id":"dd5983ceb5ffbe6bee6da1664485d1948a5e952b","path":"stderr"},"type":"ACTION"},
"stdout": {"data":{"id":"dd5983ceb5ffbe6bee6da1664485d1948a5e952b","path":"stdout"},"type":"ACTION"},
"time-start": {"data":{"id":"dd5983ceb5ffbe6bee6da1664485d1948a5e952b","path":"time-start"},"type":"ACTION"},
"time-stop": {"data":{"id":"dd5983ceb5ffbe6bee6da1664485d1948a5e952b","path":"time-stop"},"type":"ACTION"},
"work/addressbook.data": {"data":{"id":"dd5983ceb5ffbe6bee6da1664485d1948a5e952b","path":"work/addressbook.data"},"type":"ACTION"},
"work/out.txt": {"data":{"id":"dd5983ceb5ffbe6bee6da1664485d1948a5e952b","path":"work/out.txt"},"type":"ACTION"}
},
"provides": {
},
"runfiles": {
"read-write-test": {"data":{"id":"92a7e0fb13fbfea251760e81e66258782800b165"},"type":"TREE"}
}
}
INFO: List of analysed targets:
{
"#": {
"acde278315be59c6bdf436efa9dc9782a6c59f36": {
"2a483a2de7f25c1bc066e47245f55ec9a2d4a719": [{"AR":null,"ARCH":null,"CC":null,"CFLAGS":null,"CXX":null,"CXXFLAGS":null,"ENV":null,"HOST_ARCH":null,"OS":null,"TARGET_ARCH":null}]
}
},
"@": {
"": {
"examples": {
"add_person": [{"AR":null,"ARCH":null,"CC":null,"CFLAGS":null,"CXX":null,"CXXFLAGS":null,"ENV":null,"HOST_ARCH":null,"OS":null,"TARGET_ARCH":null}],
"address": [{}],
"list_people": [{"AR":null,"ARCH":null,"CC":null,"CFLAGS":null,"CXX":null,"CXXFLAGS":null,"ENV":null,"HOST_ARCH":null,"OS":null,"TARGET_ARCH":null}],
"test": [{"AR":null,"ARCH":null,"CC":null,"CFLAGS":null,"CXX":null,"CXXFLAGS":null,"ENV":null,"HOST_ARCH":null,"OS":null,"RUNS_PER_TEST":null,"TARGET_ARCH":null,"TEST_ENV":null}],
"test.sh": [{}]
}
},
"protobuf": {
"": {
"C++ runtime": [{"AR":null,"ARCH":null,"CXX":null,"ENV":null,"HOST_ARCH":null,"OS":null,"TARGET_ARCH":null}],
"protoc": [{"AR":null,"ARCH":null,"CXX":null,"ENV":null,"HOST_ARCH":null,"OS":null,"TARGET_ARCH":null}],
"well_known_protos": [{}]
}
},
"rules": {
"CC": {
"defaults": [{}]
}
}
}
}
INFO: Target tainted ["test"].
$Finally, the test passes and the output is as expected.
$ just-mr build examples test -Pwork/out.txt
INFO: Requested target is [["@","","examples","test"],{}]
INFO: Analysed target [["@","","examples","test"],{}]
INFO: Export targets found: 3 cached, 0 uncached, 0 not eligible for caching
INFO: Target tainted ["test"].
INFO: Discovered 8 actions, 4 trees, 1 blobs
INFO: Building [["@","","examples","test"],{}].
INFO: Processed 8 actions, 5 cache hits.
INFO: Artifacts built, logical paths are:
result [7ef22e9a431ad0272713b71fdc8794016c8ef12f:5:f]
stderr [e69de29bb2d1d6434b8b29ae775ad8c2e48c5391:0:f]
stdout [7fab9dd1ee66a1e76a3697a27524f905600afbd0:196:f]
time-start [9488d8109ff186e7b9ffd7bdfe9f0cc11e99f781:11:f]
time-stop [9488d8109ff186e7b9ffd7bdfe9f0cc11e99f781:11:f]
work/addressbook.data [040e76802da97fab00070bb4dbca50d91f43ac7f:41:f]
work/out.txt [a47b62aba8783b8f923218a6838972c77ac082f2:101:f]
(1 runfiles omitted.)
Person ID: 12345
Name: John Doe
E-mail address: jdoe@example.org
Updated: 2022-06-22T13:03:29Z
INFO: Target tainted ["test"].
$