Third-party projects usually ship with their own build description, which often happens to be not compatible with justbuild. Nevertheless, it is highly desireable to include external projects via their source code base, instead of relying on the integration of out-of-band binary distributions. justbuild offers a flexible approach to provide the required build description via an overlay layer without the need to touch the original code base.
In the remainder of this section, we will use the open-source project fmtlib as an example for integrating third-party software to a justbuild project.
Before we construct the overlay layer for fmtlib, we need to determine its file structure (tag 8.1.1). The relevant header and source files are structured as follows:
fmt | +--include | +--fmt | +--*.h | +--src +--format.cc +--os.cc
The public headers can be found in include/fmt, while the library’s source
files are located in src. For the overlay layer, the TARGETS files should be
placed in a tree structure that resembles the original code base’s structure.
It is also good practice to provide a top-level TARGETS file, leading to the
following structure for the overlay:
fmt-layer | +--TARGETS +--include | +--fmt | +--TARGETS | +--src +--TARGETS
Let’s create the overlay structure:
$ mkdir -p fmt-layer/include/fmt
$ mkdir -p fmt-layer/srcThe directory include/fmt contains only header files. As we want all files in
this directory to be included in the "header directory" target, we can safely
use the explicit TREE reference[fn:1], which collects, in a single
artifact (describing a directory) all directory contents
from "." of the workspace root. Note that the TARGETS file is only part of
the overlay, and
therefore will not be part of this tree. Furthermore, this tree should be staged
to "fmt", so that any consumer can include those headers via <fmt/...>. The
resulting header directory target "hdrs" in include/fmt/TARGETS should be
described as:
[fn:1] Explicit TREE references are always a list of length 3, to distinguish
them from target references of length 2 (module and target name). Furthermore,
the second list element is always null as we only want to allow tree
references from the current module.
{ "hdrs":
{ "type": ["@", "rules", "CC", "header directory"]
, "hdrs": [["TREE", null, "."]]
, "stage": ["fmt"]
}
}The actual library target is defined in the directory src. For the public
headers, it refers to the previously created header directory via its
fully-qualified target name (["include/fmt", "hdrs"]). Source files are the
two local files format.cc, and os.cc. The final target description in
src/TARGETS will look like this:
{ "fmt":
{ "type": ["@", "rules", "CC", "library"]
, "name": ["fmt"]
, "hdrs": [["include/fmt", "hdrs"]]
, "srcs": ["format.cc", "os.cc"]
}
}Finally, the top-level TARGETS file can be created. While it is technically
not strictly required, it is considered good practice to export every target
that may be used by another project. Exported targets are subject to high-level
target caching, which allows to skip the analysis and traversal of entire
subgraphs in the action graph. Therefore, we create an export target that
exports the target ["src", "fmt"], with only the variables in the field
"flexible_config" being propagated. The top-level TARGETS file contains the
following content:
{ "fmt":
{ "type": "export"
, "target": ["src", "fmt"]
, "flexible_config": ["CXX", "CXXFLAGS", "ADD_CXXFLAGS", "AR", "ENV"]
}
}Based on the hello world tutorial, we can extend the existing repos.json by
the layer definition "fmt-targets-layer" and the repository "fmtlib", which
is based on the Git repository with its target root being overlayed.
Furthermore, we want to use "fmtlib" in the repository "tutorial", and
therefore need to introduce an additional binding "format" for it:
{ "main": "tutorial"
, "repositories":
{ "just-rules":
{ "repository": {"type": "file", "path": "/tmp/justbuild/rules"}
}
, "tutorial":
{ "repository": {"type": "file", "path": "."}
, "bindings": {"rules": "just-rules", "format": "fmtlib"}
}
, "tutorial-defaults":
{ "repository": {"type": "file", "path": "./tutorial-defaults"}
}
, "fmt-targets-layer":
{ "repository": {"type": "file", "path": "./fmt-layer"}
}
, "fmtlib":
{ "repository":
{ "type": "git"
, "branch": "8.1.1"
, "commit": "b6f4ceaed0a0a24ccf575fab6c56dd50ccf6f1a9"
, "repository": "https://github.com/fmtlib/fmt.git"
}
, "target_root": "fmt-targets-layer"
, "bindings": {"rules": "just-rules"}
}
}
}This "format" binding can you be used to add a new dependency in
greet/TARGETS:
{ "greet":
{ "type": ["@", "rules", "CC", "library"]
, "name": ["greet"]
, "hdrs": ["greet.hpp"]
, "srcs": ["greet.cpp"]
, "stage": ["greet"]
, "deps": [["@", "format", "", "fmt"]]
}
}Consequently, the fmtlib library can now be used by greet/greet.cpp:
#include "greet.hpp"
#include <fmt/format.h>
void greet(std::string const& s) {
fmt::print("Hello {}!\n", s);
}Due to changes made to repos.json, building this tutorial requires to rerun
just-mr, which will fetch the necessary sources for the external repositories:
$ just-mr build helloworld
INFO: Requested target is [["@","tutorial","","helloworld"],{}]
INFO: Analysed target [["@","tutorial","","helloworld"],{}]
INFO: Export targets found: 0 cached, 0 uncached, 1 not eligible for caching
INFO: Discovered 7 actions, 4 trees, 0 blobs
INFO: Building [["@","tutorial","","helloworld"],{}].
INFO: Processed 7 actions, 0 cache hits.
INFO: Artifacts built, logical paths are:
helloworld [ccca0c1aa443e5b2adce16949de24b50d6826636:324432:x]
$Note to build the fmt target alone, its containing repository fmtlib must be
specified via the --main option:
$ just-mr --main fmtlib build fmt
INFO: Requested target is [["@","fmtlib","","fmt"],{}]
INFO: Analysed target [["@","fmtlib","","fmt"],{}]
INFO: Export targets found: 0 cached, 0 uncached, 1 not eligible for caching
INFO: Discovered 3 actions, 2 trees, 0 blobs
INFO: Building [["@","fmtlib","","fmt"],{}].
INFO: Processed 3 actions, 3 cache hits.
INFO: Artifacts built, logical paths are:
libfmt.a [14c5156a6a988f417928970e750c8c8f928460ab:863452:f]
(1 runfiles omitted.)
$The make use of high-level target caching for exported targets, we need to
ensure that all inputs to an export target are transitively content-fixed. This
is automatically the case for "type":"git" repositories. However, the libfmt
repository also depends on "fmt-target-layer", "just-rules", and
"tutorial-defaults". As those are "type":"file" repositories, they must be
put under Git versioning first:
$ git init .
$ git add tutorial-defaults fmt-layer
$ git commit -m"fix compile flags and fmt targets layer"Note that just-rules already is under Git versioning as it is a subtree of the
checked out justbuild repository.
Now, to instruct just-mr to use the content-fixed, committed source trees of
those "type":"file" repositories the pragma "to_git" must be set for them in
repos.json:
{ "main": "tutorial"
, "repositories":
{ "just-rules":
{ "repository":
{ "type": "file"
, "path": "/tmp/justbuild/rules"
, "pragma": {"to_git": true}
}
, "target_root": "tutorial-defaults"
, "rule_root": "just-rules"
}
, "tutorial":
{ "repository": {"type": "file", "path": "."}
, "bindings": {"rules": "just-rules", "format": "fmtlib"}
}
, "tutorial-defaults":
{ "repository":
{ "type": "file"
, "path": "./tutorial-defaults"
, "pragma": {"to_git": true}
}
}
, "fmt-targets-layer":
{ "repository":
{ "type": "file"
, "path": "./fmt-layer"
, "pragma": {"to_git": true}
}
}
, "fmtlib":
{ "repository":
{ "type": "git"
, "branch": "master"
, "commit": "b6f4ceaed0a0a24ccf575fab6c56dd50ccf6f1a9"
, "repository": "https://github.com/fmtlib/fmt.git"
}
, "target_root": "fmt-targets-layer"
, "bindings": {"rules": "just-rules"}
}
}
}Due to changes in the repository configuration, we need to rebuild and the benefits of the target cache should be visible on the second build:
$ just-mr build helloworld
INFO: Requested target is [["@","tutorial","","helloworld"],{}]
INFO: Analysed target [["@","tutorial","","helloworld"],{}]
INFO: Export targets found: 0 cached, 1 uncached, 0 not eligible for caching
INFO: Discovered 7 actions, 4 trees, 0 blobs
INFO: Building [["@","tutorial","","helloworld"],{}].
INFO: Processed 7 actions, 0 cache hits.
INFO: Artifacts built, logical paths are:
helloworld [0ec4e36cfb5f2c3efa0fff789349a46694a6d303:132736:x]
$
$ just-mr build helloworld
INFO: Requested target is [["@","tutorial","","helloworld"],{}]
INFO: Analysed target [["@","tutorial","","helloworld"],{}]
INFO: Export targets found: 1 cached, 0 uncached, 0 not eligible for caching
INFO: Discovered 4 actions, 2 trees, 0 blobs
INFO: Building [["@","tutorial","","helloworld"],{}].
INFO: Processed 4 actions, 4 cache hits.
INFO: Artifacts built, logical paths are:
helloworld [0ec4e36cfb5f2c3efa0fff789349a46694a6d303:132736:x]
$Note that in the second run the export target "fmt" was taken from cache and
its 3 actions were eliminated, as their result has been recorded to the
high-level target cache during the first run.
Projects typically depend on multiple external repositories. Creating an overlay
layer for each external repository might unnecessarily clutter up the repository
configuration and the file structure of your repository. One solution to
mitigate this issue is to combine the TARGETS files of multiple external
repositories in a single overlay layer. To avoid conflicts, the TARGETS files
can be assigned different file names per repository. As an example, imagine a
common overlay layer with the files TARGETS.fmt and TARGETS.gsl for the
repositories "fmtlib" and "gsl-lite", respectively:
common-layer | +--TARGETS.fmt +--TARGETS.gsl +--include | +--fmt | | +--TARGETS.fmt | +--gsl | +--TARGETS.gsl | +--src +--TARGETS.fmt
Such a common overlay layer can be used as the target root for both repositories
with only one difference: the "target_file_name" field. By specifying this
field, the dispatch where to find the respective target description for each
repository is implemented. For the given example, the following repos.json
defines the overlay "common-targets-layer", which is used by "fmtlib" and
"gsl-lite":
{ "main": "tutorial"
, "repositories":
{ "just-rules":
{ "repository":
{ "type": "file"
, "path": "/tmp/justbuild/rules"
, "pragma": {"to_git": true}
}
, "target_root": "tutorial-defaults"
, "rule_root": "just-rules"
}
, "tutorial":
{ "repository": {"type": "file", "path": "."}
, "bindings": {"rules": "just-rules", "format": "fmtlib"}
}
, "tutorial-defaults":
{ "repository":
{ "type": "file"
, "path": "./tutorial-defaults"
, "pragma": {"to_git": true}
}
}
, "common-targets-layer":
{ "repository":
{ "type": "file"
, "path": "./common-layer"
, "pragma": {"to_git": true}
}
}
, "fmtlib":
{ "repository":
{ "type": "git"
, "branch": "8.1.1"
, "commit": "b6f4ceaed0a0a24ccf575fab6c56dd50ccf6f1a9"
, "repository": "https://github.com/fmtlib/fmt.git"
}
, "target_root": "common-targets-layer"
, "target_file_name": "TARGETS.fmt"
, "bindings": {"rules": "just-rules"}
}
, "gsl-lite":
{ "repository":
{ "type": "git"
, "branch": "v0.40.0"
, "commit": "d6c8af99a1d95b3db36f26b4f22dc3bad89952de"
, "repository": "https://github.com/gsl-lite/gsl-lite.git"
}
, "target_root": "common-targets-layer"
, "target_file_name": "TARGETS.gsl"
, "bindings": {"rules": "just-rules"}
}
}
}