This is the author's personal copy of the January 2023 Roku Engineering Blog Post, You Need a Build System. Please read it there!
RokuOS is our single operating system that runs on all of our streaming players, smart TVs, and soundbars. It is a highly customized user experience built on top of an embedded Linux kernel that runs "channels" such as Netflix, Prime Video, and The Roku Channel, which are displayed on a grid on our iconic Roku home screen.
Today's article focuses on the system we use to build RokuOS. The question is: How do we translate the source code for all of RokuOS's components to something that can run on every unique device? "A compiler, of course!" you say. Easy answer! But it's a bit more complicated: You need a build system
Since RokuOS runs on a wide variety of hardware devices, it is able to control various aspects of each. For example, it knows how to talk to a platform's specific HDMI controller to change the HDMI output resolution. Our software stack, therefore, has many libraries which abstract the ways user-facing applications control physical hardware. So while the user experience will be the same regardless which Roku device you use, the underlying software can have vast differences due to changes in the physical hardware and available features!
When we say build system, what do we actually mean? GNU Make is a build system, but so is Bazel and Buildroot, and these are all very different applications.
Let's simplify this by sorting different kinds of build systems into three distinct classes:
- GCBS: Generic Component Build Systems
- MBS: Meta Build Systems
- EXBS: Exotic/Specialty Build Systems
Let's look at each in turn.
A Generic Component Build System (GCBS) is what people most commonly think of when you say "build system." This class includes applications like GNU Make, Autotools (incl. Automake, Libtool), CMake, Ninja, Gradle, Bazel, etc.
GCBSs are very flexible in what they can do -- they're not specifically tailored towards one specific goal. They are essentially a scripting language for build commands. You can do pretty much anything you want with them, including taking Make and creating a massive toolchain-compiling, cross-compiling-focused, embedded-rootfs-and-image-creating system. Buildroot, a MBS, does this. However just because you can do something and it will work, it doesn't mean you should.
GCBSs have their place and purpose, and some GCBSs do things better than others. Make is universal: The Autotools suite and CMake are similar in that they generate Makefiles -- their whole purpose is to relieve developers from the work of hand-crafting Makefiles by using translation language. Gradle is more geared towards Java and Android applications and can automate or make some things easier for developers using those technologies. They all do different things but solve the same goal: to make compiling your code easier.
Overall, if you are working with a single component, it will most likely use a GCBS class build system.
A Meta Build System (MBS) is purpose-built to accomplish two primary goals:
- Compile multiple linked codebases, a.k.a. "components" (also called packages, recipes, or elements)
- From compiled packages, assemble a Root Linux Filesystem, or "RootFS"
Within a MBS, each "component" has its own individual codebase with its own GCBS. The MBS doesn't invoke compiler commands directly, but indirectly by talking to that component's GCBS instead. For example the MBS Buildroot might include a component called Busybox which uses GCBS Make, and Make will run the compiler.
MBSs can also offer features that are geared toward Embedded Linux development, such as cross-compiling, compiling the Linux kernel, and generating flashable images for use in embedded bootloaders like U-Boot (which also might be compiled at the same time). Buildroot and OpenEmbedded/Yocto are examples of MBSs that do cross-compiling for Embedded Linux in this manner.
GCBSs can be made to do what MBSs are designed to do, but they aren't specifically designed for this purpose. It's normally better to choose an MBS than to force-fit a GCBS.
Exotic/Specialty Build Systems, or EXBSs, are typically custom-made and tailored towards specific use-cases for their applications/platforms.
An example of an EXBS is the Arch Build System, which contains many different pieces to help build the Arch Linux distribution. It works like this: Pacman the package manager uses packages ("components") from their official package repos and the Arch User Repository. Packages are built from source by Makepkg, where the packages use their PKGBUILD system to define source and how-to-build. Arch Linux "releases" are created by using Archiso, which essentially uses a defined set of packages and uses Pacman to download pre-made packages from their package repos and install them in a packaged RootFS.
We classify the Arch Build System as an EXBS rather than an MBS because it is not a single application that compiles and packages multiple codebases at one time, then uses those packages to build a RootFS. It's an EXBS because it compiles and packages at multiple stages under very different timing.
Most popular Linux distributions use some type of EXBS, and their systems are rarely usable beyond their intended scope unless significant development effort is made to "fork" and modify the system for another distribution (which actually makes it a different EXBS).
So why are we talking about this? Recently, to meet the scaling challenges of the growing number of Roku TVs and players, we ran an internal review to evaluate whether to migrate RokuOS onto a new build system.
For our use cases, we clearly want to pick a Meta Build System since this type is meant for cross-compiling embedded environments. We wouldn't, for example, want to choose a GCBS like CMake, then have to implement a lot of features which come standard in a MBS.
What options do we have when picking a Meta Build System? There are actually only four:
- BitBake/OpenEmbedded from the Yocto project.
- Buildroot
- OpenWrt's Buildroot, which is distinctly different.
- BuildStream
Not all MBSs are created equal. Since we are actively developing RokuOS and need to test the build results on real devices, the MBS needs to be fast and efficient when iteratively building code changes. This situation is called Active Development, and contrary to what you may think, MBSs are not particularly very good at it! Instead, the primary users of MBSs are system integrators, not software developers. MBSs were designed to manipulate software components that are already released and known to be working, combining them into a full operating system.
Some tests to determine friendliness to Active Development are: If you make a code change in one file of one component, how well can the build system handle that change in a global scope? Can it only rebuild that area? Is the time it takes to do so proportional to the amount of code changed, or does it need to perform all sorts of other tasks too? What about dependencies and dependent (and dependent of dependent) components? What about changing how the build tool is building the source -- is that easy to do, or does the developer need to open a hundred-page document to figure it out?
We needed to pick a MBS that was the least hostile to Active Development, and Buildroot was the first to be eliminated. Its primary use-case is strictly for system integrators. It downloads items during build-time without developer-focused tools to allow dynamically-updating component source code. OpenWrt's Buildroot, which is an evolved fork of Buildroot, fares a bit better here via their more modular Packages system, however it still has the same issue.
Well, there goes 50% of the choices already! The remaining two were selected for our evaluation:
- BitBake/OpenEmbedded, part of the Yocto project, was quickly chosen as a contender given it is the de facto standard build system for the embedded Linux industry. It might not the best for Active Development, but it is by far the best-supported build system out there, and many of our vendors already have BitBake meta layers available. Check them out at yoctoproject.org
- BuildStream. Who's this? You may not have heard about this system before -- we certainly hadn't. However, the introduction on their website was quite promising, and the system appeared to accomplish the same goals as BitBake while being easier to use and friendlier to Active Development. We selected BuildStream as the only other contender. Check them out at buildstream.build
In our Meta Build Evaluation group, we split into two teams -- BitBake and BuildStream. Each team had the goal of creating something that could evolve to eventually support the full RokuOS:
- Create a Linux distribution using no pre-built binaries, with platform-level exceptions
- Run it on that build system's first new "Roku device:" a Raspberry Pi
- The build system must create a GCC 12 cross-compiler from scratch
- Cross-compile our "base layer" of Busybox, Boost, Flatbuffers, and rostd
- Build a platform-selectable Linux kernel
- Cross-compile our "platform layer" with platform-selectable items. For the RPi, that would be the Broadcom binaries.
- Have a separate, user-selectable "debug layer" with GDBServer and Dropbear
- Finally, build the final flashable image, which is also platform-selectable. For the RPi, that would be an SD card image.
In evaluating, the teams paid attention to various developer-focused preferences, such as the ease of editing source code and build recipes, how well build accelerators like caching and distributed builds work, and their overall sentiment towards their build system.
After a few weeks, both teams were able to successfully accomplish the evaluation goals!
The BitBake team's evaluation was relatively straightforward, considering that the OpenEmbedded project and many other meta layers already provide recipes for all of the above tasks! It was only a matter of putting it all together.
The BuildStream team's evaluation had a bit more work since it doesn't really have an equivalent OpenEmbedded project. The closest existing BuildStream project to it is FreedesktopSDK. However, this project is more focused on natively built outputs while cross-compiling as little as possible, and wasn't as suitable for the more classical cross-compilation approach for embedded environments that we needed, so it was decided early on that it wouldn't quite be appropriate to use FreedesktopSDK as-is. Additionally, since we know of BitBake projects that do not use the OpenEmbedded project (Isar for example), we thought it would be fun to also evaluate how well BuildStream does without FreedesktopSDK!
What the team here created is essentially a minimalist BuildStream framework project that specifically focuses on cross-compiling for embedded environments, much like OpenEmbedded, making it a much better analogy than we created earlier with FreedesktopSDK. The only item that was still utilized from them was their prebuilt sandbox.
Each build system had unique strengths and weaknesses.
On the positive side: BitBake/OpenEmbedded is a mature project with a lot of industry adoption, has a massive catalog of well-maintained recipes of open-source software, and has been building embedded Linux distributions for 20 years.
BuildStream, which quotes BitBake/OE as inspiration, has a much better user experience. It has a polished console utility, clean and concise usage documentation, uses enforced sandboxing for all builds, has out-of-the-box remote caching, utilizes Bazel's Remote Execution API, and has a much more readable recipe/element syntax.
On the negative side, the above is reversed. BitBake recipes aren't very friendly; it uses older technologies for caching and distributed builds; and it doesn't have a level of build sandboxing that we'd like.
BuildStream, meanwhile, is a newer project that does not have a lot of adoption or activity surrounding it, as evidenced by our need to create an OpenEmbedded equivalent to focus on cross-compiling for embedded environments. We would have to do more work on our end to get to the same level. Unfortunately, since a new build system is a serious long-term commitment, we did have concerns about the long-term viability of pioneering BuildStream and how we'd be on our own.
It was a close vote! BitBake/OpenEmbedded was the winner!
Despite not being selected in the end, BuildStream gave BitBake quite a run for the money and has certainly impacted the direction our Meta Build team will be taking. Even though we loved some of the new ideas, we found BuildStream involved a bit of risk due to a general lack of industry support. However, we at Roku would like to give back a little bit and thank them for their project by open sourcing our BuildStream evaluation project! Check it out here
Moving forward with BitBake/OpenEmbedded, our Meta Build team is excited for this next step in our adventure! We believe we have found a much more scalable solution for the anticipated increase in build complexity that will come from adding more and more Roku products to our lineup! Enabling our RokuOS developers to work faster, our new BitBake build system is going to be a major driver for Roku in keeping our lead as the #1 streaming platform!
Linux Foundation® and Yocto Project® are registered trademarks of the Linux Foundation. https://www.yoctoproject.org/
Linux® is a registered trademark of Linus Torvalds. https://kernel.org/
BuildStream is a free software project hosted by the Apache Software Foundation (ASF). The project logo is the blue waterwheel created specially for BuildStream. https://buildstream.build/
The Buildroot Association is a non-profit organization registered as a legal entity in France as an association loi 1901. https://buildroot.org/
OpenWrt is a registered United States trademark of Software Freedom Conservancy (SFC), managed by the OpenWrt project. https://openwrt.org/