Buildbot configuration

Julia runs a buildbot instance at build.julialang.org, the configuration for which is stored in this repository. Buildbot follows a client-server architecture, the configuration in this repository defines all buildbot jobs and the steps within those jobs to perform various build-related tasks such as building nightly unstable versions of Julia, building the RC and final binaries for stable releases, performing GC stress runs, etc...

Actually provisioning the buildbot instance and its slaves is done through ansible scripts, located in this repository. This gives us a quick, easy, and scalable method of setting up buildslaves in what is hopefully a well-automated process. The julia-ansible-scripts repository has a wiki page that goes into more depth on what deployment entails.

The layout of this repository is very simple: all buildbot configuration is placed in the master/ directory, and a few miscellaneous scripts are located in the commands/ directory.

The commands/ directory contains scripts that are run on buildslaves (such as launchpad.sh, which is used to submit a julia version to launchpad for packaging on the julianightlies PPA) as well as scripts that are not currently run on buildslaves, but are instead run manually once in a blue moon (such as cleanup_bottles.jl which parses all bottles stored in juliabottles on AWS and removes those that are older than newest for each platform).

The master/ directory contains configuration files for the buildbot, and consists primarily of python scripts. The top-level file is master.cfg, which defines the BuildmasterConfig, a dictionary that completely controls the buildbot behavior. Further configuration is broken down into smaller files. inventory.py contains configuration for each buildslave, (each buildslave must be explicitly registered with the buildbot, otherwise it will refuse connections from that buildslave) including extra make flags that need to be set for a particular OS/architecture. Individual build jobs are broken out into lists of commands for packaging, building special versions of julia (e.g. the nightly threading job) as well as cleaning up the buildbots (nuclear_arsenal.py).

Jobs are initiated by "schedulers", as an example, automatic builds (performed upon a webhook from GitHub) are triggered by the AnyBranchScheduler, whereas the nightly threading build is triggered at a specific time by the Nightly scheduler. We also have ForceScheduler configurations, which allow users from the web interface to manually initiate some builds.