hlb v0.4.0

Welcome to HLB v0.4! A robust debugger was implemented with breakpoints, forward / reverse execution, and also allowing you to exec into the container at any step. Not only that, its only supported by IDE debuggers via DAP support.

Highlights

Debugger

Access the debugger when running any HLB module by adding --debug to the hlb run command. The debugger halts before program starts, so you can take a look at all the commands supported currently:

❯ hlb run --debug
Type help for a list of commands
(hlb) help
# Running the program
    continue (alias: c) - run until breakpoint or program termination
    next (alias: n) - step over to next source line
    step (alias: s) - single step through program
    stepout - step out of current function
    rev (alias: r) <movement> - reverses execution of program for movement specified
    restart - restart program from the start

# Manipulating breakpoints
    break (alias: b) <symbol | linespec> - sets a breakpoint
    breakpoints (alias: bp) - prints out active breakpoints
    clear <breakpoint-index> - deletes breakpoint
    clearall - deletes all breakpoints

# Viewing program variables and functions
    args - print function arguments
    funcs - print functions in this module

# Viewing the call stack and selecting frames
    backtrace (alias: bt) - prints backtrace at this step

# Filesystem only commands
    pwd - print working directory at this step
    environ - print environment at this step
    network - print network mode at this step
    security - print security mode at this step

# Other commands
    help - prints this help message
    list (alias: ls) - prints source code at this step
    exit (alias: quit) - exits the debugger

Exec at any step of container building

When the debugger is in a filesystem block, the command exec is available allowing you to exec into the container at any step. When a breakpoint is defined in source, continue will halt when it is reached, while supporting all the option::run attached to the statement.

For example, we will debug an alpine image that mounts the openllb/hlb git repo at /in. You can exit the exec back into the debugger repl and continue debugging.

❯ hlb run --debug foo.hlb
Type help for a list of commands
(hlb) continue
#1 resolve image config for docker.io/library/alpine:latest
#1 DONE 0.4s

#2 docker-image://docker.io/library/alpine:latest
#2 CACHED
foo.hlb:3:2:
1 │ fs default() {
2 │ 	image "alpine"
3 │ 	run "echo foo" with option {
  │ 	^^^
4 │ 		breakpoint
5 │ 		mount src "/in"
6 │ 	}
7 │ }
(hlb) exec
Starting process "/bin/sh"
#1 docker-image://docker.io/library/alpine:latest
#1 CACHED

#2 git://github.com/openllb/hlb.git#master
#2 0.452 1bc8728460226ff81fb8562e0b022dc8978ed322	refs/heads/master
#2 0.767 From https://github.com/openllb/hlb
#2 0.767  t [tag update]      master     -> master
#2 0.767  + 6130d0f...1bc8728 master     -> origin/master  (forced update)
#2 DONE 0.8s
/ # cat /etc/os-release
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.15.0
PRETTY_NAME="Alpine Linux v3.15"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://bugs.alpinelinux.org/"
/ # ls /in
LANGSERVER.md  builtin        codegen        examples       hlb.go         mkdocs.hlb     pkg            version.go
LICENSE        checker        diagnostic     go.hlb         language       mkdocs.yml     rpc
README.md      client.go      docs           go.mod         linter         module         scripts
build.hlb      cmd            errdefs        go.sum         local          parser         solver
/ # exit
(hlb) ls
foo.hlb:3:2:
1 │ fs default() {
2 │ 	image "alpine"
3 │ 	run "echo foo" with option {
  │ 	^^^
4 │ 		breakpoint
5 │ 		mount src "/in"
6 │ 	}
7 │ }
(hlb)

DAP

An initial implementation for DAP over stdio has landed. Using vscode-hlb you can debug HLB from Visual Code, or your favorite editor if it supports DAP.

Module URIs

HLB now supports running modules from URIs, whether it is a local file or from a remote git repository.

❯ hlb run --help
NAME:
   hlb run - compiles and runs a hlb program

USAGE:
   hlb run [command options] <uri>

The following schemes are supported:

• file:// or empty scheme for local files (same as before)
• git-https:// git repository over https
• git+ssh:// git repository over ssh
• git:// git repository, automatically detecting if ssh is supported based on your SSH agent.

For git based module URIs, an optional branch or commit ref can be provided as a suffix with @<ref> followed by an optional path :/path/to/file.hlb. (e.g. git://github.com/openllb/hlb@master:/build.hlb)

For example, we can run the lint target of this repository remotely like this:

❯ hlb run -t lint git://github.com/openllb/hlb
[+] Building 18.2s (9/9) FINISHED
 => git://github.com/openllb/hlb                                                                                               9.9s
 => resolve image config for docker.io/library/golang:1.17.5-alpine                                                            0.9s
 => docker-image://docker.io/library/golang:1.17.5-alpine                                                                      0.1s
 => => resolve docker.io/library/golang:1.17.5-alpine                                                                          0.1s
 => https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh                                                 0.2s
 => copy / /                                                                                                                   0.0s
 => CACHED /bin/sh -c 'apk add -U git gcc libc-dev'                                                                            0.0s
 => CACHED /bin/sh -c 'sh /golangci/install.sh -b /usr/bin v1.31.0'                                                            0.0s
 => /bin/sh -c 'go get'                                                                                                        2.8s
 => /bin/sh -c '/usr/bin/golangci-lint run'                                                                                    8.4s

Merge, Diff

Thanks to the work of @sipsma, merge and diff are new builtins that allow you to merge and diff filesystems.

Here's an example using both:

fs alpine() {
        image "alpine"
}

fs busybox() {
        image "busybox"
}

fs workspace() {
        busybox
        run "echo this is a test > /root/file1"
        run "dd if=/dev/urandom bs=1024 count=100 of=/var/random"
        run "rm /bin/zcat"
}

fs differences() {
        workspace
        diff busybox
}

fs workspaceRebased() {
        alpine
        merge differences
}

fs default() {
        busybox
        run "ls" with option {
                breakpoint
                mount workspace "/workspace"
                mount differences "/differences"
                mount workspaceRebased "/rebased"
        }
}

Performance

The internals of the HLB compiler is now fully asynchronous, which should improve build performance overall. In particular builds that use a lot of remote sources like docker images, should have their builds speed up significantly.

hlb v0.3

Welcome to HLB v0.3! Its been a few months since our last release, and we've managed to land important refactoring and bug fixes to make your HLB development experience even better.

Highlights

Call expressions

Previously, if you wanted to pass the results of a function as an argument to another, you have to wrap it in a function literal. For example:

fs build() {
    image "alpine"
    run "make" with option {
        dir "/in"
        mount fs {
            local "."
        } "/in"
    }
}

If the function had no arguments like scratch, you could call it without the function literal like mount scratch "/in".
Since HLB has arguments delimited by spaces, it is ambiguous to pass arguments to functions invoked as arguments themselves.

Call expressions allow you to invoke functions as expressions.

fs build() {
    image "alpine"
    run "make" with option {
        dir "/in"
        mount local(".") "/in"
    }
}

You can also do multi-line call expressions, along with comments.

fs build() {
    image "alpine"
    copy git(
        # HLB's git repo
        "git@github.com:openllb/hlb.git",
        "master",
    ) "/" "/src"
}

You may also want to convert your remote imports to one-liners now.

import go from image("openllb/go.hlb")

String interpolation

In HLB v0.2, whenever you needed to interpolate a string, you had to use format and usually in a string { ... } function literal. We built in string interpolation to greatly simplify this use case.

Before, you may have multiple levels of indentations to run a interpolated command:

fs build(string cmd) {
    image "alpine"
    run string {
        format "npm run %s" cmd
    }
}

Now you can interpolate it directly using ${...}:

fs build(string cmd) {
    image "alpine"
    run "npm run ${cmd}"
}

Note that this also works for heredocs:

fs build(string cmd) {
    image "alpine"
    run <<~CMD
        npm run ${cmd}
    CMD
}

In order to not interpolate, we introduced raw string literals and raw heredoc literals. Raw string literals uses backticks instead of double quotes like so:

fs build(string cmd) {
    image "alpine"
    run `npm run ${cmd}` # interpreted literally
}

And for raw heredocs you must wrap the starting identifier with backticks. If you're familiar with bash heredocs, this may look familiar.

fs build(string cmd) {
    image "alpine"
    run <<~`CMD`
        npm run ${cmd}
    CMD
}

Unified imports

When importing relative HLB modules and remote modules the syntax differed.

# Local import
import go "./go.hlb"

# Remote import
import go from fs {
    image "openllb/go.hlb"
}

In HLB v0.3, we've rewritten codegen for first-class value types, so we're able to collapse this into one style:

# Local import
import go from "./go.hlb"

# Remote import
import go from fs {
    image "openllb/go.hlb"
}

Not to worry, we still support the deprecated style of local imports and we do AST transformation for any of your dependencies recursively. However, if you'd like to fix this automatically, we've also introduced the HLB linter.

Linter

Starting from v0.3 we've introduced a builtin linter that can automatically address issues.

❯ hlb lint
build.hlb:2:11: import without keyword "from" infront of the expression is deprecated
Run `hlb lint --fix build.hlb` to automatically fix lint errors

HLB lint warnings never stop a build from running, and you are able to run hlb lint --fix to have the linter fix your code automatically. Through this feature, we hope to be able to extend the linter beyond deprecation warnings and do data flow analysis and help optimize builds.

hlb 0.2

Welcome to HLB 0.2! We landed many important features to the language, so we're starting to stabilize and focus heavily on testing and polishing the whole experience.

Highlights

Heredocs

Thanks to @coryb, we landed heredocs in this release. Let's look into how we can use it.

A common pattern to start from a base image and install system packages required to build or run an application. In hlb v0.1, we had to write everything in one line.

fs default() {
	image "ubuntu"
	run "apt-get update && apt-get --no-install-recommends install -y git make gcc g++"
}

Using heredocs, we can make it easier to read and maintain. A heredoc begins with a prefix << followed by an identifier. This identifier is used to indicate the start and end of the heredoc and you can pick this identifier to ensure that it doesn't appear in the heredoc body. We recommend picking an identifier related to the contents of the heredoc body. In this case, we will use APT as our heredoc identifier.

There are three types of heredoc prefixes which are useful in different scenarios: <<, <<-, and <<~

The << prefix means everything after the identifier is literal, and that includes any white spaces, newlines, and tabs. Since the shell eliminates white spaces too, a better example is if you are writing a file where whitespace matters.

fs default() {
	scratch
	mkfile "/deployment.yaml" 0o644 <<YAML
apiVersion: apps/v1
kind: Deployment
metadata:
	name: nginx-deployment
spec:
	selector:
		matchLabels:
			app: nginx
	YAML
}

The <<- prefix means all leading tabs are stripped. So we can go back to the apt-get example and improve upon it. Using <<- allows us to write a reasonably tabulated and multi-line version.

fs default() {
	image "ubuntu"
	run <<-APT
		apt-get update && \
		apt-get install --no-install-recommends install -y \
			git \
			make \
			gcc \
			g++
	APT
}

The <<~ prefix means that all sequences of white spaces, newlines, and tabs are replaced with a single white space. This means that we no longer have to suffix \ at the end of every line! Note that we still need && to run multiple commands on the shell.

fs default() {
	image "ubuntu"
	run <<~APT
		apt-get update &&
		apt-get install --no-install-recommends install -y
			git
			make
			gcc
			g++
	APT
}

Multi-targets

We landed support for multi-targets! This means that you can specify multiple targets from the command line and it'll build all the targets in parallel!

hlb run -t unitTest -t integrationTest -t lint build.hlb

Outputs

In HLB 0.1, the only way to retrieve data from a build is to specify it from the command line. For example, if we wanted to download the result of a build as a tarball we can run:

hlb run -t myBuild --downloadTarball=myBuild.tar build.hlb

In HLB 0.2, now that we have landed multi-targets, we need to associate an output like downloadTarball to a specific target. After specifying the target name, you can optionally specify multiple outputs in a CSV fashion:

hlb run -t myBuild,downloadTarball=myBuild.tar \
	-t myOtherBuild,download=./other-build,dockerPush=openllb/other-build:latest \
	build.hlb

However, specifying outputs from the command line is usually kept for one-off tasks. In 0.2, you can now execute outputs in the middle of a build using the same outputs.

fs default() {
	image "alpine"
	run <<~APK
		apk add -U
			git
			curl
			alpine-sdk
	APK
	dockerPush "openllb/alpine:build-essentials"
}

Note that command dockerPush does not modify the fs, so you can continue chaining commands in the same function:

fs default() {
	image "alpine"
	run <<~APK
		apk add -U
			git
			curl
			alpine-sdk
	APK
	dockerPush "openllb/myimage:base"
	run "make && make install" with option {
		dir "/in"
		mount src "/in"
	}
	dockerPush "openllb/myimage:built"
}

Groups

When running multi-targets from the command line, the only choice is to run them in parallel. We landed the group type in 0.2 that allows us to define multi-targets in HLB itself, as well as the ability to control which targets to run sequentially, and which to run in parallel.

We can define a group function just like any other, and it can invoke other group functions too. When invoked one after another, the first group will finish executing before the second will start. This means that if any targets in test fails, then publish will never run.

group default() {
	test
	publish
}

Groups also have a builtin group parallel(variadic group groups) that allow you to define parallel multi-targets.

group test() {
	parallel unitTest integrationTest
}

Functions with fs type can be coerced to group (but not the other way around), so unitTest and integrationTest can be regular fs functions or other group functions.

For example, we can define a group that will publish all the microservices in parallel.

group publish() {
	parallel fs {
		buildApp
		dockerPush "openllb/app"
	} fs {
		buildDB
		dockerPush "openllb/db"
	} fs {
		buildUI
		dockerPush "openllb/ui"
	}
}

Import/export

One of the main themes of HLB is to write reusable containerized builds. We landed the import and export features to allow users to write and consume HLB as libraries.

HLB programs are file scoped, this means that functions defined in one file are not directly accessible by other files. In 0.2, you can now import another file so that you can break down your HLB files by function.

import go "./go.hlb"

fs default() {
	go.build fs { local "."; }
}

In the example above, the default target runs the function build from the module go. However, in order for this to work, go.hlb must also export the build function. This is how HLB modules declare their public API.

export build

fs build(fs input) {
	...
}

We also added the ability to consume external modules. Since we already have powerful primitives in HLB, it's only like HLB to import external modules with a target.

import go from fs {
	image "openllb/go.hlb"
}

In this example, we pushed to DockerHub an image openllb/go.hlb that contains a single module.hlb file. This file has a few exported functions, so we are able to import a HLB module from a container image. Since this is a regular fs function, you can retrieve the module from pretty much anywhere.

import gist from fs {
	http "https://gist.githubusercontent.com/hinshun/fdffbc3a1acb21558022f87b2f530817/raw/549cab5a5a467b34b30187ffc6db434afc575b4d/module.hlb"
}

fs default() {
	gist.message
}

We started a new repository openllb/modules to author some high-quality HLB modules for the community. You can think of it as the start of a docker/library equivalent but for Dockerfiles. Maybe soon enough, you'll see a registry of HLB modules that is indexed and searchable!

hlb 0.1

Welcome to the first pre-release of hlb!

The language is functional but has not stabilized yet, beware!

For your convenience, there are darwin-amd64 and linux-amd64 static binaries available. Since BuildKit isn't functional on Windows yet, hlb is also not supported on Windows atm.