diff --git a/doc/freebsd.ml b/doc/freebsd.ml new file mode 100644 index 00000000..eceb1b44 --- /dev/null +++ b/doc/freebsd.ml @@ -0,0 +1,437 @@ +# Porting OBuilder to FreeBSD + +## The problem + +OBuilder is a tool used to perform arbitrary, reproduceable builds of +OCaml-related software within a sandboxed environment. + +It has been written for Linux, with support for Windows and MacOS systems being +added later. Porting to FreeBSD is the next logical step, since FreeBSD +(at least on amd64 and arm64 hardware) is a Tier 1 platform in the OCaml +ecosystem. + +## The challenge + +Being initially Linux-centric, OBuilder is architected around three major +requirements: + +- initial build environments are `docker` images. +- sandboxing is performed using the Open Container Initiative tool `runc`. +- a filesystem with snapshot capabilities is needed, and acts as a cache of + identical build steps. + +Neither of the first two items are available under FreeBSD (a `docker` client +is available but quite useless as there is no native `docker` server), +therefore alternative solutions must be found. As for the filesystem +requirement, FreeBSD has been supporting Sun's ZFS filesystem out of the box for +many releases now. + +Fortunately, the existing archicture in OBuilder encapsulates these needs as +`Fetcher`, `Sandbox` and `Store` modules, respectively, so the only work +required would be write FreeBSD-specific `Fetcher` and `Sandbox` modules. + +## Porting to FreeBSD + +### The fetcher + +An initial attempt was made to fetch Docker images without using the `docker` +command. An existing script, `download-frozen-image`, can be found in the +`moby` Github project (the open source parts of `docker`) to that effect. + +However, although using that script to fetch the various layers of the `docker` +image and apply them in order, the result will be useless, from a FreeBSD +perspective, as all the `docker` images available are filled with Linux +binaries, which can run under FreeBSD with the help of the compatibility module, +but would mislead the OCaml toolchain into believing it is running under Linux, +and thus would build Linux binaries. + +Until `docker` is available under FreeBSD, there won't be a repository of +FreeBSD images suitable for use for OBuilder. Such images will, at least +in the beginning, be built locally in the Tarides CI network. It therefore +makes sense to expect `.tar.gz` archives to be available from the CI network, +and simply download and extract them to implement the `Fetcher` module. +Moreover, FreeBSD provides its own `fetch` command which is able to download +files over `http` and `https`, and can also use `file://` URIs, which turned +out to be very helpful during development. + +There is currently no attempt to support aliases or canonical names, so all the +`(from ...)` stanza in OBuilder command files will need to be adjusted for use +with FreeBSD. This limitation can be overcome by pre-populating the OBuilder +cache with the most used images under their expected names on the OBuilder +worker systems. + +### The sandbox + +FreeBSD comes with its own sandboxing mechanism, named `jail`, since the late +1990s. In addition to only having access to a subset of the filesystem, jails +can also be denied network access, which fits the OBuilder usage pattern, where +network access is only allowed to fetch build dependencies. + +In order to start a jail, the `jail` command is invoked with either a plain text +configuration file providing its configuration, or with the configuration +parameters (in the "name=value" form) on its commandline. + +In order to keep things simple in OBuilder, and since the jail configuration +will only need a few parameters, they are all passed on the commandline. This +might be a problem, should the length of the command line to run, as specified +in the OBuilder command file, reach the FreeBSD command line size limit, but +as this limit is a few hundred kilobytes, this does not seem to be a serious +concern. + +The `jail` invocation will provide: + +- a unique jail name. +- the absolute path of the jail filesystem. +- the command (or shell script) to run in the jail. +- the user on behalf of which the command will be run. This requires the user to + exist within the jail filesystem (`/etc/passwd` and `/etc/group` entries). + +More options may be used to allow for network access, or specify commands to run +on the host or within the jail at various states of the jail lifecycle. + +Also, for processes running under the jail to behave correctly, a stripped-down +`devfs` pseudo-filesystem needs to be mounted on the `/dev` directory within +the jail, and while this can be done automatically by `jail(8)` using the proper +`mount.devfs` option, care must be taken to correctly unmount this directory +after the command run within the jail has exited. In order to be sure there will +be no leftover `devfs` mounts, which would prevent removal of the jail +filesystem at cleanup time, an `umount` command is run unconditionally by +OBuilder after the `jail` command exits. + +Lastly, since most, if not all, OBuilder commands will expect a proper `opam` +environment configuration, it is necessary to run the commands within a login +shell, and such a shell can only be run as root. Therefore the command which +will run within the jail is: + +``` + /usr/bin/su -l obuilder_user_name -c "cd obuilder_directory && obuilder_command" +``` + +# Solving the chicken-and-egg problem + +With the fetcher and the sandbox modules written, a complete OBuilder run can be +attempted. But in order to do this, two more pieces are needed: + +- a base FreeBSD image, to be used by OBuilder runs. +- a native FreeBSD build of OBuilder. + +The base FreeBSD image will require a proper `opam switch` to be created. +The commands used to create it can also be turned into an OBuilder command +file, in order to be able to quickly build any particular switch version. + +### FreeBSD setup + +The process of setting up a FreeBSD system, combined with the use of zfs +snapshots, allows various stages of the FreeBSD setup to be archived and +used as starting point for OBuilder operations. + +We can take snapshots at the following points: + +- after an initial simple FreeBSD installation, with a few external packages + required by OBuilder itself added, and an `opam` user created. This will be + known henceforth as "stage0". +- after `opam` has been installed and an initial `opam switch` created. This + setup can be directly used as a starting point for OBuilder operations, and + will be known henceforth as "stage1". + +### Optional: setting up a FreeBSD Virtual Machine + +If no physical machine is available to install FreeBSD on, one may use `qemu` +to run a virtual machine instead. + +Simply download the installation dvd, and setup a disk image: + +``` +qemu-img create -f qcow2 da0.qcow2 10G +``` + +Then launch a virtual machine with a few CPUs and a few gigabytes of memory: + +``` +qemu-system-x86_64 \ + -smp 4 \ + -m 4G \ + -drive file=da0.qcow2,format=qcow2 \ + -drive file=FreeBSD-13.2-RELEASE-amd64-dvd1.iso,media=cdrom +``` + +Since the disk image has been freshly created, this will boot into the +installation dvd. + +### Stage 0 + +FreeBSD installation is straightforward. There is nothing special to choose +within the installer, except for the use of ZFS as the default filesystem +(which is the default choice nowadays) and, of course, proper network +configuration (although the defaults of DHCP for IPv4 and SLAAC for IPv6 ought +to work in most networks). + +The default ZFS setup creates one single pool for the disk, and separate +filesystems in it. OBuilder will however require its own work pool, so the +default ZFS settings ("guided root on zfs" disk layout) in the installer can't +be used. It will be easier to build base images anyway if there is no separate +zfs pool for `/usr/home`. + +In order to save space, one might want to disable all optional system components +when asked which ones to install. + +After the installation completes and the system reboots, one may log in as root. + +A quick check of `/etc/rc.conf` should confirm that: + +- there is a `zfs_enable="YES"` line. +- network configuration is similar to + ``` + ifconfig_DEFAULT="DHCP inet6 accept_rtadv" + ``` + or + ``` + ifconfig_em0="DHCP" + ifconfig_em0_ipv6="inet6 accept_rtadv" + ``` + +At this point, it is possible to add an `opam` user with `adduser`: + +``` +echo 'opam:1000:::::::/bin/sh:' | adduser -q -w random -f - +``` +(that's seven colons between the numerical uid and the shell.) + +Note that the assigned random password of the `opam` user won't be displayed, +but this does not really matter since all uses of this account will be performed +through `/usr/bin/su -l opam`. + +A few packages need to be installed at this point: + +``` +pkg install -y bash curl git gmake patch rsync sudo zstd +``` + +Note that `zstd` is an optional dependency when building an OCaml 4 switch, +but a required dependency when building an OCaml 5 switch. + +Now that `sudo` has been installed, it should be configured to let the `opam` +user be able to use `sudo` for any command without entering any password, as +OBuilder depends on this: + +``` +echo "opam ALL=(ALL:ALL) NOPASSWD: ALL" > /usr/local/etc/sudoers.d/opam +chmod 440 /usr/local/etc/sudoers.d/opam +``` + +It is then time to take a snapshot of the system. Assuming the name of the +zfs pool for the root directory is `zroot`, run: + +``` +zfs snapshot zroot@stage0 +``` + +The first snapshot can be cloned in order to build a filesystem archive of +`stage0`, suitable to be used by OBuilder. + +``` +zfs clone zroot@stage0 zroot/clone +``` + +However, some of the files have been made immutable during the installation +(for security reasons), and would cause errors while attempting to clean up, +so it is preferrable to remove these flags: + +``` +chflags -R 0 /zroot/clone +``` + +Then, the image can be made significantly smaller by removing several file sets: +- rescue binaries and kernel modules (won't be used by OBuilder) +- profiling libraries +- manual pages + +``` +cd /zroot/clone +rm -fR rescue boot usr/share/games +rm usr/bin/fortune usr/lib/lib*_p.a +find usr/share/man -type f -delete +cd / +``` + +The last step consists of setting the usual permissions on the `/tmp` directory, +since there will be no specific filesystem mounted there. + +``` +chmod 1777 /zroot/clone/tmp +``` + +It is now possible to create the `stage0` archive and destroy the snapshot +clone: + +``` +mkdir /archive +tar -C /zroot/clone -czf /archive/stage0.tar.gz . +zfs destroy zroot/clone +``` + +The warnings regarding the inability to archive sockets in `/var` can be safely +ignored. + +### Stage 1 + +Once an archive of `stage0` is available, it is possible to install `opam` and +build an initial switch with the following OBuilder command file: + +``` +((build dev + ((from file:///path/to/stage0.tar.gz) + (workdir /home/opam) + (user (name opam)) + (run + (network host) + (shell "fetch -q https://github.com/ocaml/opam/releases/download/2.1.4/opam-full-2.1.4.tar.gz")) + (run + (shell "tar xzf opam-full-2.1.4.tar.gz")) + (run + (network host) ; needed to fetch compiler and libraries + (shell + "cd opam-full-2.1.4 && \ + gmake compiler && + ./configure --prefix=/home/opam && + gmake lib-ext")) + (run + (shell "cd opam-full-2.1.4 && gmake && gmake install")) + (run + (network host) + (shell "opam init -y -a --bare")) + (run + (network host) + (shell "opam switch create 4.14.1")) +)) + ; nothing to do after build, but a valid from stanza is required + (from file:///path/to/stage0.tar.gz) +) +``` + +The first time, of course, OBuilder is not available yet, so these commands need +to be run manually: + +``` +/usr/bin/su -l opam +``` + +Then: + +``` +fetch -q https://github.com/ocaml/opam/releases/download/2.1.4/opam-full-2.1.4.tar.gz && +tar xzf opam-full-2.1.4.tar.gz && +cd opam-full-2.1.4 && +gmake compiler && +./configure --prefix=/home/opam && +gmake lib-ext && +gmake && +gmake install && +opam init -y -a --bare +. ~/.opam/opam-init/init.sh +opam switch create 4.14.1 +exit +``` + +Once this is done, stage1 is complete; a zfs snapshot can be created, then +a new archive can be built; one may consider removing sources +in order to make the archive smaller. + +``` +rm /usr/home/opam/opam-full-2.1.4.tar.gz +rm -fR /usr/home/opam/opam-full-2.1.4 +rm -fR /usr/home/opam/.opam/*/.opam-switch/sources +rm -fR /usr/home/opam/.opam/download-cache +zfs snapshot zroot@stage1 +mkdir /archive/stage1 +cd /archive/stage1 +tar xzpf /archive/stage0.tar.gz +tar -C /usr/home -cf - opam | tar -C ./usr/home -xpf - +tar czf /archive/stage1.tar.gz . +cd /archive +rm -fR stage1 +``` + +### Building OBuilder under FreeBSD + +This step is quite straightforward (and will be even simpler once the FreeBSD +support bits are merged into the main repository): + +``` +pkg install -y pkgconf sqlite3 +/usr/bin/su -l opam +git clone https://github.com/dustanddreams/obuilder.git +cd obuilder +git checkout freebsdjail-sandbox +sed -i.orig 's/(Docker_extract)/(Archive_extract)/' main.ml +opam install -y dune +opam install -y --deps-only -t obuilder +opam install -y crunch extunix fpath # missed by the above step +dune build && dune install +``` + +or, using an existing OBuilder setup, adapted from `obuilder.spec` found in +the OBuilder source repository: + +``` +((build dev + ((from file:///path/to/stage1.tar.gz) + (workdir /src) + (user (uid 1000) (gid 1000)) + (run (shell "sudo chown opam /src")) + ; Copy just the opam file first (helps caching) + (copy (src obuilder-spec.opam obuilder.opam) (dst ./)) + (run (shell "opam pin add -yn .")) + ; Install OCaml dependencies + (run + (network host) + (shell "sudo pkg install -y pkgconf sqlite3")) + (run + (network host) + (cache (opam-archives (target /home/opam/.opam/download-cache))) + (shell "opam install -y --deps-only -t obuilder")) + (copy + (src .) + (dst /src/) + (exclude .git _build _opam)) + ; Build and test + (run (shell "opam exec -- dune build @install @runtest")))) + ; Now generate a small runtime image with just the resulting binary: + (from file:///path/to/stage0.tar.gz) + (run + (network host) + (shell "pkg install -y libedit pkgconf sqlite3")) + (copy (from (build dev)) + (src /src/_build/default/main.exe) + (dst /usr/local/bin/obuilder)) + (run (shell "obuilder --help"))) +``` + +## Integrating with OCluster + +OCluster is a larger framework which processes build requests on a cluster +of systems, each running OBuilder. In order to make the FreeBSD systems +compatible with OCluster needs, a few more adjustments are necessary: + +- a `docker` client needs to be installed on the OBuilder machine, even if it + will not be used, as part of OCluster checks. Fortunately the `docker` client + is available as a FreeBSD package, `pkg install -y docker` will do. +- an `obuilder` zfs pool needs to be created (on a separate disk or a separate + partition). + +In addition to this, the `stage0` and `stage1` snapshots can be used to set up +an initial image cache on each OBuilder worker machine, using `zfs send` on the +source machine and `zfs recv` on the build machine, to make the `stage0` and +`stage1` snapshots available as `/obuilder/base-image/busybox` and +`/obuilder/base-image/ocaml-4.14.1` respectively. + +Since the cache will take precedence over the fetcher action, this will allow +OBuilder spec files to keep referring to the names from the docker registry. + + +## Conclusion + +The modular design of OBuilder has allowed for it to be easily adapted to run +under FreeBSD. A few FreeBSD systems are currently being set up as OBuilder +workers within the OCluster orchestrator used by Tarides for automated OCaml +package testing, and will hopefully benefit the OCaml FreeBSD community. diff --git a/doc/index.mld b/doc/index.mld index 0f6ec091..c59b1fdc 100644 --- a/doc/index.mld +++ b/doc/index.mld @@ -4,6 +4,7 @@ The entry point of this library is the module: {!module-Obuilder}. {1 Design and implementation of OBuilder} -- {{!page-README}OBuilder's manual}; -- {{!page-macOS}macOS implementation documentation}; +- {{!page-README}OBuilder's manual}. +- {{!page-macOS}macOS implementation documentation}. - {{!page-windows}Windows implementation documentation}. +- {{!page-freebsd}FreeBSD implementation documentation}. diff --git a/lib/archive_extract.ml b/lib/archive_extract.ml new file mode 100644 index 00000000..2c84bd0d --- /dev/null +++ b/lib/archive_extract.ml @@ -0,0 +1,28 @@ +open Lwt.Infix + +let invoke_fetcher base destdir = + Os.with_pipe_between_children @@ fun ~r ~w -> + let stdin = `FD_move_safely r in + let stdout = `FD_move_safely w in + let stderr = stdout in + let fetcher = + Os.exec ~stdout ~stderr ["fetch"; "-q" ; "-o" ; "-" ; base ] + in + let extracter = + Os.sudo ~stdin [ "tar" ; "-C"; destdir ; "-xzpf"; "-" ] + in + fetcher >>= fun () -> + extracter + +let fetch ~log ~rootfs base = + let _ = log in + Lwt.catch + (fun () -> + invoke_fetcher base rootfs >>= fun () -> + let env = [] in + Lwt.return env) + (function + | Sys_error s -> + Fmt.failwith "Archive fetcher encountered a system error: %s" s + | e -> Lwt.fail e) + diff --git a/lib/archive_extract.mli b/lib/archive_extract.mli new file mode 100644 index 00000000..2d3ad7c2 --- /dev/null +++ b/lib/archive_extract.mli @@ -0,0 +1,3 @@ +(** Fetching of base images as .tar.gz archives *) + +include S.FETCHER diff --git a/lib/dune b/lib/dune index 9b18fdd1..3410deb1 100644 --- a/lib/dune +++ b/lib/dune @@ -4,10 +4,16 @@ (enabled_if (= %{system} macosx)) (action (copy %{deps} %{target}))) +(rule + (deps sandbox.jail.ml) + (target sandbox.ml) + (enabled_if (= %{system} freebsd)) + (action (copy %{deps} %{target}))) + (rule (deps sandbox.runc.ml) (target sandbox.ml) - (enabled_if (<> %{system} macosx)) + (enabled_if (and (<> %{system} macosx) (<> %{system} freebsd))) (action (copy %{deps} %{target}))) (rule diff --git a/lib/obuilder.ml b/lib/obuilder.ml index ed3c9ea1..b3e19893 100644 --- a/lib/obuilder.ml +++ b/lib/obuilder.ml @@ -16,8 +16,9 @@ module Store_spec = Store_spec module Docker_store = Docker_store (** {2 Fetchers} *) -module User_temp = User_temp +module Zfs_clone = Zfs_clone module Docker_extract = Docker.Extract +module Archive_extract = Archive_extract (** {2 Sandboxes} *) diff --git a/lib/sandbox.jail.ml b/lib/sandbox.jail.ml new file mode 100644 index 00000000..80e9819a --- /dev/null +++ b/lib/sandbox.jail.ml @@ -0,0 +1,174 @@ +open Lwt.Infix +open Sexplib.Conv + +let ( / ) = Filename.concat + +type t = { + jail_name_prefix : string; +} + +type config = unit [@@deriving sexp] + +(* Find out the user name to use within the jail, by parsing the + /etc/passwd file within the jail filesystem. This is roughly + equivalent to what Unix.getpwuid would do. + Note that the gid is currently ignored. *) +let jail_username rootdir config = + match config.Config.user with + | `Windows w -> w.name + | `Unix { uid; _ } -> + let pwdfile = rootdir / "etc" / "passwd" in + let uidstr = string_of_int uid in + let rec parse_line ch = + match In_channel.input_line ch with + | None -> None + | Some line -> + let fields = String.split_on_char ':' line in begin + match fields with + | name :: _pass :: uid :: _ -> + if uid = uidstr then Some name else parse_line ch + | _ -> parse_line ch + end + in + match In_channel.with_open_text pwdfile parse_line with + | None -> Fmt.failwith "No user found for uid %d" uid + | Some name -> name + +(* Compute the complete set of arguments passed to the jail(8) command: + jail username, jail path, command to run, etc. *) +let jail_options config rootdir tmp_dir = + let cache = match List.length config.Config.mounts with + | 0 -> [] + | _ -> + let path = tmp_dir / "fstab" in + let rec print_fstab oc = function + | [] -> close_out oc + | { Config.Mount.src; dst; readonly; _ } :: tl -> + let full = rootdir ^ dst in + Os.ensure_dir full; + Printf.fprintf oc "%s %s nullfs %s 0 0\n" src full (if readonly then "ro" else "rw"); + print_fstab oc tl in + let oc = open_out path in + print_fstab oc config.Config.mounts; + [ "mount.fstab=" ^ path ] in + let username = jail_username rootdir config in + let commandline = + let env = List.rev_map (fun (k, v) -> k ^ "='" ^ v ^ "'") config.env in + let commandline = List.rev (List.rev_map Filename.quote config.argv) in + let commandline = + match env with + | [] -> commandline + | _ -> "env" :: List.rev_append env commandline + in + let commandline = + String.concat " " + ([ "cd" ; Filename.quote config.cwd ; "&&" ] @ commandline) + in + (* Ask for a login shell in order to properly source opam settings. *) + [ "command=/usr/bin/su" ; "-l" ; username ; "-c" ; commandline ] + in + let path = "path=" ^ rootdir in + let devfs_setup = "mount.devfs" in + let options = + let options = [ path ; devfs_setup ] @ cache in + match config.network with + | [ "host" ] -> + "ip4=inherit" :: "ip6=inherit" :: "host=inherit" :: options + | _ -> + "exec.start=/sbin/ifconfig lo0 127.0.0.1/8" :: "vnet" :: options + in + List.rev_append options commandline + +let copy_to_log ~src ~dst = + let buf = Bytes.create 4096 in + let rec aux () = + Lwt_unix.read src buf 0 (Bytes.length buf) >>= function + | 0 -> Lwt.return_unit + | n -> Build_log.write dst (Bytes.sub_string buf 0 n) >>= aux + in + aux () + +let jail_id = ref 0 + +let run ~cancelled ?stdin:stdin ~log (t : t) config rootdir = + Lwt_io.with_temp_dir ~prefix:"obuilder-jail-" @@ fun tmp_dir -> + let zfs_volume = String.sub rootdir 1 (String.length rootdir - 1) in (* remove / from front *) + Os.sudo [ "zfs"; "inherit"; "mountpoint"; zfs_volume ^ "/rootfs" ] >>= fun () -> + let cwd = rootdir in + let jail_name = t.jail_name_prefix ^ "_" ^ string_of_int !jail_id in + incr jail_id; + Os.with_pipe_from_child @@ fun ~r:out_r ~w:out_w -> + let rootdir = rootdir / "rootfs" in + let workdir = rootdir / config.Config.cwd in + (* Make sure the work directory exists prior to starting the jail. *) + begin + match Os.check_dir workdir with + | `Present -> Lwt.return_unit + | `Missing -> Os.sudo [ "mkdir" ; "-p" ; workdir ] + end >>= fun () -> + let stdout = `FD_move_safely out_w in + let stderr = stdout in + let copy_log = copy_to_log ~src:out_r ~dst:log in + let proc = + let cmd = + let options = jail_options config rootdir tmp_dir in + "jail" :: "-c" :: ("name=" ^ jail_name) :: options + in + let stdin = Option.map (fun x -> `FD_move_safely x) stdin in + let pp f = Os.pp_cmd f ("", cmd) in + (* This is similar to + Os.sudo_result ~cwd ?stdin ~stdout ~stderr ~pp cmd + but also unmounting the in-jail devfs if necessary, see below. *) + let cmd = if Os.running_as_root then cmd else "sudo" :: "--" :: cmd in + Logs.info (fun f -> f "Exec %a" Os.pp_cmd ("", cmd)); + !Os.lwt_process_exec ~cwd ?stdin ~stdout ~stderr ~pp + ("", Array.of_list cmd) >>= function + | Ok 0 -> + let fstab = tmp_dir / "fstab" in + (if Sys.file_exists fstab + then + let cmd = [ "sudo" ; "/sbin/umount" ; "-a" ; "-F" ; fstab ] in + Os.exec ~is_success:(fun _ -> true) cmd + else Lwt.return_unit) >>= fun () -> + (* If the command within the jail completes, the jail is automatically + removed, but without performing any of the stop and release actions, + thus we can not use "exec.stop" to unmount the in-jail devfs + filesystem. Do this here, ignoring the exit code of umount(8). *) + let cmd = [ "sudo" ; "/sbin/umount" ; rootdir / "dev" ] in + Os.exec ~is_success:(fun _ -> true) cmd >>= fun () -> + Lwt_result.ok Lwt.return_unit + | Ok n -> Lwt.return @@ Fmt.error_msg "%t failed with exit status %d" pp n + | Error e -> Lwt_result.fail e + in + Lwt.on_termination cancelled (fun () -> + let rec aux () = + if Lwt.is_sleeping proc then ( + let pp f = Fmt.pf f "jail -r obuilder" in + Os.sudo_result ~cwd [ "jail" ; "-r" ; jail_name ] ~pp >>= function + | Ok () -> Lwt.return_unit + | Error (`Msg _) -> + Lwt_unix.sleep 10.0 >>= aux + ) else Lwt.return_unit (* Process has already finished *) + in + Lwt.async aux + ); + proc >>= fun r -> + copy_log >>= fun () -> + if Lwt.is_sleeping cancelled then + Lwt.return (r :> (unit, [`Msg of string | `Cancelled]) result) + else + Lwt_result.fail `Cancelled + +let create ~state_dir:_ _c = + Lwt.return { + (* Compute a unique (across obuilder instances) name prefix for the jail. *) + jail_name_prefix = "obuilder_" ^ (Int.to_string (Unix.getpid ())); + } + +let finished () = + Lwt.return () + +open Cmdliner + +let cmdliner : config Term.t = + Term.(const ()) diff --git a/lib/user_temp.ml b/lib/user_temp.ml deleted file mode 100644 index 4792b725..00000000 --- a/lib/user_temp.ml +++ /dev/null @@ -1,9 +0,0 @@ -open Lwt.Infix - -let ( / ) = Filename.concat - -let fetch ~log:_ ~rootfs base = - let zfs_volume = String.sub rootfs 9 (String.length rootfs - 16) in (* remove /Volume/ from front and /rootfs from the end *) - Os.sudo [ "zfs"; "clone"; "-o"; "mountpoint=none"; "obuilder" / "base-image" / base / "home@snap"; zfs_volume / "home" ] >>= fun () -> - Os.sudo [ "zfs"; "clone"; "-o"; "mountpoint=none"; "obuilder" / "base-image" / base / "brew@snap"; zfs_volume / "brew" ] >>= fun () -> - Lwt.return [] diff --git a/lib/user_temp.mli b/lib/user_temp.mli deleted file mode 100644 index 9db5e07e..00000000 --- a/lib/user_temp.mli +++ /dev/null @@ -1,4 +0,0 @@ -include S.FETCHER -(** The user template fetcher assumes given some [base] "image" that - there is a corresponding directory [/Users/] and [rsync]s - this directory over to the provided [rootfs] directory. *) diff --git a/lib/zfs_clone.ml b/lib/zfs_clone.ml new file mode 100644 index 00000000..e7f930ec --- /dev/null +++ b/lib/zfs_clone.ml @@ -0,0 +1,29 @@ +open Lwt.Infix + +let ( / ) = Filename.concat + +(* On FreeBSD the input is + rootfs = "/obuilder/result/522fb2a0e81ba278bc1ae7314bd754201505e6493f4f2f40a166c416624a4005/rootfs" + with base = "busybox", or base = "freebsd-ocaml-4.14" -> just clone rootfs + + On macOS the input is + rootfs = "/Volumes/obuilder/result/522fb2a0e81ba278bc1ae7314bd754201505e6493f4f2f40a166c416624a4005/rootfs" + with base = "busybox", or base = "macos-homebrew-ocaml-4.14" -> clone home and brew subvolumes *) + +let fetch ~log:_ ~rootfs base = + let path = + let remove_on_match s lst = if List.hd lst = s then List.tl lst else lst in + String.split_on_char '/' rootfs + |> List.filter (fun x -> String.length x > 0) + |> remove_on_match "Volumes" |> List.rev + |> remove_on_match "rootfs" |> List.rev in + let zfs_rootfs = String.concat "/" path in + let base_image = (List.hd path) / "base-image" / base in + Lwt_process.pread ("", [| "zfs"; "list"; "-H"; "-r"; "-o"; "name"; base_image |]) >>= fun output -> + let len = String.length base_image in + String.split_on_char '\n' output |> List.map (fun s -> (s, String.length s)) |> + List.filter (fun (_, l) -> l > len) |> List.map (fun (s, l) -> String.sub s (len + 1) (l - len - 1)) |> + Lwt_list.iter_s (fun subvolume -> + Os.sudo ["zfs"; "clone"; base_image / subvolume ^ "@snap"; zfs_rootfs / subvolume ]) >>= fun () -> + Lwt.return [] + diff --git a/lib/zfs_clone.mli b/lib/zfs_clone.mli new file mode 100644 index 00000000..48826a99 --- /dev/null +++ b/lib/zfs_clone.mli @@ -0,0 +1,4 @@ +include S.FETCHER +(** The ZFS Clone fetcher assumes given some [base] "image" that + there is a corresponding ZFS volume [obuilder/base-image/] + and [zfs clones] this directory over to the provided [rootfs]. *) diff --git a/main.ml b/main.ml index 183dc023..76e0c735 100644 --- a/main.ml +++ b/main.ml @@ -6,6 +6,7 @@ module Native_sandbox = Obuilder.Native_sandbox module Docker_sandbox = Obuilder.Docker_sandbox module Docker_store = Obuilder.Docker_store module Docker_extract = Obuilder.Docker_extract +module Archive_extract = Obuilder.Archive_extract module Store_spec = Obuilder.Store_spec type builder = Builder : (module Obuilder.BUILDER with type t = 'a) * 'a -> builder