Skip to content

Commit

Permalink
Updated gitian builder documentation
Browse files Browse the repository at this point in the history
  • Loading branch information
wagerr-builder committed Mar 4, 2023
1 parent 882eb48 commit 6a2319d
Show file tree
Hide file tree
Showing 16 changed files with 43 additions and 73 deletions.
116 changes: 43 additions & 73 deletions doc/gitian-building.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,46 +41,48 @@ Debian Linux was chosen as the host distribution because it has a lightweight in
Any kind of virtualization can be used, for example:
- [VirtualBox](https://www.virtualbox.org/) (covered by this guide)
- [KVM](http://www.linux-kvm.org/page/Main_Page)
- [LXC](https://linuxcontainers.org/), see also [Gitian host docker container](https://github.com/gdm85/tenku/tree/master/docker/gitian-bitcoin-host/README.md).
- [Docker](https://docker.com/), see also [Gitian host docker container](https://github.com/gdm85/tenku/tree/master/docker/gitian-bitcoin-host/README.md).

You can also install Gitian on actual hardware instead of using virtualization.

Create a new VirtualBox VM
---------------------------
In the VirtualBox GUI click "New" and choose the following parameters in the wizard:

Get the [Debian 11.x net installer](https://cdimage.debian.org/mirror/cdimage/release/current/amd64/iso-cd/debian-11.6.0-amd64-netinst.iso) (a more recent minor version should also work, see also [Debian Network installation](https://www.debian.org/CD/netinst/)).
This DVD image can be [validated](https://www.debian.org/CD/verify) using a SHA256 hashing tool, for example on
Unixy OSes by entering the following in a terminal:

echo "e482910626b30f9a7de9b0cc142c3d4a079fbfa96110083be1d0b473671ce08d debian-11.6.0-amd64-netinst.iso" | sha256sum -c

# (must return OK)
In the VirtualBox GUI click "New" and set the following parameters in the wizard:

![](gitian-building/create_new_vm.png)

- Type: Linux, Debian (64-bit)

- Make sure that `Skip Unattended Install` is checked.

![](gitian-building/create_vm_memsize.png)

- Memory Size: at least 3000MB, anything less and the build might not complete.
- Memory Size: at least 4096MB, anything less and the build might not complete.
- Increase the number of processors to the number of cores on your machine if you want builds to be faster.

![](gitian-building/create_vm_hard_disk.png)

- Hard Disk: Create a virtual hard disk now

![](gitian-building/create_vm_hard_disk_file_type.png)
- File location and size: at least 120GB; as low as 20GB *may* be possible, but better to err on the safe side

- Hard Disk file type: Use the default, VDI (VirtualBox Disk Image)

![](gitian-building/create_vm_storage_physical_hard_disk.png)
- Storage on physical hard disk: Dynamically Allocated (the default)

- Storage on physical hard disk: Dynamically Allocated

![](gitian-building/create_vm_file_location_size.png)

- File location and size: at least 40GB; as low as 20GB *may* be possible, but better to err on the safe side
- Click `Create`
- Click `Finish`

After creating the VM, we need to configure it.

- Click the `Settings` button, then go to `System` tab and `Processor` sub-tab. Increase the number of processors to the number of cores on your machine if you want builds to be faster.

![](gitian-building/system_settings.png)

- Go to the `Network` tab. Adapter 1 should be attached to `NAT`.
- Click the `Settings` button, then go to `System` tab and `Network` Adapter 1 should be attached to `NAT`.

![](gitian-building/network_settings.png)

Expand All @@ -99,22 +101,14 @@ After creating the VM, we need to configure it.

- Click `Ok` twice to save.

Get the [Debian 8.x net installer](http://cdimage.debian.org/mirror/cdimage/archive/8.5.0/amd64/iso-cd/debian-8.5.0-amd64-netinst.iso) (a more recent minor version should also work, see also [Debian Network installation](https://www.debian.org/CD/netinst/)).
This DVD image can be [validated](https://www.debian.org/CD/verify) using a SHA256 hashing tool, for example on
Unixy OSes by entering the following in a terminal:

echo "ad4e8c27c561ad8248d5ebc1d36eb172f884057bfeb2c22ead823f59fa8c3dff debian-8.5.0-amd64-netinst.iso" | sha256sum -c
# (must return OK)

Then start the VM. On the first launch you will be asked for a CD or DVD image. Choose the downloaded ISO.

![](gitian-building/select_startup_disk.png)

Installing Debian
------------------

This section will explain how to install Debian on the newly created VM.

- Start the VM

- Choose the non-graphical installer. We do not need the graphical environment; it will only increase installation time and disk usage.

![](gitian-building/debian_install_1_boot_menu.png)
Expand All @@ -141,8 +135,8 @@ To select a different button, press `Tab`.

![](gitian-building/debian_install_6a_set_up_root_password.png)

- Name the new user `debian` (the full name doesn't matter, you can leave it empty)
- Set the account username as `debian`
- Name the new user `gitianbuilder` (the full name doesn't matter, you can leave it empty)
- Set the account username as `gitianbuilder`

![](gitian-building/debian_install_7_set_up_user_fullname.png)
![](gitian-building/debian_install_8_set_up_username.png)
Expand Down Expand Up @@ -218,11 +212,12 @@ You'll be presented with a screen similar to this.
Type:

```
systemctl ssh enable
sed -i 's/^PermitRootLogin.*/PermitRootLogin yes/' /etc/ssh/sshd_config
```
and press enter. Then,
```
/etc/init.d/ssh restart
systemctl ssh restart
```
and enter to restart SSH. Logout by typing 'logout' and pressing 'enter'.

Expand All @@ -248,9 +243,9 @@ For example, to connect as `root` from a Linux command prompt use

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
root@debian:~#
root@gitianbuiilder:~#

Replace `root` with `debian` to log in as user.
Replace `root` with `gitianbuilder` to log in as user.

Setting up Debian for Gitian building
--------------------------------------
Expand All @@ -261,39 +256,18 @@ First we need to log in as `root` to set up dependencies and make sure that our
user can use the sudo command. Type/paste the following in the terminal:

```bash
apt-get install git ruby sudo apt-cacher-ng qemu-utils debootstrap lxc python-cheetah parted kpartx bridge-utils make ubuntu-archive-keyring curl
adduser debian sudo
```

Then set up LXC and the rest with the following, which is a complex jumble of settings and workarounds:

```bash
# the version of lxc-start in Debian needs to run as root, so make sure
# that the build script can execute it without providing a password
echo "%sudo ALL=NOPASSWD: /usr/bin/lxc-start" > /etc/sudoers.d/gitian-lxc
echo "%sudo ALL=NOPASSWD: /usr/bin/lxc-execute" >> /etc/sudoers.d/gitian-lxc
# make /etc/rc.local script that sets up bridge between guest and host
echo '#!/bin/sh -e' > /etc/rc.local
echo 'brctl addbr lxcbr0' >> /etc/rc.local
echo 'ifconfig lxcbr0 10.0.3.1/24 up' >> /etc/rc.local
echo 'iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE' >> /etc/rc.local
echo 'echo 1 > /proc/sys/net/ipv4/ip_forward' >> /etc/rc.local
echo 'exit 0' >> /etc/rc.local
# make sure that USE_LXC is always set when logging in as debian,
# and configure LXC IP addresses
echo 'export USE_LXC=1' >> /home/debian/.profile
echo 'export GITIAN_HOST_IP=10.0.3.1' >> /home/debian/.profile
echo 'export LXC_GUEST_IP=10.0.3.5' >> /home/debian/.profile
apt-get install git ruby sudo apt-cacher-ng qemu-utils debootstrap docker docker.io parted kpartx bridge-utils make curl wget python2-minimal
usermod -aG sudo gitianbuilder
usermod -aG docker gitianbuilder
reboot
```

At the end the VM is rebooted to make sure that the changes take effect. The steps in this
section only need to be performed once.

Installing Gitian
------------------

Re-login as the user `debian` that was created during installation.
Re-login as the user `gitianbuilder` that was created during installation.
The rest of the steps in this guide will be performed as that user.

There is no `python-vm-builder` package in Debian, so we need to install it from source ourselves,
Expand All @@ -304,7 +278,7 @@ echo "76cbf8c52c391160b2641e7120dbade5afded713afaa6032f733a261f13e6a8e vm-build
# (verification -- must return OK)
tar -zxvf vm-builder_0.12.4+bzr494.orig.tar.gz
cd vm-builder-0.12.4+bzr494
sudo python setup.py install
sudo python2 setup.py install
cd ..
```

Expand All @@ -327,23 +301,19 @@ This image will be copied and used every time that a build is started to
make sure that the build is deterministic.
Creating the image will take a while, but only has to be done once.

Execute the following as user `debian`:
Execute the following as user `gitianbuilder`:

```bash
cd gitian-builder
bin/make-base-vm --lxc --arch amd64 --suite bionic
bin/make-base-vm --docker --arch amd64 --distro debian --suite bullseye --disksize 22287
```

There will be a lot of warnings printed during the build of the image. These can be ignored.
There may be some warnings printed during the build of the image. These can be ignored.

**Note**: When sudo asks for a password, enter the password for the user *debian* not for *root*.
**Note**: When sudo asks for a password, enter the password for the user *gitianbuilder* not for *root*.

**Note**: Repeat this step when you have upgraded to a newer version of Gitian.

**Note**: if you get the error message *"bin/make-base-vm: mkfs.ext4: not found"* during this process you have to make the following change in file *"gitian-builder/bin/make-base-vm"* at line 117:
```bash
# mkfs.ext4 -F $OUT-lxc
/sbin/mkfs.ext4 -F $OUT-lxc # (some Gitian environents do NOT find mkfs.ext4. Some do...)
```
Getting and building the inputs
Expand Down Expand Up @@ -383,7 +353,7 @@ Output from `gbuild` will look something like
Resolving deltas: 100% (41590/41590), done.
From https://github.com/wagerr/wagerr
... (new tags, new branch etc)
--- Building for bionic amd64 ---
--- Building for bullseye amd64 ---
Stopping target if it is up
Making a new image copy
stdin: is not a tty
Expand Down Expand Up @@ -426,20 +396,20 @@ To configure apt-cacher-ng as an offline cacher, you will need to first populate
files. You must additionally patch target-bin/bootstrap-fixup to set its apt sources to something other than
plain archive.ubuntu.com: us.archive.ubuntu.com works.
So, if you use LXC:
So, if you use Docker:
```bash
export PATH="$PATH":/path/to/gitian-builder/libexec
export USE_LXC=1
export USE_DOCKER=1
cd /path/to/gitian-builder
./libexec/make-clean-vm --suite bionic --arch amd64
libexec/make-ckean-vm --docker --arch amd64 --distro debian --suite bullseye --disksize 22287
LXC_ARCH=amd64 LXC_SUITE=bionic on-target -u root apt-get update
LXC_ARCH=amd64 LXC_SUITE=bionic on-target -u root \
DOCKER_ARCH=amd64 DOCKER_SUITE=bullseye on-target -u root apt-get update
DOCKER_ARCH=amd64 DOCKER_SUITE=bullseye on-target -u root \
-e DEBIAN_FRONTEND=noninteractive apt-get --no-install-recommends -y install \
$( sed -ne '/^packages:/,/[^-] .*/ {/^- .*/{s/"//g;s/- //;p}}' ../wagerr/contrib/gitian-descriptors/*|sort|uniq )
LXC_ARCH=amd64 LXC_SUITE=bionic on-target -u root apt-get -q -y purge grub
LXC_ARCH=amd64 LXC_SUITE=bionic on-target -u root -e DEBIAN_FRONTEND=noninteractive apt-get -y dist-upgrade
DOCKER_ARCH=amd64 DOCKER_SUITE=bullseye on-target -u root apt-get -q -y purge grub
DOCKER_ARCH=amd64 DOCKER_SUITE=bullseye on-target -u root -e DEBIAN_FRONTEND=noninteractive apt-get -y dist-upgrade
```
And then set offline mode for apt-cacher-ng:
Expand Down
Binary file modified doc/gitian-building/create_new_vm.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified doc/gitian-building/create_vm_hard_disk.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file not shown.
Binary file modified doc/gitian-building/create_vm_memsize.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file not shown.
Binary file modified doc/gitian-building/debian_install_12_choose_disk.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified doc/gitian-building/debian_install_16_choose_a_mirror.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified doc/gitian-building/debian_install_1_boot_menu.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified doc/gitian-building/debian_install_5_configure_the_network.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified doc/gitian-building/debian_install_7_set_up_user_fullname.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified doc/gitian-building/debian_install_8_set_up_username.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified doc/gitian-building/debian_root_login.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified doc/gitian-building/network_settings.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified doc/gitian-building/port_forwarding_rules.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file removed doc/gitian-building/select_startup_disk.png
Binary file not shown.

0 comments on commit 6a2319d

Please sign in to comment.