Supported environment to develop Blockbook is Linux. Although it is possible build and run Blockbook on macOS or Windows our build process is not prepared for it. But you can still build Blockbook manually.
The only dependency required to build Blockbook is Docker. You can see how to install Docker here. Manual build require additional dependencies that are described in appropriate section.
All build operations run in Docker container in order to keep build environment isolated. Makefile in root of repository define few targets used for building, testing and packaging of Blockbook. With Docker image definitions and Debian package templates in build/docker and build/templates respectively, they are only inputs that make build process.
Docker build images are created at first execution of Makefile and that information is persisted. (Actually there are
created two files in repository – .bin-image and .deb-image – that are used as tags.) Sometimes it is necessary to
rebuild Docker images, it is possible by executing make build-images.
Just run make and that is it. Output binary is stored in build directory. Note that although Blockbook is Go application
it is dynamically linked with RocksDB dependencies and ZeroMQ. Therefore operating system where Blockbook will be
executed still need that dependencies installed. See Manual build instructions below or install
Blockbook via Debian packages.
Standard binary contains no debug symbols. Execute make build-debug to get binary for debugging.
How to execute tests is described in separate document here.
Blockbook and particular coin back-end are usually deployed together. They are defined in same place as well. So typical way to build Debian packages is build Blockbook and back-end deb packages by single command. But it is not mandatory, of course.
Early releases of Blockbook weren't so friendly for extending. One had to define back-end package, Blockbook package, back-end configuration and Blockbook configuration as well. There were many options that were duplicated across configuration files and therefore error prone.
Actually all configuration options and also build options for both Blockbook and back-end are defined in single JSON file and all stuff required during build is generated dynamically.
Makefile targets follow simple pattern, there are few prefixes that define what to build.
-
deb-blockbook-<coin> – Build Blockbook package for given coin.
-
deb-backend-<coin> – Build back-end package for given coin.
-
deb-<coin> – Build both Blockbook and back-end packages for given coin.
-
all-<coin> – Similar to deb-<coin> but clean repository and rebuild Docker image before package build. It is useful for production deployment.
-
all – Build both Blockbook and back-end packages for all coins.
Which coins are possible to build is defined in configs/coins. Particular coin has to have JSON config file there.
For example we want to build some packages for Bitcoin and Bitcoin Testnet.
# make all-bitcoin deb-backend-bitcoin_testnet
...
# ls build/*.deb
build/backend-bitcoin_0.16.1-satoshilabs-1_amd64.deb build/backend-bitcoin-testnet_0.16.1-satoshilabs-1_amd64.deb build/blockbook-bitcoin_0.0.6_amd64.debWe have built two back-end packages – for Bitcoin and Testnet – and Blockbook package for Bitcoin. Before build have been performed there was cleaned build directory and rebuilt Docker image.
There are few variables that can be passed to make in order to modify build process.
In general, build of Blockbook binary require some dependencies. They are downloaded automatically during build process
but if you need to build binary repeatedly it consumes a lot of time. Here comes variable UPDATE_VENDOR that if is
unset says that build process uses vendor (i.e. dependencies) from your local repository. For example:
make deb-bitcoin UPDATE_VENDOR=0. But before the command is executed there must be vendor directory populated,
you can do it by calling dep ensure --vendor-only. See Manual build instructions below.
All build targets allow pass additional parameters to underlying command inside container. It is possible via ARGS
variable. For example if you want run only subset of unit-tests, you will perform it by calling:
make test ARGS='-run TestBitcoinRPC' UPDATE_VENDOR=0
Common behaviour of Docker image build is that build steps are cached and next time they are executed much faster.
Although this is a good idea, when something went wrong you will need to override this behaviour somehow. Execute this
command: make build-images NO_CACHE=true.
All configuration keys described below are in coin definition file in configs/coins.
install and data directories
Both Blockbook and back-end have separated install and data directories. They use common preffix and are defined in configs/environ.json and all templates use them.
- back-end install directory is /opt/coins/nodes/<coin>.
- back-end data directory is /opt/coins/data/<coin>/backend.
- Blockbook install directory is /opt/coins/blockbook/<coin>.
- Blockbook data directory is /opt/coins/data/<coin>/blockbook.
coin used above is defined in coin.alias in coin definition file.
package names
Package names are defined in backend.package_name and blockbook.package_name in coin definition file. We use simple pattern <prefix>-<coin> to name packages where prefix is either blockbook or backend and coin is made similarly to coin.alias. We use convention that coin name uses lowercase characters and dash '-' as a word delimiter. Testnet versions of coins must have -testnet suffix. That differs from coin.alias because underscore has a special meaning in Debian packaging. For example there are packages backend-bitcoin and blockbook-bitcoin-testnet.
user names
User names are defined in backend.system_user and blockbook.system_user in coin definition file. We follow common Linux conventions, user names use lowercase characters and dash '-' as a word delimiter.
Back-end user name use coin name only, including testnet services. For example there is bitcoin user for both backend-bitcoin and backend-bitcoin-testnet packages.
Blockbook user name has blockbook- prefix and coin name (made same as back-end version). For example there is blockbook-bitcoin user for both blockbook-bitcoin and blockbook-bitcoin-testnet packages.
back-end versioning
Since we have to distinguish version of coin distribution and version of our configuration we follow standard Debian package versioning rules (for details see Debian policy). There is upstream version and revision both defined in coin definition file in backend.version and backend.package_revision, respectively.
blockbook versioning
Blockbook versioning is much simpler. There is only one version defined in configs/environ.json.
Because we don't keep back-end archives inside out repository we download them during build process. Build steps are these: download, verify and extract archive, prepare distribution and make package.
All configuration keys described below are in coin definition file in configs/coins.
download archive
URL from where is archive downloaded is defined in backend.binary_url.
verify archive
There are three different approaches how is archive verification done. Some projects use PGP sign of archive, some have signed sha256 sums and some don't care about verification at all. So there is option backend.verification_type that could be gpg, gpg-sha256 or sha256 and chooses particular method.
gpg type require file with digital sign and maintainer's public key imported in Docker build image (see below). Sign file is downloaded from URL defined in backend.verification_source. Than is passed to gpg in order to verify archvie.
gpg-sha256 type require signed checksum file and maintainer's public key imported in Docker build image (see below). Checksum file is downloaded from URL defined in backend.verification_source. Then is verified by gpg and passed to sha256sum in order to verify archive.
sha256 type is used for coins that don't support verification at all. In backend.verification_source is defined hexadecimal string that is compared with output of sha256sum. Although this solution is not secure, it avoid download errors and other surprises at least.
gpg and gpg-sha256 types require maintainer's public key imported in Docker build image. It is not expected that
maintainer's key will change requently while sing or checksum files are changed every release, so it is ideal to
store maintainer's key within image definition. Public keys are stored in build/docker/deb/gpg-keys directory. Docker
image must be rebuilt by calling make build-images.
extract archive
Extraction command is defined in backend.extract_command. Content of archive must be extracted to ./backend directory.
See bitcoin.json and vertcoin.json for different approaches.
prepare distribution
There are two steps in this stage – exclude unnecessary files and generate configuration.
Some files are not required for server deployment, some binaries have unnecessary dependencies, so it is good idea to extract these files from output package. Files to extract are listed in backend.exclude_files. Note that paths are relative to backend directory where archive is extracted.
Configuration is described in config.md.
Instructions below are focused on Debian 9 (Stretch). If you want to use another Linux distribution or operating system like macOS or Windows, please read instructions specific for each project.
Setup go environment:
wget https://dl.google.com/go/go1.10.3.linux-amd64.tar.gz && tar xf go1.10.3.linux-amd64.tar.gz
sudo mv go /opt/go
sudo ln -s /opt/go/bin/go /usr/bin/go
# see `go help gopath` for details
mkdir $HOME/go
export GOPATH=$HOME/go
export PATH=$PATH:$GOPATH/bin
Install RocksDB: https://github.com/facebook/rocksdb/blob/master/INSTALL.md and compile the static_lib and tools
sudo apt-get update && sudo apt-get install -y \
build-essential git wget pkg-config libzmq3-dev libgflags-dev libsnappy-dev zlib1g-dev libbz2-dev liblz4-dev
git clone https://github.com/facebook/rocksdb.git
cd rocksdb
CFLAGS=-fPIC CXXFLAGS=-fPIC make release
Setup variables for gorocksdb: https://github.com/tecbot/gorocksdb
export CGO_CFLAGS="-I/path/to/rocksdb/include"
export CGO_LDFLAGS="-L/path/to/rocksdb -lrocksdb -lstdc++ -lm -lz -lbz2 -lsnappy -llz4"
Install ZeroMQ: https://github.com/zeromq/libzmq
Install go-dep tool:
go get github.com/golang/dep/cmd/dep
Get blockbook sources, install dependencies, build:
cd $GOPATH/src
git clone https://github.com/trezor/blockbook.git
cd blockbook
dep ensure -vendor-only
go build
Blockbook require full node daemon as its back-end. You are responsible for proper installation. Port numbers and daemon configuration are defined in configs/coins and build/templates/backend/config directories. You should use specific installation process for particular coin you want run (e.g. https://bitcoin.org/en/full-node#other-linux-distributions for Bitcoin).
When you have running back-end daemon you can start Blockbook. It is highly recommended use ports described in ports.md for both Blockbook and back-end daemon. You can use contrib/scripts/build-blockchaincfg.sh that will generate Blockbook's blockchain configuration from our coin definition files.
Example for Bitcoin:
contrib/scripts/build-blockchaincfg.sh
./blockbook -sync -blockchaincfg=build/blockchaincfg.json -internal=:9030 -public=:9130 -certfile=server/testcert -logtostderr
This command starts Blockbook with parallel synchronization and providing HTTP and Socket.IO interface, with database in local directory data and established ZeroMQ and RPC connections to back-end daemon specified in configuration file passed to -blockchaincfg option.
Blockbook logs to stderr (option -logtostderr) or to directory specified by parameter -log_dir . Verbosity of logs can be tuned by command line parameters -v and -vmodule, for details see https://godoc.org/github.com/golang/glog.
You can check that Blockbook is running by simple HTTP request: curl https://localhost:9130. Returned data is JSON with some
run-time information. If port is closed, Blockbook is syncing data.