Skip to content

Commit

Permalink
Build optimizations for Docker (#60)
Browse files Browse the repository at this point in the history 
This commit removes the need of keeping two Dockerfiles for
different architectures. 

Similarly as before, the Dockerfiles are kept in two folders, a stable 
for builds based on https://github.com/wirepas/manifest and a dev folder
for builds based on the current content of the repo (+ master of c-mesh-api). 
A commons folder contains templates and the docker-entrypoint script.

Release builds are now only triggered when a tag is given to CI system, which
is meant to prepare the future push from CI to dockerhub.

There is still a need in the future to accommodate other OSes inside
the container folder, such as Alpine.

* Update readme (#63) Closes #62
* Update wirepas_messaging version Bump to rc3. See #38
  • Loading branch information
@PFigs
PFigs authored Jul 16, 2019
1 parent 7f27efa commit a979f37
Show file tree
Hide file tree
Showing 17 changed files with 1,350 additions and 193 deletions.
90 changes: 62 additions & 28 deletions .ci/build-images.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,55 +3,77 @@

set -e

DOCKERFILE_PATH="./container"
GIT_REPO_FOLDER="_repo"

export LXGW_VERSION
export VERSION
export BUILD_DATE
export IMAGE_NAME
export REGISTRY_NAME
export DOCKER_BASE
export CROSS_BUILD_START_CMD
export CROSS_BUILD_END_CMD
export GIT_MANIFEST_FILE
export GIT_MANIFEST_URL
export GIT_MANIFEST_BRANCH

LXGW_VERSION=$(< python_transport/wirepas_gateway/__init__.py awk '/__version__/{print $NF}'| tr -d '\"')
VERSION=$(< python_transport/wirepas_gateway/__init__.py awk '/__version__/{print $NF}'| tr -d '\"')
BUILD_DATE=$(date -u +'%Y-%m-%dT%H:%M:%SZ')
REGISTRY_NAME="wirepas/"
REGISTRY_NAME="wirepas"

DOCKERFILE_PATH="./container"
GIT_REPO_FOLDER="./.ci/_repo"
BUILD_TAG=${TRAVIS_TAG:-}

##
## @brief Changes to the corresponding target and builds the image
##
function _build
{
TARGET=${1:-"dev"}
CURRENT_PATH="$(pwd)"
_PATH=${1:-"${DOCKERFILE_PATH}/x86"}
_ARCH=${2:-"x86"}
_CACHE=${3:-}

if [[ ${TARGET} == "dev" ]]
# build based on architecture
IMAGE_NAME="${REGISTRY_NAME}/gateway-${_ARCH}:${BUILD_TAG}"

if [[ ${_ARCH} == "arm" ]]
then
_pull_dev_dependencies
DOCKER_BASE=wirepas/base:1.1-raspbian
CROSS_BUILD_START_CMD=cross-build-start
CROSS_BUILD_END_CMD=cross-build-end
else
DOCKER_BASE=wirepas/base:1.1-ubuntu
CROSS_BUILD_START_CMD=:
CROSS_BUILD_END_CMD=:
fi

cd "${DOCKERFILE_PATH}/${TARGET}"
IMAGE_NAME="${REGISTRY_NAME}/gateway-${TARGET}"
docker-compose build --no-cache
cd "${CURRENT_PATH}"
echo "building ${_PATH}: ${IMAGE_NAME} (from: ${DOCKER_BASE})"
#shellcheck disable=SC2086
docker-compose -f "${_PATH}" \
build ${_CACHE} \
--compress \
--parallel \

}

##
## @brief Setup repo and syncs the repositories
##
function _pull_dev_dependencies

function _fetch_dependencies
{
GIT_MANIFEST_FILE=${GIT_MANIFEST_FILE:-"gateway.xml"}
GIT_MANIFEST_URL=${GIT_MANIFEST_URL:-"https://github.com/wirepas/manifest.git"}
# pull repository dependency
GIT_MANIFEST_FILE=gateway.xml
GIT_MANIFEST_URL=https://github.com/wirepas/manifest.git
GIT_MANIFEST_BRANCH=master
_ROOT_PATH=$(pwd)

rm -rf "${GIT_REPO_FOLDER}"
mkdir "${GIT_REPO_FOLDER}"
cd "${GIT_REPO_FOLDER}"
pipenv run --two repo init -u "${GIT_MANIFEST_URL}" \
-m "${GIT_MANIFEST_FILE}" \
--no-clone-bundle
pipenv run --two repo init \
-u "${GIT_MANIFEST_URL}" \
-m "${GIT_MANIFEST_FILE}" \
-b "${GIT_MANIFEST_BRANCH}" \
--no-clone-bundle
pipenv --rm
pipenv run --two repo sync

cd ..
cd "${_ROOT_PATH}"
cp -r "${GIT_REPO_FOLDER}/sink_service/" .
}

Expand All @@ -61,9 +83,21 @@ function _pull_dev_dependencies
##
function _main
{
_build "arm"
_build "x86"
_build "dev"

# builds x86 and arm images based on manifest files
if [[ ! -z ${BUILD_TAG} ]]
then
GIT_MANIFEST_BRANCH=refs/tags/${BUILD_TAG}
_build "${DOCKERFILE_PATH}/arm/docker-compose.yml" "arm" "--no-cache"
_build "${DOCKERFILE_PATH}/x86/docker-compose.yml" "x86" "--no-cache"
else
BUILD_TAG="edge"
fi

# builds x86 and arm images based on top of current revision
_fetch_dependencies
_build "${DOCKERFILE_PATH}/dev/docker-compose.yml" "arm"
_build "${DOCKERFILE_PATH}/dev/docker-compose.yml" "x86"
}


Expand Down
9 changes: 5 additions & 4 deletions .gitignore
View file
Original file line number Diff line number Diff line change
@@ -1,11 +1,7 @@
deliverable/*
*.7z
*.so
*.whl
*.pyc
sink_service/build/
sink_service/c-mesh-api/

**/*.idea/*
*/*.wm-hell*
public-apis/
Expand All @@ -18,6 +14,10 @@ build-*.txt
**/build/
**/dist/

sink_service/build/
sink_service/c-mesh-api/
python_transport/docs/source/api/*

public/
packets/
release/
Expand All @@ -26,6 +26,7 @@ release/
.flake8
.clang*
.ci/manifest/*
.ci/_repo/*
dev-requirements.txt

_repo/*
Expand Down
129 changes: 104 additions & 25 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,27 +1,53 @@
# Wirepas Linux Gateway

This repository contains the Wirepas reference implementation for a gateway
device which offloads Wirepas Mesh data to a host.
[![Build Status](https://travis-ci.org/wirepas/gateway.svg?branch=master)](https://travis-ci.org/wirepas/gateway) [![Documentation Status](https://readthedocs.org/projects/wirepas-gateway/badge/?version=latest)](https://wirepas-gateway.readthedocs.io/en/latest/?badge=latest)

This repository contains Wirepas' reference gateway implementation, which
relies on a set of services to exchange data from/to a Wirepas Mesh network
from/to a MQTT broker or host device.

The services will be known from now on as sink service and transport service.
The sink service is responsible to interface locally with a Wirepas device
running its Dual MCU API. The transport service packs network
messages on protobuffers and publishes them on top of MQTT according to
Wirepas Backend API.

Figure 1, provides an overview of the gateway implementation and the
apis involved at each step.

![Wirepas gateway architecture][here_img_overview]

**Figure 1 -** Gateway services overview.


## Cloning this repository

This repository has a hard dependency on [c-mesh-api](https://github.com/wirepas/c-mesh-api)
and a soft dependency on the [backend-apis](https://github.com/wirepas/backend-client).
This repository depends on two other projects, [c-mesh-api][wirepas_c_mesh_api]
and [backend-apis][wirepas_backend_apis].
The [c-mesh-api][wirepas_c_mesh_api] contains the library used by the sink service, which interfaces
with the sink devices. The backend-apis contains api and message wrapper
over the protocol buffers that are transmitted over MQTT.

When cloning this repository and its dependencies you can opt for:

- Using [repo tool](https://source.android.com/setup/develop/repo)
and the [manifest repository](https://github.com/wirepas/manifest)
- Using [repo tool][repo_tool]
and the [manifest repository][wirepas_manifest]

> repo init -u https://github.com/wirepas/manifest.git -m gateway.xml
or

> repo init -u git@github.com:wirepas/manifest.git -m gateway.xml
> repo init -u https://github.com/wirepas/manifest.git
or
> repo init -u git@github.com:wirepas/manifest.git
afterwards download the repositories with

> repo sync
- Clone each repo separately (see [pull_repos.sh](./utils/pull_repos.sh))
To clone a particular tag, vX.Y.Z, please use repo's *-b* switch as follows

> repo init (...) -b refs/tags/vX.Y.Z
Usage of repo is also documented in the release Dockerfiles (see [Dockerfile][here_container_dockerfile])

## Linux Requirements

Expand Down Expand Up @@ -116,15 +142,19 @@ To get an exhausted list of parameters, please run:
wm-gw --help
```


#### From cmd line

Here is an example to start the transport module from cmd line:

```shell
wm-gw --mqtt_hostname "<server>" --mqtt_port <port> --mqtt_username <user> --mqtt_password <password> \
--gateway_id <gwid> [--ignored_endpoints_filter <ignored endpoints list>] \
[--whitened_endpoints_filter <whitened endpoints list>]
wm-gw \
--mqtt_hostname "<server>" \
--mqtt_port <port> \
--mqtt_username <user> \
--mqtt_password <password> \
--gateway_id <gwid> \
[--ignored_endpoints_filter <ignored endpoints list>] \
[--whitened_endpoints_filter <whitened endpoints list>]
```

where:
Expand Down Expand Up @@ -196,28 +226,60 @@ It can be launched from command line:
wm-dbus-print
```

## Docker build instructions
## Building and running with Docker

In the container folder you will find three folders, one for
[development][here_container]
purposes and two other for architecture images, x86 and arm.

The development folder builds an image based on the contents of the repository,
whereas the other two folder will provide you a build for what is specified
in the repo tool's manifest.

To make a development build type:

```bash
[IMAGE_NAME=wirepas/gateway-x86:edge] docker-compose -f container/dev/docker-compose.yml build
```

If you want to build a stable image for x86 type:

```bash
docker-compose -f container/stable/x86/docker-compose.yml build
```

Alternatively you can use our [ci tool][here_ci_docker_build].

We also have pre-built images available from docker hub under the
following registries:

We are changing our method to build images. We will update this section when the work is complete.
[wirepas/gateway][dockerhub_wirepas]: multi architecture registry
[wirepas/gateway-x86][dockerhub_wirepas_x86]: x86 architecture registry
[wirepas/gateway-rpi][dockerhub_wirepas_rpi]: arm/rpi architecture registry

## Starting docker services

In the container folder, you will find the wm_gateway.env file, where you
need to place the MQTT credentials for your gateway user.
When running the gateway over docker, the composition will mount your host's
system dbus inside each container. This is necessary to allow exchanging data
between both services. For that reason you will have to copy the
[dbus manifest][here_dbus_manifest] to your host's and change the policy
user to reflect what is used within the docker-compose.

Beside the MQTT credentials, you need to define the ip or hostname where
to connect the transport service to.
After configuring your host's dbus, review the [service settings file.][here_container_env]
The environment parameters will be evaluated by the [container's entry point][here_container_entrypoint]
and passed on to the sink and transport service.

Once the settings are correct, you can launch the services with:
Please ensure that you define the correct password and MQTT credentials and
launch the services with:

```shell
docker-compose up [-d]
[IMAGE_NAME=wirepas/gateway-x86:edge] docker-compose -f container/dev/docker-compose.yml up [-d]
```

To view the logs, use

```shell
docker-compose logs
docker-compose -f container/dev/docker-compose.yml logs
```

or specify which container you want to view the logs from with
Expand All @@ -230,3 +292,20 @@ or specify which container you want to view the logs from with

Wirepas Oy licensed under Apache License, Version 2.0 See file LICENSE for
full license details.

[here_img_overview]: https://github.com/wirepas/gateway/tree/master/container/img/wm-gateway-overview.png
[here_ci_docker_build]: https://github.com/wirepas/gateway/blob/master/.ci/build-images.sh
[here_container]: https://github.com/wirepas/gateway/tree/master/container/
[here_container_dockerfile]: https://github.com/wirepas/gateway/tree/master/container/Dockerfile
[here_container_env]: https://github.com/wirepas/gateway/tree/master/container/wm_gateway.env
[here_dbus_manifest]: https://github.com/wirepas/gateway/blob/master/sink_service/com.wirepas.sink.conf
[here_container_entrypoint]: https://github.com/wirepas/gateway/blob/master/container/docker-entrypoint.sh

[repo_tool]: https://source.android.com/setup/develop/repo
[wirepas_manifest]: https://github.com/wirepas/manifest
[wirepas_c_mesh_api]: https://github.com/wirepas/c-mesh-api
[wirepas_backend_apis]: https://github.com/wirepas/backend-client

[dockerhub_wirepas]: https://hub.docker.com/r/wirepas/gateway
[dockerhub_wirepas_x86]: https://hub.docker.com/r/wirepas/gateway-x86
[dockerhub_wirepas_rpi]: https://hub.docker.com/r/wirepas/gateway-rpi
Loading

0 comments on commit a979f37

Please sign in to comment.