Snowplow Docker

Introduction

This repository contains the Docker images for the following Snowplow components:

• Scala Stream Collector
• Stream Enrich
• Elasticsearch Loader
• S3 Loader
• Iglu Server
• Igluctl
• Piinguin Server
• EmrEtlRunner

They are published in the snowplow-docker-registry.bintray.io docker registry.

Pulling

You can pull the images from the registry directly:

# NSQ Scala Stream Collector image, there are others available for Kinesis, Kafka and Google PubSub
docker pull snowplow-docker-registry.bintray.io/snowplow/scala-stream-collector-nsq:0.15.0

# NSQ Stream Enrich image, there are others available for Kinesis, Kafka and Google PubSub
docker pull snowplow-docker-registry.bintray.io/snowplow/stream-enrich-nsq:0.20.0

# Elasticsearch Loader image
docker pull snowplow-docker-registry.bintray.io/snowplow/elasticsearch-loader:0.10.2

# S3 Loader image
docker pull snowplow-docker-registry.bintray.io/snowplow/s3-loader:0.6.0

# Iglu Server image
docker pull snowplow-docker-registry.bintray.io/snowplow/iglu-server:0.4.0

# Iglu Igluctl image
docker pull snowplow-docker-registry.bintray.io/snowplow/igluctl:0.6.0

# Piinguin Server image
docker pull snowplow-docker-registry.bintray.io/snowplow/piinguin-server:0.1.1

# EmrEtlRunner image
docker pull snowplow-docker-registry.bintray.io/snowplow/emr-etl-runner:r113_filitosa

Building

Alternatively, you can build them yourself:

# All images are based on the base images
docker pull snowplow-docker-registry.bintray.io/snowplow/base-alpine:0.2.0
docker pull snowplow-docker-registry.bintray.io/snowplow/base-debian:0.1.0

# NSQ Scala Stream Collector image, there are others available for Kinesis, Kafka and Google PubSub
docker build -t snowplow/scala-stream-collector-nsq:0.15.0 scala-stream-collector/0.15.0/nsq

# NSQ Stream Enrich image, there are others available for Kinesis, Kafka and Google PubSub
docker build -t snowplow/stream-enrich-nsq:0.20.0 stream-enrich/0.20.0/nsq

# Elasticsearch Loader image
docker build -t snowplow/elasticsearch-loader:0.10.2 elasticsearch-loader/0.10.2

# S3 Loader image
docker build -t snowplow/s3-loader:0.6.0 s3-loader/0.6.0

# Iglu Server image
docker build -t snowplow/iglu-server:0.4.0 iglu-server/0.4.0

# Igluctl
docker build -t snowplow/igluctl:0.6.0 igluctl/0.6.0

# Piinguin Server image
docker build -t snowplow/piinguin-server:0.1.1 piinguin-server/0.1.1

# EmrEtlRunner image
docker build -t snowplow/emr-etl-runner:r113_filitosa emr-etl-runner/r113_filitosa

Running

Create your own config file filling it according to your setup, you can find examples at the following locations:

• Scala Stream Collector configuration
• Stream Enrich configuration
• Elasticsearch Loader configuration
• S3 Loader configuration
• Iglu Server configuration
• EmrEtlRunner configuration

Next, you can run a container for each component by mounting your configuration directory:

# NSQ Scala Stream Collector container, there are others available for Kinesis, Kafka and Google PubSub
docker run \
  -v $PWD/scala-stream-collector-config:/snowplow/config \
  -p 8080:8080 \
  snowplow/scala-stream-collector-nsq:0.15.0 \ # if you have built the image
  # snowplow-docker-registry.bintray.io/snowplow/scala-stream-collector-nsq:0.15.0 if you have pulled the image
  --config /snowplow/config/config.hocon

# NSQ Stream Enrich container, there are others available for Kinesis, Kafka and Google PubSub
docker run \
  -v $PWD/stream-enrich-config:/snowplow/config \
  snowplow/stream-enrich-nsq:0.20.0 \ # if you have built the image
  # snowplow-docker-registry.bintray.io/snowplow/stream-enrich-nsq:0.20.0 if you have pulled the image
  --config /snowplow/config/config.hocon \
  --resolver file:/snowplow/config/resolver.json \
  --enrichments file:/snowplow/config/enrichments/ \
  --force-cached-files-download

# Elasticsearch Loader
docker run \
  -v $PWD/elasticsearch-loader-config:/snowplow/config \
  snowplow/elasticsearch-loader:0.10.2 \ # if you have built the image
  # snowplow-docker-registry.bintray.io/snowplow/elasticsearch-loader:0.10.2 if you have pulled the image
  --config /snowplow/config/config.hocon

# S3 Loader
docker run \
  -v $PWD/s3-loader-config:/snowplow/config \
  snowplow/s3-loader:0.6.0 \ # if you have built the image
  # snowplow-docker-registry.bintray.io/snowplow/s3-loader:0.6.0 if you have pulled the image
  --config /snowplow/config/config.hocon

# Iglu Server
docker run \
  -v ${PWD}/iglu-server-config:/snowplow/config \
  snowplow/iglu-server:0.4.0 \ # if you have built the image
  # snowplow-docker-registry.bintray.io/snowplow/iglu-server:0.4.0 if you have pulled the image
  --config /snowplow/config/application.conf

# EmrEtlRunner
docker run \
  -v ${PWD}/emr-etl-runner-config:/snowplow/config \
  snowplow/emr-etl-runner:r113_filitosa \ # if you have built the image
  # snowplow-docker-registry.bintray.io/snowplow/emr-etl-runner:r113_filitosa if you have pulled the image
  run \
  --config /snowplow/config/config.yaml \
  --resolver file:/snowplow/config/resolver.json \
  --enrichments file:/snowplow/config/enrichments/

In the case of the piinguin server there is no configuration file so you can run it simply with (if changing the port for the server, you may also need to publish a different port in the docker host):

$ docker run \
  -d \
  piinguin-server:${VERSION} \
  -h <interface e.g. 0.0.0.0>
  -p 8080
  -t <piinguin-dynamo-table>

You can find more information in the readme for each image:

• Scala Stream Collector readme
• Stream Enrich readme
• Elasticsearch Loader readme
• S3 Loader readme
• Iglu Server readme
• Igluctl
• Piinguin Server readme
• EmrEtlRunner readme

There is a Docker Compose example in the example folder. Iglu Server also has a Docker Compose example in a separate example folder.

Find out more

Technical Docs	Setup Guide	Roadmap & Contributing
Technical Docs	Setup Guide	coming soon

Copyright and license

Copyright 2017-2019 Snowplow Analytics Ltd.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this software except in compliance with the License.

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.