Release 2019 09 30

CHANGELOG.md

@@ -0,0 +1,26 @@
+# 2019-09-30 #162
+
+## Features
+
+ - Helm charts:
+   - Support for multiple helm repos (#151)
+   - Default to using DaemonSet and externalTrafficPolicy as Local for the ingress (#121)
+   - JSON logging for brig and galley, parser for nginz, making it friendly for kibana (#142)
+
+ - Ansible:
+   - Support multiple bucket bucket creation when provisioning minio (#153)
+   - Host static files on minio to allow clients to point to custom backends (#155)
+
+## Other updates
+- Update script takes a path now (#140)
+- Super simple k8s bootstrap (#150)
+
+## Breaking changes / known issues when upgrading
+
+- If using a kubernetes cluster installed with kubespray version kubespray master from 2018-10-09, i.e. commit 2ab2f3a0a3aeffdd9862bb485495b0c1e77a1ed8, the new DaemonSet default configuration will not work. See https://github.com/kubernetes-sigs/kubespray/issues/4036 for a detailed explanation and workaround.
+
+## Bug fixes
+- Fixed policy setting on minio for public files (#158)
+
+## Internal Changes
+- Lower default resource requirements (#152)

CONTRIBUTING.md

@@ -12,7 +12,7 @@ If submitting pull requests, please follow these guidelines:
 
 * if you want to make larger changes, it might be best to first open an issue to discuss the change.
 * if helm charts are involved, 
-    * use the `./bin/update.sh <chart-name>` script, to ensure changes in a subchart (e.g. brig) are correctly propagated to the parent chart (e.g. wire-server) before linting or installing.
+    * use the `./bin/update.sh ./charts/<chart-name>` script, to ensure changes in a subchart (e.g. brig) are correctly propagated to the parent chart (e.g. wire-server) before linting or installing.
     * ensure they pass linting, you can check with `helm lint -f path/to/extra/values-file.yaml charts/mychart`. 
     * If you can, try to also install the chart to see if they work the way you intended.
 

ansible/Makefile

@@ -9,7 +9,7 @@ download-kubespray:
 	poetry run ansible-playbook -i localhost, -c local download_kubespray.yml
 
 .PHONY: download-cli-binaries
-download-cli-binaries:
+download-cli-binaries: download-ansible-roles
 	# assumes /usr/local/bin is writable and on your PATH
 	# assumes amd64 (change variables if needed)
 	poetry run ansible-playbook -i localhost, -c local download_cli_binaries.yml

ansible/README.md

@@ -50,17 +50,22 @@ This document assumes
 
 ## Dependencies
 
-* Install 'poetry' (python dependency management). See also the [poetry documentation](https://poetry.eustace.io/).
-
-This assumes you're using python 2.7 (if you only have python3 available, you may need to find some workarounds):
+### Poetry
+First, we're going to install [Poetry](https://poetry.eustace.io/). We'll be using it to run ansible playbooks later.
+These directions assume you're using python 2.7 (if you only have python3 available, you may need to find some workarounds):
 
+To install poetry:
 ```
 sudo apt install -y python2.7 python-pip
 curl -sSL https://raw.githubusercontent.com/sdispater/poetry/master/get-poetry.py > get-poetry.py
 python2.7 get-poetry.py
 source $HOME/.poetry/env
 ln -s /usr/bin/python2.7 $HOME/.poetry/bin/python
 ```
+During the installation, answer 'Y' to allow the Path variable for this user to be modified.
+
+
+### Ansible
 
 * Install the python dependencies to run ansible.
 ```
@@ -71,12 +76,13 @@ cd wire-server-deploy/ansible
 poetry install
 ```
 
-* download the ansible roles necessary to install databases and kubernetes
+Note: the 'make download-cli-binaries' part of 'make download' requires either that you have run this all as root, or that the user you are running these scripts can 'sudo' without being prompted for a password. I run 'sudo ls', get prompted for a password, THEN run 'make download'.
+* Download the ansible roles necessary to install databases and kubernetes:
 ```
 make download
 ```
 
-## Provision virtual machines
+## Provisioning machines
 
 Create the following:
 
@@ -89,14 +95,13 @@ Create the following:
 | kubernetes    | 3      | 4   | 8 GB   | 20 GB  |
 | turn          | 2      | 1   | 2 GB   | 10 GB  |
 
-It's up to you how you create these VMs - kvm on a bare metal machine, VM on a cloud provider, etc. Make sure they run ubuntu 16.04/18.04.
+It's up to you how you create these machines - kvm on a bare metal machine, VM on a cloud provider, a real physical machine, etc. Make sure they run ubuntu 16.04/18.04.
 
-Ensure that your VMs have IP addresses that do not change.
+Ensure that the machines have IP addresses that do not change.
 
 ## Preparing to run ansible
 
-### All VMs
-
+### Adding IPs to hosts.ini
 Copy the example hosts file:
 
 `cp hosts.example.ini hosts.ini`
@@ -109,7 +114,7 @@ There are more settings in this file that we will set in later steps.
 
 #### WARNING: host re-use
 
-Some of these playbooks mess with the hostnames of their targets.  You MUST pick different hosts for playbooks that rename the host.  If you e.g. attempt to run Cassandra and k8s on the same 3 machines, the hostnames will be overwritten by the second installation playbook, corrupting the first.
+Some of these playbooks mess with the hostnames of their targets.  You MUST pick different hosts for playbooks that rename the host.  If you e.g. attempt to run Cassandra and k8s on the same 3 machines, the hostnames will be overwritten by the second installation playbook, breaking the first.
 
 At the least, we know that the cassandra and kubernetes playbooks are both guilty of hostname manipulation.
 
@@ -124,6 +129,7 @@ sudo apt install sshpass
  * in hosts.ini, uncomment the 'ansible_ssh_pass = ...' line, and change '...' to the password for the user you are logging in as.
  * in hosts.ini, uncomment the 'ansible_become_pass = ...' line, and change the ... to the password you'd enter to sudo.
 
+
 ##### Configuring SSH keys
 (from https://linoxide.com/how-tos/ssh-login-with-public-key/)
 If you want a bit higher security, you can copy SSH keys between the machine you are administrating with, and the machines you are managing with ansible.
@@ -150,10 +156,13 @@ If you want ansible to not be prompted for any administrative command (a differe
 Replace `<ANSIBLE_LOGIN_USERNAME>` with the username of the account you set up when you installed the machine.
 
 #### Ansible pre-kubernetes
+
 Now that you have a working hosts.ini, and you can access the host, run any ansible scripts you need, in order for the nodes to have internet (proxy config, ssl certificates, etc).
 
 ### Installing kubernetes
+Kubernetes is installed via ansible.
 
+* To deploy kubernetes:
 ```
 poetry run ansible-playbook -i hosts.ini kubernetes.yml -vv
 ```

ansible/files/registry/mk-sub-certificate

@@ -9,8 +9,8 @@ CERT_DIR="${SCRIPT_DIR}/certs"
 mkdir -p "$CERT_DIR"
 
 
-CA_cert=wire.com.crt
-CA_key=private.pem
+CA_cert=${2:?wire.com.crt}
+CA_key=${2:?private.pem}
 
 cd "$CERT_DIR" || exit
 

ansible/files/registry/registry-run.sh

@@ -5,9 +5,9 @@ docker run \
   --network=host \
   --restart=always \
   --name registry \
-  -v /mnt/registry:/var/lib/registry \
+  -v $(pwd)/../../mnt/registry:/var/lib/registry \
   -v "$(pwd)/certs:/certs" \
-  -e REGISTRY_HTTP_ADDR=0.0.0.0:443 \
+  -e REGISTRY_HTTP_ADDR=0.0.0.0:5001 \
   -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/client.crt \
   -e REGISTRY_HTTP_TLS_KEY=/certs/client.key \
   registry:2

ansible/files/registry/upload_image.sh

@@ -0,0 +1,16 @@
+#!/usr/bin/env bash
+
+registry_name="localhost:5001"
+
+function mirror() {
+    prefix=$1
+    image=$2
+    rmafter=$3
+    docker pull $prefix/$image
+    docker tag $prefix/$image $registry_name/$image
+    docker push $registry_name/$image
+    [ -n "$rmafter" ] && docker image remove $registry_name/$image
+    [ -n "$rmafter" ] && docker image remove $prefix/$image
+}
+
+mirror $1 $2 yes

ansible/hosts.example-demo.ini

@@ -2,9 +2,8 @@
 
 [all]
 # * 'ansible_host' is the IP to ssh into
-# * 'ip' is the IP to bind to (if multiple network interfaces are in use)
-#   omit 'ip' if you only have one network interface
-kubenode01        ansible_host=X.X.X.X ip=Y.Y.Y.Y etcd_member_name=etcd1
+# * (optional) 'ip' is the IP to bind to (if multiple network interfaces are in use)
+kubenode01    ansible_host=X.X.X.X etcd_member_name=etcd1
 
 [kube-master]
 kubenode01
@@ -38,9 +37,9 @@ ansible_python_interpreter = /usr/bin/python3
 ### KUBERNETES section (see kubespray documentation for details) ###
 
 bootstrap_os = ubuntu
-docker_dns_servers_strict = false
+docker_dns_servers_strict = False
 
 [k8s-cluster:vars]
 helm_enabled = True
 kube_network_plugin = flannel
-kubeconfig_localhost = true
+kubeconfig_localhost = True

ansible/minio.yml

@@ -11,7 +11,10 @@
   gather_facts: true
   vars:
     minio_server_env_extra: "MINIO_BROWSER=off"
-    bucket_name: "dummy-bucket"
+    bucket_names:
+      - "dummy-bucket"
+      - "assets"
+      - "public"
     layouts:
       # The first minio instance on this server.
       layout1:
@@ -35,7 +38,7 @@
       tags:
         - minio
   tasks:
-    - name: "check if bucket {{ bucket_name }} exists"
+    - name: "check which buckets exists"
       shell: "mc ls def"
       environment:
         MC_HOST_def: "http://{{ minio_access_key }}:{{ minio_secret_key }}@localhost{{ layouts.layout1.minio_server_addr }}"
@@ -44,21 +47,42 @@
       tags: bucket-create
 
     - name: create bucket
-      shell: "mc mb def/{{ bucket_name }}"
+      shell: "mc mb def/{{ item }}"
       environment:
         MC_HOST_def: "http://{{ minio_access_key }}:{{ minio_secret_key }}@localhost{{ layouts.layout1.minio_server_addr }}"
       run_once: true
-      when: bucket_name not in check_bucket.stdout
+      with_items: "{{ bucket_names }}"
+      when: item not in check_bucket.stdout
       tags: bucket-create
 
-    - name: "add local mc config alias"
+    - name: "add 'local' mc config alias with correct credentials"
       shell: "mc config host add local http://localhost{{ layouts.layout1.minio_server_addr }} {{ minio_access_key }} {{ minio_secret_key }}"
       tags: mc-config
 
-    - name: "remove dummy config aliases added by default"
+    - name: "make the 'public' bucket world-accessible"
+      shell: "mc policy set public local/public"
+      run_once: true
+      tags: bucket-create
+
+    - name: "remove unneeded config aliases added by default"
       shell: "mc config host rm {{ item }}"
       with_items:
         - gcs
         - s3
         - play
       tags: mc-config
+
+  # This play has to run after minio is installed and buckets are configured
+- hosts: minio
+  any_errors_fatal: true
+  become: true
+  gather_facts: true
+  tags: static-files
+  roles:
+    - role: minio-static-files
+      # Override these variables!
+      # FUTUREWORK: parse them from a configuration file shared with helm
+      # (as the domain needs to be known in helm override values.yaml)
+      prefix: "{{ minio_deeplink_prefix | default('example-') }}"
+      domain: "{{ minio_deeplink_domain | default('example.com') }}"
+      deeplink_title: "{{ minio_deeplink_domain | default('example.com environment') }}"

ansible/registry.yml

@@ -12,8 +12,15 @@
 # registry01
 #
 # [all:vars]
+# FIXME: do we need an interface binding? we're abusing apache, and localhost.
 # registry_network_interface = eth0
-#
+# # the CA key on the registry host
+# registry_ca_cert = /home/wire/docker-squid4/mk-ca-cert/certs/wire.com.crt
+# registry_ca_key  = /home/wire/docker-squid4/mk-ca-cert/certs/private.pem
+# create_ca_cert = false
+# create_ca_domain = wire.com
+# contenthome = /home/wire/docker-squid4/docker-squid/docker_registry
+# registry_dns_name = proxybox.wire.com
 #
 # Be sure to run `docker login` from the registry node if you wish to mirror any private images.
 #
@@ -24,66 +31,56 @@
 - hosts: localhost
   gather_facts: false
   tasks:
-    - name: check CA certificate existance (run files/registry/mk-cert if this fails)
-      shell: ls files/registry/certs/wire.com.crt
+  - name: create_ca
+    command: "files/registry/mk-certs"
+    when: create_ca_cert
 
-    - name: check client certificate existance (run files/registry/mk-sub-cert if this fails)
-      shell: ls files/registry/certs/client.crt
+  - name: copy fake ca certificate
+    copy:
+      src: "files/registry/certs/{{ create_ca_domain }}.crt"
+      dest: "/usr/local/share/ca-certificates/{{ create_ca_domain }}/{{ create_ca_domain }}.crt"
+    when: create_ca_cert
 
 - hosts: registry
   tasks:
     - name: create dir
       file:
-        path: "{{ item }}"
+        path: "{{ contenthome }}{{ item }}"
         state: directory
       with_items:
         - /opt/registry
         - /opt/registry/certs
         - /mnt/registry
 
-    - name: copy cert files
-      copy:
-        src: "files/registry/certs/{{ item }}"
-        dest: "/opt/registry/certs/{{ item }}"
-      with_items:
-        - client.crt
-        - client.key
-
-    - name: copy bash
+    - name: copy bash scripts
       copy:
         src: files/registry/{{ item }}
-        dest: /opt/registry/{{ item }}
+        dest: {{ contenthome }}/opt/registry/{{ item }}
         mode: 0755
       with_items:
         - images.sh
+        - upload_image.sh
         - registry-run.sh
         - list_of_docker_images.txt
+        - mk-sub-certificate
+
+    - name: copy fake ca certificate
+      copy:
+        src: "files/registry/certs/{{ create_ca_domain }}.crt"
+        dest: "/usr/local/share/ca-certificates/{{ create_ca_domain }}/{{ create_ca_domain }}.crt"
+      when: create_ca_cert
+
+    - name: create sub certificate
+      command: "{{ contenthome }}/opt/registry/mk-sub-certificate proxybox.org {{ registry_ca_cert }} {{ registry_ca_key }}"
+      args:
+        chdir: "{{ contenthome }}/opt/registry"
 
 - hosts: k8s-cluster
   tags: trust
   vars:
-    cert_local_dir: "files/registry/certs"
-    cert_dir: "/usr/local/share/ca-certificates/wire.com"
-    cert_name: "wire.com.crt"
-    registry_dns_name: "{{ lookup('file', 'files/registry/certs/dns_name') }}"
   tasks:
     - debug: var=registry_dns_name
 
-    - name: create man-in-the-middle certificate directory
-      file:
-        path: "{{ cert_dir }}"
-        state: directory
-        mode: 0755
-
-    - name: copy certificate
-      copy:
-        src: "{{ cert_local_dir }}/{{ cert_name }}"
-        dest: "{{ cert_dir }}/{{ cert_name }}"
-        mode: 0644
-
-    - name: update ca certificates
-      shell: update-ca-certificates
-
     - name: Add IP/dns of registry and of upstream docker registries to /etc/hosts
       lineinfile:
         dest: /etc/hosts