Release 2019 11 06

.github/ISSUE_TEMPLATE/question-installation.md

@@ -0,0 +1,37 @@
+---
+name: Question-Installation
+about: Question about my installation of wire-server
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+* [ ] I have seen https://docs.wire.com/ and https://github.com/wireapp/wire-server-deploy - the documentation there does not answer my question.
+
+## My question:
+
+## Context:
+
+> Please provide sufficent context about your problem:
+
+### How did you install wire-server?
+
+> On kubernetes? With docker-compose? by manually compiling and running?
+
+### How many servers are involved?
+
+### What is installed on which servers?
+
+> E.g Server A has component X and server B has component Y
+
+### Provide details about networking
+
+> We don't need to know any specific IP address, but it helps if you provide information whether an IP is ipv4 or ipv6, whether is is publicly reachable from the global internet or not, and if you installed any component of wire-server, which network interfaces are processes listening on?
+
+### How did you configure wire-server?
+
+> *Note: only the configuration from helm charts in wire-server-deploy is what we support, like [these defaults](https://github.com/wireapp/wire-server-deploy/blob/develop/charts/brig/values.yaml) applied [here](https://github.com/wireapp/wire-server-deploy/blob/develop/charts/brig/templates/configmap.yaml) in the case of `brig`. If you have used other configuration files, please post them (or the relevant parts of them).*
+> Did you use the helm charts from wire-server-deploy?
+> Did you use and adapt configuration files from wire-server? If so, which ones?
+> Are there any overrides?

.gitignore

@@ -63,14 +63,14 @@ swagger-ui
 deploy/services-demo/resources/templates/*
 deploy/services-demo/conf/nginz/zwagger-ui/*
 
-deploy/docker-ephemeral/build/airdock_base-all/
-deploy/docker-ephemeral/build/airdock_base/
-deploy/docker-ephemeral/build/airdock_fakesqs-all/
-deploy/docker-ephemeral/build/airdock_fakesqs/
-deploy/docker-ephemeral/build/airdock_rvm-all/
-deploy/docker-ephemeral/build/airdock_rvm/
-deploy/docker-ephemeral/build/dynamodb_local/
-deploy/docker-ephemeral/build/smtp/
+deploy/dockerephemeral/build/airdock_base-all/
+deploy/dockerephemeral/build/airdock_base/
+deploy/dockerephemeral/build/airdock_fakesqs-all/
+deploy/dockerephemeral/build/airdock_fakesqs/
+deploy/dockerephemeral/build/airdock_rvm-all/
+deploy/dockerephemeral/build/airdock_rvm/
+deploy/dockerephemeral/build/dynamodb_local/
+deploy/dockerephemeral/build/smtp/
 
 # Ignore cabal files; use package.yaml instead
 *.cabal

CHANGELOG.md

@@ -1,3 +1,27 @@
+# 2019-11-06 #901
+
+## Relevant for self-hosters
+
+- New configuration options available (none mandatory). See #895 #900 #869
+
+## Relevant for client developers
+
+- Support HEAD requests for `/sso/initiate-bind` (#878)
+
+## Bug fixes
+
+- Do not send conversation delete events to team members upon team deletion (#897)
+- Support SNI for bot registrations (by bumping http-client version) (#899)
+
+## Internal changes
+
+- Make gundeck handle AWS outages better. (#869, #890, #892)
+- Improve performance by avoiding unbounded intra-service traffic spikes on team deletions (#900)
+- Add optional native push connection throttling (#895)
+- New backoffice/stern endpoint (#896)
+- SAML: Store raw idp metadata with typed details in c* (#872)
+- documentation/script updates
+
 # 2019-09-30 #868
 
 ## Relevant for self-hosters

README.md

@@ -143,7 +143,7 @@ Integration tests require all of the haskell services (brig, galley, cannon, gun
 Setting up these real, but in-memory internal and "fake" external dependencies is done easiest using [`docker-compose`](https://docs.docker.com/compose/install/). Run the following in a separate terminal (it will block that terminal, C-c to shut all these docker images down again):
 
 ```
-deploy/docker-ephemeral/run.sh
+deploy/dockerephemeral/run.sh
 ```
 
 Then, to run all integration tests:

deploy/services-demo/README.md

@@ -7,7 +7,7 @@ Use 2 different terminals and run:
 ```
 # On terminal 1, start the dependencies. Note that you should turn up the max memory
 # limit of docker. More on https://github.com/wireapp/wire-server/issues/326
-deploy/docker-ephemeral/run.sh
+deploy/dockerephemeral/run.sh
 ```
 
 ```

deploy/services-demo/create_team_members.sh

@@ -2,10 +2,10 @@
 
 set -e
 
-ADMIN_UUID="a09e9521-e14e-4285-ad71-47caa97f4a16"
-TEAM_UUID="9e57a378-0dca-468f-9661-7872f5f1c910"
-BRIG_HOST="http://localhost:8082"
-CSV_FILE="myfile.csv"
+ADMIN_UUID="n/a"
+TEAM_UUID="n/a"
+BRIG_HOST="http://localhost:8080"
+CSV_FILE="n/a"
 
 USAGE="
 This bash script can be used to invite members to a given team.  Input
@@ -19,6 +19,17 @@ USAGE: $0
     -t <team uuid>: ID of the inviting team.  default: ${TEAM_UUID}
     -h <host>: Base URI of brig. default: ${BRIG_HOST}
     -c <input file>: file containing info on the invitees in format 'Email,UserName'.  default: ${CSV_FILE}
+
+If you tee(1) stdout, stderr of this script into a log file, you can
+grep that log file for errors like this:
+
+$ grep code out.log | grep email-exists  # the most common case
+$ grep code out.log | grep -v email-exists
+
+If you are in a hurry, you may want to change the sleep(1) at the end
+of the invite loop to less than a second.  If you want to give up on
+the first error, add an exit(1) where we check the $INVIDATION_ID.
+
 "
 
 # Option parsing:
@@ -73,8 +84,7 @@ do
 
     if ( ( echo "$INVITATION_ID" | grep -q '"code"' ) &&
          ( echo "$INVITATION_ID" | grep -q '"label"' ) ) ; then
-      echo "Got an error, aborting: $INVITATION_ID"
-      exit 1
+      echo "failed inviting $USER_NAME <$EMAIL>: $INVITATION_ID"
     fi
 
     echo "Sleeping 1 second..." 1>&2

deploy/services-demo/demo.sh

@@ -73,7 +73,7 @@ function check_prerequisites() {
     nc -z 127.0.0.1 9042 \
         && nc -z 127.0.0.1 9200 \
         && nc -z 127.0.0.1 6379 \
-        || { echo "Databases not up. Maybe run 'deploy/docker-ephemeral/run.sh' in a separate terminal first?";  exit 1; }
+        || { echo "Databases not up. Maybe run 'deploy/dockerephemeral/run.sh' in a separate terminal first?";  exit 1; }
     if [ "$docker_deployment" = "false" ]; then
         test -f ${DIR}/../dist/brig \
             && test -f ${DIR}/../dist/galley \
@@ -127,7 +127,7 @@ function copy_nginz_configs() {
 }
 
 # brig,gundeck,galley use the amazonka library's 'Discover', which expects AWS credentials
-# even if those are not used/can be dummy values with the fake sqs/ses/etc containers used (see deploy/docker-ephemeral/docker-compose.yaml)
+# even if those are not used/can be dummy values with the fake sqs/ses/etc containers used (see deploy/dockerephemeral/docker-compose.yaml)
 export AWS_REGION=${AWS_REGION:-eu-west-1}
 export AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID:-dummy}
 export AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY:-dummy}

deploy/services-demo/docker-compose.yaml

@@ -1,5 +1,5 @@
 networks:
-  docker-ephemeral_demo_wire:
+  dockerephemeral_demo_wire:
     external: true
 
 version: '2'
@@ -30,7 +30,7 @@ services:
       - demo_wire_sqs:sqs
       - demo_wire_smtp:smtp
     networks:
-      - docker-ephemeral_demo_wire
+      - dockerephemeral_demo_wire
 
   galley:
     image: quay.io/wire/galley
@@ -50,7 +50,7 @@ services:
     external_links:
       - demo_wire_cassandra:cassandra
     networks:
-      - docker-ephemeral_demo_wire
+      - dockerephemeral_demo_wire
 
   gundeck:
     image: quay.io/wire/gundeck
@@ -72,7 +72,7 @@ services:
       - demo_wire_sqs:sqs
       - demo_wire_localstack:sns
     networks:
-      - docker-ephemeral_demo_wire
+      - dockerephemeral_demo_wire
 
   cannon:
     image: quay.io/wire/cannon
@@ -86,7 +86,7 @@ services:
       - /configs/conf/cannon.demo-docker.yaml
     working_dir: /configs
     networks:
-      - docker-ephemeral_demo_wire
+      - dockerephemeral_demo_wire
 
   cargohold:
     image: quay.io/wire/cargohold
@@ -102,7 +102,7 @@ services:
     external_links:
       - demo_wire_s3:s3
     networks:
-      - docker-ephemeral_demo_wire
+      - dockerephemeral_demo_wire
 
   proxy:
     image: quay.io/wire/proxy
@@ -116,7 +116,7 @@ services:
       - /configs/conf/proxy.demo.yaml
     working_dir: /configs
     networks:
-      - docker-ephemeral_demo_wire
+      - dockerephemeral_demo_wire
 
   spar:
     image: quay.io/wire/spar
@@ -132,7 +132,7 @@ services:
     external_links:
       - demo_wire_cassandra:cassandra
     networks:
-      - docker-ephemeral_demo_wire
+      - dockerephemeral_demo_wire
 
   nginz:
     image: quay.io/wire/nginz
@@ -157,4 +157,4 @@ services:
       - /configs/conf/nginz/nginx-docker.conf
     working_dir: /configs
     networks:
-      - docker-ephemeral_demo_wire
+      - dockerephemeral_demo_wire

deploy/services-demo/register_idp_internal.sh

@@ -0,0 +1,25 @@
+#!/usr/bin/env bash
+
+set -e
+
+# server-side variant of ./register_idp.sh; use if you have ssh access to one of your spar instances.
+# usage: ./register_idp_internal.sh <admin id> <metadata file>
+
+backend="http://localhost:8080"
+
+metadata_file=$1
+if [ ! -e "${metadata_file}" ]; then
+    echo "*** no metadata: '$1'"
+    exit 80
+fi
+
+z_user=$2
+if [ ! -n "${z_user}" ]; then
+    echo "*** no z_user uuid"
+    exit 80
+fi
+
+which curl >/dev/null || ( echo "*** please install https://curl.haxx.se/ in your path."; exit 81 )
+curl_exe=$(which curl)
+
+${curl_exe} -is -v -XPOST ${backend}/identity-providers -H"Z-User: ${z_user}" -H'Content-type: application/xml' -d@"${metadata_file}"

docs/reference/provisioning/scim-via-curl.md

@@ -8,6 +8,11 @@ This page shows you how to communicate with the wire backend through
 the [SCIM API](http://www.simplecloud.info/) by example.  All examples
 are [curl](https://curl.haxx.se/) (in bash syntax).
 
+We support setting the handle and user name in wire (the thing with
+`@` and the longer thing without `@`).  There is also support for
+setting rich-info.  Group provisioning is planned, but the release
+date hasn't been fixed yet.
+
 If you want to dive into the backend code, start [reading here in our
 backend](https://github.com/wireapp/wire-server/blob/develop/services/spar/src/Spar/Scim.hs)
 and [our hscim library](https://github.com/wireapp/hscim).
@@ -87,19 +92,34 @@ A minimal definition of a user looks like this:
 ```bash
 export SCIM_USER='{
   "schemas"     : ["urn:ietf:params:scim:schemas:core:2.0:User"],
-  "externalId"  : "f8c4ffde-4592-11e9-8600-afe11dc7d07b",
+  "externalId"  : "nick@example.com",
   "userName"    : "nick",
   "displayName" : "The Nick"
 }'
 ```
 
+The `externalId` is used to construct a saml identity.  Two cases are
+currently supported:
+
+1. `externalId` contains a valid email address.  The SAML `NameID` has
+the form `<NameID
+Format="urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress">me@example.com</NameID>`.
+2. `externalId` contains anything that is *not* an email address.  The
+SAML `NameID` has the form `<NameID
+Format="urn:oasis:names:tc:SAML:1.1:nameid-format:unspecified">...</NameID>`.
+
+*NOTE: It is important to configure your SAML provider to use
+`nameid-format:emailAddress` or `nameid-format:unspecified`.  Other
+nameid formats are not supported at this moment*.
+See also: https://github.com/wireapp/wire-server/blob/c507ed64a7d4f0af2bffe2f9c3eb4b5f89a477c0/services/spar/src/Spar/Scim/User.hs#L149-L158
+
 We also support custom fields that are used in rich profiles in this
 form [see {#RefRichInfo}](../user/rich-info.md):
 
 ```bash
 export SCIM_USER='{
   "schemas"     : ["urn:ietf:params:scim:schemas:core:2.0:User", "urn:wire:scim:schemas:profile:1.0"],
-  "externalId"  : "f8c4ffde-4592-11e9-8600-afe11dc7d07b",
+  "externalId"  : "rnick@example.com",
   "userName"    : "rnick",
   "displayName" : "The Rich Nick",
   "urn:wire:scim:schemas:profile:1.0": {
@@ -155,7 +175,7 @@ up-to-date user present, just `GET` one right before the `PUT`.)
 ```bash
 export SCIM_USER='{
   "schemas"     : ["urn:ietf:params:scim:schemas:core:2.0:User"],
-  "externalId"  : "updated-user-id",
+  "externalId"  : "rnick@example.com",
   "userName"    : "newnick",
   "displayName" : "The New Nick"
 }'