How we set up continuous integration on Jenkins

At https://cloud-images.ubuntu.com/locator/ec2/, pick the latest hvm:ebs-ssd instance for the region (us-east-1 in our case)
1. Choose the dedicated CI VPC, with IAM role Jenkins-CI.
2. Tag it Environment: ci
3. Give it the jenkins-ci security group (inbound port 443)
4. Launch it!
Name it Jenkins-CI
Give yourself temporary SSH access (via the security group) and SSH in.

Install Jenkins:

sudo apt-get update && sudo apt-get dist-upgrade

Follow the instructions at https://pkg.jenkins.io/debian-stable/ to install:

wget -q -O - https://pkg.jenkins.io/debian-stable/jenkins.io.key | sudo apt-key add -
sudo apt-add-repository 'deb https://pkg.jenkins.io/debian-stable binary/'
sudo apt-get update
sudo apt-get install openjdk-11-jre-headless jenkins

Set up GitHub for authentication: at https://jenkins-ci.overviewdocs.com/securityRealm/finishLogin set up overview-jenkins-ci with a URL of https://jenkins-ci.overviewdocs.com.
Setup Jenkins:
1. From your computer, ssh -L 8080:localhost:8080 ubuntu@[JenkinsIP] and browse to http://localhost:8080
2. Copy/paste the administrator password in (as prompted by Jenkins)
3. "Select plugins to install" => choose defaults, ...
  - plus: embeddable-build-status, JUnit Plugin, GitHub Plugin, GitHub Authentication Plugin, Amazon EC2 Plugin, Kubernetes CLI
  - minus: Ant Plugin, Gradle Plugin
4. Skip creating the administrator user. Click "Start Using Jenkins"
5. "Manage Jenkins" => "Configure System":
  - # of executors: 0
  - Jenkins URL: https://jenkins-ci.overviewdocs.com
  - System Admin e-mail address: admin@overviewdocs.com
  - SMTP Server: email-smtp.us-east-1.amazonaws.com
  - Check User SMTP Authentication and enter Amazon's SMTP settings
6. "Manage Jenkins" => "Configure Global Security"
  - Security realm => Github Authentication Plugin
  - Enter Client ID and Client Secret from the overview-jenkins-ci app page on GitHub
  - Authorization => GitHub Committer Authorization Strategy
  - Enter comma-separated Admin User Names
  - Check Use GitHub repository permissions, and don't fill in Participant in Organization
  - Check Grant READ permissions for Anonymous Users
  - Check Grant ViewStatus permissions for Anonymous Users
  - Save. You'll be locked out.
Adjust DNS to point to your new server.

Set up the HTTPS proxy:

Install programs

sudo apt-get install haproxy
sudo add-apt-repository ppa:certbot/certbot
sudo apt-get update
sudo apt-get install certbot
sudo certbot certonly --standalone -d jenkins-ci.overviewdocs.com -m admin@overviewdocs.com --agree-tos -n

Fill in /etc/haproxy/haproxy.cfg:

global
  log /dev/log  local0
  log /dev/log  local1 notice
  chroot /var/lib/haproxy
  user haproxy
  group haproxy
  daemon
  # https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=haproxy-1.5.14&openssl=1.0.1e&hsts=yes&profile=modern
  ssl-default-bind-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
  ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
  ssl-default-server-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
  ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets

defaults
  log global
  mode http
  option httplog
  option dontlognull
  option forwardfor
  option http-server-close
  timeout connect 5000
  timeout client 50000
  timeout server 50000
  errorfile 502 /etc/haproxy/errors/502.http

frontend jenkins-ci.overviewdocs.com
  bind :80
  bind :443 ssl crt /etc/haproxy/ssl.pem
  redirect scheme https if !{ ssl_fc }
  reqadd X-Forwarded-Proto:\ https
  use_backend jenkins

backend jenkins
  server jenkins01 127.0.0.1:8080

Fill in /etc/letsencrypt/post-renew.sh:

#!/bin/sh

cat \
    /etc/letsencrypt/live/jenkins-ci.overviewdocs.com/privkey.pem \
    /etc/letsencrypt/live/jenkins-ci.overviewdocs.com/fullchain.pem \
    > /etc/haproxy/ssl.pem
chown haproxy:haproxy /etc/haproxy/ssl.pem
chmod 0600 /etc/haproxy/ssl.pem
systemctl restart haproxy

Fill in /etc/letsencrypt/pre-renew.sh:
```
#!/bin/sh
systemctl stop haproxy
```
chmod +x /etc/letsencrypt/post-renew.sh /etc/letsencrypt/pre-renew.sh
/etc/letsencrypt/post-renew.sh (will actually start haproxy)

Fill in /etc/cron.daily/letsencrypt:

#!/bin/sh

exec /usr/bin/certbot renew \
    --pre-hook /etc/letsencrypt/pre-renew.sh \
    --post-hook /etc/letsencrypt/post-renew.sh \
    --quiet

chmod +x /etc/cron.daily/letsencrypt

Set up EC2 spot-instance nodes
1. Browse to Jenkins, "Manage Jenkins" -> "Configure System"
2. "Add New Cloud" => "Amazon EC2"
3. Check Use EC2 instance profile to obtain credentials
4. Choose region us-east-1
5. Create a private key jenkins-ci in AWS (EC2 => Key Pairs => Create key pair called jenkins-ci. This will download a jenkins-ci.pem in your browser)
6. Enter the private key in Jenkins by copy/pasting from jenkins-ci.pem.
7. Advanced -> "Add" an AMI....
8. Description: Jenkins-CI
9. AMI ID: ami-026c8acd92718196b (from https://cloud-images.ubuntu.com/locator/ec2/ amd64 ebs-ssd us-east-1 bionic)
10. Instance Type: something big, Availability Zone: whatever's best. At time of writing, C48xlarge and us-east-1d have a good price. (Use the AWS console to find a combination that seems reliable.)
11. Spot Max Bid Price: whatever -- 0.5 means 50 cents per hour. (Expect 3 builds per hour. Jenkins won't build every commit, if you commit lots of commits at the same time.)
12. Choose Bid Type: persistent
13. Security group names: test-slave (which grants SSH access to Jenkins-CI security group: Jenkins will SSH into it)
14. Remote FS root: /home/ubuntu
15. Remote user: ubuntu
16. Root command prefix: sudo
17. Slave command prefix: sudo
18. Labels: test-slave
19. Idle termination time: 30
20. Advanced => Number of Executors: 1
21. Check Stop/Disconnect on Idle Timeout
22. Tags: role:jenkins-test-slave, Name:jenkins-test-slave, Environment:ci
23. IAM Profile: The description of your IAM "instance-profile" Role. This starts with arn: and ends with instance-profile/jenkins-test-slave -- a role with permission to write to s3://overview-builds.
24. Add this Init script:
```
#!/bin/bash

set -ex

DEBIAN_FRONTEND=noninteractive apt-get -y -qq update
DEBIAN_FRONTEND=noninteractive apt-get -y -qq dist-upgrade
DEBIAN_FRONTEND=noninteractive apt-get -y -qq update
DEBIAN_FRONTEND=noninteractive apt-get -y -qq install apt-transport-https awscli ca-certificates curl openjdk-11-jre-headless software-properties-common

# https://docs.docker.com/engine/installation/linux/docker-ce/ubuntu/
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable"        
DEBIAN_FRONTEND=noninteractive apt-get -y -qq update
DEBIAN_FRONTEND=noninteractive apt-get -y -qq install docker-ce
systemctl start docker
usermod -G docker ubuntu

curl -L https://github.com/kubernetes/kops/releases/download/1.12.2/kops-linux-amd64 > /usr/local/bin/kops
curl -L https://storage.googleapis.com/kubernetes-release/release/v1.15.1/bin/linux/amd64/kubectl > /usr/local/bin/kubectl
curl -L "https://github.com/docker/compose/releases/download/1.28.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/{docker-compose,kops,kubectl}
```
25. Set Block device mapping to /dev/sda1=:24 (because Docker images often grow to greater than 8GB)
26. Check 'Associate Public IP' (or it won't be able to download things)
Add Docker Hub credentials
1. Make sure you have access to https://hub.docker.com/r/overview/
2. Browse to Jenkins and click Credentials (left sidebar), (global), and then Add Credentials (left sidebar)
3. Add a Username with Password with Scope System, ID and Description both docker-hub, and the username and password that you used to log in to https://hub.docker.com.
Add overview-server project
1. Browse to Jenkins and click Create new jobs
2. Name overview-server, Pipeline, OK
3. Check GitHub project and enter URL https://github.com/overview/overview-server
4. Check GitHub hook trigger for GITScm polling
5. Check Quiet period and enter 120 seconds (builds cost time and money; this can save us a few)
6. Make the Pipeline script Pipeline script from SCM. Select Git as a source, and enter https://github.com/overview/overview-server
7. Make */master and */feature-* are the branch specifiers
Add Kubernetes credentials
1. Browse to Jenkins and click Credentials (left sidebar), (global), and then Add Credentials (left sidebar)
2. Add a Secret text with Scope Global, ID and Description both overview-production-kubernetes, and the secret text you get following the instructions in overview-server/kubernetes/README.md
Hook up to GitHub
1. Add [![Build Status](http://jenkins-ci.overviewdocs.com/job/overview-server/badge/icon)](http://jenkins-ci.overviewdocs.com/job/overview-server/) to README.md
2. In overview-server project settings on GitHub, add "Integrations and Services" -> "Jenkins (GitHub plugin)" -> https://jenkins-ci.overviewdocs.com/github-webhook/
Hook up to Slack
1. On Jenkins, add "Slack" plugin in Manage Plugins
2. On Slack, add "Jenkins" plugin
3. Copy/paste the Team Domain and Integration Token from Slack into Jenkins
4. (That's it -- the Jenkinsfile will handle the rest.)

How we set up continuous integration on Jenkins

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Clone this wiki locally