From 222763134b991739ab5a4d413d16101a47ce3833 Mon Sep 17 00:00:00 2001 From: Ziyang Li Date: Wed, 18 May 2022 15:51:36 +0430 Subject: [PATCH 1/8] ci: build an ubuntu image with packer to gcp --- packer/README.md | 14 ++++++++++++++ packer/ubuntu-2204.pkr.hcl | 24 ++++++++++++++++++++++++ 2 files changed, 38 insertions(+) create mode 100644 packer/README.md create mode 100644 packer/ubuntu-2204.pkr.hcl diff --git a/packer/README.md b/packer/README.md new file mode 100644 index 0000000..30b0c92 --- /dev/null +++ b/packer/README.md @@ -0,0 +1,14 @@ +1. Create a service account at [GCP Service Accounts](https://console.cloud.google.com/iam-admin/serviceaccounts) with roles: + +- Compute Instance Admin (v1) +- Service Account User roles + +2. Open the created service account -> Keys tab, create and download a service account key in JSON. + +## Init and Build + +This command builds an image to GCP + +```bash +packer init . && packer build . +``` diff --git a/packer/ubuntu-2204.pkr.hcl b/packer/ubuntu-2204.pkr.hcl new file mode 100644 index 0000000..07c8f0b --- /dev/null +++ b/packer/ubuntu-2204.pkr.hcl @@ -0,0 +1,24 @@ +packer { + required_plugins { + googlecompute = { + version = ">= 1.0.0" + source = "github.com/hashicorp/googlecompute" + } + } +} + +source "googlecompute" "ubuntu-2204" { + project_id = "containerssh" + account_file = file("./gcp.key.json") + source_image_family = "ubuntu-pro-2204-lts" + ssh_username = "admin" + zone = "europe-west3-c" +} + +build { + name = "ubuntu-2204-with-docker" + sources = [ + "source.googlecompute.ubuntu-2204" + ] +} + From 10b7d27b8e0b3f4474172e5d03fd39364ace3f71 Mon Sep 17 00:00:00 2001 From: Ziyang Li Date: Wed, 18 May 2022 19:09:30 +0430 Subject: [PATCH 2/8] ci: fix gcp auth for Packer --- packer/.gitignore | 2 ++ packer/README.md | 17 +++++++++++++---- packer/ubuntu-2204.pkr.hcl | 3 +-- terraform/README.md | 10 ++++++---- 4 files changed, 22 insertions(+), 10 deletions(-) create mode 100644 packer/.gitignore diff --git a/packer/.gitignore b/packer/.gitignore new file mode 100644 index 0000000..9606de4 --- /dev/null +++ b/packer/.gitignore @@ -0,0 +1,2 @@ +gcp.key.json + diff --git a/packer/README.md b/packer/README.md index 30b0c92..3ce6b5b 100644 --- a/packer/README.md +++ b/packer/README.md @@ -1,9 +1,18 @@ -1. Create a service account at [GCP Service Accounts](https://console.cloud.google.com/iam-admin/serviceaccounts) with roles: +# Packer -- Compute Instance Admin (v1) -- Service Account User roles +We use Packer to create a VM image with latest software, so that Terraform can use the image to create VM instances. -2. Open the created service account -> Keys tab, create and download a service account key in JSON. +## Set up GCP credentials for Packer + +1. Set up GCP service account for Packer following [Packer - Running outside of Google Cloud](https://www.packer.io/plugins/builders/googlecompute#running-outside-of-google-cloud) + +2. Move the downloaded service account key file to `./gcp.key.json`. + +3. Set up credentials env for Packer + + ```bash + export GOOGLE_APPLICATION_CREDENTIALS="$(pwd)/gcp.key.json" + ``` ## Init and Build diff --git a/packer/ubuntu-2204.pkr.hcl b/packer/ubuntu-2204.pkr.hcl index 07c8f0b..c07e73f 100644 --- a/packer/ubuntu-2204.pkr.hcl +++ b/packer/ubuntu-2204.pkr.hcl @@ -9,9 +9,8 @@ packer { source "googlecompute" "ubuntu-2204" { project_id = "containerssh" - account_file = file("./gcp.key.json") source_image_family = "ubuntu-pro-2204-lts" - ssh_username = "admin" + ssh_username = "packer" zone = "europe-west3-c" } diff --git a/terraform/README.md b/terraform/README.md index 32e81f5..9d89f32 100644 --- a/terraform/README.md +++ b/terraform/README.md @@ -7,21 +7,23 @@ Install it as follows: 2. Install [Terraform CLI](https://learn.hashicorp.com/tutorials/terraform/install-cli?in=terraform/gcp-get-started) -3. Create and download a GCP _service account key_ (in JSON) following [_Set up GCP_ in this guide](https://learn.hashicorp.com/tutorials/terraform/google-cloud-platform-build?in=terraform/gcp-get-started).\ - Terraform will use it to manage your GCP resources. Move the key file to `./.gcp-key.json` +3. Create and download a GCP _service account key_ (in JSON) following [Terraform - Set Up GCP](https://learn.hashicorp.com/tutorials/terraform/google-cloud-platform-build?in=terraform/gcp-get-started).\ + Terraform will use it to manage your GCP resources. Move the key file to current folder as `./gcp-key.json` 4. Update `terraform/terraform.tfvars` file with the following content ```bash - project = "" - credentials_file = "" + project = "" + credentials_file = "gcp-key.json" ``` 5. Verify if your Terraform is successfully set up. + ```bash cd terraform terraform init # initialize the working directory terraform plan # preview the changes ``` + You should not see any error message in the output. From f4ec4d541faac67f7fec69190bf1c1af3899d1ff Mon Sep 17 00:00:00 2001 From: Ziyang Li Date: Wed, 18 May 2022 22:39:25 +0430 Subject: [PATCH 3/8] ci: use keyfile for packer; fix vm image name --- packer/README.md | 11 ++++------- packer/ubuntu-2204.pkr.hcl | 2 ++ 2 files changed, 6 insertions(+), 7 deletions(-) diff --git a/packer/README.md b/packer/README.md index 3ce6b5b..58656c5 100644 --- a/packer/README.md +++ b/packer/README.md @@ -7,17 +7,14 @@ We use Packer to create a VM image with latest software, so that Terraform can u 1. Set up GCP service account for Packer following [Packer - Running outside of Google Cloud](https://www.packer.io/plugins/builders/googlecompute#running-outside-of-google-cloud) 2. Move the downloaded service account key file to `./gcp.key.json`. - -3. Set up credentials env for Packer - - ```bash - export GOOGLE_APPLICATION_CREDENTIALS="$(pwd)/gcp.key.json" - ``` + Note: if you want to use a different file name or location, change `account_file` accordingly. ## Init and Build This command builds an image to GCP ```bash -packer init . && packer build . +packer init . && packer build -force . ``` + +Note: `-force` to overwrite previously built image. diff --git a/packer/ubuntu-2204.pkr.hcl b/packer/ubuntu-2204.pkr.hcl index c07e73f..31df860 100644 --- a/packer/ubuntu-2204.pkr.hcl +++ b/packer/ubuntu-2204.pkr.hcl @@ -12,6 +12,8 @@ source "googlecompute" "ubuntu-2204" { source_image_family = "ubuntu-pro-2204-lts" ssh_username = "packer" zone = "europe-west3-c" + account_file = "./gcp.key.json" + image_name = "sacrificial-vm-image" } build { From a97797d6d1e090f11cd50316b0bccfae7d29ab42 Mon Sep 17 00:00:00 2001 From: Ziyang Li Date: Wed, 18 May 2022 22:44:12 +0430 Subject: [PATCH 4/8] ci: allow ssh for VPC; use packer image in terraform --- terraform/main.tf | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) diff --git a/terraform/main.tf b/terraform/main.tf index 56ed6db..0c380e5 100644 --- a/terraform/main.tf +++ b/terraform/main.tf @@ -11,6 +11,22 @@ resource "google_compute_network" "main" { auto_create_subnetworks = false } +resource "google_compute_firewall" "benchmark_vpc_rules" { + name = "benchmark-vpc-rules" + network = google_compute_network.main.self_link + + allow { + protocol = "icmp" + } + + allow { + protocol = "tcp" + ports = ["22"] + } + + source_ranges = ["0.0.0.0/0"] +} + resource "google_compute_subnetwork" "gateway_subnet" { name = "gateway-subnet" ip_cidr_range = "10.0.0.0/24" @@ -43,12 +59,12 @@ resource "google_compute_instance" "gateway_vm" { } resource "google_compute_instance" "sacrificial_vm" { - name = "sacrificial" + name = "sacrificial-vm" machine_type = "e2-micro" boot_disk { initialize_params { - image = "ubuntu-os-pro-cloud/ubuntu-pro-2204-lts" + image = "sacrificial-vm-image" } } From aee97ba7a26c035dc0cd63cebf9291b055614473 Mon Sep 17 00:00:00 2001 From: Ziyang Li Date: Thu, 19 May 2022 20:07:51 +0430 Subject: [PATCH 5/8] ci(packer): update pkgs and install docker --- packer/scripts/install_docker.sh | 36 ++++++++++++++++++++++++ packer/scripts/update.sh | 48 ++++++++++++++++++++++++++++++++ packer/ubuntu-2204.pkr.hcl | 13 +++++---- 3 files changed, 92 insertions(+), 5 deletions(-) create mode 100755 packer/scripts/install_docker.sh create mode 100644 packer/scripts/update.sh diff --git a/packer/scripts/install_docker.sh b/packer/scripts/install_docker.sh new file mode 100755 index 0000000..c4e1c6b --- /dev/null +++ b/packer/scripts/install_docker.sh @@ -0,0 +1,36 @@ +#!/bin/bash + +# solves apt lock issue +# https://github.com/geerlingguy/packer-boxes/issues/7#issuecomment-425641793 +function apt-get() { + while fuser -s /var/lib/apt/lists/lock; + do echo 'apt-get is waiting for the lock release ...'; + sleep 1; + done; + /usr/bin/apt-get "$@"; +} + +export DEBIAN_FRONTEND=noninteractive + +set -euxo pipefail + +apt-get update +apt-get upgrade -y +apt-get install -y \ + ca-certificates \ + curl \ + gnupg \ + lsb-release + +# add Docker's official GPG key +curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg + +# set up a stable repo +echo \ + "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \ + $(lsb_release -cs) stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null + +# install docker engine +apt-get update +apt-get install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin + diff --git a/packer/scripts/update.sh b/packer/scripts/update.sh new file mode 100644 index 0000000..5558816 --- /dev/null +++ b/packer/scripts/update.sh @@ -0,0 +1,48 @@ +#!/bin/bash -eux +# This script is adapted from +# https://github.com/chef/bento/blob/main/packer_templates/ubuntu/scripts/update.sh + +# solves apt lock issue +function apt-get() { + while fuser -s /var/lib/apt/lists/lock; + do echo 'apt-get is waiting for the lock release ...'; + sleep 1; + done; + /usr/bin/apt-get "$@"; +} + +export DEBIAN_FRONTEND=noninteractive + +echo "disable release-upgrades" +sed -i.bak 's/^Prompt=.*$/Prompt=never/' /etc/update-manager/release-upgrades; + +echo "disable systemd apt timers/services" +systemctl stop apt-daily.timer; +systemctl stop apt-daily-upgrade.timer; +systemctl disable apt-daily.timer; +systemctl disable apt-daily-upgrade.timer; +systemctl mask apt-daily.service; +systemctl mask apt-daily-upgrade.service; +systemctl daemon-reload; + +# Disable periodic activities of apt to be safe +cat </etc/apt/apt.conf.d/10periodic; +APT::Periodic::Enable "0"; +APT::Periodic::Update-Package-Lists "0"; +APT::Periodic::Download-Upgradeable-Packages "0"; +APT::Periodic::AutocleanInterval "0"; +APT::Periodic::Unattended-Upgrade "0"; +EOF + +echo "remove the unattended-upgrades and ubuntu-release-upgrader-core packages" +rm -rf /var/log/unattended-upgrades; +apt-get -y purge unattended-upgrades ubuntu-release-upgrader-core; + +echo "update the package list" +apt-get -y update; + +echo "upgrade all installed packages incl. kernel and kernel headers" +apt-get -y dist-upgrade -o Dpkg::Options::="--force-confnew"; + +reboot + diff --git a/packer/ubuntu-2204.pkr.hcl b/packer/ubuntu-2204.pkr.hcl index 31df860..645c999 100644 --- a/packer/ubuntu-2204.pkr.hcl +++ b/packer/ubuntu-2204.pkr.hcl @@ -8,12 +8,12 @@ packer { } source "googlecompute" "ubuntu-2204" { - project_id = "containerssh" + project_id = "containerssh" source_image_family = "ubuntu-pro-2204-lts" - ssh_username = "packer" - zone = "europe-west3-c" - account_file = "./gcp.key.json" - image_name = "sacrificial-vm-image" + ssh_username = "root" + zone = "europe-west3-c" + account_file = "./gcp.key.json" + image_name = "sacrificial-vm-image" } build { @@ -21,5 +21,8 @@ build { sources = [ "source.googlecompute.ubuntu-2204" ] + provisioner "shell" { + scripts = ["./scripts/update.sh", "./scripts/install_docker.sh"] + } } From 50c16262be5f9d14f2b1f7ddd10307893d8c2bd1 Mon Sep 17 00:00:00 2001 From: Ziyang Li Date: Thu, 19 May 2022 22:53:52 +0430 Subject: [PATCH 6/8] docs: update packer readme --- packer/README.md | 37 ++++++++++++++++---- packer/{ubuntu-2204.pkr.hcl => main.pkr.hcl} | 0 2 files changed, 31 insertions(+), 6 deletions(-) rename packer/{ubuntu-2204.pkr.hcl => main.pkr.hcl} (100%) diff --git a/packer/README.md b/packer/README.md index 58656c5..523aaf8 100644 --- a/packer/README.md +++ b/packer/README.md @@ -1,20 +1,45 @@ # Packer -We use Packer to create a VM image with latest software, so that Terraform can use the image to create VM instances. +We use [Packer](https://www.packer.io/) to create a [VM image on GCP](https://cloud.google.com/compute/docs/images) with the latest software and Docker installed. +The image can be used to create secure VMs. -## Set up GCP credentials for Packer +## How it works + +Packer will + +1. create a VM on Google Cloud Platform (GCP) +2. run our scripts to update software and install Docker on the VM +3. take a snapshot of the VM and store it as an image called `sacrificial-vm` on GCP +4. delete the VM + +## Getting Started + +### Prerequisite + +What you need: + +- `gcloud` installed locally +- A GCP project +- [Packer installed](https://www.packer.io/downloads) locally + +### Set up Packer 1. Set up GCP service account for Packer following [Packer - Running outside of Google Cloud](https://www.packer.io/plugins/builders/googlecompute#running-outside-of-google-cloud) -2. Move the downloaded service account key file to `./gcp.key.json`. - Note: if you want to use a different file name or location, change `account_file` accordingly. +2. Move the downloaded service account key file to `./gcp.key.json` + + > Note: if you want to use a different file name or location, change `account_file` in [`./main.pkr.hcl`](./main.pkr.hcl) accordingly -## Init and Build +3. Update `project-id` in `main.pkr.hcl` to match yours -This command builds an image to GCP +### Build the image + +Run ```bash packer init . && packer build -force . ``` +An image should be built to your GCP project + Note: `-force` to overwrite previously built image. diff --git a/packer/ubuntu-2204.pkr.hcl b/packer/main.pkr.hcl similarity index 100% rename from packer/ubuntu-2204.pkr.hcl rename to packer/main.pkr.hcl From eddf14695f5a265b7545ccf56f49d0f233c35e1e Mon Sep 17 00:00:00 2001 From: Ziyang Li Date: Thu, 19 May 2022 23:04:32 +0430 Subject: [PATCH 7/8] style: move pipefail to top --- packer/scripts/install_docker.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/packer/scripts/install_docker.sh b/packer/scripts/install_docker.sh index c4e1c6b..d242361 100755 --- a/packer/scripts/install_docker.sh +++ b/packer/scripts/install_docker.sh @@ -1,5 +1,7 @@ #!/bin/bash +set -euxo pipefail + # solves apt lock issue # https://github.com/geerlingguy/packer-boxes/issues/7#issuecomment-425641793 function apt-get() { @@ -12,8 +14,6 @@ function apt-get() { export DEBIAN_FRONTEND=noninteractive -set -euxo pipefail - apt-get update apt-get upgrade -y apt-get install -y \ From 9c182391395e9dd6c673ede0412e6c3769f51ff3 Mon Sep 17 00:00:00 2001 From: Ziyang Li Date: Thu, 19 May 2022 23:24:31 +0430 Subject: [PATCH 8/8] refactor: extract function from bash --- packer/scripts/install_docker.sh | 10 +------- packer/scripts/update.sh | 9 +------ packer/scripts/util_fn | 11 +++++++++ terraform/Deploy ContainerSSH.md | 40 ++++++++++++++++++++++++++++++++ terraform/README.md | 10 +++++++- 5 files changed, 62 insertions(+), 18 deletions(-) create mode 100644 packer/scripts/util_fn create mode 100644 terraform/Deploy ContainerSSH.md diff --git a/packer/scripts/install_docker.sh b/packer/scripts/install_docker.sh index d242361..a9d9c83 100755 --- a/packer/scripts/install_docker.sh +++ b/packer/scripts/install_docker.sh @@ -2,15 +2,7 @@ set -euxo pipefail -# solves apt lock issue -# https://github.com/geerlingguy/packer-boxes/issues/7#issuecomment-425641793 -function apt-get() { - while fuser -s /var/lib/apt/lists/lock; - do echo 'apt-get is waiting for the lock release ...'; - sleep 1; - done; - /usr/bin/apt-get "$@"; -} +[ -f ./util_fn ] && source ./util_fn export DEBIAN_FRONTEND=noninteractive diff --git a/packer/scripts/update.sh b/packer/scripts/update.sh index 5558816..bf0ec19 100644 --- a/packer/scripts/update.sh +++ b/packer/scripts/update.sh @@ -2,14 +2,7 @@ # This script is adapted from # https://github.com/chef/bento/blob/main/packer_templates/ubuntu/scripts/update.sh -# solves apt lock issue -function apt-get() { - while fuser -s /var/lib/apt/lists/lock; - do echo 'apt-get is waiting for the lock release ...'; - sleep 1; - done; - /usr/bin/apt-get "$@"; -} +[ -f ./util_fn ] && source ./util_fn export DEBIAN_FRONTEND=noninteractive diff --git a/packer/scripts/util_fn b/packer/scripts/util_fn new file mode 100644 index 0000000..16f6c71 --- /dev/null +++ b/packer/scripts/util_fn @@ -0,0 +1,11 @@ +#!/bin/bash + +# This apt-get waits until lock is released +# https://github.com/geerlingguy/packer-boxes/issues/7#issuecomment-425641793 +function apt-get() { + while fuser -s /var/lib/apt/lists/lock; + do echo 'apt-get is waiting for the lock release ...'; + sleep 1; + done; + /usr/bin/apt-get "$@"; +} diff --git a/terraform/Deploy ContainerSSH.md b/terraform/Deploy ContainerSSH.md new file mode 100644 index 0000000..054dfa6 --- /dev/null +++ b/terraform/Deploy ContainerSSH.md @@ -0,0 +1,40 @@ +Deploy ContainerSSH + +## Goal + +Implement step 1-3 of https://containerssh.io/guides/honeypot/ + +## Components and Requirements + +> Extracted from https://containerssh.io/guides/honeypot/ + +### Gateway VM x1 + +- [ ] have sufficient disk space to hold audit logs and containers +- [ ] firewall rules + - [ ] Port 22 should be open to the Internet. + - [ ] Ports 9100 and 9101 should be open from your Prometheus instance. These will be used by the Prometheus node exporter and the ContainerSSH metrics server respectively. + - [ ] Outbound rules to your S3-compatible object storage. + +### Sacrificial VM x1 + +- [ ] Use a prebuilt VM image with Docker installed to keep the host up to date. + - [ ] Use tools like [Packer](https://www.packer.io/) to keep the VM image updated +- [ ] run on its own dedicated physical hardware +- [ ] have sufficient disk space to hold audit logs and containers +- [ ] Firewall rules + - [ ] Only allows connection with the gateway host + - [ ] Only allow inbound connections on TCP port 2376 from the gateway host + +### S3-compatible object storage x1 + +Maybe set up MINIO on GCP? +For uploading audit logs + +- [ ] decide what S3 object to use + +### Prometheus x1 + +For monitoring audit logs. + +- [ ] get familiar with prometheus diff --git a/terraform/README.md b/terraform/README.md index 9d89f32..67dce1f 100644 --- a/terraform/README.md +++ b/terraform/README.md @@ -14,7 +14,7 @@ Install it as follows: ```bash project = "" - credentials_file = "gcp-key.json" + credentials_file = "gcp.key.json" ``` 5. Verify if your Terraform is successfully set up. @@ -27,3 +27,11 @@ Install it as follows: You should not see any error message in the output. +## Trouble Shooting + +1. `terraform apply` failed with `Error creating Network: googleapi: Error 403: Required 'compute.networks.create' permission for '', forbidden` + +Possible Issue: + +1. `project-id` might be wrong. Check Deployment step 4. +2. Did you grant the _Project Editor_ permission to the service account in step 3?