Set up CA #19

bafread · 2022-06-06T14:45:40Z

Description

This PR will

close Set up CA for sacrificial VM and gateway VM #15
fix Flaky packer build #10 and
fix Error with Packer (before adding Expansion) #18 with https://github.com/paseaf/containerSSH-honeypot/blob/be287d6c6fa6640fc3b049fa3ce0ee291697fc02/packer/main.pkr.hcl#L87

Testing the PR
You can verify the PR by following /terraform/README.md#Set-up-CA and set up the CA manually.

Future improvements

Automate CA set up #20

Developers notes

For CA's hostname, use sacrificial-vm ($ hostname) instead of $ hostname -f.
Reason: $ hostname -f is different for different GCP accounts, so we cannot hardcode the hostname, which makes the configuration more cumbersome.

Takeaways

How to connect to a remote docker engine

Method 1: Create a docker context

Term (Docker Context): a full environment for docker.

# create a context
docker context create --docker host=ssh://username@host remote-engine
# connect to the context
docker context use remote-engine

Now your docker will work directly on the remote host.

Method 2 (Our case): set `DOCKER_HOST`

Docker supports remote connection via SSH, TCP, TCP with TLS (our case)

export DOCKER_HOST=ssh://username@host remote-engine

This will directly send requests to the remote host using the current context.

Others

see unresolved comments below

packer & terraform variable declaration vs definition
...

Q&A

Regarding Container SSH documentation - Step 4:
Answers from Janos:

Q1: SSH vs. TLS

The document recommended us to connect to remote Docker engine over TLS. Why not over SSH? (SSH seems much easier to set up)
A1: SSH is unstable and should only be used for human access.

Janos: SSH is an additional layer of complexity. You would need to keep the SSH tunnel alive all the time for this to work, which in practice doesn't really work all that well. I once ran an SSHFS mount for a longer period of time and it frequently crashed or became unresponsive since SSH is generally meant for human use. There are very few mechanisms to handle connection failures transparently.

[...] you need ContainerSSH to be able to connect to the remote Docker socket whenever a connection comes in. We have a limited set of retries for the initial connection and no retries afterwards. If the SSH tunnel breaks, the client gets disconnected. I can't precisely tell you why, but my experience is that using SSH for anything but human access is a bad idea.

Q2: `docker run -H` doesn't work

The command on documentation doesn't work.

$ docker run -H tcp://your-sacrificial-host:2376 -ti ubuntu
unknown shorthand flag: 'H' in -H
See 'docker run --help'.

Is the command wrong? It works with

docker --tlsverify -H=tcp://sacrificial-vm:2376 run -ti ubuntu

when ~/.docker is set up as Docker - Secure by default

A2: Janos: It should work with docker -H <command>

# GOOD 1
docker --tlsverify -H tcp://sacrificial-vm:2376 version

# GOOD 2 (with -H=)
docker --tlsverify -H=tcp://sacrificial-vm:2376 version

# GOOD 3
export TLSVERIFY=1
docker -H=tcp://sacrificial-vm:2376 version

# BAD
docker -H=tcp://sacrificial-vm:2376 version
> Error response from daemon: Client sent an HTTP request to an HTTPS server.

Q3: Do we need Docker daemon to also run on port `2375`?

Step 4 gives two ports for Docker connection, but only 2376 seems to be used in other places in the doc.
A3: Janos: typically 2376 for encrypted connection, 2375 for unencrypted.

Action: Our 2375 is not open. We can probably ignore it.

1. Rename and add Firewall-rules with tags 2. Change the image of gateway-vm to ubuntu with docker 3. Build containerssh-guest-image on Sacrificial-vm

This reverts commit 292877b.

1. install ssh-container-guest-image directly with docker pull. 2. add ca_cert for client and server (need to be tested, if it works with another server with different fqdn) 3. unclean!

added some small changes

paseaf · 2022-06-06T14:47:55Z

packer/scripts/update.sh

+sudo shutdown -r +1
+sleep 5m


changed it because ...

it will fix the disconnected issue :)

Just changed it back to reboot as it's much faster and works now... (I tested it for > 10 times)
(maybe because of

provisioner "shell" { script = "./scripts/update_apt_packages.sh" + expect_disconnect = true }

paseaf

some TODOs for us: - clean up what needs to be cleaned - try to automate everything (especially key file generation. we should not commit key files to github in the end) - finish the whole setup - add some description to the PR if needed
Notes:

hostname -f: get the fqdn
Use FQDN for $HOST in docker manual commands to generate .pem

config.yaml not working yet

docker key gen still manual process. Need to automate it.

Decided to make CA work first to control the PR size.

packer/main.pkr.hcl

packer/README.md

terraform/files/prometheus.yml

terraform/main.tf

- make small changes in firewall rule (open port 9090 and 9091 only for logger-vm) - add new script to run container in gateway-vm as Janos's Suggestion (not work correctly yet; the container keep exit instead of running in background)

the script run docker container inside our VM (container will run in background) and will mount the settings for containerssh

packer/main.pkr.hcl

bafread · 2022-06-08T17:53:48Z

packer/variables.pkr.hcl

+
+variable "project_id" {
+  type =  string
+  default = "containerssh-352007"


Enter your project ID here

Will move it to variables.auto.pkrvars.hclas recommended by
https://www.packer.io/guides/hcl/variables#from-a-file

terraform/main.tf

bafread · 2022-06-08T17:56:13Z

terraform/main.tf

  allow {
    protocol = "tcp"
-    ports    = ["22", "9091"]
+    ports    = ["22"]


Open port 22 for all VM (for SSH), so that it's easier to control
TODO Later: open port 22 only for gateway-vm

We can finalize firewall rules in #5 after everything works

terraform/main.tf

bafread · 2022-06-08T17:58:57Z

terraform/main.tf

      "./scripts/download_node_exporter.sh",
-      "./scripts/run_node_exporter.sh"
+      "./scripts/run_node_exporter.sh",
+      "./scripts/run_docker_container.sh"


as per Janos's suggestion, running container inside gateway-vm to run sshcontainer

terraform/scripts/run_docker_container.sh

bafread · 2022-06-08T18:03:43Z

packer/README.md

@@ -30,7 +30,7 @@ What you need:

   > Note: if you want to use a different file name or location, change `account_file` in [`./main.pkr.hcl`](./main.pkr.hcl) accordingly

-3. Update `project-id` in `main.pkr.hcl` to match yours
+3. Update `project-id` in `variables.pkr.hcl` to match yours


Update readme file to match the changes

moved to *.auto.pkr.hcl file as packer recommended

paseaf

Finished CA set up with manual configuration.
Will do next steps in future PRs.

TODOs before merge:

check out answers to open questions and update our code if needed
remove ca*.tar files and relevant config
verify CA set up again with readme

paseaf · 2022-06-10T08:42:50Z

.gitignore

+terraform/variables.tf
+terraform/.terraform.lock.hcl
+packer/files/Neuer Ordner/
+terraform/variables.tf


Removed this file for the following reasons:

Maybe we shouldn't ignore these files.

for .terraform.lock.hcl file, refer to
https://stackoverflow.com/a/67975490

We should update the file instead of ignore it. (done in this PR)

terraform providers lock -platform=windows_amd64 -platform=darwin_amd64 -platform=linux_amd64

for Neuer Ordner, it's neither a common file to ignore nor a project relevant file, so maybe a bit off topic here?

variables.tf declares variables and their types, and should be checked into the repo.

paseaf · 2022-06-10T10:03:33Z

packer/variables.pkr.hcl

+
+variable "project_id" {
+  type =  string
+  default = "containerssh-352007"


Will move it to variables.auto.pkrvars.hclas recommended by
https://www.packer.io/guides/hcl/variables#from-a-file

paseaf · 2022-06-10T10:04:46Z

packer/variables.pkr.hcl

+  type    = string
+  default = "containerssh-352007"
+  // Sensitive vars are hidden from output as of Packer v1.6.5
+  sensitive = true


Removed sensitive as project id should not be sensitive.

paseaf · 2022-06-10T10:12:11Z

packer/README.md

@@ -30,7 +30,7 @@ What you need:

   > Note: if you want to use a different file name or location, change `account_file` in [`./main.pkr.hcl`](./main.pkr.hcl) accordingly

-3. Update `project-id` in `main.pkr.hcl` to match yours
+3. Update `project-id` in `variables.pkr.hcl` to match yours


moved to *.auto.pkr.hcl file as packer recommended

paseaf · 2022-06-11T17:21:18Z

packer/scripts/update.sh

+sudo shutdown -r +1
+sleep 5m


Just changed it back to reboot as it's much faster and works now... (I tested it for > 10 times)
(maybe because of

provisioner "shell" { script = "./scripts/update_apt_packages.sh" + expect_disconnect = true }

paseaf · 2022-06-11T17:25:48Z

terraform/main.tf

  allow {
    protocol = "tcp"
-    ports    = ["22", "9091"]
+    ports    = ["22"]


We can finalize firewall rules in #5 after everything works

terraform/scripts/run_docker_container.sh

packer/main.pkr.hcl

bafread added 7 commits June 4, 2022 14:41

fix error, where instance is not reachable after restart

85045a5

TLDR: Fix Issue with Packer and add Guest Image on Sacrificial

4f8d75c

update main.tf

90d35c9

1. Rename and add Firewall-rules with tags 2. Change the image of gateway-vm to ubuntu with docker 3. Build containerssh-guest-image on Sacrificial-vm

add machine type

292877b

Revert "add machine type"

a4bbd3f

This reverts commit 292877b.

Updated Packer

ed2dc0b

1. install ssh-container-guest-image directly with docker pull. 2. add ca_cert for client and server (need to be tested, if it works with another server with different fqdn) 3. unclean!

add script to prepare folder for config

6530a55

added some small changes

This comment was marked as off-topic.

Sign in to view

paseaf reviewed Jun 6, 2022

View reviewed changes

bafread added 5 commits June 7, 2022 11:59

Apply some changes and add a fix for docker daemon in sacrificial-vm

264f9bd

small changes, and add script

7832f4c

- make small changes in firewall rule (open port 9090 and 9091 only for logger-vm) - add new script to run container in gateway-vm as Janos's Suggestion (not work correctly yet; the container keep exit instead of running in background)

re-add the mount of folder

1e97591

small fix and small change

18f5ac5

add script: run container on gateway-vm

98a5647

the script run docker container inside our VM (container will run in background) and will mount the settings for containerssh