Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Monitor VMs and containers with Prometheus and Grafana #26

Merged
merged 10 commits into from
Jun 29, 2022
Merged

Monitor VMs and containers with Prometheus and Grafana #26

merged 10 commits into from
Jun 29, 2022

Conversation

bafread
Copy link
Collaborator

@bafread bafread commented Jun 23, 2022 •
edited
Loading

Description

What changes will the PR bring? Refer to relevant issues if applicable

This PR will

Testing the PR

How do your team test if the PR is valid?

  • After building image with Packer and apply using Terraform,
  • go tohttp://<logger-vm>:3000, here we can monitor all VMs and Containers.
  • User: admin
  • Password: admin
  • After Login, go to Dashboard, and choose containerSSH-Monitoring.

Takeaways

What have you learned from the PR and want to share with your team?\

  • Building the project with Windows is unstable.
  • Always use the latest Repo! Tried cadvisor with older version -> there's bug, it fixed with the newest
  • Sometimes new Build Image has new Features, Parameters and stuff.

Q&A

Your open questions to answer before merging the PR

  • we need to test, if we can monitor the container running guest-image as well

Screenshots:

Screenshot from 2022-06-27 00-03-28
Screenshot from 2022-06-27 00-04-00

bafread
bafread commented Jun 23, 2022
View reviewed changes
packer/main.pkr.hcl
Comment on lines +82 to +90
"./scripts/download_node_exporter.sh",
"./scripts/create_node_exporter_service.sh",
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move download_node_exporter to packer while building image.
If done on TF -> Issue: no connection from github server.

@bafread bafread changed the title initial commit Final Touch Jun 26, 2022
@bafread bafread self-assigned this Jun 26, 2022
@bafread bafread changed the title Final Touch Monitoring Jun 26, 2022
@bafread bafread marked this pull request as ready for review June 27, 2022 06:16
@bafread bafread requested a review from paseaf June 27, 2022 06:16
paseaf
paseaf requested changes Jun 28, 2022
View reviewed changes
Copy link
Owner

@paseaf paseaf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks great! 🚀
added some questions and suggestions.

terraform/variables.tf Outdated Show resolved Hide resolved
packer/variables.pkr.hcl Outdated Show resolved Hide resolved
packer/scripts/create_node_exporter_service.sh Outdated Show resolved Hide resolved
packer/main.pkr.hcl Show resolved Hide resolved
.gitignore Outdated Show resolved Hide resolved
terraform/grafana/docker-compose.yml Outdated Show resolved Hide resolved
terraform/files/prometheus.yml
# expose cadvisor metrics
- job_name: "cadvisor-container-on-sacrificial-vm"
static_configs:
- targets: ["sacrificial-vm:8080"]
Copy link
Owner

@paseaf paseaf Jun 28, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is this cadvisor on 8080 while others 8088? typo?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cadvisor running on port 8080 as default.
but on our gateway port 8080 are being used by auth-test-config.
that's why i install cadvisor as container on gateway-vm and logger-vm.
because with docker it's possible & easier to do port-forwarding by using 8088:8080

terraform/scripts/run_cadvisor.sh Outdated Show resolved Hide resolved
terraform/scripts/run_cadvisor.sh Outdated Show resolved Hide resolved
terraform/grafana/provisioning/datasources/datasource.yml Outdated Show resolved Hide resolved
@paseaf paseaf added monitor Monitoring system for our honeypot (e.g., Prometheus) deployment packer labels Jun 28, 2022
bafread
bafread commented Jun 29, 2022
View reviewed changes
packer/main.pkr.hcl Show resolved Hide resolved
packer/variables.pkr.hcl Outdated Show resolved Hide resolved
terraform/files/prometheus.yml Show resolved Hide resolved
terraform/main.tf Outdated Show resolved Hide resolved
terraform/main.tf Outdated Show resolved Hide resolved
terraform/main.tf Show resolved Hide resolved
terraform/main.tf Outdated Show resolved Hide resolved
terraform/scripts/run_cadvisor.sh Outdated Show resolved Hide resolved
packer/variables.pkr.hcl Outdated Show resolved Hide resolved
terraform/files/prometheus.yml
# expose cadvisor metrics
- job_name: "cadvisor-container-on-sacrificial-vm"
static_configs:
- targets: ["sacrificial-vm:8080"]
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cadvisor running on port 8080 as default.
but on our gateway port 8080 are being used by auth-test-config.
that's why i install cadvisor as container on gateway-vm and logger-vm.
because with docker it's possible & easier to do port-forwarding by using 8088:8080

paseaf
paseaf approved these changes Jun 29, 2022
View reviewed changes
Copy link
Owner

@paseaf paseaf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! added one more question but we could do it in future PRs!

README.md
Comment on lines +26 to +28
- Cadvisor on Gateway-VM: `8088`
- Cadvisor on Logger-VM: `8088`
- Cadvisor on Sacrificial-VM: `8080`
Copy link
Owner

@paseaf paseaf Jun 29, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe use the same port for same services if possible. it could reduce future confusion and avoid mistakes.
(On /diagrams/infra.drawio.svg it shows 8088)

But we can definitely update it in the next PRs :)
Have added a card for it
https://github.com/paseaf/ContainerSSH-honeypot/projects/1#card-83701924

@bafread bafread merged commit 8d94940 into main Jun 29, 2022
@bafread bafread deleted the adding-#25 branch June 29, 2022 16:23
@paseaf paseaf changed the title Monitoring Monitor VMs and containers with Prometheus and Grafana Jun 29, 2022
@paseaf paseaf added the documentation Improvements or additions to documentation label Jun 30, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@paseaf paseaf paseaf approved these changes

Assignees

@bafread bafread

Labels
deployment documentation Improvements or additions to documentation monitor Monitoring system for our honeypot (e.g., Prometheus) packer
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Set up honeypot monitoring with Prometheus (and Grafana)
2 participants
@bafread @paseaf