Skip to content
This repository was archived by the owner on Oct 10, 2025. It is now read-only.

Commit 8a33e42

Browse files
josecelanojosecelano
committed
feat: add local KVM/libvirt testing infrastructure with automated fixes
- Add complete local VM testing setup using OpenTofu and KVM/libvirt - Create infrastructure automation with Makefile targets for setup and testing - Add comprehensive libvirt troubleshooting and permission fixes - Implement automated AppArmor override for libvirt-qemu storage access - Add cloud-init configuration for Ubuntu 22.04 VMs with Docker setup - Create test suite for infrastructure validation and Torrust Tracker integration - Add detailed documentation with quick-start and troubleshooting guides - Configure markdownlint for consistent documentation formatting - Fix all markdown linting issues in core infrastructure documentation Infrastructure includes: - OpenTofu configuration for local VM deployment - Cloud-init setup with torrust user, Docker, and security hardening - Automated scripts for libvirt permission and storage pool fixes - Comprehensive test coverage for prerequisites, deployment, and integration - GitHub Actions workflow for CI/CD validation This enables reliable local testing before cloud deployment and provides automated solutions for common libvirt permission issues on Ubuntu/Debian.
1 parent 0ad886e commit 8a33e42

25 files changed

+3594
-22
lines changed

‎.github/copilot-instructions.md‎

Lines changed: 266 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -1,20 +1,266 @@
1-
This repository contains all the configuration needed to run the live Torrust Tracker demo.
2-
3-
The main goal is to provide a simple and easy-to-use setup for the Torrust Tracker, which can be deployed on a single server.
4-
5-
The current major initiative is to migrate the tracker to a new infrastructure on Hetzner. This involves:
6-
- Running the tracker binary directly on the host for performance.
7-
- Using Docker for supporting services like Nginx, Prometheus, Grafana and MySQL.
8-
- Migrating the database from SQLite to MySQL.
9-
10-
When providing assistance, please act as an experienced open-source developer and system administrator.
11-
12-
Follow these conventions:
13-
- Use Conventional Commits for commit messages. Include the issue number in this format `#1` in the commit message if applicable, e.g., `feat: [#1] add new feature`.
14-
- The issue number should be the branch prefix, e.g., `feat: [#1] add new feature` for branch `1-add-new-feature`.
15-
- We use the proposed GitHub branch naming convention:
16-
- Starts with a number indicating the issue number.
17-
- Followed by a hyphen and a short description of the feature or fix.
18-
- Uses hyphens to separate words, e.g., `1-add-new-feature`.
19-
- Ensure that shell scripts are POSIX-compliant.
20-
- Provide clear and concise documentation for any new features or changes.
1+
# Torrust Tracker Demo - Contributor Guide
2+
3+
## 🎯 Project Overview
4+
5+
**Torrust Tracker Demo** is the complete production deployment configuration for running a live [Torrust Tracker](https://github.com/torrust/torrust-tracker) instance. This repository provides:
6+
7+
- **Production deployment** configurations for Hetzner cloud infrastructure
8+
- **Local testing environment** using KVM/libvirt virtualization
9+
- **Infrastructure as Code** approach using OpenTofu/Terraform and cloud-init
10+
- **Monitoring setup** with Grafana dashboards and Prometheus metrics
11+
- **Automated deployment** scripts and Docker Compose configurations
12+
13+
### Current Major Initiative
14+
15+
We are migrating the tracker to a new infrastructure on Hetzner, involving:
16+
17+
- Running the tracker binary directly on the host for performance
18+
- Using Docker for supporting services (Nginx, Prometheus, Grafana, MySQL)
19+
- Migrating the database from SQLite to MySQL
20+
- Implementing Infrastructure as Code for reproducible deployments
21+
22+
## 📁 Repository Structure
23+
24+
```text
25+
torrust-tracker-demo/
26+
├── .github/
27+
│ ├── workflows/ # GitHub Actions CI/CD pipelines
28+
│ └── copilot-instructions.md # This contributor guide
29+
├── docs/
30+
│ ├── infrastructure/ # Local testing documentation
31+
│ │ ├── quick-start.md # Fast setup guide
32+
│ │ ├── local-testing-setup.md # Detailed setup
33+
│ │ └── libvirt-setup.md # Troubleshooting guide
34+
│ ├── setup.md # Production deployment docs
35+
│ ├── deployment.md # Deployment procedures
36+
│ ├── firewall.md # Security configuration
37+
│ └── *.md # Additional operational docs
38+
├── infrastructure/ # Infrastructure as Code
39+
│ ├── terraform/ # OpenTofu/Terraform configurations
40+
│ │ ├── main.tf # VM and infrastructure definition
41+
│ │ └── local.tfvars # Local configuration (git-ignored)
42+
│ ├── cloud-init/ # VM provisioning templates
43+
│ │ ├── user-data.yaml.tpl # Main system configuration
44+
│ │ ├── user-data-minimal.yaml.tpl # Debug configuration
45+
│ │ ├── meta-data.yaml # VM metadata
46+
│ │ └── network-config.yaml # Network setup
47+
│ └── scripts/ # Infrastructure automation scripts
48+
├── share/
49+
│ ├── bin/ # Deployment and utility scripts
50+
│ └── container/default/config/ # Docker service configurations
51+
├── tests/infrastructure/ # Infrastructure validation tests
52+
├── compose.yaml # Docker Compose for services
53+
├── Makefile # Main automation interface
54+
└── *.md # Project documentation
55+
```
56+
57+
### Key Components
58+
59+
#### 🏗️ Infrastructure (`infrastructure/`)
60+
61+
- **OpenTofu/Terraform**: Declarative infrastructure configuration
62+
- **Cloud-init**: Automated VM provisioning and setup
63+
- **Scripts**: Automation helpers for libvirt, monitoring, etc.
64+
65+
#### 🐳 Docker Services (`compose.yaml`, `share/container/`)
66+
67+
- **Nginx**: Reverse proxy and SSL termination
68+
- **Grafana**: Metrics visualization and dashboards
69+
- **Prometheus**: Metrics collection and storage
70+
- **Certbot**: Automated SSL certificate management
71+
72+
#### 🧪 Testing (`tests/infrastructure/`)
73+
74+
- **Validation scripts**: Ensure infrastructure works correctly
75+
- **Integration tests**: End-to-end deployment verification
76+
- **CI/CD pipelines**: Automated testing in GitHub Actions
77+
78+
#### 📚 Documentation (`docs/`)
79+
80+
- **Production**: Setup, deployment, and operational guides
81+
- **Local testing**: Development and testing instructions
82+
- **Troubleshooting**: Common issues and solutions
83+
84+
## 🛠️ Development Workflow
85+
86+
### Quick Start for Contributors
87+
88+
```bash
89+
# 1. Clone and setup
90+
git clone https://github.com/torrust/torrust-tracker-demo.git
91+
cd torrust-tracker-demo
92+
93+
# 2. Install dependencies (Ubuntu/Debian)
94+
make install-deps
95+
96+
# 3. Setup SSH key for VMs
97+
make setup-ssh-key
98+
99+
# 4. Test infrastructure locally
100+
make apply # Deploy test VM
101+
make ssh # Connect to VM
102+
make destroy # Cleanup
103+
104+
# 5. Run tests
105+
make test # Full infrastructure test
106+
make test-syntax # Syntax validation only
107+
```
108+
109+
### Main Commands
110+
111+
| Command | Purpose |
112+
| ------------------------- | --------------------------------- |
113+
| `make help` | Show all available commands |
114+
| `make install-deps` | Install OpenTofu, libvirt, KVM |
115+
| `make test` | Run complete infrastructure tests |
116+
| `make apply` | Deploy VM with full configuration |
117+
| `make apply-minimal` | Deploy VM with minimal config |
118+
| `make ssh` | Connect to deployed VM |
119+
| `make destroy` | Remove deployed VM |
120+
| `make monitor-cloud-init` | Watch VM provisioning progress |
121+
122+
## 📋 Conventions and Standards
123+
124+
### Git Workflow
125+
126+
#### Branch Naming
127+
128+
- **Format**: `{issue-number}-{short-description}`
129+
- **Examples**: `42-add-mysql-support`, `15-fix-ssl-renewal`
130+
- Always start with the GitHub issue number
131+
132+
#### Commit Messages
133+
134+
- **Format**: Conventional Commits with issue references
135+
- **Structure**: `{type}: [#{issue}] {description}`
136+
- **Examples**:
137+
```
138+
feat: [#42] add MySQL database support
139+
fix: [#15] resolve SSL certificate renewal issue
140+
docs: [#8] update deployment guide
141+
ci: [#23] add infrastructure validation tests
142+
```
143+
144+
#### Commit Types
145+
146+
- `feat`: New features
147+
- `fix`: Bug fixes
148+
- `docs`: Documentation changes
149+
- `ci`: CI/CD pipeline changes
150+
- `refactor`: Code refactoring
151+
- `test`: Test additions/changes
152+
- `chore`: Maintenance tasks
153+
154+
### Code Quality Standards
155+
156+
#### Shell Scripts
157+
158+
- **POSIX Compliance**: All shell scripts must be POSIX-compliant
159+
- **Linting**: Use [ShellCheck](https://github.com/koalaman/shellcheck)
160+
- **Error Handling**: Use `set -euo pipefail` for strict error handling
161+
- **Documentation**: Include help functions and clear comments
162+
163+
#### YAML Files
164+
165+
- **Linting**: Use [yamllint](https://yamllint.readthedocs.io/en/stable/)
166+
- **Configuration**: Follow `.yamllint-ci.yml` rules
167+
- **Formatting**: 2-space indentation, 120-character line limit
168+
- **Comments**: Use `# ` (space after hash) for comments
169+
170+
#### Markdown Documentation
171+
172+
- **Linting**: Follow [markdownlint](https://github.com/DavidAnson/markdownlint) conventions
173+
- **Structure**: Use consistent heading hierarchy
174+
- **Links**: Prefer relative links for internal documentation
175+
- **Code blocks**: Always specify language for syntax highlighting
176+
177+
#### Infrastructure as Code
178+
179+
- **Validation**: All Terraform/OpenTofu must pass `tofu validate`
180+
- **Planning**: Test with `tofu plan` before applying
181+
- **Variables**: Use `local.tfvars` for sensitive/local config (git-ignored)
182+
- **Templates**: Use `.tpl` extension for templated files
183+
184+
### Testing Requirements
185+
186+
#### Infrastructure Tests
187+
188+
- **Syntax validation**: All configurations must pass linting
189+
- **Local deployment**: Must successfully deploy and provision VMs
190+
- **Service validation**: All services must start and be accessible
191+
- **Network testing**: Ports and firewall rules must be correct
192+
193+
#### CI/CD Requirements
194+
195+
- **GitHub Actions**: All PRs must pass CI validation
196+
- **No secrets**: Never commit SSH keys, passwords, or tokens
197+
- **Documentation**: Update docs for any infrastructure changes
198+
199+
### Security Guidelines
200+
201+
#### Secrets Management
202+
203+
- **SSH Keys**: Use template variables, store in `local.tfvars`
204+
- **Git Ignore**: All sensitive files must be in `.gitignore`
205+
- **Environment Variables**: Use environment variables for CI/CD secrets
206+
- **Review**: All security-related changes require review
207+
208+
#### Infrastructure Security
209+
210+
- **UFW Firewall**: Only open required ports
211+
- **SSH Access**: Key-based authentication only
212+
- **Updates**: Enable automatic security updates
213+
- **Monitoring**: Log security events and access
214+
215+
## 🚀 Getting Started
216+
217+
### For New Contributors
218+
219+
1. **Read the documentation**:
220+
221+
- [Quick Start Guide](../docs/infrastructure/quick-start.md)
222+
- [Complete Setup Guide](../docs/infrastructure/local-testing-setup.md)
223+
224+
2. **Set up your environment**:
225+
226+
```bash
227+
make install-deps # Install dependencies
228+
make setup-ssh-key # Configure SSH access
229+
make test-prereq # Verify setup
230+
```
231+
232+
3. **Test a simple change**:
233+
234+
```bash
235+
make apply # Deploy test VM
236+
make ssh # Verify access
237+
make destroy # Clean up
238+
```
239+
240+
4. **Review existing issues**: Check [GitHub Issues](https://github.com/torrust/torrust-tracker-demo/issues) for good first contributions
241+
242+
### For Infrastructure Changes
243+
244+
1. **Local testing first**: Always test infrastructure changes locally
245+
2. **Validate syntax**: Run `make test-syntax` before committing
246+
3. **Document changes**: Update relevant documentation
247+
4. **Test end-to-end**: Ensure the full deployment pipeline works
248+
249+
### For AI Assistants
250+
251+
When providing assistance:
252+
253+
- Act as an experienced open-source developer and system administrator
254+
- Follow all conventions listed above
255+
- Prioritize security and best practices
256+
- Test infrastructure changes locally before suggesting them
257+
- Provide clear explanations and documentation
258+
- Consider the migration to Hetzner infrastructure in suggestions
259+
260+
## 📖 Additional Resources
261+
262+
- **Torrust Tracker**: <https://github.com/torrust/torrust-tracker>
263+
- **OpenTofu Documentation**: <https://opentofu.org/docs/>
264+
- **Cloud-init Documentation**: <https://cloud-init.io/>
265+
- **libvirt Documentation**: <https://libvirt.org/>
266+
- **Repomix Tool**: <https://repomix.com/> (for generating project summaries)

‎.github/workflows/infrastructure.yml‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
name: Infrastructure Tests
2+
3+
on:
4+
push:
5+
branches: [main, develop]
6+
paths:
7+
- "infrastructure/**"
8+
- "tests/infrastructure/**"
9+
- "Makefile"
10+
- ".github/workflows/infrastructure.yml"
11+
pull_request:
12+
branches: [main, develop]
13+
paths:
14+
- "infrastructure/**"
15+
- "tests/infrastructure/**"
16+
- "Makefile"
17+
- ".github/workflows/infrastructure.yml"
18+
19+
jobs:
20+
validate-infrastructure:
21+
runs-on: ubuntu-22.04
22+
23+
steps:
24+
- name: Checkout code
25+
uses: actions/checkout@v4
26+
27+
- name: Install OpenTofu
28+
run: |
29+
curl -fsSL https://get.opentofu.org/install-opentofu.sh -o install-opentofu.sh
30+
chmod +x install-opentofu.sh
31+
sudo ./install-opentofu.sh --install-method deb
32+
rm install-opentofu.sh
33+
tofu version
34+
35+
- name: Install yamllint
36+
run: |
37+
sudo apt-get update
38+
sudo apt-get install -y yamllint
39+
40+
- name: Test script permissions
41+
run: |
42+
test -x tests/infrastructure/test-local-setup.sh
43+
test -x tests/infrastructure/test-integration.sh
44+
45+
- name: Run syntax tests
46+
env:
47+
CI: true
48+
run: |
49+
make ci-test-syntax
50+
51+
- name: Validate Makefile targets
52+
run: |
53+
make help
54+
make workflow-help
55+
56+
# Note: Full VM testing is not run in CI as it requires KVM/nested virtualization
57+
# which is not available in GitHub Actions runners
58+
documentation-check:
59+
runs-on: ubuntu-22.04
60+
61+
steps:
62+
- name: Checkout code
63+
uses: actions/checkout@v4
64+
65+
- name: Check documentation links
66+
run: |
67+
# Verify documentation files exist
68+
test -f docs/infrastructure/quick-start.md
69+
test -f docs/infrastructure/local-testing-setup.md
70+
test -f infrastructure/README.md
71+
72+
# Check for common documentation issues
73+
grep -q "Quick Start" docs/infrastructure/quick-start.md
74+
grep -q "OpenTofu" docs/infrastructure/local-testing-setup.md
75+
grep -q "make" README.md

‎.markdownlint.json‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
{
2+
"default": true,
3+
"MD013": {
4+
"line_length": 80
5+
},
6+
"MD031": true,
7+
"MD032": true,
8+
"MD040": true,
9+
"MD022": true,
10+
"MD009": true,
11+
"MD007": {
12+
"indent": 2
13+
},
14+
"MD026": false,
15+
"MD041": false,
16+
"MD034": false,
17+
"MD024": false,
18+
"MD033": false
19+
}

‎.yamllint-ci.yml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
extends: default
2+
3+
rules:
4+
line-length:
5+
max: 120 # More reasonable for infrastructure code
6+
comments:
7+
min-spaces-from-content: 1 # Allow single space before comments
8+
document-start: disable # Cloud-init files don't need --- start
9+
truthy:
10+
allowed-values: ["true", "false", "yes", "no"] # Allow cloud-init common values

0 commit comments

Comments
 (0)