diff --git a/CS MOC.md b/CS MOC.md index b2ef8d2..bd764b5 100644 --- a/CS MOC.md +++ b/CS MOC.md @@ -45,6 +45,11 @@ Fundamental CS concepts, data structures, algorithms, and system design. - [[Concurrency Patterns]] - [[Distributed Systems]] - [[System Design]] +- [[Systems Language Performance]] — C, C++, Rust, Go, WASM + +### Execution + +- [[WebAssembly]] — WASM, WASI, Component Model ### Networking @@ -59,6 +64,10 @@ Fundamental CS concepts, data structures, algorithms, and system design. - [[Architectural Patterns]] - [[API Design]] +### Software Engineering + +- [[Testing Strategies]] — Unit, integration, E2E, BDD, property-based + ### Reference - [[Technical Measurements]] diff --git a/Tools MOC.md b/Tools MOC.md index 1710278..8e265b7 100644 --- a/Tools MOC.md +++ b/Tools MOC.md @@ -19,14 +19,41 @@ Development tools, libraries, and infrastructure across languages. - [[Logging Libraries]] — Structured logging, observability - [[HTTP Clients]] — REST clients, request handling - [[Debugging Tools]] — Debuggers, profilers, tracers +- [[Version Managers]] — nvm, pyenv, mise, asdf, rustup +- [[Environment Management]] — direnv, dotenv, mise +- [[Remote Development]] — SSH, VS Code Remote, Codespaces ### Build & Deploy - [[Build Systems]] — Compilers, bundlers, task runners - [[Cross-Compilation]] — Building for multiple targets -- [[Runtimes]] — Execution models, VMs, interpreters - [[Deployment]] — Containers, serverless, platforms +### Shells & CLI + +- [[Shells]] — Bash vs Zsh vs Fish vs PowerShell +- [[Terminal Emulators]] — Ghostty, Alacritty, Kitty, Warp, iTerm2 +- [[Terminal Multiplexers]] — tmux vs screen vs Zellij +- [[WSL]] — Windows Subsystem for Linux + +### Operating Systems + +- [[Linux Distributions]] — Debian, Fedora, Arch, Alpine, NixOS +- [[Package Managers]] — apt, dnf, pacman, brew, winget +- [[Process Managers]] — systemd, launchd, supervisord, PM2 + +### Runtimes + +- [[JavaScript Runtimes]] — Node.js vs Deno vs Bun +- [[Language Runtimes]] — Python, Ruby, JVM, .NET, PHP runtimes +- [[WebAssembly Runtimes]] — Wasmtime, Wasmer, WasmEdge + +### Containers + +- [[Container Runtimes]] — Docker, Podman, Colima, Rancher Desktop +- [[Container Tools]] — Portainer, Traefik, Watchtower, registries +- [[Kubernetes]] — K8s, K3s, kind, minikube, Helm + ### Infrastructure & Security - [[API Gateways]] — Traffic routing, rate limiting, auth @@ -43,10 +70,11 @@ Development tools, libraries, and infrastructure across languages. ### Observability - [[Observability Stack]] — Metrics, logs, traces comparison +- [[Log Aggregation]] — Loki, Elasticsearch, Splunk, Fluentd +- [[Distributed Tracing]] — Jaeger, Zipkin, Tempo - [[OpenTelemetry]] — Unified telemetry collection - [[Prometheus]] — Metrics monitoring - [[Grafana]] — Visualization and dashboards -- [[Distributed Tracing]] — Request flow tracking ### UI diff --git a/cs/Systems Language Performance.md b/cs/Systems Language Performance.md new file mode 100644 index 0000000..6bc8835 --- /dev/null +++ b/cs/Systems Language Performance.md @@ -0,0 +1,404 @@ +--- +title: Systems Language Performance +aliases: + - Language Performance + - C vs Rust vs Go + - Native Performance +tags: + - concept + - performance + - systems + - comparison +type: comparison +status: complete +created: 2025-12-04 +--- + +# Systems Language Performance + +Performance characteristics of C, C++, Rust, Go, and WebAssembly. + +## Overview + +| Aspect | C | C++ | Rust | Go | WASM | +|--------|---|-----|------|-----|------| +| Memory management | Manual | Manual/RAII | Ownership | GC | Linear memory | +| Runtime | None | Minimal | None | GC + scheduler | VM/JIT | +| Zero-cost abstractions | ✅ | ✅ | ✅ | ❌ | ✅ | +| Compile time | Fast | Slow | Slow | Fast | Varies | +| Binary size | Tiny | Medium | Medium | Large | Small | +| Startup time | Instant | Instant | Instant | Fast | Fast-Instant | + +--- + +## Performance Model + +### C + +The baseline. Direct hardware mapping. + +``` +Source → Compiler → Machine Code → CPU + +┌─────────────────────────────────────────┐ +│ Your Code │ +├─────────────────────────────────────────┤ +│ C Runtime │ +│ (minimal: crt0, libc) │ +├─────────────────────────────────────────┤ +│ Operating System │ +├─────────────────────────────────────────┤ +│ Hardware │ +└─────────────────────────────────────────┘ +``` + +**Characteristics:** +- No hidden costs +- Predictable memory layout +- Direct syscalls possible +- Undefined behavior risks + +### C++ + +C with abstractions. Zero-cost when done right. + +``` +Source → Compiler → Machine Code → CPU + │ + ├── Templates (compile-time) + ├── RAII (no runtime cost) + └── Virtual dispatch (vtable indirection) +``` + +**Characteristics:** +- Zero-cost abstractions (templates, RAII, constexpr) +- Costs when using: virtual, RTTI, exceptions +- More optimization opportunities than C +- Complexity can hide performance issues + +### Rust + +Zero-cost abstractions with safety guarantees. + +``` +Source → Compiler → LLVM IR → Machine Code → CPU + │ + ├── Borrow checker (compile-time only) + ├── Monomorphization (like C++ templates) + └── No GC, no runtime +``` + +**Characteristics:** +- Same performance ceiling as C/C++ +- Safety checks at compile time, not runtime +- Predictable: no hidden allocations +- LLVM backend (same optimizations as Clang) + +### Go + +Simplicity over raw performance. GC + runtime. + +``` +Source → Compiler → Machine Code → CPU + │ + ┌──────────┴──────────┐ + │ Go Runtime │ + ├─────────────────────┤ + │ Goroutine scheduler │ + │ Garbage collector │ + │ Stack management │ + └─────────────────────┘ +``` + +**Characteristics:** +- GC pauses (sub-millisecond, but present) +- Goroutine overhead (~2KB initial stack) +- Escape analysis reduces allocations +- No generics until 1.18 (some runtime costs before) + +### WebAssembly + +Portable bytecode. Performance depends on runtime. + +``` +Source → Compiler → .wasm → Runtime → (JIT/AOT) → CPU + │ + ┌─────────┴─────────┐ + │ WASM Runtime │ + ├───────────────────┤ + │ Linear memory │ + │ Bounds checking │ + │ Sandboxing │ + └───────────────────┘ +``` + +**Characteristics:** +- ~0.8-1.0x native speed (JIT/AOT) +- Mandatory bounds checking +- No direct syscalls (WASI abstraction) +- Portable but not free + +--- + +## Memory Management + +| Approach | Languages | Overhead | Predictability | +|----------|-----------|----------|----------------| +| Manual | C | None | High (if correct) | +| RAII | C++, Rust | None | High | +| Ownership | Rust | None | High | +| Ref counting | Swift, Rust (Arc) | Counter updates | Medium | +| Tracing GC | Go, Java, C# | Pause + scan | Lower | + +### Memory Layout Control + +``` +C/C++/Rust: +┌─────────────────────────────────────┐ +│ struct Point { float x, y, z; } │ +│ Memory: [x][y][z] (12 bytes, packed)│ +└─────────────────────────────────────┘ + +Go: +┌─────────────────────────────────────┐ +│ type Point struct { X, Y, Z float32 }│ +│ Memory: [X][Y][Z] (12 bytes) │ +│ But slices: pointer + len + cap │ +└─────────────────────────────────────┘ +``` + +### Allocation Patterns + +| Pattern | C/C++ | Rust | Go | +|---------|-------|------|-----| +| Stack allocation | Default | Default | Escape analysis | +| Heap allocation | malloc/new | Box, Vec | make, new, literals | +| Arena/pool | Manual | bumpalo | sync.Pool | +| Zero-copy | Possible | Possible | Harder (GC moves) | + +--- + +## Runtime Overhead + +### Startup Time + +| Language | Cold Start | Notes | +|----------|------------|-------| +| C | ~1ms | Minimal init | +| C++ | ~1-5ms | Static constructors | +| Rust | ~1ms | No runtime init | +| Go | ~5-10ms | Runtime, GC init | +| WASM (JIT) | ~5-50ms | Parse + compile | +| WASM (AOT) | ~1-5ms | Pre-compiled | + +### Memory Footprint + +| Language | Minimal Binary | Hello World | Runtime Overhead | +|----------|----------------|-------------|------------------| +| C | ~10KB | ~20KB | libc only | +| C++ | ~50KB | ~100KB+ | libstdc++ | +| Rust | ~200KB | ~300KB | std library | +| Go | ~1.2MB | ~1.8MB | Runtime + GC | +| WASM | ~10KB | ~50KB | + runtime memory | + +*Static linking increases binary size but reduces dependencies* + +--- + +## Benchmark Patterns + +### Compute-Bound + +Raw number crunching, SIMD-friendly. + +| Task | C | C++ | Rust | Go | WASM | +|------|:-:|:---:|:----:|:--:|:----:| +| Matrix multiply | 1.0x | 1.0x | 1.0x | 1.3-1.5x | 1.0-1.2x | +| Image processing | 1.0x | 1.0x | 1.0x | 1.2-1.5x | 1.0-1.3x | +| Crypto | 1.0x | 1.0x | 1.0x | 1.1-1.3x | 1.0-1.2x | + +*SIMD support: C/C++/Rust (intrinsics), Go (limited), WASM (SIMD proposal)* + +### Memory-Bound + +Cache efficiency, allocation patterns. + +| Task | C | C++ | Rust | Go | +|------|:-:|:---:|:----:|:--:| +| Large array traversal | 1.0x | 1.0x | 1.0x | 1.0-1.1x | +| Pointer chasing | 1.0x | 1.0x | 1.0x | 1.0x | +| Many small allocations | 1.0x | 1.0-1.2x | 1.0-1.1x | 1.5-3.0x | + +*Go's GC adds overhead for allocation-heavy workloads* + +### I/O-Bound + +Syscalls, async, concurrency. + +| Task | C | C++ | Rust | Go | +|------|:-:|:---:|:----:|:--:| +| File I/O | 1.0x | 1.0x | 1.0x | 1.0-1.1x | +| Network (sync) | 1.0x | 1.0x | 1.0x | 1.0x | +| Network (async) | Complex | Complex | 1.0x (tokio) | 1.0x (goroutines) | +| 10K connections | Hard | Hard | Easy | Easy | + +*Go and Rust async excel at concurrent I/O* + +--- + +## Latency vs Throughput + +### GC Impact (Go) + +``` +Go GC characteristics: +├── Sub-millisecond pauses (typically <500μs) +├── Concurrent marking +├── Write barriers during GC +└── GOGC controls frequency vs memory + +Tuning: +GOGC=100 (default) — GC when heap doubles +GOGC=200 — Less frequent, more memory +GOGC=off — Disable GC (memory grows forever) +``` + +### Predictability Spectrum + +``` +Most predictable ◄──────────────────────► Least predictable + + C Rust C++ Go + │ │ │ │ + │ │ │ └── GC pauses + │ │ └── Exceptions, virtual + │ └── Safe, but deterministic + └── Manual, undefined behavior risk +``` + +--- + +## Concurrency Performance + +### Threading Models + +| Language | Model | Overhead per Thread/Task | +|----------|-------|-------------------------| +| C | pthreads | ~8MB stack (configurable) | +| C++ | std::thread | ~8MB stack | +| Rust | std::thread | ~8MB stack | +| Rust | tokio task | ~few KB | +| Go | goroutine | ~2KB initial, grows | + +### Scaling + +``` +10,000 concurrent connections: + +C/C++ (threads): 10,000 × 8MB = 80GB (impossible) +C/C++ (epoll/io_uring): Works, complex code +Rust (tokio): Works, ~20-50MB +Go (goroutines): Works, ~20-50MB +``` + +--- + +## Compiler Optimizations + +### Optimization Comparison + +| Optimization | C (GCC/Clang) | C++ | Rust | Go | +|--------------|:-------------:|:---:|:----:|:--:| +| Inlining | ✅ | ✅ | ✅ | ✅ (limited) | +| LTO | ✅ | ✅ | ✅ | ❌ | +| PGO | ✅ | ✅ | ✅ | ✅ | +| Auto-vectorization | ✅ | ✅ | ✅ | Limited | +| Devirtualization | N/A | ✅ | ✅ (traits) | ✅ (interfaces) | +| Escape analysis | N/A | N/A | N/A | ✅ | + +### Build Time vs Runtime + +| Language | -O0 | -O2/-O3 | LTO | PGO | +|----------|-----|---------|-----|-----| +| C | Fast | Medium | Slow | Slowest | +| C++ | Medium | Slow | Very slow | Slowest | +| Rust | Slow | Slower | Very slow | Slowest | +| Go | Fast | Fast | N/A | Slow | + +--- + +## WASM Performance Details + +### Overhead Sources + +| Source | Cost | Mitigation | +|--------|------|------------| +| Bounds checking | 1-5% | Sometimes elided | +| Indirect calls | 2-10% | Speculative | +| Memory model | 0-5% | Linear memory helps | +| 32-bit pointers | Can help | Less memory traffic | +| No SIMD (old) | 2-4x slower | SIMD proposal | + +### WASM vs Native + +``` +Typical performance (compute-bound): + +Native (C/Rust): ████████████████████ 100% +WASM (AOT): ██████████████████ 90% +WASM (JIT): ████████████████ 80% +WASM (interpret): ████████ 40% +``` + +### Best WASM Performance + +1. Use Rust or C/C++ as source +2. AOT compile if possible +3. Minimize host calls +4. Use SIMD when available +5. Batch operations + +--- + +## Decision Guide + +| Priority | Best Choice | +|----------|-------------| +| Maximum performance, control | C or Rust | +| Performance + safety | Rust | +| Performance + simplicity | Go | +| Team knows C++ | C++ | +| Portability + sandbox | WASM (from Rust/C) | +| Fast compilation | Go or C | +| Predictable latency | Rust or C | +| High concurrency | Go or Rust (async) | +| Embedded/minimal | C or Rust (no_std) | + +### Trade-offs Summary + +| Language | Gives Up | Gets | +|----------|----------|------| +| C | Safety | Maximum control | +| C++ | Simplicity | Abstractions + control | +| Rust | Compile time | Safety + control | +| Go | Raw speed | Simplicity + productivity | +| WASM | Some speed | Portability + sandbox | + +--- + +## Benchmarking Resources + +- [Benchmarks Game](https://benchmarksgame-team.pages.debian.net/benchmarksgame/) +- [TechEmpower Web Frameworks](https://www.techempower.com/benchmarks/) +- [Are We Fast Yet](https://arewefastyet.com/) (JS/WASM) + +--- + +## Related + +- [[WebAssembly Runtimes]] +- [[JavaScript Runtimes]] +- [[Big O Notation]] +- [[Rust]] +- [[Go]] +- [[C++]] diff --git a/cs/Testing Strategies.md b/cs/Testing Strategies.md new file mode 100644 index 0000000..3c8385a --- /dev/null +++ b/cs/Testing Strategies.md @@ -0,0 +1,720 @@ +--- +title: Testing Strategies +aliases: + - Testing Types + - Test Pyramid +tags: + - concept + - testing + - quality +type: reference +status: complete +created: 2025-12-04 +--- + +# Testing Strategies + +Types of software testing, when to use each, and how they fit together. + +## Test Pyramid + +``` + ┌───────────┐ + ╱ ╲ Fewer, slower, expensive + ╱ E2E ╲ Full system tests + ╱ ╲ + ├───────────────────┤ + ╱ ╲ + ╱ Integration ╲ Service boundaries + ╱ ╲ + ├───────────────────────────┤ + ╱ ╲ + ╱ Unit ╲ Many, fast, cheap + ╱ ╲ + └───────────────────────────────────┘ +``` + +| Level | Speed | Cost | Scope | Quantity | +|-------|-------|------|-------|----------| +| Unit | Fast (ms) | Cheap | Single function/class | Many | +| Integration | Medium (s) | Medium | Multiple components | Some | +| E2E | Slow (min) | Expensive | Entire system | Few | + +--- + +## Unit Tests + +Test individual functions, methods, or classes in isolation. + +### Characteristics + +- **Fast** — Milliseconds per test +- **Isolated** — No external dependencies +- **Focused** — One behavior per test +- **Deterministic** — Same result every time + +### Example + +```javascript +// function +function calculateTotal(items, taxRate) { + const subtotal = items.reduce((sum, item) => sum + item.price, 0); + return subtotal * (1 + taxRate); +} + +// unit test +describe('calculateTotal', () => { + it('calculates total with tax', () => { + const items = [{ price: 10 }, { price: 20 }]; + expect(calculateTotal(items, 0.1)).toBe(33); + }); + + it('returns 0 for empty cart', () => { + expect(calculateTotal([], 0.1)).toBe(0); + }); + + it('handles zero tax rate', () => { + const items = [{ price: 100 }]; + expect(calculateTotal(items, 0)).toBe(100); + }); +}); +``` + +```python +# Python with pytest +def calculate_total(items, tax_rate): + subtotal = sum(item['price'] for item in items) + return subtotal * (1 + tax_rate) + +def test_calculate_total_with_tax(): + items = [{'price': 10}, {'price': 20}] + assert calculate_total(items, 0.1) == 33 + +def test_calculate_total_empty_cart(): + assert calculate_total([], 0.1) == 0 +``` + +### When to Use + +- Pure functions +- Business logic +- Data transformations +- Utility functions +- Algorithm verification + +### When Not to Use + +- Database queries (use integration) +- API calls (use integration) +- UI interactions (use E2E) + +--- + +## Integration Tests + +Test how multiple components work together. + +### Characteristics + +- **Broader scope** — Multiple units together +- **Real dependencies** — Database, APIs, queues +- **Slower** — Seconds per test +- **Test boundaries** — Service interfaces + +### Types + +| Type | Tests | +|------|-------| +| Database integration | Queries, transactions | +| API integration | HTTP endpoints | +| Service integration | Microservice communication | +| Message queue | Pub/sub, async processing | + +### Example + +```javascript +// API integration test +describe('POST /users', () => { + beforeEach(async () => { + await db.clear('users'); + }); + + it('creates a user and stores in database', async () => { + const response = await request(app) + .post('/users') + .send({ name: 'Alice', email: 'alice@example.com' }); + + expect(response.status).toBe(201); + expect(response.body.id).toBeDefined(); + + // Verify in database + const user = await db.users.findById(response.body.id); + expect(user.name).toBe('Alice'); + }); + + it('returns 400 for invalid email', async () => { + const response = await request(app) + .post('/users') + .send({ name: 'Alice', email: 'invalid' }); + + expect(response.status).toBe(400); + }); +}); +``` + +```python +# Python with pytest and test database +import pytest +from app import create_app, db + +@pytest.fixture +def client(): + app = create_app({'TESTING': True, 'DATABASE_URL': 'sqlite:///:memory:'}) + with app.test_client() as client: + with app.app_context(): + db.create_all() + yield client + +def test_create_user(client): + response = client.post('/users', json={ + 'name': 'Alice', + 'email': 'alice@example.com' + }) + + assert response.status_code == 201 + assert response.json['id'] is not None + + # Verify in database + user = User.query.get(response.json['id']) + assert user.name == 'Alice' +``` + +### Test Containers + +Use real services in Docker containers. + +```java +// Java with Testcontainers +@Testcontainers +class UserRepositoryTest { + @Container + static PostgreSQLContainer postgres = new PostgreSQLContainer<>("postgres:15"); + + @Test + void shouldSaveUser() { + UserRepository repo = new UserRepository(postgres.getJdbcUrl()); + User user = repo.save(new User("Alice")); + + assertThat(user.getId()).isNotNull(); + assertThat(repo.findById(user.getId())).isPresent(); + } +} +``` + +--- + +## End-to-End (E2E) Tests + +Test the entire application from user's perspective. + +### Characteristics + +- **Full stack** — Frontend → Backend → Database +- **User flows** — Complete scenarios +- **Slowest** — Minutes per test +- **Brittle** — Many failure points + +### Tools + +| Tool | Language | Best For | +|------|----------|----------| +| Playwright | JS/TS, Python | Modern, cross-browser | +| Cypress | JS/TS | Developer experience | +| Selenium | Many | Legacy, wide support | +| Puppeteer | JS/TS | Chrome/Chromium | + +### Example (Playwright) + +```typescript +import { test, expect } from '@playwright/test'; + +test.describe('User Registration', () => { + test('user can sign up and log in', async ({ page }) => { + // Sign up + await page.goto('/signup'); + await page.fill('[name="email"]', 'alice@example.com'); + await page.fill('[name="password"]', 'securepassword'); + await page.click('button[type="submit"]'); + + // Verify redirect to dashboard + await expect(page).toHaveURL('/dashboard'); + await expect(page.locator('h1')).toContainText('Welcome, Alice'); + + // Log out + await page.click('button:text("Logout")'); + await expect(page).toHaveURL('/login'); + + // Log back in + await page.fill('[name="email"]', 'alice@example.com'); + await page.fill('[name="password"]', 'securepassword'); + await page.click('button[type="submit"]'); + await expect(page).toHaveURL('/dashboard'); + }); +}); +``` + +### Example (Cypress) + +```javascript +describe('Shopping Cart', () => { + beforeEach(() => { + cy.login('testuser@example.com', 'password'); + }); + + it('adds item to cart and checks out', () => { + cy.visit('/products'); + cy.contains('Widget').click(); + cy.get('[data-testid="add-to-cart"]').click(); + + cy.get('[data-testid="cart-count"]').should('contain', '1'); + + cy.visit('/cart'); + cy.contains('Widget').should('be.visible'); + + cy.get('[data-testid="checkout"]').click(); + cy.get('[name="card-number"]').type('4242424242424242'); + cy.get('[name="expiry"]').type('12/25'); + cy.get('[name="cvc"]').type('123'); + cy.get('button[type="submit"]').click(); + + cy.url().should('include', '/order-confirmation'); + cy.contains('Thank you for your order').should('be.visible'); + }); +}); +``` + +--- + +## Behavior-Driven Development (BDD) + +Write tests in natural language using Given-When-Then. + +### Gherkin Syntax + +```gherkin +# features/login.feature +Feature: User Login + As a user + I want to log in to my account + So that I can access my dashboard + + Scenario: Successful login with valid credentials + Given I am on the login page + And I have a registered account with email "alice@example.com" + When I enter my email "alice@example.com" + And I enter my password "correctpassword" + And I click the login button + Then I should be redirected to the dashboard + And I should see "Welcome, Alice" + + Scenario: Failed login with wrong password + Given I am on the login page + When I enter my email "alice@example.com" + And I enter my password "wrongpassword" + And I click the login button + Then I should see an error message "Invalid credentials" + And I should remain on the login page +``` + +### Step Definitions (Cucumber.js) + +```javascript +const { Given, When, Then } = require('@cucumber/cucumber'); +const { expect } = require('@playwright/test'); + +Given('I am on the login page', async function() { + await this.page.goto('/login'); +}); + +When('I enter my email {string}', async function(email) { + await this.page.fill('[name="email"]', email); +}); + +When('I enter my password {string}', async function(password) { + await this.page.fill('[name="password"]', password); +}); + +When('I click the login button', async function() { + await this.page.click('button[type="submit"]'); +}); + +Then('I should be redirected to the dashboard', async function() { + await expect(this.page).toHaveURL('/dashboard'); +}); + +Then('I should see {string}', async function(text) { + await expect(this.page.locator('body')).toContainText(text); +}); +``` + +### BDD Frameworks + +| Language | Framework | +|----------|-----------| +| JavaScript | Cucumber.js | +| Python | Behave, pytest-bdd | +| Java | Cucumber-JVM | +| Ruby | Cucumber | +| .NET | SpecFlow | + +--- + +## Contract Tests + +Verify API contracts between services. + +### Consumer-Driven Contracts + +``` +┌──────────────┐ ┌──────────────┐ +│ Consumer │────────▶│ Provider │ +│ (Client) │ │ (Server) │ +└──────────────┘ └──────────────┘ + │ │ + ▼ ▼ + Consumer Test Provider Test + "I expect this" "I provide this" + │ │ + └────────┬───────────────┘ + │ + ┌────▼────┐ + │ Contract │ + │ Broker │ + └──────────┘ +``` + +### Example (Pact) + +```javascript +// Consumer test +const { Pact } = require('@pact-foundation/pact'); + +describe('User API', () => { + const provider = new Pact({ + consumer: 'Frontend', + provider: 'UserService', + }); + + beforeAll(() => provider.setup()); + afterAll(() => provider.finalize()); + + it('gets user by id', async () => { + await provider.addInteraction({ + state: 'user 123 exists', + uponReceiving: 'a request for user 123', + withRequest: { + method: 'GET', + path: '/users/123', + }, + willRespondWith: { + status: 200, + headers: { 'Content-Type': 'application/json' }, + body: { + id: 123, + name: 'Alice', + email: 'alice@example.com', + }, + }, + }); + + const response = await fetch(`${provider.mockService.baseUrl}/users/123`); + const user = await response.json(); + + expect(user.name).toBe('Alice'); + }); +}); +``` + +--- + +## Snapshot Tests + +Compare output against saved "snapshot." + +### UI Snapshots + +```javascript +// React component snapshot +import { render } from '@testing-library/react'; +import UserCard from './UserCard'; + +test('UserCard matches snapshot', () => { + const { container } = render( + + ); + expect(container).toMatchSnapshot(); +}); +``` + +### API Response Snapshots + +```javascript +test('GET /users returns expected structure', async () => { + const response = await request(app).get('/users'); + expect(response.body).toMatchSnapshot(); +}); +``` + +### When to Use + +- UI components +- API responses +- Generated code/config +- Serialized data + +### When Not to Use + +- Frequently changing data +- Timestamps, random IDs +- Large outputs + +--- + +## Property-Based Tests + +Test with randomly generated inputs. + +### Example (fast-check) + +```javascript +import fc from 'fast-check'; + +// Test properties that should always hold +describe('sort function', () => { + it('should return same length', () => { + fc.assert( + fc.property(fc.array(fc.integer()), (arr) => { + return sort(arr).length === arr.length; + }) + ); + }); + + it('should return sorted array', () => { + fc.assert( + fc.property(fc.array(fc.integer()), (arr) => { + const sorted = sort(arr); + for (let i = 1; i < sorted.length; i++) { + if (sorted[i] < sorted[i - 1]) return false; + } + return true; + }) + ); + }); + + it('should contain same elements', () => { + fc.assert( + fc.property(fc.array(fc.integer()), (arr) => { + const sorted = sort(arr); + return arr.every(x => sorted.includes(x)) && + sorted.every(x => arr.includes(x)); + }) + ); + }); +}); +``` + +### Frameworks + +| Language | Framework | +|----------|-----------| +| JavaScript | fast-check | +| Python | Hypothesis | +| Scala | ScalaCheck | +| Haskell | QuickCheck | +| Rust | proptest | + +--- + +## Mutation Tests + +Verify test quality by introducing bugs. + +``` +Original Code Mutant (bug introduced) +───────────── ───────────────────── +if (x > 0) if (x >= 0) // Changed > to >= +return x + y return x - y // Changed + to - +``` + +If tests still pass → **weak tests** (mutant survived) +If tests fail → **good tests** (mutant killed) + +### Tools + +| Language | Tool | +|----------|------| +| JavaScript | Stryker | +| Python | mutmut | +| Java | PITest | +| Ruby | mutant | + +--- + +## Performance Tests + +### Load Testing + +```javascript +// k6 load test +import http from 'k6/http'; +import { check, sleep } from 'k6'; + +export const options = { + vus: 100, // Virtual users + duration: '5m', // Test duration + thresholds: { + http_req_duration: ['p(95)<500'], // 95% under 500ms + }, +}; + +export default function() { + const res = http.get('http://api.example.com/users'); + + check(res, { + 'status is 200': (r) => r.status === 200, + 'response time < 500ms': (r) => r.timings.duration < 500, + }); + + sleep(1); +} +``` + +### Tools + +| Tool | Type | +|------|------| +| k6 | Load testing | +| Artillery | Load testing | +| Locust | Load testing (Python) | +| Apache Benchmark | Simple load | +| wrk | HTTP benchmarking | + +--- + +## Test Organization + +### Naming Conventions + +``` +Pattern: should_ExpectedBehavior_When_Condition + +Examples: +- should_ReturnNull_When_UserNotFound +- should_ThrowException_When_InvalidInput +- should_SendEmail_When_OrderConfirmed +``` + +### Arrange-Act-Assert (AAA) + +```javascript +test('calculates order total correctly', () => { + // Arrange + const items = [ + { name: 'Widget', price: 25.00, quantity: 2 }, + { name: 'Gadget', price: 15.00, quantity: 1 }, + ]; + const order = new Order(items); + + // Act + const total = order.calculateTotal(); + + // Assert + expect(total).toBe(65.00); +}); +``` + +### Test Fixtures + +```python +# pytest fixtures +@pytest.fixture +def database(): + db = create_test_database() + db.seed_test_data() + yield db + db.cleanup() + +@pytest.fixture +def authenticated_user(database): + user = database.create_user(email='test@example.com') + return user + +def test_user_can_create_post(authenticated_user, database): + post = database.create_post(author=authenticated_user, title='Test') + assert post.id is not None +``` + +--- + +## Test Coverage + +### Types of Coverage + +| Type | Measures | +|------|----------| +| Line | Lines executed | +| Branch | Decision branches taken | +| Function | Functions called | +| Statement | Statements executed | + +### Coverage Goals + +| Level | Recommendation | +|-------|----------------| +| <50% | Insufficient | +| 50-70% | Minimum viable | +| 70-80% | Good | +| 80-90% | Very good | +| >90% | Excellent (but diminishing returns) | + +### Don't Aim for 100% + +```javascript +// This code doesn't need testing +if (process.env.NODE_ENV === 'development') { + console.log('Debug mode'); +} +``` + +--- + +## Decision Guide + +| Goal | Test Type | +|------|-----------| +| Function correctness | Unit | +| Component integration | Integration | +| User workflows | E2E | +| Stakeholder communication | BDD | +| API compatibility | Contract | +| Regression prevention | Snapshot | +| Edge cases | Property-based | +| Test quality | Mutation | +| Performance | Load | + +### Test Ratio (Suggested) + +``` +Typical ratio: +├── Unit tests: 70% +├── Integration tests: 20% +└── E2E tests: 10% +``` + +--- + +## Related + +- [[Testing Frameworks]] +- [[Debugging Tools]] +- [[Deployment]] diff --git a/cs/WebAssembly.md b/cs/WebAssembly.md new file mode 100644 index 0000000..00ef95c --- /dev/null +++ b/cs/WebAssembly.md @@ -0,0 +1,682 @@ +--- +title: WebAssembly +aliases: + - WASM + - Wasm +tags: + - concept + - wasm + - runtime + - portable +type: reference +status: complete +created: 2025-12-04 +--- + +# WebAssembly + +Portable binary instruction format for a stack-based virtual machine. + +## Overview + +| Aspect | Details | +|--------|---------| +| Type | Binary instruction format | +| Execution | Stack machine | +| Memory | Linear (flat array) | +| Created | 2015 (W3C standard 2019) | +| Use cases | Browser, server, edge, embedded, plugins | + +--- + +## What is WASM? + +``` +┌─────────────────────────────────────────────────────────┐ +│ Source Languages │ +│ Rust, C, C++, Go, Zig, AssemblyScript │ +└──────────────────────┬──────────────────────────────────┘ + │ Compile + ▼ + ┌─────────────────┐ + │ .wasm binary │ ← Portable bytecode + └────────┬────────┘ + │ Execute + ▼ +┌─────────────────────────────────────────────────────────┐ +│ Runtimes │ +│ Browser │ Wasmtime │ Wasmer │ WasmEdge │ Node/Deno │ +└─────────────────────────────────────────────────────────┘ +``` + +### Key Properties + +| Property | Description | +|----------|-------------| +| **Portable** | Same binary runs on any conforming runtime | +| **Fast** | Near-native speed (JIT/AOT compiled) | +| **Safe** | Sandboxed, memory-safe by default | +| **Compact** | Binary format smaller than text | +| **Language-agnostic** | Compile from many source languages | +| **Deterministic** | Same inputs → same outputs | + +--- + +## Binary Format + +### Module Structure + +``` +WASM Module (.wasm): +┌─────────────────────────────────────────┐ +│ Magic number: 0x00 0x61 0x73 0x6D │ ← "\0asm" +│ Version: 0x01 0x00 0x00 0x00 │ ← version 1 +├─────────────────────────────────────────┤ +│ Section: Types (function signatures)│ +│ Section: Imports (from host/other) │ +│ Section: Functions (indices to types) │ +│ Section: Tables (indirect calls) │ +│ Section: Memory (linear memory) │ +│ Section: Globals (mutable/const) │ +│ Section: Exports (exposed to host) │ +│ Section: Start (initialization) │ +│ Section: Elements (table init) │ +│ Section: Code (function bodies) │ +│ Section: Data (memory init) │ +└─────────────────────────────────────────┘ +``` + +### Text Format (WAT) + +```wasm +;; add.wat - Human-readable WebAssembly text format +(module + ;; Function that adds two i32 values + (func $add (param $a i32) (param $b i32) (result i32) + local.get $a + local.get $b + i32.add + ) + + ;; Export the function + (export "add" (func $add)) +) +``` + +```bash +# Convert WAT to WASM +wat2wasm add.wat -o add.wasm + +# Convert WASM to WAT +wasm2wat add.wasm -o add.wat +``` + +### Value Types + +| Type | Description | +|------|-------------| +| `i32` | 32-bit integer | +| `i64` | 64-bit integer | +| `f32` | 32-bit float | +| `f64` | 64-bit float | +| `v128` | 128-bit SIMD vector | +| `funcref` | Function reference | +| `externref` | External (host) reference | + +--- + +## Memory Model + +### Linear Memory + +``` +WASM Linear Memory: +┌─────────────────────────────────────────────────────────┐ +│ 0x0000 │ 0x0004 │ 0x0008 │ ... │ 0xFFFF │ │ +├─────────┴─────────┴─────────┴─────┴─────────┴───────────┤ +│ Flat byte array │ +│ (grows in 64KB pages) │ +└─────────────────────────────────────────────────────────┘ + ▲ ▲ + │ │ + Heap start Current size +``` + +### Memory Operations + +```wasm +;; Load 32-bit int from address +(i32.load (i32.const 0)) + +;; Store 32-bit int at address +(i32.store (i32.const 0) (i32.const 42)) + +;; Grow memory by 1 page (64KB) +(memory.grow (i32.const 1)) + +;; Get current memory size in pages +(memory.size) +``` + +### Bounds Checking + +``` +Every memory access is bounds-checked: + +load(address) { + if (address + size > memory.length) { + trap("out of bounds memory access") + } + return memory[address:address+size] +} +``` + +--- + +## Execution Model + +### Stack Machine + +``` +Expression: (i32.add (i32.const 1) (i32.const 2)) + +Stack evolution: +┌───┐ ┌───┐ ┌───┐ ┌───┐ +│ │ │ 1 │ │ 2 │ │ 3 │ +│ │ → │ │ → │ 1 │ → │ │ +│ │ │ │ │ │ │ │ +└───┘ └───┘ └───┘ └───┘ + init push 1 push 2 add → pop 2, push result +``` + +### Control Flow + +```wasm +;; Blocks and branches +(block $outer + (block $inner + (br_if $outer (i32.eq (local.get $x) (i32.const 0))) + ;; code if x != 0 + ) + ;; code after inner block +) + +;; Loops +(loop $continue + ;; loop body + (br_if $continue (local.get $condition)) +) + +;; If/else +(if (result i32) (local.get $cond) + (then (i32.const 1)) + (else (i32.const 0)) +) +``` + +### Traps + +Execution stops on: +- Out of bounds memory access +- Divide by zero +- Invalid indirect call +- Stack overflow +- Unreachable instruction + +--- + +## WASI + +WebAssembly System Interface — syscalls for WASM outside browsers. + +### Why WASI? + +``` +Problem: WASM has no built-in I/O + +Browser: WASM ← → JavaScript ← → Web APIs + (import/export) + +Server (WASI): WASM ← → WASI ← → Operating System + (standardized interface) +``` + +### WASI Preview 1 + +Original WASI. POSIX-like syscalls. + +```wasm +;; WASI imports +(import "wasi_snapshot_preview1" "fd_write" + (func $fd_write (param i32 i32 i32 i32) (result i32))) + +(import "wasi_snapshot_preview1" "fd_read" + (func $fd_read (param i32 i32 i32 i32) (result i32))) + +(import "wasi_snapshot_preview1" "path_open" + (func $path_open ...)) +``` + +### Capabilities + +```bash +# WASI is capability-based (explicit permissions) + +# No permissions (sandboxed) +wasmtime app.wasm + +# Grant file access +wasmtime --dir=./data app.wasm + +# Grant specific paths +wasmtime --dir=/input::/app/input --dir=/output::/app/output app.wasm + +# Environment variables +wasmtime --env KEY=value app.wasm + +# Network (preview2) +wasmtime --wasi tcp app.wasm +``` + +### WASI Preview 2 + +New version based on Component Model. + +| Preview 1 | Preview 2 | +|-----------|-----------| +| POSIX-like | Higher-level | +| Functions | Interface types | +| Linear memory passing | Rich types | +| Single module | Components | + +--- + +## Component Model + +The future of WASM composition and interop. + +### Core Concepts + +``` +Component Model: +┌─────────────────────────────────────────────────────────┐ +│ Component │ +│ ┌─────────────────────────────────────────────────┐ │ +│ │ Core Module │ │ +│ │ (.wasm) │ │ +│ └─────────────────────────────────────────────────┘ │ +│ │ │ +│ ┌──────────┴──────────┐ │ +│ ▼ ▼ │ +│ ┌─────────────────┐ ┌─────────────────┐ │ +│ │ WIT Imports │ │ WIT Exports │ │ +│ │ (interfaces) │ │ (interfaces) │ │ +│ └─────────────────┘ └─────────────────┘ │ +└─────────────────────────────────────────────────────────┘ +``` + +### WIT (WASM Interface Types) + +```wit +// http.wit - Define an HTTP interface + +package example:http; + +interface types { + record request { + method: string, + uri: string, + headers: list>, + body: option>, + } + + record response { + status: u16, + headers: list>, + body: option>, + } +} + +interface handler { + use types.{request, response}; + + handle: func(req: request) -> response; +} + +world http-handler { + export handler; +} +``` + +### Component Composition + +``` +┌─────────────────────────────────────────────────────────┐ +│ Composed Component │ +│ │ +│ ┌──────────────┐ ┌──────────────┐ │ +│ │ Component A │───────▶│ Component B │ │ +│ │ (logging) │ │ (business) │ │ +│ └──────────────┘ └──────┬───────┘ │ +│ │ │ +│ ┌──────┴───────┐ │ +│ │ Component C │ │ +│ │ (database) │ │ +│ └──────────────┘ │ +└─────────────────────────────────────────────────────────┘ +``` + +```bash +# Compose components +wasm-tools compose -o composed.wasm \ + --config config.yaml \ + main.wasm +``` + +--- + +## Compiling to WASM + +### Rust + +```bash +# Add target +rustup target add wasm32-wasip1 # WASI +rustup target add wasm32-unknown-unknown # Browser + +# Build for WASI +cargo build --target wasm32-wasip1 --release + +# Build for browser +cargo build --target wasm32-unknown-unknown --release + +# With wasm-bindgen (browser interop) +wasm-pack build --target web +``` + +```rust +// src/lib.rs - Browser WASM with wasm-bindgen +use wasm_bindgen::prelude::*; + +#[wasm_bindgen] +pub fn greet(name: &str) -> String { + format!("Hello, {}!", name) +} +``` + +### C/C++ + +```bash +# With Emscripten (browser-focused) +emcc main.c -o main.js -s WASM=1 + +# With WASI SDK (standalone) +$WASI_SDK/bin/clang main.c -o main.wasm + +# Optimization +emcc -O3 -s WASM=1 main.c -o main.wasm +``` + +### Go + +```bash +# Standard Go (larger output) +GOOS=wasip1 GOARCH=wasm go build -o main.wasm + +# TinyGo (smaller, better for WASM) +tinygo build -o main.wasm -target=wasi main.go + +# For browser +GOOS=js GOARCH=wasm go build -o main.wasm +``` + +### Zig + +```bash +# Build for WASI +zig build-exe main.zig -target wasm32-wasi + +# Optimize +zig build-exe main.zig -target wasm32-wasi -O ReleaseFast +``` + +### Other Languages + +| Language | Tool | Notes | +|----------|------|-------| +| AssemblyScript | `asc` | TypeScript-like, WASM-first | +| Kotlin | Kotlin/Wasm | Experimental | +| Swift | SwiftWasm | Community toolchain | +| C# | Blazor, NativeAOT | Browser or WASI | +| Python | Pyodide, py2wasm | Interpreter or compile | +| Ruby | ruby.wasm | Interpreter in WASM | + +--- + +## Toolchain + +### Core Tools + +| Tool | Purpose | +|------|---------| +| `wasm-tools` | Parse, validate, compose | +| `wasm-opt` | Optimize WASM binaries | +| `wasm2wat` | Binary → text | +| `wat2wasm` | Text → binary | +| `wit-bindgen` | Generate bindings from WIT | +| `wasm-pack` | Rust → npm package | + +### wasm-tools + +```bash +# Install +cargo install wasm-tools + +# Validate +wasm-tools validate app.wasm + +# Print info +wasm-tools print app.wasm + +# Parse to JSON +wasm-tools dump app.wasm + +# Component operations +wasm-tools component new core.wasm -o component.wasm +wasm-tools component wit component.wasm +``` + +### wasm-opt (Binaryen) + +```bash +# Install (via binaryen) +# macOS: brew install binaryen +# Linux: apt install binaryen + +# Optimize +wasm-opt -O3 input.wasm -o output.wasm + +# Aggressive size optimization +wasm-opt -Oz input.wasm -o output.wasm + +# Strip debug info +wasm-opt --strip-debug input.wasm -o output.wasm +``` + +--- + +## Browser Usage + +### Loading WASM + +```javascript +// Modern async approach +const response = await fetch('app.wasm'); +const { instance } = await WebAssembly.instantiateStreaming(response); + +// Call exported function +const result = instance.exports.add(1, 2); +``` + +### With JavaScript Glue + +```javascript +// Import object provides functions to WASM +const imports = { + env: { + log: (ptr, len) => { + const bytes = new Uint8Array(memory.buffer, ptr, len); + console.log(new TextDecoder().decode(bytes)); + }, + memory: new WebAssembly.Memory({ initial: 1 }) + } +}; + +const { instance } = await WebAssembly.instantiateStreaming( + fetch('app.wasm'), + imports +); +``` + +### With wasm-bindgen (Rust) + +```javascript +import init, { greet } from './pkg/my_wasm.js'; + +await init(); // Load and initialize WASM +console.log(greet("World")); // "Hello, World!" +``` + +--- + +## Use Cases + +### Plugin Systems + +``` +Host Application +┌─────────────────────────────────────────────┐ +│ ┌─────────────┐ ┌─────────────┐ │ +│ │ Plugin A │ │ Plugin B │ ... │ +│ │ (.wasm) │ │ (.wasm) │ │ +│ └─────────────┘ └─────────────┘ │ +│ │ │ │ +│ └────────┬───────┘ │ +│ ▼ │ +│ ┌─────────────────┐ │ +│ │ WASM Runtime │ │ +│ │ (sandboxed) │ │ +│ └─────────────────┘ │ +└─────────────────────────────────────────────┘ + +Benefits: +- Language agnostic plugins +- Sandboxed (can't crash host) +- Portable across platforms +``` + +**Examples:** Envoy filters, Figma plugins, VS Code extensions + +### Edge Computing + +``` +┌─────────────┐ ┌─────────────┐ ┌─────────────┐ +│ Cloudflare │ │ Fastly │ │ Fermyon │ +│ Workers │ │ Compute │ │ Spin │ +└──────┬──────┘ └──────┬──────┘ └──────┬──────┘ + │ │ │ + └───────────────────┴───────────────────┘ + │ + ┌──────▼──────┐ + │ Your WASM │ + │ Module │ + └─────────────┘ + +Benefits: +- Cold start < 1ms +- Memory isolation per request +- Same code everywhere +``` + +### Serverless / FaaS + +```rust +// Spin serverless function +use spin_sdk::http::{IntoResponse, Request, Response}; +use spin_sdk::http_component; + +#[http_component] +fn handle(req: Request) -> anyhow::Result { + Ok(Response::builder() + .status(200) + .body("Hello from WASM!") + .build()) +} +``` + +### Embedded / IoT + +- Scripting layer in constrained devices +- OTA-updatable logic +- Hardware-agnostic business logic + +### Browser Applications + +- High-performance web apps +- Porting native apps (Figma, AutoCAD, Photoshop) +- Games (Unity, Unreal) +- Video/audio processing + +--- + +## Performance Tips + +### Minimize Host Calls + +``` +❌ Slow: Many small calls + for item in items: + host.process(item) # Cross-boundary call + +✅ Fast: Batch operations + host.process_all(items) # Single call +``` + +### Use Linear Memory Efficiently + +```rust +// Pre-allocate in WASM, pass pointer to host +let buffer = vec![0u8; 1024]; +let ptr = buffer.as_ptr(); +let len = buffer.len(); +unsafe { host_fill_buffer(ptr, len); } +``` + +### Enable SIMD + +```rust +// Rust with SIMD +#[target_feature(enable = "simd128")] +fn sum_simd(a: &[f32]) -> f32 { + // SIMD implementation +} +``` + +### AOT Compile + +```bash +# Wasmtime AOT +wasmtime compile app.wasm -o app.cwasm + +# Run pre-compiled +wasmtime run app.cwasm +``` + +--- + +## Related + +- [[WebAssembly Runtimes]] +- [[Systems Language Performance]] +- [[JavaScript Runtimes]] +- [[Rust]] diff --git a/tools/Container Runtimes.md b/tools/Container Runtimes.md new file mode 100644 index 0000000..f265cb0 --- /dev/null +++ b/tools/Container Runtimes.md @@ -0,0 +1,407 @@ +--- +title: Container Runtimes +aliases: + - Docker Alternatives + - Container Engines +tags: + - tool + - comparison + - containers + - docker + - devops +type: comparison +status: complete +created: 2025-12-04 +--- + +# Container Runtimes + +Tools for building, running, and managing containers. + +## Overview + +| Tool | Type | Daemon | Rootless | Kubernetes | Platform | +|------|------|:------:|:--------:|:----------:|----------| +| Docker | Engine + CLI | ✅ | ✅ | Via K8s | All | +| Podman | Engine + CLI | ❌ | ✅ | Via K8s | Linux, macOS, Windows | +| containerd | Low-level runtime | ✅ | ✅ | ✅ Native | Linux, Windows | +| Rancher Desktop | GUI + Engine | ✅ | ✅ | ✅ Built-in | macOS, Windows, Linux | +| Colima | CLI wrapper | ✅ | ❌ | ✅ Optional | macOS, Linux | +| Lima | VM manager | ✅ | ❌ | ❌ | macOS, Linux | +| OrbStack | GUI + Engine | ✅ | ❌ | ✅ Built-in | macOS | + +--- + +## Architecture Layers + +``` +┌─────────────────────────────────────────────────────────┐ +│ User Interface │ +│ Docker CLI / Podman / nerdctl / Rancher Desktop │ +├─────────────────────────────────────────────────────────┤ +│ Container Engine │ +│ dockerd / Podman / containerd │ +├─────────────────────────────────────────────────────────┤ +│ Low-Level Runtime │ +│ runc / crun / kata / gVisor │ +├─────────────────────────────────────────────────────────┤ +│ Linux Kernel │ +│ namespaces, cgroups, seccomp │ +└─────────────────────────────────────────────────────────┘ +``` + +--- + +## Docker + +The original container platform. Industry standard. + +### Key Characteristics + +- **Docker Engine** — Daemon-based container runtime +- **Docker CLI** — Primary interface +- **Docker Compose** — Multi-container orchestration +- **Docker Hub** — Default registry +- **Docker Desktop** — GUI for macOS/Windows + +### Architecture + +``` +┌──────────────┐ ┌──────────────┐ ┌──────────────┐ +│ Docker CLI │────▶│ dockerd │────▶│ containerd │ +└──────────────┘ │ (daemon) │ └──────┬───────┘ + └──────────────┘ │ + ▼ + ┌──────────────┐ + │ runc │ + └──────────────┘ +``` + +### Common Commands + +```bash +# Images +docker pull nginx:latest +docker build -t myapp . +docker push myregistry/myapp:v1 + +# Containers +docker run -d -p 8080:80 nginx +docker ps +docker exec -it container_id /bin/sh +docker logs -f container_id +docker stop container_id + +# Compose +docker compose up -d +docker compose down +docker compose logs -f +``` + +### Docker Desktop + +- GUI for container management +- Built-in Kubernetes +- Volume management +- Extension marketplace +- **Licensing**: Free for personal/small business, paid for enterprise + +### Considerations + +- Requires daemon (security surface) +- Docker Desktop licensing for enterprises +- Resource usage on macOS/Windows (VM) + +--- + +## Podman + +Daemonless container engine. Docker-compatible CLI. + +### Key Characteristics + +- **Daemonless** — No background service required +- **Rootless** — Run containers without root +- **Docker-compatible** — Same CLI commands +- **Pod support** — Group containers like Kubernetes +- **Systemd integration** — Generate unit files + +### Architecture + +``` +┌──────────────┐ ┌──────────────┐ +│ Podman CLI │────▶│ Container │ (no daemon) +└──────────────┘ │ Process │ + └──────┬───────┘ + │ + ▼ + ┌──────────────┐ + │ crun/runc │ + └──────────────┘ +``` + +### Common Commands + +```bash +# Alias for Docker compatibility +alias docker=podman + +# Same commands work +podman pull nginx +podman run -d -p 8080:80 nginx +podman ps +podman exec -it container_id /bin/sh + +# Pods (Podman-specific) +podman pod create --name mypod -p 8080:80 +podman run -d --pod mypod nginx +podman run -d --pod mypod redis + +# Generate Kubernetes YAML +podman generate kube mypod > pod.yaml + +# Generate systemd unit +podman generate systemd --new --name mycontainer +``` + +### Podman Desktop + +GUI application similar to Docker Desktop: +- Container/pod management +- Image building +- Kubernetes integration +- Extension support + +### Podman vs Docker + +| Aspect | Docker | Podman | +|--------|--------|--------| +| Daemon | Required | None | +| Root required | Default (rootless available) | Rootless default | +| CLI compatibility | Native | `alias docker=podman` | +| Compose | Docker Compose | podman-compose or Compose v2 | +| Pods | No | Yes (like K8s) | +| Systemd integration | Limited | Native | +| macOS/Windows | Docker Desktop | Podman Desktop + VM | +| License | Docker Desktop paid for enterprise | Free | + +### Considerations + +- Some Docker Compose features may differ +- macOS/Windows requires VM (like Docker) +- Smaller ecosystem than Docker +- Some edge cases with Docker compatibility + +--- + +## containerd + +Industry-standard low-level container runtime. + +### Key Characteristics + +- **Low-level** — Used by Docker, Kubernetes, others +- **CNCF graduated** — Production-ready, widely adopted +- **Kubernetes native** — CRI implementation +- **Minimal** — Runtime only, no build tools + +### Usage + +```bash +# nerdctl = Docker-compatible CLI for containerd +nerdctl run -d nginx +nerdctl build -t myapp . +nerdctl compose up + +# ctr = low-level containerd CLI +ctr images pull docker.io/library/nginx:latest +ctr run docker.io/library/nginx:latest nginx-container +``` + +### When to Use Directly + +- Kubernetes nodes (containerd as CRI) +- Minimal container runtime needs +- Building custom container platforms +- Embedded systems + +--- + +## Rancher Desktop + +GUI application providing container runtime and Kubernetes. + +### Key Characteristics + +- **GUI-first** — Visual container/K8s management +- **Choice of runtime** — containerd or dockerd +- **Built-in Kubernetes** — K3s distribution +- **Cross-platform** — macOS, Windows, Linux +- **Free** — Open source, no licensing fees + +### Features + +``` +Rancher Desktop provides: +├── Container runtime (dockerd or containerd) +├── Kubernetes (K3s) +├── nerdctl or docker CLI +├── Helm dashboard +├── Image management +└── Port forwarding UI +``` + +### Configuration + +```yaml +# Settings via GUI or ~/.rd/settings.json +{ + "kubernetes": { + "version": "1.28.3", + "enabled": true + }, + "containerEngine": { + "name": "containerd" # or "moby" for dockerd + } +} +``` + +### Rancher Desktop vs Docker Desktop + +| Aspect | Rancher Desktop | Docker Desktop | +|--------|-----------------|----------------| +| License | Free (open source) | Free personal, paid enterprise | +| Kubernetes | K3s (built-in) | Docker's K8s | +| Container engine | containerd or dockerd | dockerd | +| CLI | nerdctl or docker | docker | +| Extensions | Limited | Marketplace | +| Resource usage | Similar | Similar | + +--- + +## Colima + +Minimal container runtime for macOS (and Linux). + +### Key Characteristics + +- **Lightweight** — Minimal VM wrapper +- **CLI-focused** — No GUI, just works +- **Lima-based** — Uses Lima VMs +- **Docker-compatible** — Works with Docker CLI +- **Kubernetes optional** — Via K3s + +### Installation & Usage + +```bash +# Install +brew install colima docker + +# Start (creates VM with Docker) +colima start + +# With resource limits +colima start --cpu 4 --memory 8 --disk 100 + +# With Kubernetes +colima start --kubernetes + +# Use Docker CLI normally +docker ps +docker run -d nginx + +# Multiple profiles +colima start --profile dev +colima start --profile test --cpu 2 --memory 4 +``` + +### Configuration + +```yaml +# ~/.colima/default/colima.yaml +cpu: 4 +memory: 8 +disk: 100 + +# VM type: qemu or vz (Virtualization.framework on macOS 13+) +vmType: vz + +# Mount type +mountType: virtiofs # or 9p, sshfs + +kubernetes: + enabled: true + version: v1.28.3 +``` + +### Colima vs Docker Desktop + +| Aspect | Colima | Docker Desktop | +|--------|--------|----------------| +| Interface | CLI only | GUI + CLI | +| Resource usage | Lower | Higher | +| Startup time | Fast | Slower | +| Features | Minimal | Full-featured | +| Price | Free | Paid for enterprise | +| Kubernetes | K3s (optional) | Docker K8s | +| File sharing | Configurable | Automatic | + +--- + +## Comparison Summary + +### Feature Matrix + +| Feature | Docker | Podman | Rancher Desktop | Colima | +|---------|:------:|:------:|:---------------:|:------:| +| Daemon required | ✅ | ❌ | ✅ | ✅ | +| Rootless | ✅ | ✅ Default | ✅ | ❌ | +| GUI | Desktop | Desktop | ✅ | ❌ | +| Kubernetes | ✅ | Via tools | ✅ K3s | ✅ K3s | +| Compose | ✅ | ✅ | ✅ | ✅ | +| Free for enterprise | ❌ | ✅ | ✅ | ✅ | +| Linux native | ✅ | ✅ | ✅ | VM | +| macOS | Via VM | Via VM | Via VM | Via VM | +| Windows | Via VM | Via VM | Via VM | ❌ | + +### Performance (macOS) + +| Tool | Startup | Memory | Disk I/O | +|------|---------|--------|----------| +| Docker Desktop | Slower | Higher | Good (VirtioFS) | +| Colima (vz) | Fast | Lower | Good (VirtioFS) | +| Podman | Medium | Medium | Medium | +| Rancher Desktop | Medium | Medium | Good | + +--- + +## Decision Guide + +| Use Case | Recommendation | +|----------|----------------| +| Enterprise, need support | Docker (paid) or Rancher | +| Free Docker Desktop alternative | Colima or Rancher Desktop | +| Rootless, daemonless | Podman | +| Minimal, CLI-only (macOS) | Colima | +| Need GUI + Kubernetes | Rancher Desktop | +| Kubernetes development | Rancher Desktop or Colima + K3s | +| CI/CD pipelines | Docker or Podman | +| Security-focused | Podman (rootless, daemonless) | + +### Quick Picks + +| Profile | Tool | +|---------|------| +| **Docker user, want free** | Colima + Docker CLI | +| **Security conscious** | Podman | +| **Need GUI + K8s** | Rancher Desktop | +| **Enterprise standard** | Docker | +| **Minimal setup** | Colima | + +--- + +## Related + +- [[Deployment]] +- [[Kubernetes]] +- [[WSL]] diff --git a/tools/Container Tools.md b/tools/Container Tools.md new file mode 100644 index 0000000..2ef7570 --- /dev/null +++ b/tools/Container Tools.md @@ -0,0 +1,677 @@ +--- +title: Container Tools +aliases: + - Docker Tools + - Container Ecosystem +tags: + - tool + - comparison + - containers + - docker + - devops +type: comparison +status: complete +created: 2025-12-04 +--- + +# Container Tools + +FOSS tools for managing, monitoring, and orchestrating containers. + +## Overview + +| Category | Tools | +|----------|-------| +| Management UI | Portainer, Dockge, Yacht | +| Reverse Proxy | Traefik, Caddy, nginx-proxy | +| Auto-Update | Watchtower, Diun | +| Orchestration | Docker Swarm, Nomad | +| Registry | Harbor, Distribution, Zot | +| Networking | Tailscale, Cloudflare Tunnel | +| Backups | Duplicati, Restic, Borgmatic | +| Secrets | Docker Secrets, Vault, SOPS | + +--- + +## Container Management + +### Portainer + +Web UI for managing Docker/Kubernetes environments. + +```yaml +# docker-compose.yaml +services: + portainer: + image: portainer/portainer-ce:latest + ports: + - "9443:9443" + - "9000:9000" + volumes: + - /var/run/docker.sock:/var/run/docker.sock + - portainer_data:/data + restart: unless-stopped + +volumes: + portainer_data: +``` + +**Features:** +- Container/stack management +- Docker Compose support +- Multi-environment (Docker, Swarm, K8s) +- User management and RBAC +- Templates and app store +- Edge agent for remote hosts + +### Dockge + +Lightweight Docker Compose stack manager. + +```yaml +# docker-compose.yaml +services: + dockge: + image: louislam/dockge:1 + ports: + - "5001:5001" + volumes: + - /var/run/docker.sock:/var/run/docker.sock + - ./data:/app/data + - /opt/stacks:/opt/stacks # Your compose files + environment: + - DOCKGE_STACKS_DIR=/opt/stacks + restart: unless-stopped +``` + +**Features:** +- Simple, clean UI +- Edit compose files in browser +- Start/stop/restart stacks +- View logs +- Terminal access + +### Yacht + +Docker container management UI. + +```yaml +services: + yacht: + image: selfhostedpro/yacht + ports: + - "8000:8000" + volumes: + - yacht:/config + - /var/run/docker.sock:/var/run/docker.sock + restart: unless-stopped +``` + +### Comparison + +| Feature | Portainer | Dockge | Yacht | +|---------|:---------:|:------:|:-----:| +| Compose support | ✅ | ✅ | ✅ | +| Kubernetes | ✅ | ❌ | ❌ | +| Multi-host | ✅ | ❌ | ❌ | +| Resource usage | Medium | Low | Low | +| Complexity | Higher | Simple | Simple | + +--- + +## Reverse Proxy / Ingress + +### Traefik + +Cloud-native reverse proxy with auto-discovery. + +```yaml +# docker-compose.yaml +services: + traefik: + image: traefik:v3.0 + command: + - "--api.dashboard=true" + - "--providers.docker=true" + - "--providers.docker.exposedbydefault=false" + - "--entrypoints.web.address=:80" + - "--entrypoints.websecure.address=:443" + - "--certificatesresolvers.letsencrypt.acme.httpchallenge=true" + - "--certificatesresolvers.letsencrypt.acme.email=admin@example.com" + - "--certificatesresolvers.letsencrypt.acme.storage=/letsencrypt/acme.json" + ports: + - "80:80" + - "443:443" + volumes: + - /var/run/docker.sock:/var/run/docker.sock:ro + - letsencrypt:/letsencrypt + labels: + - "traefik.enable=true" + - "traefik.http.routers.dashboard.rule=Host(`traefik.example.com`)" + - "traefik.http.routers.dashboard.service=api@internal" + + whoami: + image: traefik/whoami + labels: + - "traefik.enable=true" + - "traefik.http.routers.whoami.rule=Host(`whoami.example.com`)" + - "traefik.http.routers.whoami.entrypoints=websecure" + - "traefik.http.routers.whoami.tls.certresolver=letsencrypt" +``` + +**Features:** +- Auto-discovery via Docker labels +- Let's Encrypt integration +- Middleware (auth, rate limit, etc.) +- Dashboard +- Metrics (Prometheus) +- Multiple backends (Docker, K8s, file, etc.) + +### Caddy + +Modern web server with automatic HTTPS. + +```yaml +services: + caddy: + image: caddy:2 + ports: + - "80:80" + - "443:443" + volumes: + - ./Caddyfile:/etc/caddy/Caddyfile + - caddy_data:/data + - caddy_config:/config +``` + +```caddyfile +# Caddyfile +app.example.com { + reverse_proxy app:3000 +} + +api.example.com { + reverse_proxy api:8080 +} + +# Automatic HTTPS by default +``` + +### nginx-proxy + +Automated nginx reverse proxy for Docker. + +```yaml +services: + nginx-proxy: + image: nginxproxy/nginx-proxy + ports: + - "80:80" + - "443:443" + volumes: + - /var/run/docker.sock:/tmp/docker.sock:ro + - certs:/etc/nginx/certs + - vhost:/etc/nginx/vhost.d + - html:/usr/share/nginx/html + + acme-companion: + image: nginxproxy/acme-companion + volumes: + - /var/run/docker.sock:/var/run/docker.sock:ro + - certs:/etc/nginx/certs + - acme:/etc/acme.sh + environment: + - DEFAULT_EMAIL=admin@example.com + - NGINX_PROXY_CONTAINER=nginx-proxy + + app: + image: myapp + environment: + - VIRTUAL_HOST=app.example.com + - LETSENCRYPT_HOST=app.example.com +``` + +### Comparison + +| Feature | Traefik | Caddy | nginx-proxy | +|---------|:-------:|:-----:|:-----------:| +| Auto-discovery | ✅ | Config file | ✅ | +| Auto HTTPS | ✅ | ✅ | With companion | +| Dashboard | ✅ | ❌ | ❌ | +| Config style | Labels | Caddyfile | Env vars | +| Middleware | Extensive | Good | Basic | +| Performance | Good | Excellent | Excellent | + +--- + +## Auto-Update + +### Watchtower + +Automatically update running containers. + +```yaml +services: + watchtower: + image: containrrr/watchtower + volumes: + - /var/run/docker.sock:/var/run/docker.sock + environment: + - WATCHTOWER_CLEANUP=true + - WATCHTOWER_POLL_INTERVAL=86400 # Daily + - WATCHTOWER_NOTIFICATIONS=email + - WATCHTOWER_NOTIFICATION_EMAIL_TO=admin@example.com + restart: unless-stopped +``` + +**Options:** +```yaml +environment: + # Update schedule + - WATCHTOWER_SCHEDULE=0 0 4 * * * # 4 AM daily + + # Selective updates (label-based) + - WATCHTOWER_LABEL_ENABLE=true + + # Rolling restarts + - WATCHTOWER_ROLLING_RESTART=true + + # Monitor only (no updates) + - WATCHTOWER_MONITOR_ONLY=true +``` + +```yaml +# Label containers to include/exclude +services: + app: + image: myapp + labels: + - "com.centurylinklabs.watchtower.enable=true" + + db: + image: postgres + labels: + - "com.centurylinklabs.watchtower.enable=false" +``` + +### Diun (Docker Image Update Notifier) + +Notify about available updates without auto-updating. + +```yaml +services: + diun: + image: crazymax/diun:latest + volumes: + - /var/run/docker.sock:/var/run/docker.sock + - ./diun.yml:/diun.yml:ro + environment: + - TZ=America/New_York + - DIUN_WATCH_SCHEDULE=0 */6 * * * +``` + +```yaml +# diun.yml +watch: + workers: 20 + schedule: "0 */6 * * *" + +notif: + discord: + webhookURL: https://discord.com/api/webhooks/xxx + slack: + webhookURL: https://hooks.slack.com/xxx + +providers: + docker: + watchByDefault: true +``` + +--- + +## Container Registry + +### Harbor + +Enterprise container registry with security scanning. + +```bash +# Install Harbor +wget https://github.com/goharbor/harbor/releases/download/v2.10.0/harbor-offline-installer-v2.10.0.tgz +tar xvf harbor-offline-installer-v2.10.0.tgz +cd harbor + +# Configure +cp harbor.yml.tmpl harbor.yml +# Edit harbor.yml (hostname, https, passwords) + +# Install +./install.sh --with-trivy # Include vulnerability scanning +``` + +**Features:** +- Vulnerability scanning (Trivy) +- RBAC and projects +- Replication between registries +- Helm chart repository +- OCI artifact support + +### Distribution (Docker Registry) + +Official Docker registry. + +```yaml +services: + registry: + image: registry:2 + ports: + - "5000:5000" + volumes: + - registry-data:/var/lib/registry + environment: + - REGISTRY_STORAGE_DELETE_ENABLED=true +``` + +### Registry UI + +```yaml +services: + registry-ui: + image: joxit/docker-registry-ui:latest + ports: + - "8080:80" + environment: + - REGISTRY_TITLE=My Registry + - REGISTRY_URL=http://registry:5000 + - DELETE_IMAGES=true + depends_on: + - registry +``` + +--- + +## Networking + +### Tailscale + +Mesh VPN for containers. + +```yaml +services: + tailscale: + image: tailscale/tailscale:latest + hostname: docker-server + environment: + - TS_AUTHKEY=tskey-xxx + - TS_STATE_DIR=/var/lib/tailscale + volumes: + - tailscale-state:/var/lib/tailscale + - /dev/net/tun:/dev/net/tun + cap_add: + - NET_ADMIN + - SYS_MODULE + restart: unless-stopped + + # Use as sidecar + app: + network_mode: service:tailscale + depends_on: + - tailscale +``` + +### Cloudflare Tunnel + +Expose services without opening ports. + +```yaml +services: + cloudflared: + image: cloudflare/cloudflared:latest + command: tunnel --no-autoupdate run + environment: + - TUNNEL_TOKEN=xxx + restart: unless-stopped + + app: + image: myapp + # No ports exposed, accessed via tunnel +``` + +```yaml +# Tunnel config (Cloudflare dashboard or CLI) +ingress: + - hostname: app.example.com + service: http://app:3000 + - hostname: api.example.com + service: http://api:8080 + - service: http_status:404 +``` + +--- + +## Backup Tools + +### Duplicati + +Backup with web UI. + +```yaml +services: + duplicati: + image: linuxserver/duplicati + ports: + - "8200:8200" + volumes: + - /opt/stacks:/source:ro # What to backup + - duplicati-config:/config + - /backups:/backups + environment: + - PUID=1000 + - PGID=1000 +``` + +### Restic + Docker + +```yaml +services: + restic-backup: + image: mazzolino/restic + environment: + - BACKUP_CRON=0 3 * * * + - RESTIC_REPOSITORY=s3:s3.amazonaws.com/my-bucket + - AWS_ACCESS_KEY_ID=xxx + - AWS_SECRET_ACCESS_KEY=xxx + - RESTIC_PASSWORD=xxx + volumes: + - /opt/stacks:/data:ro +``` + +### docker-volume-backup + +Backup Docker volumes directly. + +```yaml +services: + backup: + image: offen/docker-volume-backup:latest + environment: + - BACKUP_CRON_EXPRESSION=0 4 * * * + - AWS_S3_BUCKET_NAME=my-backups + - AWS_ACCESS_KEY_ID=xxx + - AWS_SECRET_ACCESS_KEY=xxx + volumes: + - /var/run/docker.sock:/var/run/docker.sock:ro + - app-data:/backup/app-data:ro + +volumes: + app-data: + labels: + - docker-volume-backup.stop-during-backup=true +``` + +--- + +## Monitoring + +### cAdvisor + +Container resource monitoring. + +```yaml +services: + cadvisor: + image: gcr.io/cadvisor/cadvisor:latest + ports: + - "8080:8080" + volumes: + - /:/rootfs:ro + - /var/run:/var/run:ro + - /sys:/sys:ro + - /var/lib/docker/:/var/lib/docker:ro +``` + +### Prometheus + cAdvisor + +```yaml +# prometheus.yml +scrape_configs: + - job_name: 'cadvisor' + static_configs: + - targets: ['cadvisor:8080'] +``` + +### Dozzle + +Real-time log viewer. + +```yaml +services: + dozzle: + image: amir20/dozzle:latest + ports: + - "8080:8080" + volumes: + - /var/run/docker.sock:/var/run/docker.sock:ro +``` + +--- + +## Development Tools + +### lazydocker + +Terminal UI for Docker. + +```bash +# Install +brew install lazydocker +# or +docker run --rm -it -v /var/run/docker.sock:/var/run/docker.sock lazyteam/lazydocker +``` + +**Features:** +- Container/image/volume management +- Log viewing +- Stats monitoring +- Keyboard-driven + +### ctop + +Top-like container metrics. + +```bash +docker run --rm -it \ + -v /var/run/docker.sock:/var/run/docker.sock \ + quay.io/vektorlab/ctop:latest +``` + +### dive + +Explore Docker image layers. + +```bash +# Analyze image +dive nginx:latest + +# Check Dockerfile efficiency +dive build -t myapp . +``` + +--- + +## Docker Compose Utilities + +### docker-compose-viz + +Visualize compose files. + +```bash +docker run --rm -it --name dcv -v $(pwd):/input pmsipilot/docker-compose-viz render -m image docker-compose.yml +``` + +### Composerize + +Convert `docker run` to compose. + +```bash +# Web: composerize.com +# CLI +npx composerize docker run -d -p 80:80 nginx +``` + +--- + +## Quick Reference + +### Essential Stack + +```yaml +# docker-compose.yaml - Self-hosted essentials +services: + traefik: + image: traefik:v3.0 + # ... reverse proxy with auto SSL + + portainer: + image: portainer/portainer-ce + labels: + - "traefik.enable=true" + - "traefik.http.routers.portainer.rule=Host(`portainer.example.com`)" + # ... management UI + + watchtower: + image: containrrr/watchtower + # ... auto-updates + + dozzle: + image: amir20/dozzle + labels: + - "traefik.enable=true" + - "traefik.http.routers.dozzle.rule=Host(`logs.example.com`)" + # ... log viewer +``` + +--- + +## Decision Guide + +| Need | Tool | +|------|------| +| Container management UI | Portainer (full) or Dockge (simple) | +| Reverse proxy + SSL | Traefik (auto-discovery) or Caddy (simple) | +| Auto-updates | Watchtower | +| Update notifications | Diun | +| Private registry | Harbor (enterprise) or Distribution (simple) | +| Remote access | Tailscale or Cloudflare Tunnel | +| Log viewing | Dozzle | +| Metrics | cAdvisor + Prometheus | +| Terminal management | lazydocker | + +--- + +## Related + +- [[Container Runtimes]] +- [[Observability Stack]] +- [[Deployment]] +- [[Distributed Tracing]] diff --git a/tools/Distributed Tracing.md b/tools/Distributed Tracing.md new file mode 100644 index 0000000..38d00ab --- /dev/null +++ b/tools/Distributed Tracing.md @@ -0,0 +1,510 @@ +--- +title: Distributed Tracing +aliases: + - Tracing + - APM Tracing +tags: + - tool + - comparison + - observability + - tracing +type: comparison +status: complete +created: 2025-12-04 +--- + +# Distributed Tracing + +Tools for tracking requests across distributed systems. + +## Overview + +| Tool | Type | License | Storage | Best For | +|------|------|---------|---------|----------| +| Jaeger | Tracing backend | OSS (Apache 2) | Cassandra, ES, Memory | CNCF standard | +| Zipkin | Tracing backend | OSS (Apache 2) | Cassandra, ES, MySQL | Simple setup | +| Tempo | Tracing backend | OSS (AGPLv3) | Object storage | Grafana users | +| SigNoz | Full observability | OSS (MIT) | ClickHouse | All-in-one OSS | +| AWS X-Ray | Managed tracing | Commercial | AWS | AWS workloads | +| Datadog APM | Managed APM | Commercial | Datadog | Full APM | + +--- + +## Tracing Concepts + +### Trace Structure + +``` +Trace: A complete request journey +├── trace_id: unique identifier for entire request +│ +├── Span: API Gateway +│ ├── span_id: abc001 +│ ├── parent_span_id: null (root) +│ ├── operation: "HTTP GET /users" +│ ├── duration: 150ms +│ ├── tags: {http.method: GET, http.status: 200} +│ │ +│ ├── Span: Auth Service +│ │ ├── span_id: abc002 +│ │ ├── parent_span_id: abc001 +│ │ ├── operation: "validate_token" +│ │ └── duration: 20ms +│ │ +│ └── Span: User Service +│ ├── span_id: abc003 +│ ├── parent_span_id: abc001 +│ ├── operation: "get_user" +│ ├── duration: 100ms +│ │ +│ ├── Span: Cache Lookup +│ │ ├── span_id: abc004 +│ │ ├── parent_span_id: abc003 +│ │ └── duration: 5ms (cache miss) +│ │ +│ └── Span: Database Query +│ ├── span_id: abc005 +│ ├── parent_span_id: abc003 +│ └── duration: 80ms +``` + +### Context Propagation + +``` +Service A Service B Service C + │ │ │ + │ traceparent: 00-xxx-yyy │ traceparent: 00-xxx-zzz │ + │ ─────────────────────────▶ │ ─────────────────────────▶ │ + │ │ │ + │ W3C Trace Context or │ Same trace_id, │ + │ B3 Headers │ new span_id │ +``` + +### Propagation Formats + +| Format | Header Example | Used By | +|--------|----------------|---------| +| W3C Trace Context | `traceparent: 00-{trace_id}-{span_id}-{flags}` | OpenTelemetry (default) | +| B3 Single | `b3: {trace_id}-{span_id}-{sampled}` | Zipkin | +| B3 Multi | `X-B3-TraceId`, `X-B3-SpanId`, etc. | Zipkin (legacy) | +| Jaeger | `uber-trace-id: {trace_id}:{span_id}:{parent}:{flags}` | Jaeger | + +--- + +## Jaeger + +CNCF graduated project. Production-ready distributed tracing. + +### Architecture + +``` +┌─────────────────────────────────────────────────────────────┐ +│ Applications │ +│ (instrumented with OpenTelemetry) │ +└──────────────────────────┬──────────────────────────────────┘ + │ OTLP / Jaeger protocol + ▼ +┌─────────────────────────────────────────────────────────────┐ +│ Jaeger Collector │ +│ (receives, validates, transforms) │ +└──────────────────────────┬──────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────┐ +│ Storage │ +│ Cassandra │ Elasticsearch │ Kafka │ Badger │ Memory │ +└──────────────────────────┬──────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────┐ +│ Jaeger Query │ +│ (UI and API) │ +└─────────────────────────────────────────────────────────────┘ +``` + +### All-in-One Deployment + +```yaml +# docker-compose.yaml +services: + jaeger: + image: jaegertracing/all-in-one:latest + ports: + - "16686:16686" # UI + - "4317:4317" # OTLP gRPC + - "4318:4318" # OTLP HTTP + - "14268:14268" # Jaeger HTTP + - "6831:6831/udp" # Jaeger compact (legacy) + environment: + - COLLECTOR_OTLP_ENABLED=true +``` + +### Production Deployment + +```yaml +# docker-compose.yaml +services: + jaeger-collector: + image: jaegertracing/jaeger-collector:latest + environment: + - SPAN_STORAGE_TYPE=elasticsearch + - ES_SERVER_URLS=http://elasticsearch:9200 + ports: + - "4317:4317" + - "4318:4318" + + jaeger-query: + image: jaegertracing/jaeger-query:latest + environment: + - SPAN_STORAGE_TYPE=elasticsearch + - ES_SERVER_URLS=http://elasticsearch:9200 + ports: + - "16686:16686" + + elasticsearch: + image: elasticsearch:8.11.0 + environment: + - discovery.type=single-node + - xpack.security.enabled=false +``` + +### Sampling Configuration + +```yaml +# sampling.json +{ + "service_strategies": [ + { + "service": "api-gateway", + "type": "probabilistic", + "param": 0.5 + }, + { + "service": "critical-service", + "type": "const", + "param": 1 + } + ], + "default_strategy": { + "type": "probabilistic", + "param": 0.1 + } +} +``` + +--- + +## Zipkin + +Simple, mature distributed tracing system. + +### Architecture + +``` +┌─────────────────────────────────────────────────────────────┐ +│ Applications │ +│ (instrumented with Zipkin/OpenTelemetry) │ +└──────────────────────────┬──────────────────────────────────┘ + │ HTTP POST /api/v2/spans + ▼ +┌─────────────────────────────────────────────────────────────┐ +│ Zipkin │ +│ ┌───────────┐ ┌───────────┐ ┌───────────────────┐ │ +│ │ Collector │──▶│ Storage │◀──│ Query API │ │ +│ └───────────┘ └───────────┘ └─────────┬─────────┘ │ +│ │ │ +│ ▼ │ +│ ┌──────────┐ │ +│ │ UI │ │ +│ └──────────┘ │ +└─────────────────────────────────────────────────────────────┘ +``` + +### Deployment + +```yaml +# docker-compose.yaml +services: + zipkin: + image: openzipkin/zipkin:latest + ports: + - "9411:9411" + environment: + - STORAGE_TYPE=elasticsearch + - ES_HOSTS=http://elasticsearch:9200 + + # Or simple in-memory + zipkin-slim: + image: openzipkin/zipkin-slim:latest + ports: + - "9411:9411" +``` + +### Storage Options + +| Storage | Use Case | +|---------|----------| +| Memory | Development, testing | +| Elasticsearch | Production, search | +| Cassandra | High-volume production | +| MySQL | Small deployments | + +--- + +## Tempo + +Grafana's distributed tracing backend. Cost-effective at scale. + +### Key Concepts + +- **Object storage** — Uses S3/GCS/Azure for traces +- **No indexing** — Relies on trace IDs +- **TraceQL** — Query language +- **Grafana native** — Deep integration + +### Architecture + +``` +┌─────────────────────────────────────────────────────────────┐ +│ Applications │ +│ (instrumented with OpenTelemetry) │ +└──────────────────────────┬──────────────────────────────────┘ + │ OTLP + ▼ +┌─────────────────────────────────────────────────────────────┐ +│ Tempo │ +│ ┌───────────────┐ ┌───────────────┐ │ +│ │ Distributor │──▶│ Ingester │ │ +│ └───────────────┘ └───────┬───────┘ │ +│ │ │ +│ ▼ │ +│ ┌───────────────┐ │ +│ │Object Storage │ │ +│ │ (S3/GCS) │ │ +│ └───────┬───────┘ │ +│ │ │ +│ ▼ │ +│ ┌───────────────┐ │ +│ │ Querier │ │ +│ └───────────────┘ │ +└─────────────────────────────────────────────────────────────┘ +``` + +### TraceQL Examples + +```traceql +// Find traces by service +{ resource.service.name = "api" } + +// Find slow spans +{ span.http.status_code >= 500 } + +// Duration filter +{ span.http.url = "/api/users" } | duration > 500ms + +// Structural queries +{ span.http.method = "POST" } >> { span.db.system = "postgresql" } + +// Aggregate +{ resource.service.name = "api" } | count() > 100 +``` + +### Deployment + +```yaml +# docker-compose.yaml +services: + tempo: + image: grafana/tempo:latest + command: ["-config.file=/etc/tempo.yaml"] + volumes: + - ./tempo.yaml:/etc/tempo.yaml + - tempo-data:/var/tempo + ports: + - "3200:3200" # Tempo query + - "4317:4317" # OTLP gRPC + - "4318:4318" # OTLP HTTP +``` + +```yaml +# tempo.yaml +server: + http_listen_port: 3200 + +distributor: + receivers: + otlp: + protocols: + grpc: + http: + +storage: + trace: + backend: local + local: + path: /var/tempo/traces + # Or for production: + # backend: s3 + # s3: + # bucket: tempo-traces + # endpoint: s3.amazonaws.com +``` + +--- + +## OpenTelemetry Integration + +All modern tracing backends support OpenTelemetry. + +### SDK Setup (Node.js) + +```javascript +// tracing.js +const { NodeSDK } = require('@opentelemetry/sdk-node'); +const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-grpc'); +const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node'); + +const sdk = new NodeSDK({ + traceExporter: new OTLPTraceExporter({ + url: 'http://jaeger:4317', // or tempo, or collector + }), + instrumentations: [getNodeAutoInstrumentations()], +}); + +sdk.start(); +``` + +### SDK Setup (Python) + +```python +# tracing.py +from opentelemetry import trace +from opentelemetry.sdk.trace import TracerProvider +from opentelemetry.sdk.trace.export import BatchSpanProcessor +from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter +from opentelemetry.instrumentation.flask import FlaskInstrumentor + +trace.set_tracer_provider(TracerProvider()) +tracer = trace.get_tracer(__name__) + +otlp_exporter = OTLPSpanExporter(endpoint="http://jaeger:4317") +trace.get_tracer_provider().add_span_processor( + BatchSpanProcessor(otlp_exporter) +) + +# Auto-instrument Flask +FlaskInstrumentor().instrument() +``` + +### SDK Setup (Go) + +```go +package main + +import ( + "go.opentelemetry.io/otel" + "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc" + "go.opentelemetry.io/otel/sdk/trace" +) + +func initTracer() (*trace.TracerProvider, error) { + exporter, err := otlptracegrpc.New(ctx, + otlptracegrpc.WithEndpoint("jaeger:4317"), + otlptracegrpc.WithInsecure(), + ) + if err != nil { + return nil, err + } + + tp := trace.NewTracerProvider( + trace.WithBatcher(exporter), + ) + otel.SetTracerProvider(tp) + return tp, nil +} +``` + +--- + +## Sampling Strategies + +### Head-Based Sampling + +Decision made at trace start. + +```yaml +# OpenTelemetry Collector config +processors: + probabilistic_sampler: + sampling_percentage: 10 # Sample 10% +``` + +### Tail-Based Sampling + +Decision made after trace completes. + +```yaml +# OpenTelemetry Collector config +processors: + tail_sampling: + decision_wait: 10s + policies: + - name: errors + type: status_code + status_code: {status_codes: [ERROR]} + - name: slow + type: latency + latency: {threshold_ms: 1000} + - name: default + type: probabilistic + probabilistic: {sampling_percentage: 5} +``` + +### Sampling Comparison + +| Strategy | Pros | Cons | +|----------|------|------| +| Head-based | Simple, low overhead | Miss interesting traces | +| Tail-based | Keep important traces | Higher resource usage | +| Probabilistic | Predictable cost | Random, might miss errors | +| Rate-limited | Controlled volume | Might miss bursts | + +--- + +## Comparison Matrix + +| Feature | Jaeger | Zipkin | Tempo | +|---------|:------:|:------:|:-----:| +| Query by trace ID | ✅ | ✅ | ✅ | +| Search by tags | ✅ | ✅ | TraceQL | +| Service dependency graph | ✅ | ✅ | Via metrics | +| Storage backends | Many | Many | Object storage | +| UI | Built-in | Built-in | Grafana | +| Scalability | High | High | Very high | +| Cost at scale | $$$ | $$$ | $ | +| Retention policies | Manual | Manual | Built-in | +| CNCF status | Graduated | Incubating | — | + +--- + +## Decision Guide + +| Scenario | Recommendation | +|----------|----------------| +| Grafana stack | Tempo | +| CNCF standard | Jaeger | +| Simple setup | Zipkin | +| Cost-sensitive, high volume | Tempo | +| Elasticsearch already in use | Jaeger | +| AWS workloads | X-Ray or Jaeger | +| Full APM needed | Datadog or SigNoz | + +--- + +## Related + +- [[Observability Stack]] +- [[Log Aggregation]] +- [[OpenTelemetry]] +- [[Prometheus]] +- [[Grafana]] diff --git a/tools/Environment Management.md b/tools/Environment Management.md new file mode 100644 index 0000000..c54430f --- /dev/null +++ b/tools/Environment Management.md @@ -0,0 +1,550 @@ +--- +title: Environment Management +aliases: + - Environment Variables + - direnv + - dotenv +tags: + - tool + - comparison + - development +type: comparison +status: complete +created: 2025-12-04 +--- + +# Environment Management + +Tools for managing environment variables, project settings, and development environments. + +## Overview + +| Tool | Type | Shell Integration | Language Agnostic | +|------|------|-------------------|-------------------| +| direnv | Directory-based env | ✅ | ✅ | +| dotenv | File loader | Library per language | Libraries | +| mise | Polyglot env + versions | ✅ | ✅ | +| asdf-direnv | asdf + direnv integration | ✅ | ✅ | +| envchain | Secure env storage | ✅ | ✅ | +| 1Password CLI | Secrets manager | ✅ | ✅ | + +--- + +## direnv + +Automatically load/unload environment variables per directory. + +### How It Works + +``` +~/projects/app1/ ~/projects/app2/ +├── .envrc ├── .envrc +│ DATABASE_URL=... │ DATABASE_URL=... +│ API_KEY=... │ API_KEY=... +│ │ +cd app1 → loads app1 env cd app2 → loads app2 env +cd .. → unloads env cd .. → unloads env +``` + +### Installation + +```bash +# macOS +brew install direnv + +# Ubuntu +sudo apt install direnv + +# Shell setup +# Bash (~/.bashrc) +eval "$(direnv hook bash)" + +# Zsh (~/.zshrc) +eval "$(direnv hook zsh)" + +# Fish (~/.config/fish/config.fish) +direnv hook fish | source +``` + +### Basic Usage + +```bash +# Create .envrc +echo 'export DATABASE_URL="postgres://localhost/dev"' > .envrc + +# Allow the file (security feature) +direnv allow + +# Now entering directory loads variables +cd myproject +echo $DATABASE_URL # postgres://localhost/dev + +# Leaving unloads them +cd .. +echo $DATABASE_URL # (empty) +``` + +### .envrc Examples + +```bash +# .envrc + +# Simple exports +export DATABASE_URL="postgres://localhost/dev" +export API_KEY="dev-key-123" +export DEBUG=true + +# Load from .env file +dotenv + +# Load from .env if exists +dotenv_if_exists + +# Load from specific file +dotenv .env.local + +# Use PATH_add for local binaries +PATH_add bin +PATH_add node_modules/.bin + +# Use layout for language-specific setup +layout python3 # Creates/activates venv +layout node # Sets up node_modules/.bin +layout ruby # Sets up bundle paths + +# Source another file +source_env .envrc.local + +# Use direnv's stdlib +use nix # Load nix-shell +use flake # Load nix flake +``` + +### Security + +```bash +# .envrc files must be explicitly allowed +direnv allow . + +# Deny a file +direnv deny . + +# Edit and re-allow +direnv edit . + +# Block patterns in ~/.config/direnv/direnv.toml +[global] +load_dotenv = false +``` + +### With mise/asdf + +```bash +# .envrc - integrate with mise +use mise + +# Or with asdf +use asdf +``` + +--- + +## dotenv (.env files) + +Standard for storing environment variables in files. + +### Format + +```bash +# .env +DATABASE_URL=postgres://localhost/dev +API_KEY=secret123 +DEBUG=true + +# Quoted values (preserves spaces) +MESSAGE="Hello World" +MULTILINE="Line 1\nLine 2" + +# Comments +# This is a comment + +# Variable expansion (some implementations) +BASE_URL=https://api.example.com +USERS_URL=${BASE_URL}/users +``` + +### File Hierarchy + +``` +project/ +├── .env # Shared defaults (commit this) +├── .env.local # Local overrides (gitignore) +├── .env.development # Development settings +├── .env.production # Production settings +├── .env.test # Test settings +└── .env.example # Template (commit this) +``` + +### Language Libraries + +#### JavaScript/Node.js + +```javascript +// npm install dotenv +require('dotenv').config(); + +// Or ES modules +import 'dotenv/config'; + +// Access +console.log(process.env.DATABASE_URL); + +// With options +require('dotenv').config({ + path: '.env.local', + override: true +}); +``` + +#### Python + +```python +# pip install python-dotenv + +from dotenv import load_dotenv +import os + +load_dotenv() # Loads .env +# or +load_dotenv('.env.local') + +database_url = os.getenv('DATABASE_URL') + +# Auto-load in Flask +# FLASK_DEBUG=1 in .env works automatically +``` + +#### Ruby + +```ruby +# gem install dotenv + +require 'dotenv/load' # Auto-loads .env + +# Or manually +require 'dotenv' +Dotenv.load('.env.local') + +ENV['DATABASE_URL'] +``` + +#### Go + +```go +// go get github.com/joho/godotenv + +import "github.com/joho/godotenv" + +func init() { + godotenv.Load() // Loads .env + // or + godotenv.Load(".env.local") +} + +os.Getenv("DATABASE_URL") +``` + +### Security Best Practices + +```bash +# .gitignore +.env.local +.env.*.local +.env.development.local +.env.production.local + +# Never commit actual secrets +# Use .env.example as template +``` + +--- + +## mise for Environment + +mise handles both versions and environment. + +### Environment Features + +```toml +# mise.toml or .mise.toml + +[env] +DATABASE_URL = "postgres://localhost/dev" +API_KEY = "dev-key" + +# From file +_.file = ".env" + +# Path manipulation +_.path = ["./bin", "./node_modules/.bin"] + +# Conditional +[env.development] +DEBUG = "true" + +[env.production] +DEBUG = "false" +``` + +### With Tools + +```toml +# .mise.toml +[tools] +node = "20" +python = "3.12" + +[env] +NODE_ENV = "development" +PYTHONPATH = "./src" +``` + +### Tasks + +```toml +# .mise.toml +[tasks.dev] +run = "npm run dev" +env = { DEBUG = "true" } + +[tasks.test] +run = "pytest" +env = { TESTING = "true" } +``` + +```bash +mise run dev +mise run test +``` + +--- + +## Secret Management + +### envchain (macOS Keychain) + +Store secrets in system keychain. + +```bash +# Install +brew install envchain + +# Store secret +envchain --set myproject AWS_ACCESS_KEY_ID +# (prompts for value, stores in Keychain) + +envchain --set myproject AWS_SECRET_ACCESS_KEY + +# Run command with secrets +envchain myproject aws s3 ls + +# List namespaces +envchain --list +``` + +### 1Password CLI + +```bash +# Install +brew install 1password-cli + +# Sign in +op signin + +# Reference in .env +DATABASE_URL="op://Vault/Database/password" + +# Run with secrets injected +op run -- npm start + +# In scripts +export API_KEY=$(op read "op://Vault/API/key") +``` + +### AWS Secrets Manager / Parameter Store + +```bash +# Fetch at runtime +export DB_PASSWORD=$(aws ssm get-parameter \ + --name /myapp/db-password \ + --with-decryption \ + --query Parameter.Value \ + --output text) +``` + +### HashiCorp Vault + +```bash +# Login +vault login + +# Read secret +vault kv get -field=password secret/myapp/db + +# In app +export DB_PASSWORD=$(vault kv get -field=password secret/myapp/db) +``` + +--- + +## Framework Integration + +### Next.js + +```bash +# Automatically loads: +.env # All environments +.env.local # Local (gitignored) +.env.development # Development only +.env.production # Production only +.env.development.local # Local dev overrides +``` + +```javascript +// Must prefix with NEXT_PUBLIC_ for browser +NEXT_PUBLIC_API_URL=https://api.example.com +SECRET_KEY=server-only // Only available server-side +``` + +### Vite + +```bash +# Automatically loads based on mode +.env +.env.local +.env.[mode] +.env.[mode].local +``` + +```bash +# Must prefix with VITE_ for client +VITE_API_URL=https://api.example.com +SECRET=server-only // Not exposed +``` + +### Rails + +```ruby +# Uses credentials or dotenv-rails gem +# config/credentials.yml.enc (encrypted) + +# With dotenv-rails gem: +# Loads in order: +# .env +# .env.development / .env.test / .env.production +# .env.local +# .env.development.local / etc. +``` + +### Django + +```python +# settings.py with django-environ +import environ + +env = environ.Env() +environ.Env.read_env() # Loads .env + +DATABASES = { + 'default': env.db('DATABASE_URL') +} +SECRET_KEY = env('SECRET_KEY') +DEBUG = env.bool('DEBUG', default=False) +``` + +--- + +## Comparison + +### direnv vs dotenv vs mise + +| Aspect | direnv | dotenv | mise | +|--------|--------|--------|------| +| Type | Shell hook | Library | Shell hook | +| Automatic | ✅ On cd | ❌ Manual load | ✅ On cd | +| Language | Any | Per language | Any | +| Version management | ❌ | ❌ | ✅ | +| Shell functions | ✅ | ❌ | ✅ | +| Task running | ❌ | ❌ | ✅ | + +### When to Use What + +| Scenario | Tool | +|----------|------| +| Quick env per project | direnv | +| App configuration | dotenv library | +| Polyglot + versions | mise | +| Secrets | envchain, 1Password, Vault | +| Team standardization | mise or direnv + dotenv | + +--- + +## Best Practices + +### Project Setup + +```bash +# Recommended structure +project/ +├── .envrc # direnv (loads .env, sets PATH) +├── .env # Shared defaults (safe to commit) +├── .env.local # Local secrets (gitignored) +├── .env.example # Template for new devs +├── .mise.toml # Tool versions + env +└── .gitignore # Ignore .env.local, .env.*.local +``` + +### .envrc Template + +```bash +# .envrc +# Load .env file +dotenv_if_exists + +# Load local overrides +dotenv_if_exists .env.local + +# Add local binaries to PATH +PATH_add bin +PATH_add node_modules/.bin + +# Use mise for tool versions +use mise + +# Project-specific vars +export PROJECT_NAME="myapp" +``` + +### .gitignore + +```bash +# .gitignore + +# Local environment files +.env.local +.env.*.local + +# But keep templates +!.env.example + +# direnv +.direnv/ +``` + +--- + +## Related + +- [[Version Managers]] +- [[Shells]] +- [[Remote Development]] diff --git a/tools/JavaScript Runtimes.md b/tools/JavaScript Runtimes.md new file mode 100644 index 0000000..9b7d225 --- /dev/null +++ b/tools/JavaScript Runtimes.md @@ -0,0 +1,342 @@ +--- +title: JavaScript Runtimes +aliases: + - JS Runtimes + - Node vs Deno vs Bun +tags: + - tool + - comparison + - javascript + - typescript + - runtime +type: comparison +status: complete +created: 2025-12-04 +--- + +# JavaScript Runtimes + +Server-side JavaScript/TypeScript execution environments: Node.js, Deno, and Bun. + +## Overview + +| Aspect | Node.js | Deno | Bun | +|--------|---------|------|-----| +| First release | 2009 | 2020 | 2022 | +| Created by | Ryan Dahl | Ryan Dahl | Jarred Sumner | +| Written in | C++ | Rust | Zig | +| Engine | V8 | V8 | JavaScriptCore | +| TypeScript | Via transpiler | Native | Native | +| Package manager | npm/yarn/pnpm | Built-in (+ npm) | Built-in (+ npm) | +| Security model | Full access | Permissions-based | Full access | +| Config files | package.json, tsconfig, etc. | deno.json (optional) | bunfig.toml (optional) | + +--- + +## Node.js + +The original server-side JavaScript runtime. Massive ecosystem. + +### Key Characteristics + +- **Mature ecosystem** — npm has 2M+ packages +- **CommonJS + ESM** — Supports both module systems +- **Event loop** — libuv-based async I/O +- **Wide adoption** — Most tutorials, SO answers, production deployments + +### Package Management + +```bash +# npm (default) +npm install express + +# yarn +yarn add express + +# pnpm (disk-efficient) +pnpm add express +``` + +### Module Systems + +```javascript +// CommonJS (traditional) +const express = require('express'); +module.exports = { handler }; + +// ESM (modern, add "type": "module" to package.json) +import express from 'express'; +export { handler }; +``` + +### Considerations + +- Fragmented tooling (bundlers, transpilers, test runners) +- `node_modules` can be massive +- TypeScript requires build step +- Permission model added in v20+ but not default + +--- + +## Deno + +"A secure runtime for JavaScript and TypeScript" — addresses Node.js design regrets. + +### Key Characteristics + +- **Secure by default** — Explicit permissions for fs, net, env +- **TypeScript native** — No config needed +- **Standard library** — Audited, versioned std lib +- **URL imports** — Import directly from URLs +- **Web-compatible** — Uses web platform APIs (fetch, Web Crypto) + +### Permissions Model + +```bash +# No permissions (sandboxed) +deno run script.ts + +# Explicit permissions +deno run --allow-read --allow-net script.ts + +# Allow specific paths/hosts +deno run --allow-read=/data --allow-net=api.example.com script.ts + +# All permissions (not recommended) +deno run --allow-all script.ts +``` + +### Import Styles + +```typescript +// URL imports (original style) +import { serve } from "https://deno.land/std@0.208.0/http/server.ts"; + +// Import maps (deno.json) +import { serve } from "@std/http"; + +// npm compatibility +import express from "npm:express@4"; +``` + +### Built-in Tools + +```bash +deno fmt # Formatter +deno lint # Linter +deno test # Test runner +deno bench # Benchmarking +deno compile # Create executable +deno doc # Documentation generator +deno jupyter # Jupyter kernel +``` + +### Considerations + +- Smaller ecosystem (but npm compat helps) +- Different mental model from Node +- Some npm packages don't work +- Less hosting/deployment support + +--- + +## Bun + +"All-in-one JavaScript runtime & toolkit" — focused on speed. + +### Key Characteristics + +- **Speed** — JavaScriptCore + Zig = fast startup, fast runtime +- **All-in-one** — Runtime + bundler + transpiler + package manager +- **Node compatible** — Drop-in replacement for many Node apps +- **Native TypeScript** — No build step +- **Fast package manager** — Installs packages 10-100x faster than npm + +### Performance Focus + +``` +Bun optimizes for: +├── Startup time (faster than Node/Deno) +├── Runtime performance (JSC optimizations) +├── Package install speed +├── Bundling speed +└── Test execution speed +``` + +### Package Management + +```bash +bun install # Install from package.json (very fast) +bun add express # Add package +bun remove lodash # Remove package +bun update # Update packages +``` + +### Built-in APIs + +```typescript +// Fast file I/O +const file = Bun.file("./data.json"); +const contents = await file.json(); + +// Built-in SQLite +import { Database } from "bun:sqlite"; +const db = new Database("mydb.sqlite"); + +// Fast HTTP server +Bun.serve({ + port: 3000, + fetch(req) { + return new Response("Hello!"); + }, +}); +``` + +### Built-in Tools + +```bash +bun run script.ts # Run (no build needed) +bun test # Test runner (Jest-compatible) +bun build # Bundler +bunx # Like npx +``` + +### Considerations + +- Youngest runtime, still maturing +- Some Node APIs not yet implemented +- Smaller community +- Less battle-tested in production + +--- + +## Feature Comparison + +| Feature | Node.js | Deno | Bun | +|---------|:-------:|:----:|:---:| +| TypeScript native | | ✅ | ✅ | +| Permissions system | Opt-in | ✅ Default | | +| npm compatibility | ✅ Native | ✅ Via npm: | ✅ Native | +| Built-in test runner | ✅ (v18+) | ✅ | ✅ | +| Built-in bundler | | | ✅ | +| Built-in formatter | | ✅ | | +| Web APIs (fetch, etc.) | ✅ (v18+) | ✅ | ✅ | +| Single executable | ✅ (experimental) | ✅ | ✅ | +| JSX support | Via bundler | ✅ | ✅ | +| Watch mode | ✅ --watch | ✅ --watch | ✅ --watch | + +--- + +## Performance Comparison + +| Benchmark | Node.js | Deno | Bun | +|-----------|---------|------|-----| +| Startup time | Baseline | ~Similar | Faster | +| HTTP throughput | Good | Good | Excellent | +| Package install | Slow (npm) | Medium | Very fast | +| Cold start | Medium | Medium | Fast | +| Memory usage | Medium | Medium | Lower | + +*Note: Benchmarks vary by workload. Always test your specific use case.* + +--- + +## Ecosystem & Tooling + +### Node.js Ecosystem + +``` +Node.js requires external tools: +├── TypeScript → tsc, ts-node, tsx +├── Bundler → webpack, esbuild, rollup, vite +├── Test → jest, mocha, vitest +├── Lint → eslint +├── Format → prettier +└── Types → @types/* packages +``` + +### Deno Ecosystem + +``` +Deno includes: +├── TypeScript → built-in +├── Bundler → deno bundle (deprecated, use esbuild) +├── Test → deno test +├── Lint → deno lint +├── Format → deno fmt +└── Types → built-in +``` + +### Bun Ecosystem + +``` +Bun includes: +├── TypeScript → built-in +├── Bundler → bun build +├── Test → bun test +├── Lint → use external (eslint, biome) +├── Format → use external (prettier, biome) +└── Types → built-in +``` + +--- + +## Migration Paths + +### Node → Deno + +```typescript +// Change imports +// Before (Node) +import fs from 'fs'; +import path from 'path'; + +// After (Deno) +import * as fs from "@std/fs"; +import * as path from "@std/path"; +// Or use Node compat +import fs from "node:fs"; +``` + +### Node → Bun + +```bash +# Often just works +bun run your-node-script.ts + +# Replace npm scripts in package.json +bun run dev +bun test +``` + +--- + +## Decision Guide + +| Use Case | Recommendation | +|----------|----------------| +| Enterprise, maximum stability | Node.js | +| Security-critical, sandboxing needed | Deno | +| Maximum performance, fast iteration | Bun | +| Existing Node.js codebase | Node.js or Bun | +| New TypeScript project | Deno or Bun | +| Edge/serverless functions | Bun or Deno | +| Scripts and tooling | Bun (fast startup) | +| Learning JavaScript server-side | Node.js (most resources) | + +### Summary + +| Runtime | Choose When | +|---------|-------------| +| **Node.js** | Need stability, ecosystem breadth, hiring, documentation | +| **Deno** | Want security-first, clean design, TypeScript-native, std lib | +| **Bun** | Want speed, all-in-one tooling, minimal config, Node compat | + +--- + +## Related + +- [[Build Systems]] +- [[TypeScript]] +- [[Deployment]] +- [[Testing Frameworks]] diff --git a/tools/Kubernetes.md b/tools/Kubernetes.md new file mode 100644 index 0000000..15a5319 --- /dev/null +++ b/tools/Kubernetes.md @@ -0,0 +1,618 @@ +--- +title: Kubernetes +aliases: + - K8s + - Container Orchestration +tags: + - tool + - comparison + - containers + - kubernetes + - devops +type: reference +status: complete +created: 2025-12-04 +--- + +# Kubernetes + +Container orchestration platform and its lightweight distributions. + +## Overview + +| Distribution | Type | Use Case | Resources | +|--------------|------|----------|-----------| +| Kubernetes (K8s) | Full | Production | High | +| K3s | Lightweight | Edge, IoT, dev | Low | +| kind | Docker-based | CI/CD, testing | Medium | +| minikube | Local dev | Learning, dev | Medium | +| MicroK8s | Snap-based | Dev, edge | Low-Medium | +| Docker Desktop | Built-in | Local dev | Medium | +| Rancher Desktop | GUI + K3s | Local dev | Medium | + +--- + +## Kubernetes Architecture + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ Control Plane │ +│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ +│ │ API Server │ │ Scheduler │ │ Controller │ │ +│ │ │ │ │ │ Manager │ │ +│ └──────┬──────┘ └─────────────┘ └─────────────┘ │ +│ │ │ +│ ┌──────┴──────┐ │ +│ │ etcd │ │ +│ │ (state) │ │ +│ └─────────────┘ │ +└─────────────────────────────────────────────────────────────────┘ + │ + ┌───────────────┼───────────────┐ + ▼ ▼ ▼ +┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ +│ Worker Node │ │ Worker Node │ │ Worker Node │ +│ ┌───────────┐ │ │ ┌───────────┐ │ │ ┌───────────┐ │ +│ │ kubelet │ │ │ │ kubelet │ │ │ │ kubelet │ │ +│ ├───────────┤ │ │ ├───────────┤ │ │ ├───────────┤ │ +│ │kube-proxy │ │ │ │kube-proxy │ │ │ │kube-proxy │ │ +│ ├───────────┤ │ │ ├───────────┤ │ │ ├───────────┤ │ +│ │ Container │ │ │ │ Container │ │ │ │ Container │ │ +│ │ Runtime │ │ │ │ Runtime │ │ │ │ Runtime │ │ +│ └───────────┘ │ │ └───────────┘ │ │ └───────────┘ │ +│ ┌───┐ ┌───┐ │ │ ┌───┐ ┌───┐ │ │ ┌───┐ ┌───┐ │ +│ │Pod│ │Pod│ │ │ │Pod│ │Pod│ │ │ │Pod│ │Pod│ │ +│ └───┘ └───┘ │ │ └───┘ └───┘ │ │ └───┘ └───┘ │ +└─────────────────┘ └─────────────────┘ └─────────────────┘ +``` + +--- + +## K3s + +Lightweight Kubernetes by Rancher. Single binary, production-ready. + +### Key Features + +- **Single binary** — ~70MB, includes everything +- **Low resources** — 512MB RAM minimum +- **Batteries included** — Traefik, ServiceLB, local-path storage +- **SQLite/etcd** — SQLite default, etcd optional +- **ARM support** — Raspberry Pi, edge devices + +### Installation + +```bash +# Server (control plane) +curl -sfL https://get.k3s.io | sh - + +# Check status +sudo k3s kubectl get nodes +sudo systemctl status k3s + +# Get kubeconfig +sudo cat /etc/rancher/k3s/k3s.yaml + +# Agent (worker node) +curl -sfL https://get.k3s.io | K3S_URL=https://server:6443 \ + K3S_TOKEN=$(sudo cat /var/lib/rancher/k3s/server/node-token) sh - +``` + +### Configuration + +```bash +# Install with options +curl -sfL https://get.k3s.io | sh -s - \ + --disable traefik \ + --disable servicelb \ + --write-kubeconfig-mode 644 + +# Or via config file +# /etc/rancher/k3s/config.yaml +write-kubeconfig-mode: "0644" +disable: + - traefik + - servicelb +cluster-init: true +``` + +### K3s vs K8s + +| Aspect | K3s | K8s | +|--------|-----|-----| +| Binary size | ~70MB | ~300MB+ | +| RAM (min) | 512MB | 2GB+ | +| Default storage | SQLite | etcd | +| Default ingress | Traefik | None | +| Default LB | ServiceLB | None | +| HA setup | Built-in | Complex | +| Edge/IoT | ✅ Excellent | ❌ Heavy | + +--- + +## kind (Kubernetes in Docker) + +Run Kubernetes clusters using Docker containers as nodes. + +### Key Features + +- **Docker-based** — Nodes are containers +- **Multi-node** — Simulate real clusters +- **CI/CD friendly** — Fast creation/deletion +- **Config-driven** — YAML cluster definitions + +### Installation + +```bash +# macOS +brew install kind + +# Linux +curl -Lo ./kind https://kind.sigs.k8s.io/dl/latest/kind-linux-amd64 +chmod +x ./kind +sudo mv ./kind /usr/local/bin/kind + +# Windows +choco install kind +``` + +### Basic Usage + +```bash +# Create cluster +kind create cluster + +# Create named cluster +kind create cluster --name my-cluster + +# List clusters +kind get clusters + +# Delete cluster +kind delete cluster --name my-cluster + +# Get kubeconfig +kind get kubeconfig --name my-cluster +``` + +### Multi-Node Cluster + +```yaml +# kind-config.yaml +kind: Cluster +apiVersion: kind.x-k8s.io/v1alpha4 +nodes: + - role: control-plane + kubeadmConfigPatches: + - | + kind: InitConfiguration + nodeRegistration: + kubeletExtraArgs: + node-labels: "ingress-ready=true" + extraPortMappings: + - containerPort: 80 + hostPort: 80 + protocol: TCP + - containerPort: 443 + hostPort: 443 + protocol: TCP + - role: worker + - role: worker + - role: worker +``` + +```bash +kind create cluster --config kind-config.yaml +``` + +### Load Images + +```bash +# Load local image into kind +docker build -t myapp:latest . +kind load docker-image myapp:latest --name my-cluster + +# Now use in pods +# image: myapp:latest +# imagePullPolicy: Never +``` + +--- + +## minikube + +Local Kubernetes for learning and development. + +### Key Features + +- **Multiple drivers** — Docker, VirtualBox, Hyper-V, etc. +- **Addons** — Dashboard, ingress, metrics-server +- **Profiles** — Multiple clusters +- **LoadBalancer** — Tunnel support + +### Installation + +```bash +# macOS +brew install minikube + +# Linux +curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 +sudo install minikube-linux-amd64 /usr/local/bin/minikube + +# Windows +choco install minikube +``` + +### Basic Usage + +```bash +# Start cluster +minikube start + +# Start with specific driver +minikube start --driver=docker + +# Start with resources +minikube start --cpus=4 --memory=8192 + +# Status +minikube status + +# Stop/delete +minikube stop +minikube delete +``` + +### Addons + +```bash +# List addons +minikube addons list + +# Enable addons +minikube addons enable ingress +minikube addons enable dashboard +minikube addons enable metrics-server + +# Access dashboard +minikube dashboard +``` + +### Access Services + +```bash +# Get service URL +minikube service myservice --url + +# Tunnel for LoadBalancer +minikube tunnel + +# SSH into node +minikube ssh +``` + +--- + +## MicroK8s + +Canonical's lightweight Kubernetes. Snap-based. + +### Installation + +```bash +# Install +sudo snap install microk8s --classic + +# Add to group +sudo usermod -a -G microk8s $USER + +# Check status +microk8s status + +# Enable addons +microk8s enable dns dashboard ingress + +# Use kubectl +microk8s kubectl get nodes +# Or alias +alias kubectl='microk8s kubectl' +``` + +### Features + +```bash +# Enable common addons +microk8s enable dns +microk8s enable dashboard +microk8s enable ingress +microk8s enable storage +microk8s enable registry +microk8s enable prometheus + +# HA cluster +microk8s add-node # On first node +# Run output command on other nodes +``` + +--- + +## Docker Desktop Kubernetes + +Built into Docker Desktop for Mac/Windows. + +### Enable + +1. Docker Desktop → Settings → Kubernetes +2. Enable Kubernetes +3. Apply & Restart + +```bash +# Switch context +kubectl config use-context docker-desktop + +# Verify +kubectl get nodes +``` + +--- + +## Comparison Matrix + +| Feature | K3s | kind | minikube | MicroK8s | +|---------|:---:|:----:|:--------:|:--------:| +| Production ready | ✅ | ❌ | ❌ | ✅ | +| Multi-node | ✅ | ✅ | ❌ | ✅ | +| CI/CD friendly | ✅ | ✅ | ⚠️ | ⚠️ | +| ARM support | ✅ | ✅ | ✅ | ✅ | +| Ingress included | ✅ | ❌ | Addon | Addon | +| Dashboard | ❌ | ❌ | Addon | Addon | +| Resource usage | Low | Medium | Medium | Low | +| Setup speed | Fast | Fast | Medium | Fast | +| Cluster upgrade | ✅ | Recreate | ✅ | ✅ | + +--- + +## Essential kubectl Commands + +### Cluster Info + +```bash +# Cluster info +kubectl cluster-info +kubectl get nodes +kubectl get namespaces + +# Context management +kubectl config get-contexts +kubectl config use-context my-cluster +kubectl config current-context +``` + +### Workloads + +```bash +# Pods +kubectl get pods +kubectl get pods -A # All namespaces +kubectl describe pod my-pod +kubectl logs my-pod +kubectl logs -f my-pod # Follow +kubectl exec -it my-pod -- /bin/sh + +# Deployments +kubectl get deployments +kubectl create deployment nginx --image=nginx +kubectl scale deployment nginx --replicas=3 +kubectl rollout status deployment nginx +kubectl rollout undo deployment nginx + +# Services +kubectl get services +kubectl expose deployment nginx --port=80 --type=LoadBalancer +kubectl port-forward svc/nginx 8080:80 +``` + +### Apply/Delete + +```bash +# Apply manifests +kubectl apply -f deployment.yaml +kubectl apply -f ./manifests/ +kubectl apply -k ./kustomize/ + +# Delete +kubectl delete -f deployment.yaml +kubectl delete pod my-pod +kubectl delete deployment nginx +``` + +### Debugging + +```bash +# Events +kubectl get events --sort-by='.lastTimestamp' + +# Resource usage +kubectl top nodes +kubectl top pods + +# Describe for details +kubectl describe node my-node +kubectl describe pod my-pod + +# Debug container +kubectl debug my-pod -it --image=busybox +``` + +--- + +## Kubernetes Manifests + +### Deployment + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: myapp + labels: + app: myapp +spec: + replicas: 3 + selector: + matchLabels: + app: myapp + template: + metadata: + labels: + app: myapp + spec: + containers: + - name: myapp + image: myapp:1.0.0 + ports: + - containerPort: 8080 + resources: + requests: + memory: "128Mi" + cpu: "100m" + limits: + memory: "256Mi" + cpu: "500m" + livenessProbe: + httpGet: + path: /health + port: 8080 + initialDelaySeconds: 10 + periodSeconds: 10 + readinessProbe: + httpGet: + path: /ready + port: 8080 + initialDelaySeconds: 5 + periodSeconds: 5 +``` + +### Service + +```yaml +apiVersion: v1 +kind: Service +metadata: + name: myapp +spec: + selector: + app: myapp + ports: + - port: 80 + targetPort: 8080 + type: ClusterIP # or LoadBalancer, NodePort +``` + +### Ingress + +```yaml +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: myapp + annotations: + nginx.ingress.kubernetes.io/rewrite-target: / +spec: + ingressClassName: nginx + rules: + - host: myapp.example.com + http: + paths: + - path: / + pathType: Prefix + backend: + service: + name: myapp + port: + number: 80 + tls: + - hosts: + - myapp.example.com + secretName: myapp-tls +``` + +### ConfigMap & Secret + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: myapp-config +data: + DATABASE_HOST: "postgres" + LOG_LEVEL: "info" +--- +apiVersion: v1 +kind: Secret +metadata: + name: myapp-secrets +type: Opaque +stringData: + DATABASE_PASSWORD: "supersecret" +``` + +--- + +## Helm + +Package manager for Kubernetes. + +```bash +# Install Helm +brew install helm + +# Add repo +helm repo add bitnami https://charts.bitnami.com/bitnami +helm repo update + +# Search charts +helm search repo nginx + +# Install chart +helm install my-nginx bitnami/nginx +helm install my-nginx bitnami/nginx -f values.yaml +helm install my-nginx bitnami/nginx --set service.type=LoadBalancer + +# List releases +helm list + +# Upgrade +helm upgrade my-nginx bitnami/nginx --set replicaCount=3 + +# Uninstall +helm uninstall my-nginx + +# Create chart +helm create mychart +``` + +--- + +## Decision Guide + +| Use Case | Recommendation | +|----------|----------------| +| Learning Kubernetes | minikube or Docker Desktop | +| Local development | kind or minikube | +| CI/CD testing | kind | +| Edge/IoT production | K3s | +| Raspberry Pi cluster | K3s | +| Single-node production | K3s or MicroK8s | +| Multi-node production | K3s or full K8s | +| Quick testing | kind | +| GUI preference | Rancher Desktop | + +--- + +## Related + +- [[Container Runtimes]] +- [[Container Tools]] +- [[Deployment]] diff --git a/tools/Language Runtimes.md b/tools/Language Runtimes.md new file mode 100644 index 0000000..1dc76c1 --- /dev/null +++ b/tools/Language Runtimes.md @@ -0,0 +1,516 @@ +--- +title: Language Runtimes +aliases: + - Interpreters + - VMs + - Runtime Comparison +tags: + - tool + - comparison + - runtime +type: comparison +status: complete +created: 2025-12-04 +--- + +# Language Runtimes + +Alternative runtimes and implementations for Python, Ruby, Java, .NET, and PHP. + +## Overview + +| Language | Reference | Alternatives | +|----------|-----------|--------------| +| Python | CPython | PyPy, GraalPy, Jython, Cython | +| Ruby | MRI/CRuby | JRuby, TruffleRuby, mruby | +| Java | OpenJDK | GraalVM, Azul, Amazon Corretto, Eclipse Temurin | +| .NET | .NET (Core) | Mono, NativeAOT, .NET Framework | +| PHP | Zend Engine | Swoole, FrankenPHP, RoadRunner | + +--- + +## Python Runtimes + +### CPython + +The reference implementation. Written in C. + +| Aspect | Details | +|--------|---------| +| Type | Interpreter + bytecode VM | +| GIL | Yes (limits parallelism) | +| Performance | Baseline | +| Ecosystem | 100% compatible | +| Use case | Default, general purpose | + +```bash +# Standard Python +python3 script.py +``` + +### PyPy + +JIT-compiled Python. Much faster for long-running code. + +| Aspect | Details | +|--------|---------| +| Type | JIT compiler | +| GIL | Yes | +| Performance | 2-10x faster than CPython | +| Compatibility | ~99% (C extensions vary) | +| Use case | CPU-bound, long-running | + +```bash +# Install +# macOS: brew install pypy3 +# Ubuntu: apt install pypy3 + +pypy3 script.py + +# Compatibility notes +# - Most pure Python works +# - C extensions need PyPy versions +# - numpy, scipy have PyPy builds +``` + +### GraalPy + +GraalVM's Python implementation. + +| Aspect | Details | +|--------|---------| +| Type | JIT (Truffle framework) | +| GIL | No (in native mode) | +| Performance | Competitive with PyPy | +| Interop | Java, JS, Ruby, R | +| Use case | Polyglot, Java integration | + +```bash +# Install GraalVM with Python +gu install python + +graalpython script.py + +# Java interop +from java.util import ArrayList +list = ArrayList() +list.add("Hello") +``` + +### Python Runtime Comparison + +| Runtime | Startup | Peak Perf | Memory | C Extensions | +|---------|---------|-----------|--------|--------------| +| CPython | Fast | Baseline | Low | ✅ Native | +| PyPy | Medium | 2-10x | Higher | ⚠️ Limited | +| GraalPy | Slow | High | Higher | ⚠️ Limited | + +--- + +## Ruby Runtimes + +### MRI / CRuby + +Matz's Ruby Interpreter. The reference implementation. + +| Aspect | Details | +|--------|---------| +| Type | Interpreter + YARV bytecode | +| GIL | Yes (GVL) | +| JIT | YJIT (Ruby 3.1+), MJIT | +| Use case | Default, general purpose | + +```bash +# Standard Ruby with YJIT (Ruby 3.1+) +ruby --yjit script.rb + +# Check YJIT status +ruby --yjit -e "p RubyVM::YJIT.enabled?" +``` + +### JRuby + +Ruby on the JVM. + +| Aspect | Details | +|--------|---------| +| Type | JVM bytecode compiler | +| GIL | No (real threads) | +| Performance | Good for threaded apps | +| Interop | Full Java interop | +| Use case | Java integration, threading | + +```bash +# Install +# macOS: brew install jruby +# sdkman: sdk install jruby + +jruby script.rb + +# Java interop +require 'java' +java_import 'java.util.ArrayList' + +list = ArrayList.new +list.add("Hello") +``` + +### TruffleRuby + +GraalVM's Ruby implementation. + +| Aspect | Details | +|--------|---------| +| Type | JIT (Truffle framework) | +| GIL | No | +| Performance | Fastest (warmed up) | +| Interop | Java, JS, Python, R | +| Use case | Performance critical | + +```bash +# Install via GraalVM +gu install ruby + +truffleruby script.rb + +# Or via ruby-build/rbenv +rbenv install truffleruby-23.1.0 +``` + +### Ruby Runtime Comparison + +| Runtime | Startup | Peak Perf | Threading | C Extensions | +|---------|---------|-----------|-----------|--------------| +| MRI | Fast | Baseline | GVL | ✅ Native | +| MRI+YJIT | Fast | 1.5-2x | GVL | ✅ Native | +| JRuby | Slow | Good | ✅ Real | ⚠️ Limited | +| TruffleRuby | Slow | Fastest | ✅ Real | ⚠️ Limited | + +--- + +## Java/JVM Runtimes + +### OpenJDK + +The open-source reference implementation. + +| Aspect | Details | +|--------|---------| +| Type | JIT (C2, Graal) | +| License | GPL v2 + Classpath | +| Vendors | Many (builds differ slightly) | +| Use case | Default, general purpose | + +### Major OpenJDK Distributions + +| Distribution | Vendor | LTS Support | Notes | +|--------------|--------|-------------|-------| +| Eclipse Temurin | Adoptium | Free | Community standard | +| Amazon Corretto | Amazon | Free | AWS optimized | +| Azul Zulu | Azul | Free/Paid | Wide platform support | +| Microsoft Build | Microsoft | Free | Azure optimized | +| Oracle OpenJDK | Oracle | 6 months | Reference builds | +| Oracle JDK | Oracle | Paid LTS | Commercial license | +| Red Hat Build | Red Hat | RHEL subscription | Enterprise | +| SAP Machine | SAP | Free | SAP workloads | + +```bash +# Install via SDKMAN +sdk list java # See all distributions +sdk install java 21.0.1-tem # Temurin +sdk install java 21.0.1-amzn # Corretto +sdk install java 21.0.1-zulu # Azul Zulu +``` + +### GraalVM + +Polyglot VM with native compilation. + +| Aspect | Details | +|--------|---------| +| Type | JIT + AOT (Native Image) | +| Languages | Java, JS, Python, Ruby, R, WASM | +| Native Image | AOT compile to binary | +| Use case | Polyglot, fast startup, cloud | + +```bash +# Install +sdk install java 21.0.1-graal + +# Native image compilation +native-image -jar myapp.jar + +# Run polyglot +js --jvm script.js +``` + +### Native Image (GraalVM) + +Ahead-of-time compilation to native binaries. + +| Aspect | Without Native Image | With Native Image | +|--------|---------------------|-------------------| +| Startup | 1-5 seconds | 10-50 ms | +| Memory | Higher (JVM) | Lower | +| Peak perf | Higher (JIT) | Lower | +| Reflection | Full | Requires config | + +```bash +# Compile to native +native-image --no-fallback -jar app.jar -o app + +# Result: standalone binary +./app +``` + +### JVM Decision Guide + +| Need | Distribution | +|------|--------------| +| Community standard | Eclipse Temurin | +| AWS deployment | Amazon Corretto | +| Long-term support | Azul Zulu or Temurin | +| Native compilation | GraalVM | +| Enterprise support | Oracle JDK or Red Hat | +| Microservices | GraalVM Native Image | + +--- + +## .NET Runtimes + +### .NET (Core → Modern .NET) + +The modern, cross-platform runtime. + +| Aspect | Details | +|--------|---------| +| Type | JIT (RyuJIT) + AOT | +| Platforms | Windows, Linux, macOS | +| Current | .NET 8 (LTS), .NET 9 | +| Use case | Everything (default) | + +```bash +# Install +# macOS: brew install dotnet +# Windows: winget install Microsoft.DotNet.SDK.8 + +dotnet new console +dotnet run +``` + +### .NET Framework + +Legacy Windows-only runtime. + +| Aspect | Details | +|--------|---------| +| Type | JIT | +| Platforms | Windows only | +| Current | 4.8.1 (final) | +| Status | Maintenance mode | +| Use case | Legacy Windows apps | + +### Mono + +Open-source .NET Framework implementation. + +| Aspect | Details | +|--------|---------| +| Type | JIT + AOT | +| Platforms | All (incl. mobile, WASM) | +| Use case | Xamarin, Unity, legacy | + +### NativeAOT + +Ahead-of-time compilation for .NET. + +| Aspect | Details | +|--------|---------| +| Type | AOT (no JIT) | +| Platforms | Windows, Linux, macOS | +| Benefits | Fast startup, smaller, no runtime | +| Trade-offs | Larger binary, no reflection | + +```bash +# Publish with NativeAOT +dotnet publish -c Release -r linux-x64 \ + -p:PublishAot=true + +# Result: self-contained native binary +``` + +### .NET Comparison + +| Runtime | Startup | Size | Reflection | Platforms | +|---------|---------|------|------------|-----------| +| .NET (JIT) | Medium | Medium | ✅ Full | Cross-platform | +| NativeAOT | Fast | Larger | ⚠️ Limited | Cross-platform | +| Mono | Medium | Small | ✅ Full | All | +| Framework | Medium | N/A | ✅ Full | Windows | + +### .NET Decision Guide + +| Need | Runtime | +|------|---------| +| General purpose | .NET 8+ | +| Fast startup (cloud) | NativeAOT | +| Mobile apps | Mono via MAUI | +| Unity games | Mono | +| Legacy Windows | .NET Framework | +| WebAssembly | Blazor (Mono) | + +--- + +## PHP Runtimes + +### Zend Engine (Standard PHP) + +The reference implementation. + +| Aspect | Details | +|--------|---------| +| Type | Interpreter + OPcache | +| Model | Process-per-request (PHP-FPM) | +| Performance | Baseline | +| Use case | Default, traditional | + +```bash +# PHP-FPM (FastCGI Process Manager) +php-fpm + +# OPcache enabled by default in production +php -d opcache.enable_cli=1 script.php + +# JIT (PHP 8+) +php -d opcache.jit=1255 script.php +``` + +### Swoole + +Coroutine-based async runtime. + +| Aspect | Details | +|--------|---------| +| Type | Extension (C) | +| Model | Event loop, coroutines | +| Performance | 5-10x traditional | +| Use case | High-concurrency, real-time | + +```php +on("request", function (Request $request, Response $response) { + $response->header("Content-Type", "text/plain"); + $response->end("Hello World"); +}); + +$server->start(); +``` + +```bash +# Install +pecl install swoole + +# Run +php server.php +# Long-running process, handles many concurrent requests +``` + +### FrankenPHP + +Modern PHP app server built on Caddy. + +| Aspect | Details | +|--------|---------| +| Type | Go + C (embedded PHP) | +| Model | Worker mode (persistent) | +| Features | HTTP/3, Early Hints, real-time | +| Use case | Modern deployment | + +```bash +# Run Symfony app +frankenphp php-server --root public/ + +# Worker mode (persistent processes) +frankenphp php-server --worker public/index.php +``` + +### RoadRunner + +High-performance PHP application server. + +| Aspect | Details | +|--------|---------| +| Type | Go + PHP workers | +| Model | Long-running workers | +| Features | gRPC, queues, WebSocket | +| Use case | Microservices, high-load | + +```yaml +# .rr.yaml +server: + command: "php worker.php" + +http: + address: 0.0.0.0:8080 + pool: + num_workers: 4 +``` + +### PHP Runtime Comparison + +| Runtime | Model | Startup | Concurrency | Memory | +|---------|-------|---------|-------------|--------| +| PHP-FPM | Process/request | Per request | Low | Per process | +| Swoole | Coroutines | Once | Very high | Shared | +| FrankenPHP | Workers | Once | High | Workers | +| RoadRunner | Workers | Once | High | Workers | + +### PHP Decision Guide + +| Need | Runtime | +|------|---------| +| Traditional hosting | PHP-FPM | +| High concurrency | Swoole | +| Modern deployment | FrankenPHP | +| Microservices | RoadRunner | +| Real-time features | Swoole | +| Easy Caddy integration | FrankenPHP | + +--- + +## Cross-Language Comparison + +### Performance Profile + +| Language | Reference | Fastest Alternative | +|----------|-----------|---------------------| +| Python | CPython | PyPy (2-10x) | +| Ruby | MRI | TruffleRuby (warm) | +| Java | OpenJDK | GraalVM Native Image (startup) | +| .NET | .NET JIT | NativeAOT (startup) | +| PHP | PHP-FPM | Swoole (concurrency) | + +### When to Use Alternatives + +| Scenario | Consider | +|----------|----------| +| Long-running compute | PyPy, JRuby, TruffleRuby | +| Fast startup needed | GraalVM Native, NativeAOT | +| High concurrency | JRuby, Swoole | +| Polyglot project | GraalVM | +| Microservices | GraalVM Native, NativeAOT, FrankenPHP | + +--- + +## Related + +- [[JavaScript Runtimes]] +- [[WebAssembly Runtimes]] +- [[Systems Language Performance]] +- [[Version Managers]] diff --git a/tools/Linux Distributions.md b/tools/Linux Distributions.md new file mode 100644 index 0000000..b9f190a --- /dev/null +++ b/tools/Linux Distributions.md @@ -0,0 +1,441 @@ +--- +title: Linux Distributions +aliases: + - Linux Distros + - Distros +tags: + - tool + - comparison + - linux + - operating-system +type: comparison +status: complete +created: 2025-12-04 +--- + +# Linux Distributions + +Major Linux distributions, their families, and use cases. + +## Overview + +| Distro | Family | Package Manager | Release Model | Best For | +|--------|--------|-----------------|---------------|----------| +| Ubuntu | Debian | apt/deb | Fixed (LTS) | Desktop, servers, beginners | +| Debian | Debian | apt/deb | Fixed (Stable) | Servers, stability | +| Fedora | Red Hat | dnf/rpm | Fixed (6mo) | Developers, new tech | +| RHEL | Red Hat | dnf/rpm | Fixed (LTS) | Enterprise servers | +| Arch | Independent | pacman | Rolling | Power users | +| openSUSE | SUSE | zypper/rpm | Leap (fixed) / Tumbleweed (rolling) | Enterprise, KDE | +| Alpine | Independent | apk | Fixed | Containers, minimal | +| NixOS | Independent | nix | Rolling | Reproducible, declarative | + +--- + +## Distribution Families + +``` + ┌─────────────┐ + │ Debian │ + └──────┬──────┘ + ┌───────────────┼───────────────┐ + ▼ ▼ ▼ + ┌─────────┐ ┌──────────┐ ┌─────────┐ + │ Ubuntu │ │ Mint │ │ Kali │ + └────┬────┘ └──────────┘ └─────────┘ + │ + ┌─────┴─────┐ + ▼ ▼ +┌─────────┐ ┌─────────┐ +│ Pop!_OS │ │ Kubuntu │ +└─────────┘ └─────────┘ + + + ┌─────────────┐ + │ Red Hat │ + └──────┬──────┘ + ┌────────┼────────┐ + ▼ ▼ ▼ +┌──────┐ ┌──────┐ ┌────────┐ +│ RHEL │ │Fedora│ │CentOS* │ +└──┬───┘ └──────┘ └────────┘ + │ + ├──▶ AlmaLinux + ├──▶ Rocky Linux + └──▶ Oracle Linux + + + ┌─────────────┐ + │ SUSE │ + └──────┬──────┘ + ┌────────┴────────┐ + ▼ ▼ +┌────────┐ ┌──────────┐ +│ SLES │ │ openSUSE │ +└────────┘ └──────────┘ + + + ┌─────────────┐ + │ Arch │ + └──────┬──────┘ + ┌────────┼────────┐ + ▼ ▼ ▼ +┌───────┐ ┌───────┐ ┌─────────┐ +│Manjaro│ │EndeavOS││SteamOS 3│ +└───────┘ └───────┘ └─────────┘ +``` + +*CentOS Stream continues, CentOS Linux ended* + +--- + +## Package Managers by Family + +| Family | Package Format | CLI Tool | GUI Tool | +|--------|---------------|----------|----------| +| Debian | .deb | apt, dpkg | Synaptic, GNOME Software | +| Red Hat | .rpm | dnf, yum, rpm | GNOME Software, dnfdragora | +| SUSE | .rpm | zypper, rpm | YaST | +| Arch | .pkg.tar.zst | pacman | pamac | +| Alpine | .apk | apk | — | +| NixOS | .nar (closures) | nix | — | + +--- + +## Debian Family + +### Debian + +The universal operating system. Rock-solid stability. + +| Aspect | Details | +|--------|---------| +| Release cycle | ~2 years, 5 year support | +| Current | Debian 12 "Bookworm" | +| Branches | Stable, Testing, Unstable (Sid) | +| Philosophy | Free software, stability | + +```bash +# Package management +apt update # Update package lists +apt upgrade # Upgrade packages +apt install nginx # Install package +apt remove nginx # Remove package +apt search nginx # Search packages +apt show nginx # Package info + +# Low-level +dpkg -i package.deb # Install .deb file +dpkg -l # List installed +dpkg -L nginx # List package files +``` + +### Ubuntu + +Debian-based, user-friendly, most popular desktop Linux. + +| Aspect | Details | +|--------|---------| +| Release cycle | 6 months (April, October) | +| LTS releases | Every 2 years, 5 year support (10 with Pro) | +| Current LTS | Ubuntu 24.04 "Noble Numbat" | +| Flavors | Ubuntu, Kubuntu, Xubuntu, Ubuntu Server | + +```bash +# Same as Debian plus: +snap install code # Snap packages +add-apt-repository ppa:... # PPAs (personal package archives) +do-release-upgrade # Upgrade to next release +ubuntu-drivers autoinstall # Install drivers +``` + +### Linux Mint + +Ubuntu-based, traditional desktop experience. + +| Aspect | Details | +|--------|---------| +| Based on | Ubuntu LTS | +| Editions | Cinnamon, MATE, Xfce | +| Focus | Desktop usability, Windows migrants | + +--- + +## Red Hat Family + +### Fedora + +Cutting-edge features, developer-focused. + +| Aspect | Details | +|--------|---------| +| Release cycle | ~6 months | +| Support | 13 months per release | +| Editions | Workstation, Server, Silverblue (immutable) | +| Upstream | RHEL derives from Fedora | + +```bash +# Package management +dnf update # Update all +dnf install nginx # Install +dnf remove nginx # Remove +dnf search nginx # Search +dnf info nginx # Package info +dnf groupinstall "Development Tools" # Install group + +# Modules (versioned streams) +dnf module list nodejs +dnf module enable nodejs:18 +``` + +### RHEL (Red Hat Enterprise Linux) + +Enterprise Linux with commercial support. + +| Aspect | Details | +|--------|---------| +| Release cycle | ~3 years major, minor every 6 months | +| Support | 10 years (full) + extended | +| Use cases | Enterprise servers, production | +| Cost | Subscription-based | + +### AlmaLinux / Rocky Linux + +RHEL-compatible, community-driven replacements for CentOS. + +| Aspect | AlmaLinux | Rocky Linux | +|--------|-----------|-------------| +| Backed by | CloudLinux | Rocky Enterprise Software Foundation | +| Binary compat | 1:1 with RHEL | 1:1 with RHEL | +| Use case | CentOS replacement | CentOS replacement | + +--- + +## SUSE Family + +### openSUSE + +Community SUSE. Two variants. + +| Variant | Description | +|---------|-------------| +| Leap | Fixed release, SLES-based, stable | +| Tumbleweed | Rolling release, bleeding edge | + +```bash +# Package management +zypper refresh # Update repos +zypper update # Upgrade packages +zypper install nginx # Install +zypper remove nginx # Remove +zypper search nginx # Search +zypper info nginx # Info + +# Patterns (package groups) +zypper install -t pattern devel_basis + +# YaST (admin tool) +yast # TUI configuration tool +``` + +### SLES (SUSE Linux Enterprise Server) + +Enterprise SUSE with commercial support. + +--- + +## Arch Family + +### Arch Linux + +DIY, bleeding-edge, minimalist. + +| Aspect | Details | +|--------|---------| +| Release | Rolling (continuous updates) | +| Philosophy | KISS, user-centric | +| Installation | Manual (archinstall helper available) | +| Wiki | Legendary documentation | + +```bash +# Package management +pacman -Syu # Full system upgrade +pacman -S nginx # Install +pacman -R nginx # Remove +pacman -Rs nginx # Remove with unused deps +pacman -Ss nginx # Search +pacman -Si nginx # Info +pacman -Qs nginx # Search installed + +# AUR (Arch User Repository) +# Use AUR helper like yay or paru +yay -S visual-studio-code-bin +paru -S spotify +``` + +### Manjaro + +Arch-based, user-friendly, delayed updates. + +| Aspect | Details | +|--------|---------| +| Based on | Arch Linux | +| Release | Rolling (with testing delay) | +| Editions | GNOME, KDE, Xfce | +| Difference | Easier install, curated repos | + +### EndeavourOS + +Arch-based, closer to pure Arch than Manjaro. + +--- + +## Specialty Distributions + +### Alpine Linux + +Minimal, security-focused. Docker standard. + +| Aspect | Details | +|--------|---------| +| Size | ~5MB base | +| libc | musl (not glibc) | +| Init | OpenRC | +| Use case | Containers, embedded | + +```bash +# Package management +apk update # Update index +apk upgrade # Upgrade packages +apk add nginx # Install +apk del nginx # Remove +apk search nginx # Search +apk info nginx # Info + +# Key difference: musl compatibility +# Some binaries compiled for glibc won't work +``` + +```dockerfile +# Docker - Alpine saves space +FROM alpine:3.19 # ~5MB +RUN apk add --no-cache nodejs # Minimal install + +# vs Debian +FROM debian:12 # ~120MB +RUN apt-get update && apt-get install -y nodejs +``` + +### NixOS + +Declarative, reproducible configuration. + +| Aspect | Details | +|--------|---------| +| Package manager | Nix (also runs on other OSes) | +| Configuration | Declarative (configuration.nix) | +| Rollbacks | Atomic, any generation | +| Reproducibility | Exact same system anywhere | + +```nix +# /etc/nixos/configuration.nix +{ config, pkgs, ... }: + +{ + environment.systemPackages = with pkgs; [ + vim + git + firefox + ]; + + services.nginx.enable = true; + + users.users.alice = { + isNormalUser = true; + extraGroups = [ "wheel" ]; + }; +} +``` + +```bash +# Apply configuration +sudo nixos-rebuild switch + +# Rollback +sudo nixos-rebuild switch --rollback + +# List generations +sudo nix-env --list-generations +``` + +--- + +## Release Models + +### Fixed Release + +``` +Fixed Release (Ubuntu LTS, RHEL, Debian Stable): + +Version 1.0 ─────────────────────▶ EOL + security patches only + +Version 2.0 ─────────────────────▶ EOL + security patches only + +Pros: Stability, predictability +Cons: Older packages +``` + +### Rolling Release + +``` +Rolling Release (Arch, openSUSE Tumbleweed, NixOS): + +─────────────────────────────────────────▶ + ▲ ▲ ▲ ▲ ▲ + │ │ │ │ │ + update update update update update + +Pros: Latest packages always +Cons: Potential breakage, constant updates +``` + +--- + +## Decision Guide + +| Use Case | Recommendation | +|----------|----------------| +| Desktop beginner | Ubuntu, Linux Mint, Pop!_OS | +| Developer workstation | Fedora, Ubuntu | +| Server (stability) | Debian, Ubuntu LTS, AlmaLinux | +| Enterprise (support) | RHEL, SLES, Ubuntu Pro | +| Containers | Alpine, Debian slim | +| Power user | Arch, EndeavourOS | +| Gaming | Pop!_OS, Nobara, Arch | +| Reproducible systems | NixOS | +| Learning Linux internals | Arch, Gentoo | +| KDE desktop | openSUSE, Fedora KDE, Kubuntu | +| Raspberry Pi | Raspberry Pi OS, Ubuntu | + +### Quick Picks + +| Profile | Distro | +|---------|--------| +| **Just works desktop** | Ubuntu or Linux Mint | +| **Developer** | Fedora or Ubuntu | +| **Server** | Debian or AlmaLinux | +| **Docker base** | Alpine | +| **I want control** | Arch | +| **Enterprise** | RHEL | +| **Declarative/reproducible** | NixOS | + +--- + +## Related + +- [[Package Managers]] +- [[WSL]] +- [[Shells]] +- [[Container Runtimes]] diff --git a/tools/Log Aggregation.md b/tools/Log Aggregation.md new file mode 100644 index 0000000..832a092 --- /dev/null +++ b/tools/Log Aggregation.md @@ -0,0 +1,568 @@ +--- +title: Log Aggregation +aliases: + - Logging Platforms + - Centralized Logging +tags: + - tool + - comparison + - observability + - logging +type: comparison +status: complete +created: 2025-12-04 +--- + +# Log Aggregation + +Centralized logging platforms for collecting, storing, and searching logs. + +## Overview + +| Tool | Type | License | Query Language | Best For | +|------|------|---------|----------------|----------| +| Loki | Log aggregation | OSS (AGPLv3) | LogQL | Grafana users, cost-effective | +| Elasticsearch | Search engine | OSS/Commercial | KQL, Lucene | Full-text search | +| Splunk | SIEM/Log platform | Commercial | SPL | Enterprise, security | +| Fluentd | Log collector | OSS (Apache 2) | — | Collection/routing | +| Fluent Bit | Log collector | OSS (Apache 2) | — | Lightweight collection | +| Vector | Log collector | OSS (MPL 2) | VRL | High-performance | +| Graylog | Log management | OSS/Commercial | — | Self-hosted alternative | + +--- + +## Architecture Patterns + +### Collection → Aggregation → Storage + +``` +┌─────────────────────────────────────────────────────────────┐ +│ Applications │ +│ stdout/stderr │ files │ syslog │ API calls │ +└────────┬────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────┐ +│ Log Collectors │ +│ Fluent Bit │ Fluentd │ Vector │ Promtail │ Beats │ +│ (parse, transform, buffer, forward) │ +└────────┬────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────┐ +│ Log Aggregation/Storage │ +│ Loki │ Elasticsearch │ Splunk │ ClickHouse │ +└────────┬────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────┐ +│ Visualization │ +│ Grafana │ Kibana │ Splunk UI │ Custom │ +└─────────────────────────────────────────────────────────────┘ +``` + +--- + +## Loki + +Grafana's log aggregation system. "Like Prometheus, but for logs." + +### Key Concepts + +- **Labels only** — Indexes labels, not log content +- **LogQL** — PromQL-like query language +- **Cost-effective** — Store in object storage (S3, GCS) +- **Grafana native** — Perfect integration + +### Architecture + +``` +┌─────────────────────────────────────────────────────────────┐ +│ Applications │ +└──────────────────────────┬──────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────┐ +│ Promtail │ +│ (or Grafana Alloy, Fluent Bit) │ +│ - Discovers log files │ +│ - Attaches labels │ +│ - Pushes to Loki │ +└──────────────────────────┬──────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────┐ +│ Loki │ +│ ┌───────────┐ ┌───────────┐ ┌───────────┐ │ +│ │Distributor│──▶│ Ingester │──▶│ Querier │ │ +│ └───────────┘ └─────┬─────┘ └───────────┘ │ +│ │ │ +│ ▼ │ +│ ┌───────────┐ │ +│ │ Storage │ │ +│ │ (S3/GCS) │ │ +│ └───────────┘ │ +└─────────────────────────────────────────────────────────────┘ +``` + +### LogQL Examples + +```logql +# Basic stream selection +{job="nginx"} + +# Filter by content +{job="nginx"} |= "error" +{job="nginx"} != "healthcheck" +{job="nginx"} |~ "status=[45].." + +# Parse and filter +{job="nginx"} | json | status >= 400 + +# Pattern parsing +{job="nginx"} | pattern ` - - [<_>] " <_>" ` + | status = "500" + +# Aggregations +count_over_time({job="nginx"} |= "error" [5m]) +rate({job="nginx"}[1m]) +sum by (status) (count_over_time({job="nginx"} | json [5m])) + +# Top errors +topk(10, sum by (error) (count_over_time({job="api"} | json [1h]))) +``` + +### Promtail Config + +```yaml +# promtail-config.yaml +server: + http_listen_port: 9080 + +positions: + filename: /tmp/positions.yaml + +clients: + - url: http://loki:3100/loki/api/v1/push + +scrape_configs: + - job_name: containers + docker_sd_configs: + - host: unix:///var/run/docker.sock + relabel_configs: + - source_labels: ['__meta_docker_container_name'] + target_label: container + + - job_name: system + static_configs: + - targets: + - localhost + labels: + job: syslog + __path__: /var/log/syslog +``` + +### Deployment + +```yaml +# docker-compose.yaml +services: + loki: + image: grafana/loki:latest + ports: + - "3100:3100" + command: -config.file=/etc/loki/local-config.yaml + volumes: + - loki-data:/loki + + promtail: + image: grafana/promtail:latest + volumes: + - /var/log:/var/log:ro + - ./promtail-config.yaml:/etc/promtail/config.yaml + command: -config.file=/etc/promtail/config.yaml +``` + +--- + +## Elasticsearch (ELK/Elastic Stack) + +Full-text search engine, powerful for log analysis. + +### Key Concepts + +- **Full-text indexing** — Indexes all log content +- **Lucene-based** — Powerful search capabilities +- **ELK Stack** — Elasticsearch + Logstash + Kibana +- **Beats** — Lightweight data shippers + +### Architecture + +``` +┌─────────────────────────────────────────────────────────────┐ +│ Applications │ +└────────┬───────────────────────────────────────┬────────────┘ + │ │ + ▼ ▼ +┌─────────────────┐ ┌─────────────────────┐ +│ Beats │ │ Logstash │ +│ (Filebeat, │ │ (transform, │ +│ Metricbeat) │ │ enrich) │ +└────────┬────────┘ └──────────┬──────────┘ + │ │ + └───────────────────┬───────────────────┘ + │ + ▼ + ┌─────────────────┐ + │ Elasticsearch │ + │ (cluster) │ + └────────┬────────┘ + │ + ▼ + ┌─────────────────┐ + │ Kibana │ + │ (visualization) │ + └─────────────────┘ +``` + +### Query Examples + +```json +// KQL (Kibana Query Language) +status:error AND service:api +response_time:>1000 +message:"connection refused" + +// Lucene syntax +status:5* AND NOT path:/healthcheck +@timestamp:[2024-01-01 TO 2024-01-31] + +// Elasticsearch Query DSL +{ + "query": { + "bool": { + "must": [ + { "match": { "level": "error" } }, + { "range": { "@timestamp": { "gte": "now-1h" } } } + ], + "filter": [ + { "term": { "service": "api" } } + ] + } + }, + "aggs": { + "errors_by_type": { + "terms": { "field": "error_type.keyword" } + } + } +} +``` + +### Filebeat Config + +```yaml +# filebeat.yml +filebeat.inputs: + - type: container + paths: + - /var/lib/docker/containers/*/*.log + processors: + - add_docker_metadata: ~ + + - type: log + paths: + - /var/log/nginx/*.log + fields: + service: nginx + +output.elasticsearch: + hosts: ["elasticsearch:9200"] + index: "logs-%{+yyyy.MM.dd}" + +setup.kibana: + host: "kibana:5601" +``` + +### Index Lifecycle Management + +```json +// ILM Policy +{ + "policy": { + "phases": { + "hot": { + "actions": { + "rollover": { + "max_size": "50GB", + "max_age": "1d" + } + } + }, + "warm": { + "min_age": "7d", + "actions": { + "shrink": { "number_of_shards": 1 }, + "forcemerge": { "max_num_segments": 1 } + } + }, + "delete": { + "min_age": "30d", + "actions": { "delete": {} } + } + } + } +} +``` + +--- + +## Splunk + +Enterprise log management and SIEM platform. + +### Key Concepts + +- **SPL** — Search Processing Language +- **Indexers** — Store and index data +- **Forwarders** — Collect and send data +- **Search Heads** — Query interface + +### Architecture + +``` +┌─────────────────────────────────────────────────────────────┐ +│ Data Sources │ +└────────┬───────────────────────────────────────┬────────────┘ + │ │ + ▼ ▼ +┌─────────────────┐ ┌─────────────────────┐ +│ Universal │ │ Heavy │ +│ Forwarder │ │ Forwarder │ +│ (lightweight) │ │ (parsing/routing) │ +└────────┬────────┘ └──────────┬──────────┘ + │ │ + └───────────────────┬───────────────────┘ + │ + ▼ + ┌─────────────────┐ + │ Indexers │ + │ (cluster) │ + └────────┬────────┘ + │ + ▼ + ┌─────────────────┐ + │ Search Heads │ + │ (Web UI/API) │ + └─────────────────┘ +``` + +### SPL Examples + +```spl +// Basic search +index=main sourcetype=nginx status>=400 + +// Stats +index=main sourcetype=nginx +| stats count by status + +// Time chart +index=main sourcetype=api +| timechart span=5m count by level + +// Top errors +index=main level=error +| top 10 error_message + +// Transaction tracking +index=main +| transaction request_id +| where duration > 5 + +// Join +index=main sourcetype=api +| join user_id [search index=users] + +// Complex analysis +index=main sourcetype=api +| eval response_time_ms = response_time * 1000 +| where response_time_ms > 1000 +| stats avg(response_time_ms) as avg_latency, + count as slow_requests + by endpoint +| sort - slow_requests +``` + +### Deployment Options + +| Option | Use Case | +|--------|----------| +| Splunk Cloud | SaaS, managed | +| Splunk Enterprise | On-premises | +| Splunk Free | <500MB/day, single user | + +--- + +## Log Collectors + +### Fluentd + +Plugin-based log collector. Ruby-based. + +```xml + + + @type tail + path /var/log/nginx/access.log + pos_file /var/log/td-agent/nginx-access.log.pos + tag nginx.access + + @type nginx + + + + + @type record_transformer + + hostname "#{Socket.gethostname}" + + + + + @type elasticsearch + host elasticsearch + port 9200 + logstash_format true + + @type file + path /var/log/td-agent/buffer/elasticsearch + flush_interval 5s + + +``` + +### Fluent Bit + +Lightweight version of Fluentd. C-based. + +```ini +# fluent-bit.conf +[SERVICE] + Flush 5 + Daemon Off + Log_Level info + +[INPUT] + Name tail + Path /var/log/containers/*.log + Parser docker + Tag kube.* + Mem_Buf_Limit 5MB + +[FILTER] + Name kubernetes + Match kube.* + Merge_Log On + K8S-Logging.Parser On + +[OUTPUT] + Name loki + Match * + Host loki + Port 3100 + Labels job=fluent-bit +``` + +### Vector + +High-performance observability data pipeline. + +```toml +# vector.toml +[sources.logs] +type = "file" +include = ["/var/log/**/*.log"] + +[transforms.parse] +type = "remap" +inputs = ["logs"] +source = ''' +. = parse_json!(.message) +.timestamp = now() +''' + +[transforms.filter_errors] +type = "filter" +inputs = ["parse"] +condition = '.level == "error"' + +[sinks.loki] +type = "loki" +inputs = ["parse"] +endpoint = "http://loki:3100" +labels.service = "{{ service }}" + +[sinks.elasticsearch] +type = "elasticsearch" +inputs = ["filter_errors"] +endpoints = ["http://elasticsearch:9200"] +``` + +### Collector Comparison + +| Aspect | Fluentd | Fluent Bit | Vector | +|--------|---------|------------|--------| +| Language | Ruby | C | Rust | +| Memory | ~40MB | ~1MB | ~10MB | +| Plugins | 1000+ | 70+ | 100+ | +| Performance | Good | Excellent | Excellent | +| Use case | Feature-rich | Resource-constrained | High-throughput | + +--- + +## Comparison Matrix + +| Feature | Loki | Elasticsearch | Splunk | +|---------|:----:|:-------------:|:------:| +| Full-text search | Slow | Fast | Fast | +| Label/field queries | Fast | Fast | Fast | +| Storage cost | Low | High | High | +| Query language | LogQL | KQL/Lucene | SPL | +| Learning curve | Low | Medium | Medium | +| Self-hosted | ✅ | ✅ | ✅ | +| Cloud hosted | Grafana Cloud | Elastic Cloud | Splunk Cloud | +| SIEM features | ❌ | ✅ | ✅ | +| Visualization | Grafana | Kibana | Splunk UI | + +### Cost Comparison + +| Storage Approach | Relative Cost | +|------------------|---------------| +| Loki (object storage) | $ | +| Elasticsearch (SSD) | $$$ | +| Splunk (indexed) | $$$$ | + +--- + +## Decision Guide + +| Scenario | Recommendation | +|----------|----------------| +| Already use Grafana | Loki | +| Full-text search critical | Elasticsearch | +| Enterprise/compliance | Splunk | +| Cost-sensitive | Loki | +| SIEM/Security | Splunk or Elastic SIEM | +| Kubernetes native | Loki + Promtail | +| High-cardinality labels | Loki | +| Complex log parsing | Elasticsearch | + +--- + +## Related + +- [[Observability Stack]] +- [[Distributed Tracing]] +- [[Prometheus]] +- [[Grafana]] +- [[OpenTelemetry]] diff --git a/tools/Package Managers.md b/tools/Package Managers.md new file mode 100644 index 0000000..945bd88 --- /dev/null +++ b/tools/Package Managers.md @@ -0,0 +1,465 @@ +--- +title: Package Managers +aliases: + - System Package Managers + - OS Package Managers +tags: + - tool + - comparison + - package-manager +type: comparison +status: complete +created: 2025-12-04 +--- + +# Package Managers + +System-level package managers for Linux, macOS, and Windows. + +## Overview by Platform + +| Platform | Package Manager | Package Format | +|----------|-----------------|----------------| +| Debian/Ubuntu | apt | .deb | +| Fedora/RHEL | dnf | .rpm | +| Arch | pacman | .pkg.tar.zst | +| openSUSE | zypper | .rpm | +| Alpine | apk | .apk | +| NixOS/Nix | nix | .nar | +| macOS | Homebrew | bottles/formulae | +| macOS | MacPorts | ports | +| Windows | winget | .msix/.exe | +| Windows | Chocolatey | .nupkg | +| Windows | Scoop | manifests | + +--- + +## Linux Package Managers + +### apt (Debian/Ubuntu) + +Advanced Package Tool. Standard on Debian-based systems. + +```bash +# Update and upgrade +apt update # Refresh package lists +apt upgrade # Upgrade all packages +apt full-upgrade # Upgrade, allowing removals + +# Install/remove +apt install nginx # Install +apt install nginx=1.18.0-1 # Specific version +apt remove nginx # Remove (keep config) +apt purge nginx # Remove with config +apt autoremove # Remove unused dependencies + +# Search/info +apt search nginx # Search packages +apt show nginx # Package details +apt list --installed # List installed +apt list --upgradable # List upgradable + +# Cache +apt clean # Clear package cache +apt autoclean # Clear old package cache + +# Low-level (dpkg) +dpkg -i package.deb # Install local .deb +dpkg -l # List all installed +dpkg -L nginx # Files in package +dpkg -S /usr/bin/nginx # Which package owns file +``` + +### dnf (Fedora/RHEL) + +Dandified YUM. Modern Red Hat package manager. + +```bash +# Update +dnf check-update # Check for updates +dnf update # Update all +dnf update nginx # Update specific + +# Install/remove +dnf install nginx # Install +dnf install nginx-1.20.0 # Specific version +dnf remove nginx # Remove +dnf autoremove # Remove unused deps + +# Search/info +dnf search nginx # Search +dnf info nginx # Package info +dnf list installed # List installed +dnf list available # List available +dnf provides /usr/bin/nginx # Find package by file + +# Groups +dnf group list # List groups +dnf group install "Development Tools" + +# Modules (version streams) +dnf module list # List modules +dnf module enable nodejs:18 # Enable stream +dnf module install nodejs:18 # Install from stream + +# History/rollback +dnf history # Transaction history +dnf history undo 5 # Undo transaction 5 + +# Repository management +dnf repolist # List repos +dnf config-manager --add-repo URL +``` + +### pacman (Arch) + +Simple, fast. Rolling release philosophy. + +```bash +# Sync database +pacman -Sy # Sync repos +pacman -Syy # Force sync + +# Install +pacman -S nginx # Install +pacman -S nginx vim git # Multiple packages +pacman -U package.pkg.tar.zst # Install local + +# Remove +pacman -R nginx # Remove +pacman -Rs nginx # Remove with unused deps +pacman -Rns nginx # Remove with deps and config + +# Update +pacman -Syu # Full system upgrade +# NEVER do pacman -Sy package (partial upgrade) + +# Search/info +pacman -Ss nginx # Search repos +pacman -Qs nginx # Search installed +pacman -Si nginx # Remote info +pacman -Qi nginx # Local info +pacman -Ql nginx # List files in package +pacman -Qo /usr/bin/nginx # Find owning package + +# Cleanup +pacman -Sc # Clear cache (keep installed) +pacman -Scc # Clear all cache +paccache -r # Keep last 3 versions + +# AUR (use helper) +yay -S package # yay +paru -S package # paru +``` + +### zypper (openSUSE) + +SUSE's package manager. Rich feature set. + +```bash +# Refresh/update +zypper refresh # Refresh repos +zypper update # Update packages +zypper dup # Distribution upgrade + +# Install/remove +zypper install nginx # Install +zypper remove nginx # Remove +zypper install -t pattern lamp_server # Install pattern + +# Search/info +zypper search nginx # Search +zypper info nginx # Info +zypper what-provides nginx # What provides + +# Repos +zypper repos # List repos +zypper addrepo URL alias # Add repo +zypper removerepo alias # Remove repo + +# Locks +zypper addlock nginx # Prevent updates +zypper removelock nginx # Remove lock +``` + +### apk (Alpine) + +Minimal, fast. Container-focused. + +```bash +# Update +apk update # Update index +apk upgrade # Upgrade packages + +# Install/remove +apk add nginx # Install +apk add --no-cache nginx # Install without cache (Docker) +apk del nginx # Remove + +# Search/info +apk search nginx # Search +apk info nginx # Info +apk info -L nginx # List files + +# Virtual packages (for build deps) +apk add --virtual .build-deps gcc musl-dev +# ... build ... +apk del .build-deps # Remove all at once +``` + +### nix (NixOS and others) + +Functional, declarative. Works on any Linux/macOS. + +```bash +# Imperative commands +nix-env -iA nixpkgs.nginx # Install +nix-env -e nginx # Remove +nix-env -u # Upgrade all +nix-env -q # List installed + +# Search +nix search nixpkgs nginx # Search packages + +# Generations (rollback) +nix-env --list-generations # List generations +nix-env --rollback # Rollback +nix-env -G 42 # Switch to generation 42 + +# Garbage collection +nix-collect-garbage # Remove unused +nix-collect-garbage -d # Delete old generations too + +# New nix command (experimental) +nix profile install nixpkgs#nginx +nix profile remove nginx +nix profile upgrade '.*' +``` + +--- + +## macOS Package Managers + +### Homebrew + +The missing package manager for macOS (also Linux). + +```bash +# Install Homebrew +/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" + +# Update +brew update # Update Homebrew +brew upgrade # Upgrade all +brew upgrade nginx # Upgrade specific + +# Install +brew install nginx # Install formula +brew install --cask firefox # Install GUI app (cask) +brew install --formula nginx # Explicitly formula + +# Remove +brew uninstall nginx # Remove +brew autoremove # Remove unused deps + +# Search/info +brew search nginx # Search +brew info nginx # Info +brew list # List installed +brew deps nginx # Show dependencies +brew uses nginx --installed # What depends on this + +# Maintenance +brew cleanup # Remove old versions +brew doctor # Check for issues + +# Taps (additional repos) +brew tap homebrew/cask-fonts # Add tap +brew untap repo # Remove tap + +# Services +brew services list # List services +brew services start nginx # Start service +brew services stop nginx # Stop service +``` + +### MacPorts + +Traditional Unix ports. More Unix-like. + +```bash +# Install MacPorts +# Download from macports.org + +# Update +sudo port selfupdate # Update MacPorts +sudo port upgrade outdated # Upgrade packages + +# Install +sudo port install nginx # Install +sudo port install nginx +ssl # With variant + +# Remove +sudo port uninstall nginx # Remove +sudo port uninstall --follow-dependents nginx + +# Search/info +port search nginx # Search +port info nginx # Info +port installed # List installed +port contents nginx # List files + +# Variants +port variants nginx # Show variants +``` + +### Homebrew vs MacPorts + +| Aspect | Homebrew | MacPorts | +|--------|----------|----------| +| Philosophy | Use system libs when possible | Self-contained | +| Install location | /opt/homebrew (M1) or /usr/local | /opt/local | +| Root required | No | Yes | +| Package count | ~6,000 formulae + casks | ~27,000 ports | +| GUI apps | Casks | Limited | +| Popularity | More popular | Traditional Unix users | + +--- + +## Windows Package Managers + +### winget (Windows Package Manager) + +Microsoft's official package manager. + +```powershell +# Search +winget search vscode # Search +winget show Microsoft.VisualStudioCode # Info + +# Install +winget install Microsoft.VisualStudioCode +winget install -e --id Git.Git # Exact ID match +winget install -h # Silent install + +# Update +winget upgrade # List upgradable +winget upgrade --all # Upgrade all +winget upgrade Git.Git # Upgrade specific + +# Remove +winget uninstall Git.Git # Uninstall + +# List +winget list # List installed +winget list --source winget # Only winget-installed + +# Export/import +winget export -o packages.json # Export installed +winget import -i packages.json # Import/install +``` + +### Chocolatey + +Community package manager. Large repository. + +```powershell +# Install Chocolatey (admin PowerShell) +Set-ExecutionPolicy Bypass -Scope Process -Force +[System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072 +iex ((New-Object System.Net.WebClient).DownloadString('https://community.chocolatey.org/install.ps1')) + +# Install +choco install git # Install +choco install git -y # Auto-confirm +choco install git --version=2.40.0 # Specific version + +# Update +choco upgrade all # Upgrade all +choco upgrade git # Upgrade specific +choco outdated # List outdated + +# Remove +choco uninstall git # Uninstall + +# Search/info +choco search git # Search +choco info git # Info +choco list # List installed + +# Features +choco feature enable -n allowGlobalConfirmation +``` + +### Scoop + +Git-based, user-level installs. No admin required. + +```powershell +# Install Scoop +Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser +Invoke-RestMethod -Uri https://get.scoop.sh | Invoke-Expression + +# Install +scoop install git # Install +scoop install extras/vscode # From bucket + +# Update +scoop update # Update Scoop + buckets +scoop update * # Update all apps +scoop update git # Update specific + +# Remove +scoop uninstall git # Uninstall + +# Search +scoop search git # Search +scoop info git # Info +scoop list # List installed + +# Buckets (additional repos) +scoop bucket add extras # Add bucket +scoop bucket list # List buckets +scoop bucket known # Known buckets + +# Cleanup +scoop cleanup * # Remove old versions +scoop cache rm * # Clear download cache +``` + +### Windows: Which to Choose? + +| Aspect | winget | Chocolatey | Scoop | +|--------|--------|------------|-------| +| Official | Yes | No | No | +| Packages | ~4,000 | ~10,000 | ~3,000+ | +| Admin required | Usually | Yes | No | +| GUI apps | Yes | Yes | Limited | +| Dev tools | Good | Excellent | Excellent | +| Scripting | Good | Mature | Good | +| Portable apps | No | Some | Yes (default) | + +--- + +## Cross-Platform Comparison + +### Common Operations + +| Operation | apt | dnf | pacman | brew | winget | +|-----------|-----|-----|--------|------|--------| +| Update index | `apt update` | `dnf check-update` | `pacman -Sy` | `brew update` | — | +| Upgrade all | `apt upgrade` | `dnf update` | `pacman -Syu` | `brew upgrade` | `winget upgrade --all` | +| Install | `apt install X` | `dnf install X` | `pacman -S X` | `brew install X` | `winget install X` | +| Remove | `apt remove X` | `dnf remove X` | `pacman -R X` | `brew uninstall X` | `winget uninstall X` | +| Search | `apt search X` | `dnf search X` | `pacman -Ss X` | `brew search X` | `winget search X` | +| Info | `apt show X` | `dnf info X` | `pacman -Si X` | `brew info X` | `winget show X` | +| List installed | `apt list --installed` | `dnf list installed` | `pacman -Q` | `brew list` | `winget list` | +| Clean cache | `apt clean` | `dnf clean all` | `pacman -Sc` | `brew cleanup` | — | + +--- + +## Related + +- [[Linux Distributions]] +- [[Version Managers]] +- [[Shells]] diff --git a/tools/Process Managers.md b/tools/Process Managers.md new file mode 100644 index 0000000..99c8bfa --- /dev/null +++ b/tools/Process Managers.md @@ -0,0 +1,546 @@ +--- +title: Process Managers +aliases: + - Service Managers + - Init Systems +tags: + - tool + - comparison + - devops + - systems +type: comparison +status: complete +created: 2025-12-04 +--- + +# Process Managers + +Tools for managing, monitoring, and keeping processes running. + +## Overview + +| Tool | Type | Platform | Language | Use Case | +|------|------|----------|----------|----------| +| systemd | Init system | Linux | C | System services | +| launchd | Init system | macOS | C | System services | +| supervisord | Process manager | Cross-platform | Python | App processes | +| PM2 | Process manager | Cross-platform | Node.js | Node.js apps | +| runit | Init system | Linux/BSD | C | Simple services | +| s6 | Init system | Linux | C | Containers | +| Circus | Process manager | Cross-platform | Python | Multi-process apps | + +--- + +## Use Case Comparison + +``` +System Init (boot services): +├── systemd (Linux standard) +├── launchd (macOS) +├── runit (lightweight Linux) +└── s6 (containers) + +Application Process Manager: +├── PM2 (Node.js focus) +├── supervisord (Python apps, general) +└── Circus (multi-process) +``` + +--- + +## systemd + +The standard Linux init system and service manager. + +### Key Features + +- **Unit files** — Declarative service definitions +- **Dependencies** — Service ordering and requirements +- **Socket activation** — Start on demand +- **Resource control** — cgroups integration +- **Journaling** — Centralized logging + +### Unit File Example + +```ini +# /etc/systemd/system/myapp.service +[Unit] +Description=My Application +After=network.target +Requires=postgresql.service + +[Service] +Type=simple +User=myapp +WorkingDirectory=/opt/myapp +ExecStart=/opt/myapp/bin/server +ExecReload=/bin/kill -HUP $MAINPID +Restart=on-failure +RestartSec=5s +StandardOutput=journal +StandardError=journal +Environment=NODE_ENV=production + +[Install] +WantedBy=multi-user.target +``` + +### Common Commands + +```bash +# Service control +sudo systemctl start myapp +sudo systemctl stop myapp +sudo systemctl restart myapp +sudo systemctl reload myapp # If supported +sudo systemctl status myapp + +# Enable/disable on boot +sudo systemctl enable myapp +sudo systemctl disable myapp + +# List services +systemctl list-units --type=service +systemctl list-units --state=failed + +# View logs +journalctl -u myapp # All logs +journalctl -u myapp -f # Follow +journalctl -u myapp --since "1 hour ago" +journalctl -u myapp -n 100 # Last 100 lines + +# Reload unit files after changes +sudo systemctl daemon-reload +``` + +### Service Types + +| Type | Behavior | +|------|----------| +| `simple` | Process stays in foreground (default) | +| `exec` | Like simple, but started after exec() | +| `forking` | Traditional daemon (forks to background) | +| `oneshot` | Run once, then exit | +| `notify` | Like simple, but notifies when ready | + +### Resource Limits + +```ini +[Service] +# Memory limits +MemoryMax=500M +MemoryHigh=400M + +# CPU limits +CPUQuota=50% +CPUWeight=100 + +# File limits +LimitNOFILE=65535 + +# Security +NoNewPrivileges=true +ProtectSystem=strict +ProtectHome=true +PrivateTmp=true +``` + +--- + +## launchd + +macOS's init and service management system. + +### Key Concepts + +- **LaunchAgents** — Per-user services +- **LaunchDaemons** — System-wide services +- **Property lists** — XML/plist configuration +- **On-demand loading** — Start when needed + +### Plist Locations + +| Path | Scope | Runs As | +|------|-------|---------| +| `~/Library/LaunchAgents` | User | Current user | +| `/Library/LaunchAgents` | All users | Current user | +| `/Library/LaunchDaemons` | System | root | +| `/System/Library/Launch*` | System (Apple) | Various | + +### Plist Example + +```xml + + + + + + Label + com.myapp.server + + ProgramArguments + + /usr/local/bin/myapp + --port=3000 + + + WorkingDirectory + /opt/myapp + + RunAtLoad + + + KeepAlive + + + StandardOutPath + /var/log/myapp/stdout.log + + StandardErrorPath + /var/log/myapp/stderr.log + + EnvironmentVariables + + NODE_ENV + production + + + +``` + +### Common Commands + +```bash +# Load/unload service +launchctl load ~/Library/LaunchAgents/com.myapp.server.plist +launchctl unload ~/Library/LaunchAgents/com.myapp.server.plist + +# Modern commands (macOS 10.10+) +launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.myapp.plist +launchctl bootout gui/$(id -u) ~/Library/LaunchAgents/com.myapp.plist + +# Start/stop +launchctl start com.myapp.server +launchctl stop com.myapp.server + +# List services +launchctl list +launchctl list | grep myapp + +# View service info +launchctl print gui/$(id -u)/com.myapp.server +``` + +### Homebrew Services + +```bash +# Homebrew wraps launchd +brew services list +brew services start postgresql +brew services stop postgresql +brew services restart postgresql + +# View generated plist +cat ~/Library/LaunchAgents/homebrew.mxcl.postgresql.plist +``` + +--- + +## supervisord + +Python-based process manager. Cross-platform. + +### Key Features + +- **Simple config** — INI file format +- **Web UI** — Optional HTTP interface +- **Process groups** — Manage related processes +- **Event system** — Hook into process lifecycle +- **Cross-platform** — Linux, macOS, Windows + +### Configuration + +```ini +; /etc/supervisor/conf.d/myapp.conf +[program:myapp] +command=/opt/myapp/venv/bin/python app.py +directory=/opt/myapp +user=myapp +autostart=true +autorestart=true +startsecs=5 +startretries=3 +redirect_stderr=true +stdout_logfile=/var/log/myapp/app.log +stdout_logfile_maxbytes=10MB +stdout_logfile_backups=5 +environment=PYTHONPATH="/opt/myapp",NODE_ENV="production" + +[program:myapp-worker] +command=/opt/myapp/venv/bin/celery -A app worker +directory=/opt/myapp +user=myapp +numprocs=4 +process_name=%(program_name)s_%(process_num)02d +autostart=true +autorestart=true +``` + +### Process Groups + +```ini +[group:myapp] +programs=myapp,myapp-worker + +; Control all at once: +; supervisorctl start myapp:* +; supervisorctl stop myapp:* +``` + +### Common Commands + +```bash +# Service control +supervisorctl status +supervisorctl start myapp +supervisorctl stop myapp +supervisorctl restart myapp +supervisorctl start all +supervisorctl stop all + +# Reload config +supervisorctl reread # Check for changes +supervisorctl update # Apply changes + +# Tail logs +supervisorctl tail myapp +supervisorctl tail -f myapp + +# Interactive shell +supervisorctl +supervisor> help +``` + +### Web Interface + +```ini +; /etc/supervisor/supervisord.conf +[inet_http_server] +port=127.0.0.1:9001 +username=admin +password=secret +``` + +--- + +## PM2 + +Node.js process manager with advanced features. + +### Key Features + +- **Cluster mode** — Multi-core load balancing +- **Zero-downtime reload** — Graceful restarts +- **Ecosystem file** — Declarative configuration +- **Monitoring** — Built-in metrics, web dashboard +- **Log management** — Rotation, viewing + +### Basic Usage + +```bash +# Start application +pm2 start app.js +pm2 start app.js --name myapp +pm2 start app.js -i max # Cluster mode, all cores +pm2 start app.js -i 4 # 4 instances + +# Control +pm2 stop myapp +pm2 restart myapp +pm2 reload myapp # Zero-downtime +pm2 delete myapp + +# Status and logs +pm2 list +pm2 status +pm2 logs +pm2 logs myapp +pm2 monit # Real-time dashboard + +# Save and restore +pm2 save # Save current processes +pm2 startup # Generate startup script +pm2 resurrect # Restore saved processes +``` + +### Ecosystem File + +```javascript +// ecosystem.config.js +module.exports = { + apps: [ + { + name: 'myapp', + script: './app.js', + instances: 'max', + exec_mode: 'cluster', + watch: false, + max_memory_restart: '1G', + env: { + NODE_ENV: 'development' + }, + env_production: { + NODE_ENV: 'production', + PORT: 3000 + } + }, + { + name: 'worker', + script: './worker.js', + instances: 2, + exec_mode: 'cluster' + } + ] +}; +``` + +```bash +# Use ecosystem file +pm2 start ecosystem.config.js +pm2 start ecosystem.config.js --env production + +# Deploy (if deploy section defined) +pm2 deploy production setup +pm2 deploy production +``` + +### Non-Node.js Apps + +```bash +# Python +pm2 start app.py --interpreter python3 + +# Bash script +pm2 start script.sh --interpreter bash + +# Binary +pm2 start ./myapp + +# General +pm2 start "command to run" --name myprocess +``` + +### PM2 Plus (Monitoring) + +```bash +# Link to PM2 Plus dashboard +pm2 plus + +# Or use local monitoring +pm2 monit +``` + +--- + +## Comparison + +### Feature Matrix + +| Feature | systemd | launchd | supervisord | PM2 | +|---------|:-------:|:-------:|:-----------:|:---:| +| Init system | ✅ | ✅ | ❌ | ❌ | +| Process manager | ✅ | ✅ | ✅ | ✅ | +| Cluster mode | ❌ | ❌ | ❌ | ✅ | +| Zero-downtime | Manual | Manual | ❌ | ✅ | +| Web UI | ❌ | ❌ | ✅ | ✅ | +| Log rotation | journald | Manual | ✅ | ✅ | +| Resource limits | ✅ | ✅ | ❌ | ✅ | +| Cross-platform | Linux | macOS | ✅ | ✅ | +| Config format | INI | Plist | INI | JS/JSON | + +### Performance Overhead + +| Tool | Memory | Startup | Complexity | +|------|--------|---------|------------| +| systemd | Low | Fast | Medium | +| launchd | Low | Fast | Medium | +| supervisord | Medium | Medium | Low | +| PM2 | Higher | Medium | Low | + +--- + +## Use Case Guide + +| Scenario | Recommendation | +|----------|----------------| +| Linux system services | systemd | +| macOS services | launchd (or Homebrew services) | +| Python/Django apps | supervisord or systemd | +| Node.js production | PM2 | +| Multi-process apps | PM2 or supervisord | +| Docker containers | s6, runit, or direct | +| Dev environment | PM2 or direct | +| Enterprise Linux | systemd | + +### Quick Setup by Language + +```bash +# Node.js → PM2 +npm install -g pm2 +pm2 start app.js +pm2 startup + +# Python → supervisord +pip install supervisor +# Create /etc/supervisor/conf.d/myapp.conf +supervisorctl update + +# Any language → systemd +# Create /etc/systemd/system/myapp.service +sudo systemctl enable myapp +sudo systemctl start myapp +``` + +--- + +## Container Considerations + +### Inside Containers + +| Scenario | Recommendation | +|----------|----------------| +| Single process | Run directly (no manager) | +| Multiple processes | s6, runit, or supervisord | +| Process supervision | s6-overlay or dumb-init | + +```dockerfile +# Single process (preferred) +CMD ["node", "app.js"] + +# With s6-overlay (multiple processes) +FROM alpine +RUN apk add s6-overlay +COPY services.d /etc/services.d +ENTRYPOINT ["/init"] +``` + +### Outside Containers + +Use systemd/launchd to manage Docker/Podman containers: + +```ini +# /etc/systemd/system/myapp-container.service +[Service] +ExecStart=/usr/bin/docker run --rm --name myapp myimage +ExecStop=/usr/bin/docker stop myapp +Restart=always +``` + +--- + +## Related + +- [[Container Runtimes]] +- [[Deployment]] +- [[Linux Distributions]] diff --git a/tools/Remote Development.md b/tools/Remote Development.md new file mode 100644 index 0000000..326dd76 --- /dev/null +++ b/tools/Remote Development.md @@ -0,0 +1,545 @@ +--- +title: Remote Development +aliases: + - Remote Dev + - Cloud Development +tags: + - tool + - comparison + - development +type: comparison +status: complete +created: 2025-12-04 +--- + +# Remote Development + +Tools and platforms for developing on remote machines and cloud environments. + +## Overview + +| Tool/Platform | Type | IDE Support | Cost | +|---------------|------|-------------|------| +| SSH + editor | Direct connection | Any | Free | +| VS Code Remote | IDE extension | VS Code | Free | +| JetBrains Gateway | IDE remote backend | JetBrains IDEs | License | +| GitHub Codespaces | Cloud environment | VS Code, JetBrains | Usage-based | +| Gitpod | Cloud environment | VS Code, JetBrains | Usage-based | +| Coder | Self-hosted platform | Any | Free/Enterprise | +| DevPod | Open-source devcontainers | Any | Free | + +--- + +## SSH Configuration + +The foundation of remote development. + +### Basic Config + +```bash +# ~/.ssh/config +Host dev-server + HostName 192.168.1.100 + User developer + Port 22 + IdentityFile ~/.ssh/id_ed25519 + +Host jump-host + HostName bastion.example.com + User admin + +Host internal-server + HostName 10.0.0.50 + User developer + ProxyJump jump-host + +Host * + AddKeysToAgent yes + IdentitiesOnly yes + ServerAliveInterval 60 + ServerAliveCountMax 3 +``` + +### Key Management + +```bash +# Generate key +ssh-keygen -t ed25519 -C "user@example.com" + +# Copy to server +ssh-copy-id dev-server + +# Agent forwarding +ssh -A dev-server + +# SSH agent +eval "$(ssh-agent -s)" +ssh-add ~/.ssh/id_ed25519 +``` + +### Port Forwarding + +```bash +# Local forward (access remote service locally) +ssh -L 8080:localhost:3000 dev-server +# localhost:8080 → remote:3000 + +# Remote forward (expose local service remotely) +ssh -R 9000:localhost:3000 dev-server +# remote:9000 → local:3000 + +# Dynamic (SOCKS proxy) +ssh -D 1080 dev-server + +# In config +Host dev-server + LocalForward 8080 localhost:3000 + RemoteForward 9000 localhost:3000 +``` + +### Multiplexing + +```bash +# ~/.ssh/config +Host * + ControlMaster auto + ControlPath ~/.ssh/sockets/%r@%h-%p + ControlPersist 600 + +# First connection establishes socket +# Subsequent connections reuse it (instant) +``` + +--- + +## VS Code Remote + +Extensions for remote development in VS Code. + +### Remote Extensions + +| Extension | Use Case | +|-----------|----------| +| Remote - SSH | Connect to any SSH host | +| Remote - Containers | Develop in containers | +| Remote - WSL | Develop in WSL | +| Remote - Tunnels | Connect via tunnel (no SSH) | +| Dev Containers | Devcontainer support | + +### Remote - SSH + +```bash +# Install extension +code --install-extension ms-vscode-remote.remote-ssh + +# Connect +# Cmd/Ctrl+Shift+P → "Remote-SSH: Connect to Host" +# Or click green button bottom-left +``` + +Settings in VS Code: + +```json +{ + "remote.SSH.remotePlatform": { + "dev-server": "linux" + }, + "remote.SSH.defaultExtensions": [ + "golang.go", + "rust-lang.rust-analyzer" + ] +} +``` + +### Dev Containers + +```json +// .devcontainer/devcontainer.json +{ + "name": "Node.js", + "image": "mcr.microsoft.com/devcontainers/javascript-node:18", + "features": { + "ghcr.io/devcontainers/features/docker-in-docker:2": {} + }, + "forwardPorts": [3000], + "postCreateCommand": "npm install", + "customizations": { + "vscode": { + "extensions": [ + "dbaeumer.vscode-eslint", + "esbenp.prettier-vscode" + ], + "settings": { + "editor.formatOnSave": true + } + } + } +} +``` + +```dockerfile +# .devcontainer/Dockerfile +FROM mcr.microsoft.com/devcontainers/base:ubuntu + +RUN apt-get update && apt-get install -y \ + build-essential \ + && rm -rf /var/lib/apt/lists/* + +# Install specific tools +RUN curl -fsSL https://deb.nodesource.com/setup_20.x | bash - \ + && apt-get install -y nodejs +``` + +### Remote Tunnels + +Connect without SSH, through Microsoft's tunnel service. + +```bash +# On remote machine +code tunnel + +# Creates a tunnel accessible via: +# - vscode.dev +# - VS Code desktop with Remote - Tunnels extension +``` + +--- + +## JetBrains Gateway + +Remote development for JetBrains IDEs. + +### How It Works + +``` +┌─────────────────┐ ┌─────────────────┐ +│ Local Machine │ │ Remote Server │ +│ │ │ │ +│ ┌───────────┐ │ SSH │ ┌───────────┐ │ +│ │ Gateway │◄─┼──────────┼─▶│ IDE │ │ +│ │ (thin) │ │ │ │ Backend │ │ +│ └───────────┘ │ │ └───────────┘ │ +└─────────────────┘ └─────────────────┘ +``` + +### Setup + +1. Download JetBrains Gateway +2. Configure SSH connection +3. Gateway installs IDE backend on remote +4. Develop with full IDE features + +### Connection Options + +| Method | Description | +|--------|-------------| +| SSH | Direct SSH connection | +| JetBrains Space | JetBrains cloud platform | +| Gitpod | Gitpod workspace | +| GitHub Codespaces | Codespace connection | +| Google Cloud | GCP workstations | + +### Project Configuration + +```yaml +# .idea/remote-dev.yaml +remoteProjectPath: /home/user/project +ide: IntelliJ IDEA Ultimate +``` + +--- + +## GitHub Codespaces + +Cloud development environments from GitHub. + +### Features + +- **Instant environments** — Pre-built from repo +- **VS Code or JetBrains** — IDE choice +- **Dotfiles** — Personalization +- **Prebuilds** — Faster startup + +### Configuration + +```json +// .devcontainer/devcontainer.json +{ + "name": "My Project", + "image": "mcr.microsoft.com/devcontainers/base:ubuntu", + "features": { + "ghcr.io/devcontainers/features/node:1": { + "version": "20" + }, + "ghcr.io/devcontainers/features/docker-in-docker:2": {} + }, + "postCreateCommand": "npm install", + "forwardPorts": [3000, 5432], + "portsAttributes": { + "3000": { + "label": "Application", + "onAutoForward": "openPreview" + } + } +} +``` + +### Prebuilds + +```yaml +# .github/workflows/codespaces-prebuild.yml +# Configured in repo settings, not as workflow +# Settings → Codespaces → Prebuilds +``` + +### Usage + +```bash +# CLI +gh codespace create +gh codespace list +gh codespace ssh +gh codespace code # Open in VS Code +gh codespace delete + +# Machine types +gh codespace create --machine largePremiumLinux +``` + +### Dotfiles + +GitHub automatically clones your dotfiles repo: + +```bash +# Create repo: github.com/username/dotfiles +# Contains: +# - .bashrc or .zshrc +# - .gitconfig +# - install.sh (runs automatically) +``` + +--- + +## Gitpod + +Open-source cloud development platform. + +### Features + +- **Ephemeral environments** — Fresh every time +- **Prebuilds** — Pre-compile on push +- **VS Code and JetBrains** — IDE choice +- **Self-hostable** — Run on your infrastructure + +### Configuration + +```yaml +# .gitpod.yml +image: + file: .gitpod.Dockerfile + +tasks: + - name: Setup + init: | + npm install + npm run build + command: npm run dev + + - name: Database + command: docker-compose up db + +ports: + - port: 3000 + onOpen: open-preview + visibility: public + - port: 5432 + onOpen: ignore + +vscode: + extensions: + - dbaeumer.vscode-eslint + - esbenp.prettier-vscode + +jetbrains: + intellij: + plugins: + - com.intellij.plugins.vscodekeymap +``` + +```dockerfile +# .gitpod.Dockerfile +FROM gitpod/workspace-full + +RUN npm install -g pnpm +``` + +### Prebuilds + +```yaml +# .gitpod.yml +github: + prebuilds: + master: true + branches: true + pullRequests: true + pullRequestsFromForks: false + addComment: true + addBadge: true +``` + +### Usage + +```bash +# Open in Gitpod +# Prefix any GitHub URL with: gitpod.io/# +# https://gitpod.io/#https://github.com/user/repo + +# CLI +gitpod workspace create +gitpod workspace list +``` + +--- + +## Coder + +Self-hosted remote development platform. + +### Features + +- **Self-hosted** — Your infrastructure +- **Any IDE** — VS Code, JetBrains, Vim, etc. +- **Templates** — Terraform-based environments +- **Enterprise** — SSO, audit logs, quotas + +### Template Example + +```hcl +# template.tf +terraform { + required_providers { + coder = { source = "coder/coder" } + docker = { source = "kreuzwerker/docker" } + } +} + +data "coder_workspace" "me" {} + +resource "docker_container" "workspace" { + name = "coder-${data.coder_workspace.me.owner}-${data.coder_workspace.me.name}" + image = "codercom/enterprise-base:ubuntu" + + env = [ + "CODER_AGENT_TOKEN=${coder_agent.main.token}" + ] + + volumes { + host_path = "/home/coder" + container_path = "/home/coder" + } +} + +resource "coder_agent" "main" { + os = "linux" + arch = "amd64" +} + +resource "coder_app" "code-server" { + agent_id = coder_agent.main.id + slug = "code-server" + display_name = "VS Code" + url = "http://localhost:13337" + icon = "/icon/code.svg" +} +``` + +--- + +## DevPod + +Open-source dev containers, anywhere. + +### Features + +- **Provider agnostic** — Local, SSH, cloud +- **Devcontainer spec** — Standard configuration +- **No vendor lock-in** — Open source +- **CLI and GUI** — Both available + +### Usage + +```bash +# Install +brew install loft-sh/tap/devpod + +# Add provider +devpod provider add docker +devpod provider add ssh +devpod provider add aws + +# Create workspace +devpod up github.com/user/repo +devpod up . --provider docker + +# Open in IDE +devpod up . --ide vscode +devpod up . --ide goland +devpod up . --ide openvscode # Browser + +# SSH into workspace +devpod ssh my-workspace + +# List +devpod list +devpod delete my-workspace +``` + +--- + +## Comparison + +### Feature Matrix + +| Feature | VS Code Remote | Gateway | Codespaces | Gitpod | +|---------|:--------------:|:-------:|:----------:|:------:| +| SSH | ✅ | ✅ | Via CLI | ❌ | +| Containers | ✅ | ✅ | ✅ | ✅ | +| Cloud hosted | Via tunnels | ❌ | ✅ | ✅ | +| Self-hostable | ❌ | ❌ | ❌ | ✅ | +| Prebuilds | ❌ | ❌ | ✅ | ✅ | +| Cost | Free | License | Usage | Usage | +| IDE | VS Code | JetBrains | Both | Both | + +### Latency Considerations + +| Approach | Latency | Best For | +|----------|---------|----------| +| SSH + terminal Vim/Neovim | Lowest | Experienced devs | +| VS Code Remote SSH | Low | General development | +| JetBrains Gateway | Low-Medium | JetBrains users | +| Cloud environments | Medium | Standardization | +| Browser-based | Higher | Quick access | + +--- + +## Decision Guide + +| Scenario | Recommendation | +|----------|----------------| +| Existing SSH server | VS Code Remote SSH | +| JetBrains user | JetBrains Gateway | +| Team standardization | Codespaces or Gitpod | +| Self-hosted requirement | Coder or Gitpod | +| Quick contribution | Codespaces (GitHub) | +| Open source, flexible | DevPod | +| Corporate firewall | Remote Tunnels | +| Low bandwidth | Terminal + Neovim | + +--- + +## Related + +- [[Terminal Emulators]] +- [[Terminal Multiplexers]] +- [[Container Runtimes]] +- [[WSL]] diff --git a/tools/Shells.md b/tools/Shells.md new file mode 100644 index 0000000..d998729 --- /dev/null +++ b/tools/Shells.md @@ -0,0 +1,454 @@ +--- +title: Shells +aliases: + - Command Line Shells + - Bash vs Zsh vs Fish + - PowerShell +tags: + - tool + - comparison + - shell + - cli + - terminal +type: comparison +status: complete +created: 2025-12-04 +--- + +# Shells + +Command-line interpreters for interacting with operating systems and scripting. + +## Overview + +| Aspect | Bash | Zsh | Fish | PowerShell | +|--------|------|-----|------|------------| +| Full name | Bourne Again Shell | Z Shell | Friendly Interactive Shell | PowerShell | +| First release | 1989 | 1990 | 2005 | 2006 | +| Default on | Linux, older macOS | macOS (10.15+) | — | Windows | +| Config file | `.bashrc`, `.bash_profile` | `.zshrc` | `config.fish` | `$PROFILE` | +| Scripting | POSIX-ish | POSIX + extensions | Own syntax | .NET-based | +| Philosophy | Compatibility | Power user features | User-friendly | Object pipeline | + +--- + +## Bash + +The standard shell on most Linux systems. POSIX-compatible. + +### Key Characteristics + +- **Ubiquitous** — Available on virtually all Unix-like systems +- **POSIX-compatible** — Scripts are portable +- **Mature** — Extensive documentation, huge community +- **Default** — What most tutorials and scripts assume + +### Configuration + +```bash +# ~/.bashrc (interactive non-login) +# ~/.bash_profile (login shells) + +# Prompt customization +export PS1="\u@\h:\w\$ " + +# Aliases +alias ll='ls -la' +alias gs='git status' + +# Functions +mkcd() { mkdir -p "$1" && cd "$1"; } +``` + +### Scripting + +```bash +#!/bin/bash + +# Variables +name="world" +echo "Hello, $name" + +# Arrays +arr=(one two three) +echo "${arr[1]}" # "two" + +# Conditionals +if [[ -f "$file" ]]; then + echo "File exists" +fi + +# Loops +for i in {1..5}; do + echo "$i" +done +``` + +### Considerations + +- Limited interactive features out of box +- No built-in autocomplete for commands +- Syntax quirks (quoting, spacing matters) +- `[[` vs `[` vs `test` confusion + +--- + +## Zsh + +Bash-compatible with powerful extensions. macOS default since Catalina. + +### Key Characteristics + +- **Bash-compatible** — Most bash scripts work +- **Better completion** — Tab completion for commands, flags, arguments +- **Glob patterns** — Extended globbing (`**/*.ts`) +- **Frameworks** — Oh My Zsh, Prezto ecosystems +- **Themes** — Powerlevel10k, etc. + +### Configuration + +```bash +# ~/.zshrc + +# Enable extended globbing +setopt extended_glob + +# History settings +HISTSIZE=10000 +SAVEHIST=10000 +setopt share_history + +# Better completion +autoload -Uz compinit && compinit +``` + +### Oh My Zsh + +```bash +# Install +sh -c "$(curl -fsSL https://raw.github.com/ohmyzsh/ohmyzsh/master/tools/install.sh)" + +# ~/.zshrc with Oh My Zsh +plugins=( + git + docker + kubectl + zsh-autosuggestions + zsh-syntax-highlighting +) +ZSH_THEME="robbyrussell" # or "powerlevel10k/powerlevel10k" +``` + +### Zsh vs Bash + +| Feature | Bash | Zsh | +|---------|------|-----| +| Tab completion | Basic | Advanced (context-aware) | +| Spelling correction | No | Yes (`setopt correct`) | +| Glob patterns | Basic | Extended (`**/*.ts`) | +| Prompt themes | DIY | Rich ecosystem | +| Right-side prompt | No | Yes (RPROMPT) | +| Array indexing | 0-based | 1-based (configurable) | +| Plugin ecosystem | Limited | Oh My Zsh, Prezto | + +### Considerations + +- Slightly slower startup (especially with plugins) +- 1-based arrays can confuse bash users +- Oh My Zsh can become bloated +- Minor incompatibilities with bash scripts + +--- + +## Fish + +User-friendly shell that works out of the box. + +### Key Characteristics + +- **Autosuggestions** — Built-in, based on history +- **Syntax highlighting** — Real-time, in the prompt +- **Web config** — `fish_config` opens browser UI +- **No configuration needed** — Great defaults +- **Not POSIX** — Own scripting syntax + +### Configuration + +```fish +# ~/.config/fish/config.fish + +# Variables (no export keyword) +set -x PATH $HOME/bin $PATH + +# Aliases (actually functions) +alias ll 'ls -la' + +# Functions +function mkcd + mkdir -p $argv[1] && cd $argv[1] +end +``` + +### Scripting Differences + +```fish +# Fish syntax differs from bash/zsh + +# Variables +set name "world" +echo "Hello, $name" + +# Conditionals (no [[ ]]) +if test -f $file + echo "File exists" +end + +# Loops +for i in (seq 1 5) + echo $i +end + +# Command substitution +set files (ls *.txt) + +# No && or ||, use and/or +command1; and command2 +command1; or command2 +``` + +### Key Features + +```fish +# Autosuggestions (accept with →) +$ git commit -m "last message" + +# Abbreviations (expand on space) +abbr -a gc 'git commit' +# Type "gc " → expands to "git commit " + +# Universal variables (persist across sessions) +set -U fish_greeting "" +``` + +### Considerations + +- Not POSIX — can't run bash scripts directly +- Fewer tutorials written for fish +- Some tools assume bash +- Abbreviations > aliases philosophy + +--- + +## PowerShell + +Object-oriented shell from Microsoft. Cross-platform since PowerShell Core. + +### Key Characteristics + +- **Object pipeline** — Pass .NET objects, not text +- **Verb-Noun commands** — `Get-Process`, `Set-Location` +- **Cross-platform** — Windows, macOS, Linux +- **.NET integration** — Full access to .NET libraries +- **Structured output** — Objects with properties + +### Configuration + +```powershell +# $PROFILE (create if doesn't exist) +# Usually: ~\Documents\PowerShell\Microsoft.PowerShell_profile.ps1 + +# Aliases +Set-Alias -Name g -Value git +Set-Alias -Name ll -Value Get-ChildItem + +# Functions +function mkcd { param($path) New-Item -ItemType Directory $path; Set-Location $path } + +# Prompt customization +function prompt { + "$($executionContext.SessionState.Path.CurrentLocation)> " +} +``` + +### Object Pipeline + +```powershell +# Objects, not text +Get-Process | Where-Object { $_.CPU -gt 10 } | Select-Object Name, CPU + +# vs Unix (parsing text) +ps aux | awk '$3 > 10 {print $11, $3}' + +# Properties are accessible +$procs = Get-Process +$procs[0].Name +$procs | ForEach-Object { $_.WorkingSet64 / 1MB } +``` + +### Common Commands + +| Unix | PowerShell | Alias | +|------|------------|-------| +| `ls` | `Get-ChildItem` | `dir`, `ls`, `gci` | +| `cd` | `Set-Location` | `cd`, `sl` | +| `cat` | `Get-Content` | `cat`, `gc` | +| `rm` | `Remove-Item` | `rm`, `ri` | +| `cp` | `Copy-Item` | `cp`, `copy` | +| `mv` | `Move-Item` | `mv`, `move` | +| `grep` | `Select-String` | `sls` | +| `echo` | `Write-Output` | `echo` | + +### PowerShell vs Unix Shells + +| Aspect | PowerShell | Unix Shells | +|--------|------------|-------------| +| Pipeline | Objects | Text streams | +| Commands | Verb-Noun | Short names | +| Scripting | C#-like | POSIX/Bash | +| Error handling | Try/Catch | Exit codes | +| Variables | `$var` | `$var` | +| Cross-platform | Yes (Core) | Native | +| Windows integration | Excellent | WSL needed | + +### Considerations + +- Verbose command names (use aliases) +- Steeper learning curve for Unix users +- Different mental model (objects vs text) +- Less common on servers (except Windows) + +--- + +## Feature Matrix + +| Feature | Bash | Zsh | Fish | PowerShell | +|---------|:----:|:---:|:----:|:----------:| +| POSIX compatible | ✅ | ✅ | ❌ | ❌ | +| Autosuggestions | ❌ | Plugin | ✅ | Plugin | +| Syntax highlighting | ❌ | Plugin | ✅ | Plugin | +| Tab completion | Basic | Excellent | Excellent | Excellent | +| Spell correction | ❌ | ✅ | ✅ | ❌ | +| Plugin ecosystem | Limited | Rich | Growing | PSGallery | +| Web-based config | ❌ | ❌ | ✅ | ❌ | +| Object pipeline | ❌ | ❌ | ❌ | ✅ | +| Cross-platform | Unix | Unix | Unix | ✅ | +| Windows native | WSL/Git Bash | WSL | WSL | ✅ | + +--- + +## Plugin Managers & Frameworks + +### Zsh + +| Tool | Purpose | +|------|---------| +| Oh My Zsh | Framework with plugins/themes | +| Prezto | Lighter alternative to OMZ | +| zinit | Fast plugin manager | +| zplug | Feature-rich plugin manager | + +### Fish + +| Tool | Purpose | +|------|---------| +| Fisher | Plugin manager | +| Oh My Fish | Framework (like OMZ) | +| Tide | Powerlevel10k-like prompt | + +### PowerShell + +| Tool | Purpose | +|------|---------| +| PSReadLine | Enhanced editing (built-in now) | +| oh-my-posh | Cross-shell prompt themes | +| Terminal-Icons | File icons in listing | +| z | Directory jumper | + +--- + +## Prompt Customization + +### Cross-Shell (Starship) + +Works with all shells. Written in Rust. + +```bash +# Install +curl -sS https://starship.rs/install.sh | sh + +# Bash: add to .bashrc +eval "$(starship init bash)" + +# Zsh: add to .zshrc +eval "$(starship init zsh)" + +# Fish: add to config.fish +starship init fish | source + +# PowerShell: add to $PROFILE +Invoke-Expression (&starship init powershell) +``` + +--- + +## Script Portability + +``` +Portability spectrum: + +Most portable ←————————————————————→ Least portable + + sh/POSIX Bash Zsh Fish PowerShell + │ │ │ │ │ + └──────────┴─────────┘ └────────────┘ + Works on Unix Own ecosystem +``` + +### Writing Portable Scripts + +```bash +#!/bin/sh +# Use /bin/sh for maximum portability +# Avoid bash-isms: [[ ]], arrays, <<<, etc. + +# Instead of [[ ]] +if [ -f "$file" ]; then + echo "exists" +fi + +# Instead of $(()) +result=$(expr 1 + 1) +``` + +--- + +## Decision Guide + +| Use Case | Recommendation | +|----------|----------------| +| Maximum compatibility | Bash | +| Power user on macOS/Linux | Zsh + Oh My Zsh | +| Best out-of-box experience | Fish | +| Windows administration | PowerShell | +| Cross-platform scripts | Bash or PowerShell Core | +| Interactive daily use | Fish or Zsh | +| CI/CD pipelines | Bash | +| Minimal configuration | Fish | +| Windows + Unix hybrid | PowerShell Core | + +### Summary + +| Shell | Choose When | +|-------|-------------| +| **Bash** | Need portability, writing scripts, Linux servers, CI/CD | +| **Zsh** | Want bash compatibility + better UX, macOS users, customization | +| **Fish** | Want great defaults, minimal config, don't need POSIX | +| **PowerShell** | Windows environment, .NET integration, object manipulation | + +--- + +## Related + +- [[Terminal UI & Language Features]] +- [[Build Systems]] +- [[Deployment]] diff --git a/tools/Terminal Emulators.md b/tools/Terminal Emulators.md new file mode 100644 index 0000000..1454b88 --- /dev/null +++ b/tools/Terminal Emulators.md @@ -0,0 +1,478 @@ +--- +title: Terminal Emulators +aliases: + - Terminals + - Terminal Apps +tags: + - tool + - comparison + - terminal + - cli +type: comparison +status: complete +created: 2025-12-04 +--- + +# Terminal Emulators + +Modern terminal applications for running shells and CLI programs. + +## Overview + +| Terminal | Platform | GPU Accel | Written In | License | +|----------|----------|:---------:|------------|---------| +| Ghostty | macOS, Linux | ✅ | Zig | MIT | +| Alacritty | Cross-platform | ✅ | Rust | Apache 2.0 | +| Kitty | macOS, Linux | ✅ | Python/C | GPL-3.0 | +| Warp | macOS, Linux | ✅ | Rust | Proprietary | +| iTerm2 | macOS | ❌ | Objective-C | GPL-2.0 | +| Windows Terminal | Windows | ✅ | C++ | MIT | +| Hyper | Cross-platform | ❌ | Electron | MIT | +| WezTerm | Cross-platform | ✅ | Rust | MIT | + +--- + +## Ghostty + +New GPU-accelerated terminal by Mitchell Hashimoto (HashiCorp founder). + +### Key Characteristics + +- **Native** — Platform-native UI (macOS Cocoa, GTK on Linux) +- **Fast** — Zig + GPU acceleration +- **Zero config** — Works great out of box +- **Ligatures** — Full font ligature support +- **Minimal** — Focused feature set + +### Configuration + +``` +# ~/.config/ghostty/config + +font-family = JetBrains Mono +font-size = 14 +theme = catppuccin-mocha + +window-padding-x = 10 +window-padding-y = 10 + +cursor-style = block +cursor-style-blink = false +``` + +### Strengths + +- Extremely fast rendering +- Native look and feel +- Simple configuration +- Low resource usage +- Active development + +### Considerations + +- Newer, smaller ecosystem +- No Windows support yet +- Fewer features than Kitty/WezTerm + +--- + +## Alacritty + +"The fastest terminal emulator in existence" — minimal, GPU-accelerated. + +### Key Characteristics + +- **Speed** — OpenGL rendering, no bloat +- **Cross-platform** — macOS, Linux, Windows, BSD +- **Minimal** — No tabs, splits, ligatures (by design) +- **Vi mode** — Built-in vi keybindings + +### Configuration + +```yaml +# ~/.config/alacritty/alacritty.toml + +[font] +normal = { family = "JetBrains Mono", style = "Regular" } +size = 14.0 + +[window] +padding = { x = 10, y = 10 } +decorations = "Buttonless" +opacity = 0.95 + +[cursor] +style = { shape = "Block", blinking = "Off" } +``` + +### Strengths + +- Blazing fast +- Simple, predictable +- Cross-platform consistency +- Low memory usage + +### Considerations + +- No tabs/splits (use tmux) +- No ligatures +- TOML config only +- Minimal feature set by philosophy + +--- + +## Kitty + +Feature-rich, GPU-accelerated terminal with unique capabilities. + +### Key Characteristics + +- **GPU rendering** — OpenGL-based +- **Kitten plugins** — Extensible via Python +- **Graphics protocol** — Display images inline +- **Layouts** — Built-in tabs, splits, layouts +- **Remote control** — Control via CLI/scripts + +### Configuration + +```conf +# ~/.config/kitty/kitty.conf + +font_family JetBrains Mono +font_size 14.0 + +background_opacity 0.95 +window_padding_width 10 + +enable_audio_bell no +tab_bar_style powerline + +# Keyboard shortcuts +map ctrl+shift+t new_tab +map ctrl+shift+enter new_window +``` + +### Kittens (Plugins) + +```bash +# View images in terminal +kitty +kitten icat image.png + +# SSH with automatic shell integration +kitty +kitten ssh user@host + +# Show diffs with syntax highlighting +kitty +kitten diff file1 file2 + +# Unicode input +kitty +kitten unicode_input +``` + +### Strengths + +- Image display in terminal +- Rich feature set +- Highly configurable +- Active development +- Good documentation + +### Considerations + +- Config syntax can be verbose +- Some features Linux-focused +- Heavier than Alacritty + +--- + +## Warp + +AI-powered terminal with modern UX. Reimagines the terminal experience. + +### Key Characteristics + +- **Block-based** — Commands grouped as blocks +- **AI integration** — Natural language to commands +- **Workflows** — Saved/shared command sequences +- **Modern input** — IDE-like text editing +- **Collaboration** — Share sessions + +### Features + +``` +Warp innovations: +├── Command blocks (select, copy, share) +├── AI command search (describe what you want) +├── Completions (context-aware) +├── Workflows (parameterized snippets) +├── Warp Drive (cloud sync) +└── IDE-style editing (multi-cursor, etc.) +``` + +### Strengths + +- Modern, intuitive UX +- AI assistance built-in +- Great for beginners +- Active development +- Beautiful default theme + +### Considerations + +- Requires account signup +- Proprietary/closed source +- Different mental model +- Cloud features may raise privacy concerns +- Resource heavier + +--- + +## iTerm2 + +The classic macOS terminal. Feature-rich, battle-tested. + +### Key Characteristics + +- **macOS native** — Deep OS integration +- **Mature** — 15+ years of development +- **Feature-complete** — Everything you might need +- **Profiles** — Multiple configurations + +### Key Features + +``` +iTerm2 features: +├── Split panes (horizontal, vertical) +├── Hotkey window (drop-down terminal) +├── Search with regex +├── Autocomplete +├── Shell integration +├── Triggers (pattern → action) +├── Password manager +├── tmux integration +└── Instant replay +``` + +### Configuration + +Preferences GUI-based, but can export/import JSON. + +### Strengths + +- Rock solid stability +- Comprehensive features +- Great tmux integration +- Shell integration (marks, navigation) +- Mature ecosystem + +### Considerations + +- macOS only +- Not GPU-accelerated +- Can feel dated compared to newer terminals +- Memory usage higher with many tabs + +--- + +## Windows Terminal + +Microsoft's modern terminal for Windows. + +### Key Characteristics + +- **Modern Windows terminal** — Replaces cmd.exe, PowerShell windows +- **Multiple profiles** — PowerShell, WSL, cmd, Azure Cloud Shell +- **GPU-accelerated** — DirectX rendering +- **Customizable** — Themes, keybindings, appearance + +### Configuration + +```json +// settings.json +{ + "defaultProfile": "{574e775e-4f2a-5b96-ac1e-a2962a402336}", + "profiles": { + "defaults": { + "font": { "face": "JetBrains Mono", "size": 14 }, + "opacity": 95, + "useAcrylic": true + }, + "list": [ + { "name": "PowerShell", "source": "Windows.Terminal.PowershellCore" }, + { "name": "Ubuntu", "source": "Windows.Terminal.Wsl" } + ] + }, + "schemes": [{ "name": "One Dark", ... }] +} +``` + +### Strengths + +- Best Windows terminal experience +- WSL integration +- Multiple shell profiles +- Quake mode (dropdown) +- Active Microsoft support + +### Considerations + +- Windows only +- JSON config can be verbose +- Some features still maturing + +--- + +## WezTerm + +Cross-platform terminal with Lua configuration. + +### Key Characteristics + +- **Lua config** — Full programming language for config +- **Multiplexer built-in** — Tabs, panes, workspaces +- **Cross-platform** — Windows, macOS, Linux +- **GPU-accelerated** — Fast rendering +- **SSH integration** — Built-in multiplexing over SSH + +### Configuration + +```lua +-- ~/.config/wezterm/wezterm.lua +local wezterm = require 'wezterm' +local config = {} + +config.font = wezterm.font 'JetBrains Mono' +config.font_size = 14.0 +config.color_scheme = 'Catppuccin Mocha' + +config.window_background_opacity = 0.95 +config.enable_tab_bar = true + +config.keys = { + { key = 't', mods = 'CTRL|SHIFT', action = wezterm.action.SpawnTab 'CurrentPaneDomain' }, +} + +return config +``` + +### Strengths + +- Lua = dynamic, programmable config +- Feature-rich like Kitty +- True cross-platform +- Built-in multiplexer +- Serial port support + +### Considerations + +- Lua config steeper learning curve +- Less mainstream than others +- Some platform differences + +--- + +## Hyper + +Electron-based terminal. Extensible via web technologies. + +### Key Characteristics + +- **Electron** — HTML/CSS/JS based +- **Plugins** — npm ecosystem +- **Themeable** — Web technologies +- **Cross-platform** — Anywhere Electron runs + +### Configuration + +```javascript +// ~/.hyper.js +module.exports = { + config: { + fontSize: 14, + fontFamily: '"JetBrains Mono", monospace', + cursorShape: 'BLOCK', + shell: '/bin/zsh', + plugins: [ + 'hyper-one-dark', + 'hyper-search', + 'hypercwd', + ], + }, +}; +``` + +### Strengths + +- Easy to theme (CSS) +- npm plugin ecosystem +- Familiar for web developers +- Cross-platform + +### Considerations + +- Electron = higher resource usage +- Slower than native terminals +- Can feel sluggish +- Battery impact on laptops + +--- + +## Feature Matrix + +| Feature | Ghostty | Alacritty | Kitty | Warp | iTerm2 | WinTerm | WezTerm | +|---------|:-------:|:---------:|:-----:|:----:|:------:|:-------:|:-------:| +| GPU accelerated | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | +| Tabs built-in | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | +| Splits built-in | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | +| Ligatures | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | +| Images in terminal | ❌ | ❌ | ✅ | ✅ | ✅ | ❌ | ✅ | +| Config language | Custom | TOML | Custom | GUI | GUI | JSON | Lua | +| Scrollback search | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | +| macOS | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | +| Linux | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ | +| Windows | ❌ | ✅ | ❌ | ❌ | ❌ | ✅ | ✅ | + +--- + +## Performance Comparison + +| Terminal | Startup | Latency | Memory | Battery | +|----------|---------|---------|--------|---------| +| Ghostty | Excellent | Excellent | Low | Good | +| Alacritty | Excellent | Excellent | Lowest | Good | +| Kitty | Good | Excellent | Medium | Good | +| Warp | Medium | Good | Higher | Medium | +| iTerm2 | Medium | Good | Higher | Medium | +| WezTerm | Good | Excellent | Medium | Good | +| Hyper | Slow | Medium | High | Poor | + +--- + +## Decision Guide + +| Use Case | Recommendation | +|----------|----------------| +| macOS, want native feel | Ghostty or iTerm2 | +| Absolute speed, minimalism | Alacritty + tmux | +| Feature-rich, images, extensible | Kitty or WezTerm | +| New to terminal, want guidance | Warp | +| Windows primary | Windows Terminal | +| Cross-platform consistency | Alacritty, WezTerm, or Kitty | +| Web developer, want plugins | Hyper | +| Programmable config | WezTerm (Lua) | + +### Quick Picks + +| Profile | Terminal | +|---------|----------| +| **Minimalist** | Alacritty | +| **Power user** | Kitty or WezTerm | +| **macOS native** | Ghostty or iTerm2 | +| **Modern UX** | Warp | +| **Windows** | Windows Terminal | + +--- + +## Related + +- [[Shells]] +- [[Terminal UI & Language Features]] diff --git a/tools/Terminal Multiplexers.md b/tools/Terminal Multiplexers.md new file mode 100644 index 0000000..2f5d74f --- /dev/null +++ b/tools/Terminal Multiplexers.md @@ -0,0 +1,489 @@ +--- +title: Terminal Multiplexers +aliases: + - tmux vs screen + - Multiplexers +tags: + - tool + - comparison + - terminal + - cli +type: comparison +status: complete +created: 2025-12-04 +--- + +# Terminal Multiplexers + +Tools for managing multiple terminal sessions within a single window. + +## Overview + +| Tool | Written In | Config | Session Persist | Pane Splits | +|------|------------|--------|-----------------|-------------| +| tmux | C | `.tmux.conf` | ✅ | ✅ | +| screen | C | `.screenrc` | ✅ | Limited | +| Zellij | Rust | KDL | ✅ | ✅ | +| byobu | Shell (wrapper) | Multiple | ✅ | ✅ | + +--- + +## Why Use a Multiplexer? + +``` +Without multiplexer: With multiplexer: +┌─────────────────────┐ ┌─────────────────────┐ +│ SSH Session │ │ SSH Session │ +│ │ │ ┌─────┬─────┬─────┐ │ +│ Single terminal │ │ │vim │logs │build│ │ +│ │ │ ├─────┴─────┴─────┤ │ +│ Disconnect = lose │ │ │ shell │ │ +│ everything │ │ └─────────────────┘ │ +└─────────────────────┘ │ Persists on │ + │ disconnect! │ + └─────────────────────┘ +``` + +**Key benefits:** +- Sessions persist when disconnected +- Multiple panes in one terminal +- Switch between workspaces +- Remote pair programming +- Consistent environment across machines + +--- + +## tmux + +Terminal MUltipleXer. The modern standard. + +### Concepts + +``` +tmux hierarchy: +┌─────────────────────────────────────────────────────────┐ +│ Server (background daemon) │ +│ ├── Session: "dev" │ +│ │ ├── Window 0: "editor" │ +│ │ │ ├── Pane 0: vim │ +│ │ │ └── Pane 1: terminal │ +│ │ └── Window 1: "server" │ +│ │ └── Pane 0: npm run dev │ +│ └── Session: "work" │ +│ └── Window 0: "main" │ +│ └── Pane 0: shell │ +└─────────────────────────────────────────────────────────┘ +``` + +### Essential Commands + +```bash +# Sessions +tmux # New session +tmux new -s name # New named session +tmux ls # List sessions +tmux attach -t name # Attach to session +tmux kill-session -t name # Kill session + +# Prefix key: Ctrl-b (default) +# All shortcuts start with prefix + +# After prefix: +d # Detach +:new # New session +$ # Rename session +s # List sessions +( # Previous session +) # Next session +``` + +### Windows (Tabs) + +```bash +# After prefix: +c # Create window +, # Rename window +n # Next window +p # Previous window +0-9 # Switch to window N +& # Kill window +w # List windows +``` + +### Panes (Splits) + +```bash +# After prefix: +% # Split vertical +" # Split horizontal +o # Switch pane +x # Kill pane +z # Toggle zoom (fullscreen) +{ # Swap pane left +} # Swap pane right +Arrow keys # Navigate panes +Ctrl+Arrow # Resize pane +Space # Cycle layouts +``` + +### Copy Mode + +```bash +# After prefix: +[ # Enter copy mode +# In copy mode (vi keys): +Space # Start selection +Enter # Copy and exit +q # Exit copy mode + +# Paste +] # Paste buffer +``` + +### Configuration + +```bash +# ~/.tmux.conf + +# Change prefix to Ctrl-a +unbind C-b +set-option -g prefix C-a +bind-key C-a send-prefix + +# Vi mode +setw -g mode-keys vi + +# Mouse support +set -g mouse on + +# Start windows at 1 +set -g base-index 1 +setw -g pane-base-index 1 + +# Split shortcuts +bind | split-window -h +bind - split-window -v + +# Vim-style pane navigation +bind h select-pane -L +bind j select-pane -D +bind k select-pane -U +bind l select-pane -R + +# Reload config +bind r source-file ~/.tmux.conf \; display "Reloaded!" + +# Status bar +set -g status-style 'bg=#333333 fg=#ffffff' +set -g status-right '%Y-%m-%d %H:%M' + +# Increase scrollback +set -g history-limit 10000 +``` + +--- + +## screen + +GNU Screen. The original multiplexer. + +### Basic Usage + +```bash +# Sessions +screen # New session +screen -S name # New named session +screen -ls # List sessions +screen -r name # Reattach +screen -d -r name # Detach elsewhere, attach here +screen -X -S name quit # Kill session + +# Prefix key: Ctrl-a (default) +``` + +### After Prefix (Ctrl-a) + +```bash +# Windows +c # Create window +n # Next window +p # Previous window +0-9 # Switch to window N +" # List windows +A # Rename window +k # Kill window + +# Splits (limited) +S # Split horizontal +| # Split vertical (newer versions) +Tab # Switch region +X # Close region + +# Other +d # Detach +? # Help +[ # Copy mode +] # Paste +``` + +### Configuration + +```bash +# ~/.screenrc + +# Change escape key +escape ^Aa + +# Visual bell +vbell on + +# Scrollback +defscrollback 10000 + +# Status line +hardstatus alwayslastline +hardstatus string '%{= kG}[ %{G}%H %{g}][%= %{= kw}%?%-Lw%?%{r}(%{W}%n*%f%t%?(%u)%?%{r})%{w}%?%+Lw%?%?%= %{g}][%{B} %m-%d %{W}%c %{g}]' + +# Start with multiple windows +screen -t shell 0 +screen -t vim 1 +screen -t logs 2 +``` + +--- + +## Zellij + +Modern, user-friendly multiplexer. Written in Rust. + +### Key Features + +- **Discoverable UI** — On-screen hints +- **Layouts** — Declarative, saveable +- **Floating panes** — Popup windows +- **Sessions** — Built-in session management +- **Plugins** — WebAssembly plugins + +### Basic Usage + +```bash +# Start +zellij # New session +zellij -s name # Named session +zellij ls # List sessions +zellij attach name # Attach +zellij kill-session name # Kill + +# Built-in help shown at bottom of screen +``` + +### Keybindings + +```bash +# Modes (press key to enter mode) +Ctrl+g # Locked (disable zellij keys) +Ctrl+p # Pane mode +Ctrl+t # Tab mode +Ctrl+n # Resize mode +Ctrl+h # Move mode +Ctrl+s # Scroll mode +Ctrl+o # Session mode +Ctrl+q # Quit + +# In Pane mode (Ctrl+p): +n # New pane (down) +d # New pane (down) +r # New pane (right) +x # Close pane +f # Toggle fullscreen +w # Toggle floating +hjkl / arrows # Navigate + +# In Tab mode (Ctrl+t): +n # New tab +x # Close tab +r # Rename tab +1-9 # Go to tab +``` + +### Layouts + +```kdl +// ~/.config/zellij/layouts/dev.kdl +layout { + pane size=1 borderless=true { + plugin location="tab-bar" + } + pane split_direction="vertical" { + pane command="nvim" + pane split_direction="horizontal" { + pane command="npm" args=["run", "dev"] + pane // empty shell + } + } + pane size=2 borderless=true { + plugin location="status-bar" + } +} +``` + +```bash +# Use layout +zellij --layout dev +``` + +### Configuration + +```kdl +// ~/.config/zellij/config.kdl + +keybinds { + normal { + bind "Alt h" { MoveFocus "Left"; } + bind "Alt l" { MoveFocus "Right"; } + bind "Alt j" { MoveFocus "Down"; } + bind "Alt k" { MoveFocus "Up"; } + } +} + +themes { + catppuccin { + fg "#CDD6F4" + bg "#1E1E2E" + // ... + } +} + +theme "catppuccin" + +default_shell "zsh" +default_layout "compact" +``` + +--- + +## byobu + +Wrapper around tmux or screen with better defaults. + +```bash +# Install +sudo apt install byobu # Ubuntu/Debian +brew install byobu # macOS + +# Start (uses tmux by default) +byobu + +# Switch backend +byobu-select-backend # Choose tmux or screen + +# Key bindings (F-keys by default) +F2 # New window +F3 # Previous window +F4 # Next window +F6 # Detach +F7 # Scrollback +F8 # Rename window +F9 # Config menu +``` + +--- + +## Comparison + +### Feature Matrix + +| Feature | tmux | screen | Zellij | +|---------|:----:|:------:|:------:| +| Vertical splits | ✅ | ✅ (newer) | ✅ | +| Horizontal splits | ✅ | ✅ | ✅ | +| Floating panes | ❌ | ❌ | ✅ | +| Mouse support | ✅ | Limited | ✅ | +| Scripting | ✅ | ✅ | Limited | +| Layouts | ✅ | ❌ | ✅ (KDL) | +| Session management | ✅ | ✅ | ✅ | +| Learning curve | Medium | Easy | Easy | +| Visual UI | Minimal | Minimal | Rich | +| Plugin system | Via scripts | ❌ | ✅ (WASM) | + +### Resource Usage + +| Tool | Memory | Startup | +|------|--------|---------| +| tmux | ~5MB | Fast | +| screen | ~3MB | Fast | +| Zellij | ~15MB | Fast | + +--- + +## Common Workflows + +### SSH + Multiplexer + +```bash +# Start persistent session +ssh server +tmux new -s work + +# Later, after disconnect +ssh server +tmux attach -t work # Everything still there! +``` + +### Development Layout + +```bash +# tmux script: ~/.local/bin/dev-session +#!/bin/bash +tmux new-session -d -s dev +tmux rename-window -t dev:0 'editor' +tmux send-keys -t dev:0 'nvim .' C-m +tmux new-window -t dev -n 'server' +tmux send-keys -t dev:1 'npm run dev' C-m +tmux new-window -t dev -n 'shell' +tmux select-window -t dev:0 +tmux attach -t dev +``` + +### Pair Programming + +```bash +# Host +tmux new -s pair + +# Guest (same machine or via SSH) +tmux attach -t pair + +# Both see and can type in same session +``` + +--- + +## Decision Guide + +| Use Case | Recommendation | +|----------|----------------| +| Servers, minimal | tmux or screen | +| New user, want discoverability | Zellij | +| Heavy customization | tmux | +| Already know screen | screen or byobu | +| Modern features, plugins | Zellij | +| Remote pair programming | tmux | +| Quick start, good defaults | byobu or Zellij | + +### Quick Picks + +| Profile | Tool | +|---------|------| +| **Power user** | tmux | +| **Beginner** | Zellij or byobu | +| **Minimal/legacy** | screen | +| **Modern, discoverable** | Zellij | + +--- + +## Related + +- [[Shells]] +- [[Terminal Emulators]] +- [[WSL]] diff --git a/tools/Version Managers.md b/tools/Version Managers.md new file mode 100644 index 0000000..52791c6 --- /dev/null +++ b/tools/Version Managers.md @@ -0,0 +1,522 @@ +--- +title: Version Managers +aliases: + - Runtime Version Managers + - Language Version Managers +tags: + - tool + - comparison + - development +type: comparison +status: complete +created: 2025-12-04 +--- + +# Version Managers + +Tools for managing multiple versions of programming languages and runtimes. + +## Overview + +| Tool | Languages | Shell Integration | Config File | +|------|-----------|-------------------|-------------| +| mise (rtx) | All (polyglot) | All shells | `.mise.toml`, `.tool-versions` | +| asdf | All (polyglot) | Bash, Zsh, Fish | `.tool-versions` | +| nvm | Node.js | Bash, Zsh | `.nvmrc` | +| fnm | Node.js | All shells | `.nvmrc`, `.node-version` | +| pyenv | Python | Bash, Zsh | `.python-version` | +| rbenv | Ruby | Bash, Zsh | `.ruby-version` | +| rustup | Rust | N/A (manages itself) | `rust-toolchain.toml` | +| sdkman | JVM (Java, Kotlin, etc.) | Bash, Zsh | — | + +--- + +## Polyglot Managers + +### mise (formerly rtx) + +Modern, fast, polyglot. Written in Rust. + +```bash +# Install +curl https://mise.run | sh +# or +brew install mise + +# Shell setup (add to shell rc) +eval "$(mise activate bash)" # or zsh, fish + +# Install runtimes +mise install node@20 # Latest 20.x +mise install node@lts # Latest LTS +mise install python@3.12 +mise install go@latest + +# Use globally +mise use -g node@20 +mise use -g python@3.12 + +# Use in project (creates .mise.toml) +mise use node@20 +mise use python@3.12 + +# List versions +mise list node # Installed versions +mise list-remote node # Available versions + +# Show current +mise current # All active versions +mise where node # Path to current node + +# Run commands +mise exec node@18 -- node -v # Run with specific version +mise run build # Run task from mise.toml +``` + +```toml +# .mise.toml +[tools] +node = "20" +python = "3.12" +go = "1.22" + +[env] +NODE_ENV = "development" + +[tasks.build] +run = "npm run build" + +[tasks.test] +run = "npm test" +``` + +### asdf + +Plugin-based, community standard. + +```bash +# Install +git clone https://github.com/asdf-vm/asdf.git ~/.asdf --branch v0.14.0 + +# Shell setup (add to shell rc) +. "$HOME/.asdf/asdf.sh" +. "$HOME/.asdf/completions/asdf.bash" + +# Add plugins +asdf plugin add nodejs +asdf plugin add python +asdf plugin add golang + +# Install versions +asdf install nodejs 20.10.0 +asdf install nodejs latest +asdf install python 3.12.1 + +# Set versions +asdf global nodejs 20.10.0 # Global default +asdf local nodejs 20.10.0 # Project (creates .tool-versions) +asdf shell nodejs 18.0.0 # Current shell only + +# List versions +asdf list nodejs # Installed +asdf list all nodejs # Available + +# Update plugins +asdf plugin update --all +``` + +``` +# .tool-versions +nodejs 20.10.0 +python 3.12.1 +golang 1.22.0 +``` + +### mise vs asdf + +| Aspect | mise | asdf | +|--------|------|------| +| Speed | Faster (Rust) | Slower (Shell) | +| Compatibility | asdf plugins + own | Plugins only | +| Config | `.mise.toml` or `.tool-versions` | `.tool-versions` | +| Tasks | Built-in | No | +| Env vars | Built-in | Via direnv | +| Shell startup | Faster | Slower | + +--- + +## Node.js + +### nvm (Node Version Manager) + +The original. Most widely used. + +```bash +# Install +curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.7/install.sh | bash + +# Shell setup (automatic in install) +export NVM_DIR="$HOME/.nvm" +[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh" + +# Install versions +nvm install 20 # Latest 20.x +nvm install --lts # Latest LTS +nvm install node # Latest + +# Use +nvm use 20 # Switch version +nvm alias default 20 # Set default + +# List +nvm list # Installed +nvm list-remote # Available + +# Run with version +nvm exec 18 node -v # Run with specific version +nvm run 18 app.js # Run script with version + +# Auto-switch (add to shell rc) +autoload -U add-zsh-hook +load-nvmrc() { + if [[ -f .nvmrc ]]; then + nvm use + fi +} +add-zsh-hook chpwd load-nvmrc +``` + +``` +# .nvmrc +20 +``` + +### fnm (Fast Node Manager) + +Rust-based, faster alternative to nvm. + +```bash +# Install +curl -fsSL https://fnm.vercel.app/install | bash +# or +brew install fnm + +# Shell setup +eval "$(fnm env --use-on-cd)" # Auto-switch on cd + +# Install +fnm install 20 +fnm install --lts + +# Use +fnm use 20 +fnm default 20 # Set default + +# List +fnm list # Installed +fnm list-remote # Available +``` + +### n + +Simple, no shell modifications needed. + +```bash +# Install +npm install -g n + +# Install versions +n 20 # Install and switch +n lts # Latest LTS +n latest # Latest + +# Interactive picker +n # Shows picker + +# List +n ls # Installed +n ls-remote # Available +``` + +### Node Version Managers Compared + +| Aspect | nvm | fnm | n | +|--------|-----|-----|---| +| Speed | Slow | Fast | Fast | +| Shell integration | Heavy | Light | None | +| Auto-switch | Via hook | Built-in | No | +| Windows | nvm-windows | Yes | No | + +--- + +## Python + +### pyenv + +Standard Python version manager. + +```bash +# Install +curl https://pyenv.run | bash +# or +brew install pyenv + +# Shell setup +export PYENV_ROOT="$HOME/.pyenv" +export PATH="$PYENV_ROOT/bin:$PATH" +eval "$(pyenv init -)" + +# Install versions +pyenv install 3.12.1 +pyenv install 3.11.7 + +# Set version +pyenv global 3.12.1 # Global default +pyenv local 3.11.7 # Project (.python-version) +pyenv shell 3.10.0 # Current shell + +# List +pyenv versions # Installed +pyenv install --list # Available + +# With virtualenv plugin +pyenv virtualenv 3.12.1 myenv # Create virtualenv +pyenv activate myenv # Activate +pyenv deactivate # Deactivate +``` + +### uv + +Fast Python package and version manager. Written in Rust. + +```bash +# Install +curl -LsSf https://astral.sh/uv/install.sh | sh +# or +brew install uv + +# Install Python +uv python install 3.12 # Install version +uv python install # Install latest + +# Use in projects +uv init # Initialize project +uv add requests # Add dependency +uv sync # Sync environment +uv run python script.py # Run with env + +# Direct usage +uv pip install requests # pip replacement +uv venv # Create venv +``` + +--- + +## Ruby + +### rbenv + +Simple Ruby version manager. + +```bash +# Install +brew install rbenv ruby-build + +# Shell setup +eval "$(rbenv init -)" + +# Install versions +rbenv install 3.3.0 +rbenv install --list # Available versions + +# Set version +rbenv global 3.3.0 # Global +rbenv local 3.2.0 # Project (.ruby-version) + +# Rehash (after installing gems with binaries) +rbenv rehash +``` + +### rvm (Ruby Version Manager) + +Feature-rich, but heavier. + +```bash +# Install +\curl -sSL https://get.rvm.io | bash -s stable + +# Install versions +rvm install 3.3.0 +rvm install ruby --latest + +# Use +rvm use 3.3.0 +rvm use 3.3.0 --default # Set default + +# Gemsets (isolated gem environments) +rvm gemset create myproject +rvm use 3.3.0@myproject +``` + +--- + +## Java/JVM + +### SDKMAN + +JVM ecosystem manager. Java, Kotlin, Gradle, Maven, etc. + +```bash +# Install +curl -s "https://get.sdkman.io" | bash + +# Shell setup (automatic) +source "$HOME/.sdkman/bin/sdkman-init.sh" + +# Install Java +sdk install java # Latest +sdk install java 21.0.1-tem # Specific (Temurin) +sdk install java 21.0.1-graal # GraalVM + +# List available +sdk list java # All Java distributions +sdk list kotlin # Kotlin versions + +# Use +sdk use java 21.0.1-tem # Current shell +sdk default java 21.0.1-tem # Default + +# Other tools +sdk install kotlin +sdk install gradle +sdk install maven +sdk install scala + +# Env file +sdk env init # Create .sdkmanrc +sdk env # Load from .sdkmanrc +``` + +```properties +# .sdkmanrc +java=21.0.1-tem +kotlin=1.9.22 +gradle=8.5 +``` + +--- + +## Rust + +### rustup + +Official Rust toolchain manager. + +```bash +# Install +curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh + +# Toolchains +rustup install stable +rustup install nightly +rustup install 1.75.0 # Specific version + +# Use +rustup default stable # Set default +rustup override set nightly # Project override + +# Components +rustup component add clippy +rustup component add rustfmt +rustup component add rust-analyzer + +# Targets (cross-compilation) +rustup target add wasm32-wasip1 +rustup target add aarch64-apple-darwin + +# Update +rustup update # Update all toolchains +``` + +```toml +# rust-toolchain.toml +[toolchain] +channel = "1.75.0" +components = ["rustfmt", "clippy"] +targets = ["wasm32-wasip1"] +``` + +--- + +## Go + +### goenv + +pyenv-style Go version manager. + +```bash +# Install +git clone https://github.com/go-nv/goenv.git ~/.goenv + +# Shell setup +export GOENV_ROOT="$HOME/.goenv" +export PATH="$GOENV_ROOT/bin:$PATH" +eval "$(goenv init -)" + +# Install +goenv install 1.22.0 + +# Use +goenv global 1.22.0 +goenv local 1.21.0 +``` + +### g + +Simple Go version manager. + +```bash +# Install +curl -sSL https://git.io/g-install | sh -s + +# Use +g install 1.22.0 # Install and switch +g # Interactive picker +``` + +--- + +## Decision Guide + +| Need | Tool | +|------|------| +| Multiple languages | mise or asdf | +| Just Node.js | fnm (fast) or nvm (standard) | +| Just Python | pyenv or uv | +| Just Ruby | rbenv | +| JVM ecosystem | SDKMAN | +| Rust | rustup (only option) | +| Speed priority | mise, fnm, uv | +| Wide compatibility | asdf | +| Simple, minimal | Language-specific tools | + +### Polyglot Setup + +```bash +# Option 1: mise (recommended) +brew install mise +echo 'eval "$(mise activate zsh)"' >> ~/.zshrc + +# Option 2: asdf +git clone https://github.com/asdf-vm/asdf.git ~/.asdf +echo '. $HOME/.asdf/asdf.sh' >> ~/.zshrc +asdf plugin add nodejs python golang ruby +``` + +--- + +## Related + +- [[JavaScript Runtimes]] +- [[Package Managers]] +- [[Shells]] diff --git a/tools/WSL.md b/tools/WSL.md new file mode 100644 index 0000000..5e6a079 --- /dev/null +++ b/tools/WSL.md @@ -0,0 +1,453 @@ +--- +title: WSL +aliases: + - Windows Subsystem for Linux + - WSL2 +tags: + - tool + - windows + - linux + - virtualization +type: reference +status: complete +created: 2025-12-04 +--- + +# WSL + +Windows Subsystem for Linux — run Linux directly on Windows. + +## Overview + +| Aspect | WSL 1 | WSL 2 | +|--------|-------|-------| +| Architecture | Translation layer | Full Linux kernel | +| File system | NTFS via translation | ext4 in virtual disk | +| System call compatibility | ~70% | 100% | +| Memory usage | Lower | Higher (VM) | +| Cross-OS file access | Fast | Slower (9P protocol) | +| Docker support | Limited | Full native | +| GPU support | No | Yes (CUDA, DirectML) | +| Networking | Shared with Windows | Virtual adapter | + +--- + +## Installation + +### Windows 10/11 (Modern) + +```powershell +# Install WSL with Ubuntu (default) +wsl --install + +# Install specific distro +wsl --install -d Debian + +# List available distros +wsl --list --online +``` + +### Available Distributions + +| Distro | Command | +|--------|---------| +| Ubuntu | `wsl --install -d Ubuntu` | +| Ubuntu 22.04 | `wsl --install -d Ubuntu-22.04` | +| Debian | `wsl --install -d Debian` | +| Fedora | Via Fedora Remix or manual | +| Arch | Via ArchWSL or manual | +| Alpine | `wsl --install -d Alpine` | +| openSUSE | `wsl --install -d openSUSE-Leap-15.5` | + +--- + +## WSL 1 vs WSL 2 + +### Architecture Comparison + +``` +WSL 1: +┌─────────────────────────────────────┐ +│ Windows Kernel │ +├─────────────────────────────────────┤ +│ Linux Syscall Translation Layer │ +├─────────────────────────────────────┤ +│ Linux User Space │ +└─────────────────────────────────────┘ + +WSL 2: +┌─────────────────────────────────────┐ +│ Windows Kernel │ +├─────────────────────────────────────┤ +│ Lightweight Hyper-V VM │ +│ ┌───────────────────────────────┐ │ +│ │ Real Linux Kernel │ │ +│ ├───────────────────────────────┤ │ +│ │ Linux User Space │ │ +│ └───────────────────────────────┘ │ +└─────────────────────────────────────┘ +``` + +### When to Use Each + +| Use Case | Recommendation | +|----------|----------------| +| Docker/containers | WSL 2 | +| Full syscall compatibility | WSL 2 | +| Working with Windows files | WSL 1 | +| Low memory usage | WSL 1 | +| GPU workloads | WSL 2 | +| Network tools (low-level) | WSL 2 | + +### Switch Versions + +```powershell +# Set WSL 2 as default +wsl --set-default-version 2 + +# Convert existing distro +wsl --set-version Ubuntu 2 + +# Check version of installed distros +wsl -l -v +``` + +--- + +## Configuration + +### Per-Distro Config + +```bash +# /etc/wsl.conf (inside Linux) + +[boot] +systemd = true # Enable systemd (WSL 2) + +[automount] +enabled = true +root = /mnt/ +options = "metadata,umask=22,fmask=11" + +[network] +hostname = dev-machine +generateHosts = true +generateResolvConf = true + +[interop] +enabled = true # Run Windows executables +appendWindowsPath = true # Include Windows PATH +``` + +### Global Config + +```ini +# %USERPROFILE%\.wslconfig (Windows side) + +[wsl2] +memory = 8GB # Limit memory +processors = 4 # Limit CPUs +swap = 2GB +localhostForwarding = true + +# Experimental features +[experimental] +sparseVhd = true # Reclaim disk space +autoMemoryReclaim = gradual +``` + +--- + +## File System + +### Path Translation + +```bash +# Windows paths from WSL +/mnt/c/Users/username/Documents + +# WSL paths from Windows +\\wsl$\Ubuntu\home\username + +# Or via wsl command +wsl --cd ~ +``` + +### Performance Considerations + +``` +File access speed: + +WSL 2: +├── Linux files (ext4) ──────────▶ Fast +├── Windows files (/mnt/c) ──────▶ Slow (9P protocol) +└── Recommendation: Keep projects in Linux FS + +WSL 1: +├── Linux files ─────────────────▶ Medium +├── Windows files (/mnt/c) ──────▶ Fast (native NTFS) +└── Good for cross-FS work +``` + +### Best Practice + +```bash +# Clone repos to Linux filesystem, not /mnt/c +cd ~ +git clone https://github.com/user/project.git + +# NOT this (slow in WSL 2) +cd /mnt/c/Users/username/projects +git clone ... +``` + +--- + +## Docker Integration + +### Docker Desktop with WSL 2 + +``` +┌──────────────────────────────────────────┐ +│ Docker Desktop │ +├──────────────────────────────────────────┤ +│ WSL 2 Backend (docker-desktop-data) │ +├────────────────────┬─────────────────────┤ +│ Ubuntu │ Debian │ +│ docker CLI works │ docker CLI works │ +└────────────────────┴─────────────────────┘ +``` + +Settings: +1. Docker Desktop → Settings → General → Use WSL 2 based engine +2. Settings → Resources → WSL Integration → Enable for your distros + +### Without Docker Desktop + +```bash +# Install Docker in WSL 2 directly +sudo apt update +sudo apt install docker.io + +# Start Docker (with systemd) +sudo systemctl start docker +sudo systemctl enable docker + +# Add user to docker group +sudo usermod -aG docker $USER +``` + +--- + +## Networking + +### WSL 2 Networking + +``` +WSL 2 has its own virtual network adapter: + +Windows ◄───────► Virtual Switch ◄───────► WSL 2 VM +192.168.x.x 172.x.x.x +``` + +```bash +# Get WSL IP +hostname -I + +# Get Windows host IP (from WSL) +cat /etc/resolv.conf | grep nameserver + +# Access Windows localhost from WSL +# Use host IP or localhost (with localhostForwarding) + +# Access WSL server from Windows +# localhost:port works with port forwarding +``` + +### Port Forwarding + +```powershell +# WSL 2 auto-forwards ports to localhost +# Access WSL service at localhost:3000 from Windows + +# Manual port proxy (if needed) +netsh interface portproxy add v4tov4 ` + listenport=3000 listenaddress=0.0.0.0 ` + connectport=3000 connectaddress=172.x.x.x +``` + +--- + +## GUI Apps (WSLg) + +Windows 11 / Windows 10 21H2+ supports Linux GUI apps. + +```bash +# Install GUI app +sudo apt install gedit + +# Just run it +gedit & + +# Apps appear in Windows Start menu +# Audio and clipboard work automatically +``` + +### GPU Support + +```bash +# NVIDIA CUDA in WSL 2 +# Install NVIDIA driver on Windows (not in WSL) + +# Verify +nvidia-smi + +# TensorFlow/PyTorch work with GPU +pip install tensorflow +python -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))" +``` + +--- + +## Development Workflow + +### VS Code Integration + +```bash +# Open VS Code from WSL +code . + +# VS Code Remote - WSL extension +# - Edit Linux files with Windows VS Code +# - Terminal runs in WSL +# - Extensions run in WSL +``` + +### JetBrains IDEs + +``` +Options: +├── JetBrains Gateway (remote dev) +├── IDE in Windows, interpreter in WSL +└── IDE running in WSL via WSLg +``` + +### Git Configuration + +```bash +# Consistent line endings +git config --global core.autocrlf input + +# Credential sharing with Windows +git config --global credential.helper "/mnt/c/Program\ Files/Git/mingw64/bin/git-credential-manager.exe" +``` + +--- + +## Common Commands + +### WSL Management + +```powershell +# List installed distros +wsl -l -v + +# Run specific distro +wsl -d Ubuntu + +# Run as specific user +wsl -u root + +# Terminate distro +wsl -t Ubuntu + +# Shutdown all WSL +wsl --shutdown + +# Export distro +wsl --export Ubuntu ubuntu-backup.tar + +# Import distro +wsl --import MyUbuntu C:\WSL\MyUbuntu ubuntu-backup.tar + +# Unregister (delete) distro +wsl --unregister Ubuntu + +# Update WSL +wsl --update +``` + +### From Inside WSL + +```bash +# Run Windows commands +cmd.exe /c dir +powershell.exe Get-Process +explorer.exe . + +# Open Windows apps +notepad.exe file.txt +code . + +# Access Windows env vars +echo $WSLENV +cmd.exe /c "echo %USERNAME%" +``` + +--- + +## Troubleshooting + +### Common Issues + +| Issue | Solution | +|-------|----------| +| Slow file access | Move files to Linux FS (`~`) | +| High memory usage | Set `memory` limit in `.wslconfig` | +| Network issues | `wsl --shutdown` and restart | +| DNS not working | Check `/etc/resolv.conf`, regenerate | +| Docker not working | Enable WSL 2 integration in Docker Desktop | +| systemd not starting | Add `systemd=true` to `/etc/wsl.conf` | + +### Reset Networking + +```powershell +# Windows side +wsl --shutdown +netsh winsock reset +netsh int ip reset +# Restart computer +``` + +### Reclaim Disk Space + +```powershell +# WSL 2 VHD doesn't shrink automatically +wsl --shutdown + +# Compact the disk (PowerShell as admin) +Optimize-VHD -Path "C:\Users\\AppData\Local\Packages\\LocalState\ext4.vhdx" -Mode Full + +# Or enable sparse VHD in .wslconfig +# [experimental] +# sparseVhd = true +``` + +--- + +## Performance Tips + +1. **Keep code in Linux FS** — `/home/user`, not `/mnt/c` +2. **Use WSL 2** — Better performance for most tasks +3. **Limit memory** — Set `.wslconfig` memory limit +4. **Exclude from antivirus** — Add WSL paths to Windows Defender exclusions +5. **Use VS Code Remote** — Better than editing `/mnt/c` files +6. **Enable sparse VHD** — Automatic disk space reclamation + +--- + +## Related + +- [[Shells]] +- [[Container Runtimes]] +- [[Terminal Emulators]] diff --git a/tools/WebAssembly Runtimes.md b/tools/WebAssembly Runtimes.md new file mode 100644 index 0000000..41c1311 --- /dev/null +++ b/tools/WebAssembly Runtimes.md @@ -0,0 +1,544 @@ +--- +title: WebAssembly Runtimes +aliases: + - WASM Runtimes + - Wasm +tags: + - tool + - comparison + - wasm + - runtime +type: comparison +status: complete +created: 2025-12-04 +--- + +# WebAssembly Runtimes + +Standalone runtimes for executing WebAssembly outside the browser. + +## Overview + +| Runtime | Written In | WASI | Component Model | Focus | +|---------|------------|:----:|:---------------:|-------| +| Wasmtime | Rust | ✅ | ✅ | Reference, production | +| Wasmer | Rust | ✅ | ✅ | Universal, ecosystem | +| WasmEdge | C++ | ✅ | ✅ | Cloud-native, AI | +| WAMR | C | ✅ | Partial | Embedded, IoT | +| wazero | Go | ✅ | ❌ | Zero dependencies | +| Spin | Rust | ✅ | ✅ | Serverless apps | + +--- + +## What is WASM? + +WebAssembly (WASM) is a binary instruction format designed as a portable compilation target. + +``` +Source Code → Compiler → .wasm → Runtime → Execution + +┌─────────────────────────────────────────────────────┐ +│ Languages │ +│ Rust, C, C++, Go, Zig, AssemblyScript, etc. │ +└──────────────────────┬──────────────────────────────┘ + ▼ + ┌─────────────────┐ + │ .wasm binary │ + └────────┬────────┘ + ▼ +┌─────────────────────────────────────────────────────┐ +│ Runtimes │ +│ Browser, Wasmtime, Wasmer, WasmEdge, etc. │ +└─────────────────────────────────────────────────────┘ +``` + +### Key Properties + +| Property | Description | +|----------|-------------| +| Portable | Same binary runs anywhere | +| Sandboxed | No direct access to host by default | +| Fast | Near-native execution speed | +| Language-agnostic | Compile from many languages | +| Compact | Small binary size | + +--- + +## WASI + +WebAssembly System Interface — standardized syscall interface for WASM outside browsers. + +``` +┌─────────────────────────────────────────────────┐ +│ WASM Module │ +│ ┌───────────────────────────────────────────┐ │ +│ │ Application Code │ │ +│ └───────────────────────────────────────────┘ │ +│ │ │ +│ ▼ │ +│ ┌───────────────────────────────────────────┐ │ +│ │ WASI Imports │ │ +│ │ fd_read, fd_write, path_open, etc. │ │ +│ └───────────────────────────────────────────┘ │ +└──────────────────────┬──────────────────────────┘ + ▼ +┌─────────────────────────────────────────────────┐ +│ WASI Implementation │ +│ (provided by runtime) │ +└─────────────────────────────────────────────────┘ +``` + +### WASI Capabilities + +```bash +# Grant file system access +wasmtime --dir=/data app.wasm + +# Grant network access (preview2) +wasmtime --wasi tcp app.wasm + +# Environment variables +wasmtime --env KEY=value app.wasm +``` + +--- + +## Wasmtime + +Bytecode Alliance reference runtime. Production-grade. + +### Key Characteristics + +- **Standards-first** — Reference WASI implementation +- **Component Model** — Full support +- **Cranelift** — Fast JIT/AOT compiler +- **Embeddings** — Rust, C, Python, Go, .NET, etc. +- **Production** — Used by Shopify, Fastly, Fermyon + +### Usage + +```bash +# Install +curl https://wasmtime.dev/install.sh -sSf | bash + +# Run WASM module +wasmtime hello.wasm + +# With WASI capabilities +wasmtime --dir=. --env NAME=World app.wasm + +# AOT compilation +wasmtime compile app.wasm -o app.cwasm +wasmtime run app.cwasm +``` + +### Embedding (Rust) + +```rust +use wasmtime::*; + +fn main() -> Result<()> { + let engine = Engine::default(); + let module = Module::from_file(&engine, "hello.wasm")?; + + let mut store = Store::new(&engine, ()); + let instance = Instance::new(&mut store, &module, &[])?; + + let hello = instance.get_typed_func::<(), ()>(&mut store, "hello")?; + hello.call(&mut store, ())?; + + Ok(()) +} +``` + +### Strengths + +- Most spec-compliant +- Excellent documentation +- Component Model leader +- Strong embedding story +- Bytecode Alliance backing + +--- + +## Wasmer + +Universal runtime with focus on developer experience. + +### Key Characteristics + +- **Multiple compilers** — Singlepass, Cranelift, LLVM +- **Package registry** — wasmer.io/registry (WAPM) +- **Wasmer Edge** — Deployment platform +- **Broad language support** — Many embedding options + +### Usage + +```bash +# Install +curl https://get.wasmer.io -sSfL | sh + +# Run WASM module +wasmer run hello.wasm + +# Run from registry +wasmer run python/python + +# Create package +wasmer init +wasmer publish +``` + +### Compiler Options + +| Compiler | Speed | Optimization | Use Case | +|----------|-------|--------------|----------| +| Singlepass | Fastest compile | Minimal | JIT, quick iteration | +| Cranelift | Fast compile | Good | General purpose | +| LLVM | Slow compile | Best | Maximum performance | + +```bash +# Choose compiler +wasmer run --singlepass app.wasm +wasmer run --cranelift app.wasm +wasmer run --llvm app.wasm +``` + +### Embedding (Python) + +```python +from wasmer import engine, Store, Module, Instance + +store = Store(engine.JIT()) +module = Module(store, open('hello.wasm', 'rb').read()) +instance = Instance(module) + +result = instance.exports.add(5, 37) +print(result) # 42 +``` + +### Strengths + +- Great developer experience +- Package registry +- Multiple compiler backends +- Edge deployment platform +- Extensive language bindings + +--- + +## WasmEdge + +Cloud-native WebAssembly runtime optimized for edge and AI. + +### Key Characteristics + +- **Cloud-native** — Kubernetes, Docker integration +- **AI inference** — TensorFlow, PyTorch, llama.cpp +- **WASI-NN** — Neural network extension +- **Networking** — Async I/O, HTTP, sockets +- **Languages** — Rust, C, Go, JS, Python + +### Usage + +```bash +# Install +curl -sSf https://raw.githubusercontent.com/WasmEdge/WasmEdge/master/utils/install.sh | bash + +# Run WASM +wasmedge hello.wasm + +# With reactor mode (library) +wasmedge --reactor lib.wasm function_name arg1 arg2 + +# With AI extension +wasmedge --dir .:. wasmedge-nn.wasm +``` + +### Docker/Kubernetes Integration + +```dockerfile +# Use WasmEdge in Docker +FROM scratch +COPY app.wasm /app.wasm +ENTRYPOINT ["/app.wasm"] +``` + +```bash +# Run WASM container with Docker +docker run --runtime=io.containerd.wasmedge.v1 myapp:latest +``` + +### AI Workloads + +```rust +// Using WASI-NN for inference +use wasi_nn::{Graph, GraphExecutionContext, TensorType}; + +let graph = Graph::load(&model_bytes, GraphEncoding::Ggml)?; +let context = graph.init_execution_context()?; +context.set_input(0, TensorType::F32, &input_dims, &input_data)?; +context.compute()?; +``` + +### Strengths + +- Best container/K8s integration +- AI/ML focus with WASI-NN +- Async networking +- Active development +- Edge computing focus + +--- + +## WAMR + +WebAssembly Micro Runtime — for embedded and IoT. + +### Key Characteristics + +- **Tiny footprint** — ~100KB runtime +- **Interpreter + JIT + AOT** — Multiple modes +- **Real-time** — Deterministic execution +- **Embedded** — MCUs, IoT devices +- **Intel backing** — Production tested + +### Execution Modes + +| Mode | Memory | Speed | Use Case | +|------|--------|-------|----------| +| Interpreter | Tiny | Slower | Very constrained devices | +| Fast JIT | Small | Fast | Resource-limited | +| LLVM JIT | Larger | Fastest | Performance critical | +| AOT | Tiny runtime | Fastest | Production deployment | + +### Usage + +```bash +# Run with interpreter +iwasm app.wasm + +# AOT compile +wamrc -o app.aot app.wasm + +# Run AOT +iwasm app.aot +``` + +### Strengths + +- Smallest footprint +- Real-time capable +- Multiple execution modes +- Battle-tested at Intel +- Ideal for embedded + +--- + +## wazero + +Pure Go WebAssembly runtime. Zero dependencies. + +### Key Characteristics + +- **Pure Go** — No CGO, no dependencies +- **Portable** — Anywhere Go runs +- **Embedder-friendly** — Simple Go API +- **Compliant** — WASI preview1 + +### Usage (Go) + +```go +import ( + "context" + "github.com/tetratelabs/wazero" + "github.com/tetratelabs/wazero/imports/wasi_snapshot_preview1" +) + +func main() { + ctx := context.Background() + + // Create runtime + r := wazero.NewRuntime(ctx) + defer r.Close(ctx) + + // Instantiate WASI + wasi_snapshot_preview1.MustInstantiate(ctx, r) + + // Load and run module + wasm, _ := os.ReadFile("app.wasm") + mod, _ := r.Instantiate(ctx, wasm) + + // Call function + fn := mod.ExportedFunction("hello") + fn.Call(ctx) +} +``` + +### Strengths + +- Zero dependencies +- Go-native experience +- Cross-compilation friendly +- Simple embedding +- Compliant implementation + +--- + +## Spin + +Serverless application framework built on WASM. + +### Key Characteristics + +- **Framework** — Not just runtime +- **Components** — WASM component model +- **Triggers** — HTTP, Redis, timers +- **Multi-language** — Rust, Go, JS, Python, etc. +- **Fermyon Cloud** — Deployment platform + +### Usage + +```bash +# Install +curl -fsSL https://developer.fermyon.com/downloads/install.sh | bash + +# Create new app +spin new -t http-rust hello-spin +cd hello-spin + +# Build +spin build + +# Run locally +spin up + +# Deploy to Fermyon Cloud +spin deploy +``` + +### Application Structure + +```rust +// Rust component +use spin_sdk::http::{IntoResponse, Request, Response}; +use spin_sdk::http_component; + +#[http_component] +fn handle_request(req: Request) -> anyhow::Result { + Ok(Response::builder() + .status(200) + .body("Hello from Spin!") + .build()) +} +``` + +### Strengths + +- Full framework, not just runtime +- Component Model native +- Sub-millisecond cold starts +- Multi-language support +- Serverless platform included + +--- + +## Feature Comparison + +| Feature | Wasmtime | Wasmer | WasmEdge | WAMR | wazero | +|---------|:--------:|:------:|:--------:|:----:|:------:| +| WASI Preview 1 | ✅ | ✅ | ✅ | ✅ | ✅ | +| WASI Preview 2 | ✅ | ✅ | ✅ | Partial | ❌ | +| Component Model | ✅ | ✅ | ✅ | ❌ | ❌ | +| JIT compilation | ✅ | ✅ | ✅ | ✅ | ❌ | +| AOT compilation | ✅ | ✅ | ✅ | ✅ | ❌ | +| Interpreter | ❌ | ❌ | ❌ | ✅ | ✅ | +| Async I/O | ✅ | ✅ | ✅ | ❌ | ❌ | +| Docker/K8s | Via containerd | Wasmer Edge | ✅ Native | ❌ | ❌ | +| AI/NN extensions | ❌ | ❌ | ✅ | ❌ | ❌ | + +--- + +## Use Cases + +| Use Case | Best Runtimes | +|----------|---------------| +| Reference/standards | Wasmtime | +| Developer experience | Wasmer | +| Cloud/Kubernetes | WasmEdge | +| Embedded/IoT | WAMR | +| Go applications | wazero | +| Serverless apps | Spin | +| AI inference | WasmEdge | +| Browser + Server | All support | +| Plugin systems | Wasmtime, Wasmer, wazero | + +--- + +## Compiling to WASM + +### Rust + +```bash +# Add target +rustup target add wasm32-wasip1 + +# Build +cargo build --target wasm32-wasip1 --release + +# Output: target/wasm32-wasip1/release/app.wasm +``` + +### Go + +```bash +# Build with TinyGo (recommended for WASM) +tinygo build -o app.wasm -target=wasi main.go + +# Or standard Go (larger output) +GOOS=wasip1 GOARCH=wasm go build -o app.wasm +``` + +### C/C++ + +```bash +# With Emscripten +emcc app.c -o app.wasm + +# With WASI SDK +$WASI_SDK/bin/clang app.c -o app.wasm +``` + +### Other Languages + +| Language | Compiler/Tool | +|----------|---------------| +| AssemblyScript | `asc` (TypeScript-like) | +| Zig | `zig build-exe -target wasm32-wasi` | +| Swift | SwiftWasm | +| Kotlin | Kotlin/Wasm | +| C# | Blazor, NativeAOT-LLVM | + +--- + +## Decision Guide + +| Need | Recommendation | +|------|----------------| +| Standards compliance | Wasmtime | +| Best developer UX | Wasmer | +| Kubernetes/containers | WasmEdge | +| Embedded systems | WAMR | +| Go project, simple embedding | wazero | +| Serverless applications | Spin | +| AI inference at edge | WasmEdge | +| Production, battle-tested | Wasmtime or Wasmer | +| Smallest footprint | WAMR | + +--- + +## Related + +- [[JavaScript Runtimes]] +- [[Container Runtimes]] +- [[Deployment]]