diff --git a/Computer Science MOC.md b/Computer Science MOC.md index 0d31f35..83972d4 100644 --- a/Computer Science MOC.md +++ b/Computer Science MOC.md @@ -34,6 +34,7 @@ Fundamental CS concepts, data structures, algorithms, and system design. - [[Dynamic Programming]] - [[Greedy Algorithms]] - [[Divide & Conquer]] +- [[Computer Science/Compression|Compression]] — gzip, zstd, LZ4, Brotli, how they work ### Complexity @@ -71,6 +72,30 @@ Fundamental CS concepts, data structures, algorithms, and system design. - [[Testing Strategies]] — Unit, integration, E2E, BDD, property-based +### Testing + +- [[Computer Science/TDD and BDD|TDD and BDD]] — Test-driven and behavior-driven development +- [[Computer Science/Property-Based Testing|Property-Based Testing]] — QuickCheck, Hypothesis, generative testing +- [[Computer Science/Load Testing|Load Testing]] — Performance and stress testing + +### Developer Practices + +- [[Computer Science/Git Internals|Git Internals]] — Objects, refs, plumbing commands +- [[Computer Science/Code Review|Code Review]] — PR reviews, feedback, best practices +- [[Computer Science/Technical Writing|Technical Writing]] — Documentation, READMEs, API docs +- [[Computer Science/Regex Reference|Regex Reference]] — Regular expression patterns + +### API Styles + +- [[Computer Science/REST vs GraphQL vs gRPC|REST vs GraphQL vs gRPC]] — API architectural comparison + +### OS Development Basics + +- [[Computer Science/XDG Base Directory|XDG Base Directory]] — Linux config/data/cache paths +- [[Computer Science/Windows Development Basics|Windows Development Basics]] — AppData, Registry, paths +- [[Computer Science/macOS Development Basics|macOS Development Basics]] — Library folders, plist, Homebrew +- [[Computer Science/Cross-Platform Development Paths|Cross-Platform Development Paths]] — Where to store files on any OS + ### Reference - [[Technical Measurements]] diff --git a/Computer Science/Code Review.md b/Computer Science/Code Review.md new file mode 100644 index 0000000..e039986 --- /dev/null +++ b/Computer Science/Code Review.md @@ -0,0 +1,354 @@ +--- +title: Code Review +aliases: + - Code Reviews + - Pull Request Review + - PR Review +tags: + - cs + - practices + - collaboration + - quality +type: reference +status: complete +created: "2025-12-18" +--- + +# Code Review + +Systematic examination of source code by peers to improve quality and share knowledge. + +## Overview + +| Aspect | Details | +|--------|---------| +| **Purpose** | Find bugs, improve code quality, share knowledge | +| **Timing** | Before merging to main branch | +| **Participants** | Author + 1-2 reviewers (typically) | +| **Tools** | GitHub, GitLab, Bitbucket, Gerrit, Phabricator | + +## Benefits + +| Benefit | Description | +|---------|-------------| +| **Bug Detection** | Catch issues before production | +| **Knowledge Sharing** | Spread understanding across team | +| **Code Quality** | Maintain standards and consistency | +| **Learning** | Junior devs learn from reviews | +| **Documentation** | PR descriptions explain changes | +| **Reduced Risk** | Multiple eyes reduce single points of failure | + +## Review Process + +### Typical Workflow + +``` +┌─────────────────────────────────────────────────────────────┐ +│ 1. Developer creates branch and makes changes │ +│ │ │ +│ ▼ │ +│ 2. Opens Pull Request with description │ +│ │ │ +│ ▼ │ +│ 3. Automated checks run (CI, linting, tests) │ +│ │ │ +│ ▼ │ +│ 4. Reviewer(s) assigned or requested │ +│ │ │ +│ ▼ │ +│ 5. Reviewer examines code, leaves comments │ +│ │ │ +│ ▼ │ +│ 6. Author addresses feedback, pushes updates │ +│ │ │ +│ ▼ │ +│ 7. Reviewer approves (or requests more changes) │ +│ │ │ +│ ▼ │ +│ 8. PR merged to main branch │ +└─────────────────────────────────────────────────────────────┘ +``` + +## What to Look For + +### Code Quality + +| Area | Check For | +|------|-----------| +| **Correctness** | Does it do what it's supposed to? | +| **Logic** | Are there edge cases missed? | +| **Performance** | Any obvious inefficiencies? | +| **Security** | Input validation, injection risks? | +| **Error Handling** | Are errors handled gracefully? | +| **Tests** | Are there adequate tests? | + +### Design + +| Area | Check For | +|------|-----------| +| **Architecture** | Does it fit the system design? | +| **Complexity** | Can it be simplified? | +| **DRY** | Is there unnecessary duplication? | +| **SOLID** | Are principles followed? | +| **Dependencies** | Are new deps justified? | + +### Maintainability + +| Area | Check For | +|------|-----------| +| **Readability** | Is it easy to understand? | +| **Naming** | Are names clear and consistent? | +| **Comments** | Are complex parts explained? | +| **Documentation** | Are public APIs documented? | +| **Consistency** | Does it match codebase style? | + +## Writing Good PR Descriptions + +### Template + +```markdown +## Summary +Brief description of what this PR does. + +## Motivation +Why is this change needed? Link to issue/ticket. + +## Changes +- Added X feature +- Fixed Y bug +- Refactored Z module + +## Testing +How was this tested? +- [ ] Unit tests added +- [ ] Manual testing done +- [ ] Integration tests pass + +## Screenshots (if UI changes) +Before / After images + +## Checklist +- [ ] Code follows style guidelines +- [ ] Tests pass locally +- [ ] Documentation updated +- [ ] No sensitive data exposed +``` + +### Good Example + +```markdown +## Summary +Add rate limiting to the authentication API endpoints. + +## Motivation +We've seen increased brute force attempts on login. +Relates to SEC-1234. + +## Changes +- Added redis-based rate limiter middleware +- Limited login attempts to 5 per minute per IP +- Added 429 response with Retry-After header +- Added rate limit metrics to Prometheus + +## Testing +- Unit tests for rate limiter logic +- Integration test for blocked requests +- Load tested with 100 req/s + +## Rollback +Disable with `RATE_LIMIT_ENABLED=false` env var +``` + +## Giving Feedback + +### Be Constructive + +| Instead Of | Say | +|------------|-----| +| "This is wrong" | "Consider X approach because..." | +| "Why did you do this?" | "Help me understand the reasoning for..." | +| "This is confusing" | "Could we add a comment explaining...?" | +| "Bad naming" | "What about `calculateTotal` for clarity?" | + +### Comment Types + +``` +# Blocking - must change +"This will cause a null pointer exception when X is empty." + +# Suggestion - consider changing +"nit: Consider extracting this to a helper function." + +# Question - seeking understanding +"Question: Why is this timeout set to 30s? Is that documented?" + +# Praise - acknowledge good work +"Nice! This is a clean solution to a tricky problem." + +# FYI - informational +"FYI: We have a similar utility in `src/utils/dates.ts`" +``` + +### Use Conventional Comments + +``` +# Prefix with category +nitpick: Trailing whitespace on line 42. +suggestion: We could use Optional.ofNullable() here. +issue: This doesn't handle the case where user is null. +question: Is this timeout value from requirements? +thought: Wonder if we should add caching here in the future. +praise: Great test coverage! +``` + +## Receiving Feedback + +### Best Practices + +| Do | Don't | +|----|-------| +| Assume good intent | Take criticism personally | +| Ask clarifying questions | Get defensive | +| Thank reviewers | Ignore comments | +| Explain your reasoning | Dismiss without consideration | +| Accept you might be wrong | Dig in on every point | + +### Responding to Comments + +``` +# Agree and fix +"Good catch! Fixed in abc123." + +# Discuss +"I considered that approach but chose X because of Y. +What do you think?" + +# Defer +"That's a good point but out of scope. +Created issue #456 to track." + +# Push back (respectfully) +"I think the current approach is better because... +but happy to discuss more." +``` + +## Review Size + +### Keep PRs Small + +| Size | Lines Changed | Review Time | +|------|---------------|-------------| +| **XS** | < 50 | 5-10 min | +| **S** | 50-200 | 15-30 min | +| **M** | 200-500 | 30-60 min | +| **L** | 500-1000 | 1-2 hours | +| **XL** | > 1000 | Consider splitting | + +### Why Small PRs + +- Faster reviews +- Better feedback +- Easier to understand +- Lower risk +- Quicker deployment +- Easier to revert + +## Automation + +### Pre-Review Checks + +```yaml +# GitHub Actions example +name: PR Checks +on: pull_request + +jobs: + checks: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - name: Lint + run: npm run lint + + - name: Type Check + run: npm run typecheck + + - name: Tests + run: npm test + + - name: Security Scan + run: npm audit +``` + +### Code Owners + +``` +# CODEOWNERS file +# Auto-assign reviewers + +*.js @frontend-team +*.go @backend-team +/docs/ @tech-writers +/security/ @security-team +*.tf @devops-team +``` + +## Review Efficiency + +### For Reviewers + +| Practice | Benefit | +|----------|---------| +| **Review promptly** | Unblock teammates | +| **Set aside focused time** | Better quality review | +| **Review in passes** | High-level first, details second | +| **Use checklists** | Don't miss common issues | +| **Trust but verify** | Run the code locally | + +### For Authors + +| Practice | Benefit | +|----------|---------| +| **Self-review first** | Catch obvious issues | +| **Small, focused PRs** | Easier to review | +| **Good descriptions** | Context for reviewers | +| **Respond quickly** | Keep momentum | +| **Pre-flight checks** | Don't waste reviewer time | + +## Common Anti-Patterns + +| Anti-Pattern | Problem | +|--------------|---------| +| **Rubber-stamping** | Approving without reading | +| **Nitpick overload** | Focusing only on style | +| **Blocking on opinions** | Preferences vs requirements | +| **Review by committee** | Too many reviewers | +| **Review avoidance** | PRs sitting for days | +| **Gatekeeping** | Using reviews as power | + +## Metrics + +| Metric | Target | Why | +|--------|--------|-----| +| **Review turnaround** | < 24 hours | Avoid blocking | +| **PR size** | < 400 LOC | Better reviews | +| **Review cycles** | < 3 | Avoid thrashing | +| **Coverage** | 100% of PRs | No exceptions | + +## Tools & Features + +| Feature | GitHub | GitLab | Bitbucket | +|---------|--------|--------|-----------| +| Inline comments | ✅ | ✅ | ✅ | +| Suggestions | ✅ | ✅ | ✅ | +| Required reviewers | ✅ | ✅ | ✅ | +| Code owners | ✅ | ✅ | ✅ | +| Auto-merge | ✅ | ✅ | ✅ | +| Draft PRs | ✅ | ✅ | ✅ | + +## Related + +- [[Git Internals]] — Version control +- [[Testing Frameworks]] — Automated testing +- [[Computer Science MOC]] — CS topics diff --git a/Computer Science/Compression.md b/Computer Science/Compression.md new file mode 100644 index 0000000..5e610e4 --- /dev/null +++ b/Computer Science/Compression.md @@ -0,0 +1,518 @@ +--- +title: Compression +aliases: + - Data Compression + - Compression Algorithms + - gzip + - zstd +tags: + - cs + - algorithms + - performance +type: reference +status: complete +created: "2025-12-18" +--- + +# Compression + +Reducing data size by encoding information more efficiently. + +## Overview + +| Type | Description | Use Cases | +|------|-------------|-----------| +| **Lossless** | Perfect reconstruction | Text, code, executables | +| **Lossy** | Approximation (smaller) | Images, audio, video | + +## Lossless vs Lossy + +``` +Original: "AAAAAABBBBCCCCCCCC" (18 bytes) + +Lossless (RLE): "6A4B8C" (6 bytes) + → Decompresses to exact original + +Lossy (example): "6A4B8C" but allows "AAAAAABBBBCCCCCCCD" + → Close enough for images/audio +``` + +## Core Techniques + +### Run-Length Encoding (RLE) + +**Replace repeated values with count + value.** + +``` +Input: AAAAAABBBBCCCCCCCC +Output: 6A4B8C + +Input: WWWWWWWWWWWWBWWWWWWWWWWWWBBBWWWWW +Output: 12W1B12W3B5W +``` + +```python +def rle_encode(data: str) -> str: + result = [] + count = 1 + for i in range(1, len(data)): + if data[i] == data[i-1]: + count += 1 + else: + result.append(f"{count}{data[i-1]}") + count = 1 + result.append(f"{count}{data[-1]}") + return ''.join(result) +``` + +**Best for:** Simple patterns, bitmap images (BMP), fax + +### Dictionary Coding (LZ77/LZ78) + +**Replace repeated sequences with references to earlier occurrences.** + +``` +Input: "abracadabra" + +LZ77 process: +Position 0-3: "abra" → output literal "abra" +Position 4: "c" → output literal "c" +Position 5-6: "ad" → output literal "ad" +Position 7-10: "abra" → already seen at position 0! + → output (offset=7, length=4) + +Compressed: "abra" + "c" + "ad" + <7,4> +``` + +``` +┌─────────────────────────────────────────────────────┐ +│ Sliding Window (LZ77) │ +│ │ +│ ←── Search Buffer ──→ ←── Look-ahead ──→ │ +│ [...already seen...] [next to encode...] │ +│ │ +│ Find longest match in search buffer │ +│ Output: (distance back, length) or literal │ +└─────────────────────────────────────────────────────┘ +``` + +**Used by:** gzip, DEFLATE, zstd, LZ4 + +### Huffman Coding + +**Variable-length codes: frequent symbols get shorter codes.** + +``` +Text: "AAAAABBBCCDE" +Frequencies: A=5, B=3, C=2, D=1, E=1 + +Fixed-length (3 bits each): 12 × 3 = 36 bits + +Huffman tree: + (12) + / \ + A(5) (7) + / \ + B(3) (4) + / \ + C(2) (2) + / \ + D(1) E(1) + +Huffman codes: +A = 0 (1 bit) - most frequent +B = 10 (2 bits) +C = 110 (3 bits) +D = 1110 (4 bits) +E = 1111 (4 bits) - least frequent + +Encoded: 5×1 + 3×2 + 2×3 + 1×4 + 1×4 = 25 bits +Savings: 36 - 25 = 11 bits (30% smaller) +``` + +**Used by:** DEFLATE (gzip), JPEG, MP3 + +### Arithmetic Coding + +**Encode entire message as single number between 0 and 1.** + +More efficient than Huffman for skewed distributions. + +``` +Message: "AABA" with P(A)=0.8, P(B)=0.2 + +Range starts as [0, 1) + +A: [0, 0.8) → narrow to A's range +A: [0, 0.64) → narrow again +B: [0.512, 0.64) → B's portion of current range +A: [0.512, 0.6144) → final range + +Output: any number in [0.512, 0.6144), e.g., 0.55 +``` + +**Used by:** JPEG 2000, H.264/H.265, modern codecs + +### Burrows-Wheeler Transform (BWT) + +**Rearrange data to group similar characters together, then use RLE/Huffman.** + +``` +Input: "banana" + +1. Create all rotations: + banana + ananab + nanaba + anaban + nabana + abanan + +2. Sort alphabetically: + abanan + anaban + ananab + banana + nabana + nanaba + +3. Take last column: "nnbaaa" + (Notice: a's grouped together!) + +4. Apply RLE: "2n1b3a" or Huffman +``` + +**Used by:** bzip2 + +## Common Algorithms + +### Comparison Table + +| Algorithm | Ratio | Speed | Use Case | +|-----------|-------|-------|----------| +| **gzip/DEFLATE** | Good | Medium | General, HTTP | +| **zstd** | Excellent | Fast | Modern default | +| **LZ4** | Fair | Very fast | Real-time, games | +| **Brotli** | Excellent | Slow | Web assets | +| **bzip2** | Excellent | Slow | Archival | +| **xz/LZMA** | Best | Very slow | Archival | +| **Snappy** | Fair | Very fast | Databases | + +### DEFLATE (gzip, zlib, zip) + +``` +┌─────────────────────────────────────────────────────┐ +│ DEFLATE │ +│ │ +│ Input → LZ77 → Huffman → Compressed Output │ +│ (find (variable │ +│ matches) length codes) │ +└─────────────────────────────────────────────────────┘ +``` + +```bash +# gzip (DEFLATE with gzip header) +gzip file.txt # → file.txt.gz +gzip -d file.txt.gz # decompress +gzip -9 file.txt # max compression +gzip -1 file.txt # fastest + +# zlib is DEFLATE with different header (used in PNG, HTTP) +``` + +```python +import gzip +import zlib + +# gzip +with gzip.open('file.txt.gz', 'wt') as f: + f.write('Hello, World!') + +# zlib (raw DEFLATE) +compressed = zlib.compress(b'Hello, World!') +original = zlib.decompress(compressed) +``` + +### Zstandard (zstd) + +**Modern algorithm: better ratio than gzip, faster than gzip.** + +```bash +# CLI +zstd file.txt # → file.txt.zst +zstd -d file.txt.zst # decompress +zstd -19 file.txt # max compression (1-19) +zstd -1 file.txt # fastest +zstd --train *.json # train dictionary for similar files +``` + +```python +import zstandard as zstd + +# Compress +cctx = zstd.ZstdCompressor(level=3) +compressed = cctx.compress(b'Hello, World!') + +# Decompress +dctx = zstd.ZstdDecompressor() +original = dctx.decompress(compressed) + +# With dictionary (for small similar files) +dict_data = zstd.train_dictionary(100000, samples) +cctx = zstd.ZstdCompressor(dict_data=dict_data) +``` + +### LZ4 + +**Extremely fast, lower ratio. Good for real-time.** + +```bash +lz4 file.txt # → file.txt.lz4 +lz4 -d file.txt.lz4 # decompress +lz4 -9 file.txt # better compression +lz4 -1 file.txt # fastest (default) +``` + +```python +import lz4.frame + +compressed = lz4.frame.compress(b'Hello, World!') +original = lz4.frame.decompress(compressed) +``` + +### Brotli + +**Designed for web. Best ratio for text/HTML/CSS/JS.** + +```bash +brotli file.txt # → file.txt.br +brotli -d file.txt.br # decompress +brotli -q 11 file.txt # max quality (0-11) +``` + +```python +import brotli + +compressed = brotli.compress(b'Hello, World!', quality=4) +original = brotli.decompress(compressed) +``` + +```nginx +# nginx config +brotli on; +brotli_types text/html text/css application/javascript; +``` + +### Snappy + +**Google's fast compressor. Used in databases.** + +```python +import snappy + +compressed = snappy.compress(b'Hello, World!') +original = snappy.uncompress(compressed) +``` + +**Used by:** LevelDB, RocksDB, Kafka, Cassandra + +## Benchmarks (Typical) + +### Compression Ratio (smaller = better) + +| Algorithm | Text | Binary | JSON | +|-----------|------|--------|------| +| None | 1.00 | 1.00 | 1.00 | +| LZ4 | 0.45 | 0.65 | 0.35 | +| Snappy | 0.48 | 0.68 | 0.38 | +| gzip -6 | 0.32 | 0.55 | 0.25 | +| zstd -3 | 0.30 | 0.52 | 0.22 | +| Brotli -4 | 0.28 | 0.50 | 0.20 | +| xz -6 | 0.25 | 0.45 | 0.18 | + +### Speed (MB/s, approximate) + +| Algorithm | Compress | Decompress | +|-----------|----------|------------| +| LZ4 | 750 | 4000 | +| Snappy | 500 | 1500 | +| zstd -1 | 500 | 1500 | +| zstd -3 | 350 | 1200 | +| gzip -6 | 50 | 400 | +| Brotli -4 | 40 | 400 | +| xz -6 | 10 | 150 | + +## HTTP Compression + +### Content-Encoding + +```http +# Request +GET /api/data HTTP/1.1 +Accept-Encoding: gzip, deflate, br + +# Response +HTTP/1.1 200 OK +Content-Encoding: gzip +Content-Length: 1234 +``` + +### Priority + +| Priority | Encoding | Support | +|----------|----------|---------| +| 1 | `br` (Brotli) | Modern browsers | +| 2 | `gzip` | Universal | +| 3 | `deflate` | Legacy | + +### Express.js Example + +```javascript +const compression = require('compression'); +const express = require('express'); + +const app = express(); +app.use(compression({ + level: 6, // gzip level + threshold: 1024, // min size to compress + filter: (req, res) => { + if (req.headers['x-no-compression']) return false; + return compression.filter(req, res); + } +})); +``` + +## File Formats + +| Format | Algorithm | Use | +|--------|-----------|-----| +| `.gz` | gzip (DEFLATE) | Single file | +| `.zip` | DEFLATE | Archive + compression | +| `.tar.gz` / `.tgz` | tar + gzip | Unix archives | +| `.zst` | Zstandard | Modern single file | +| `.br` | Brotli | Web assets | +| `.xz` | LZMA2 | High compression | +| `.bz2` | bzip2 | Legacy high compression | +| `.lz4` | LZ4 | Fast compression | +| `.7z` | LZMA/LZMA2 | Best ratio archive | + +## When to Use What + +| Scenario | Recommendation | +|----------|----------------| +| **HTTP responses** | Brotli (or gzip fallback) | +| **Log files** | zstd or gzip | +| **Database storage** | LZ4 or Snappy | +| **Real-time streaming** | LZ4 | +| **Archival** | xz or zstd -19 | +| **General purpose** | zstd | +| **Maximum compatibility** | gzip | +| **Container images** | zstd (OCI 1.1) | + +## Dictionaries + +**Pre-shared patterns for better small-file compression.** + +```bash +# Train dictionary on sample files +zstd --train samples/*.json -o dict + +# Compress with dictionary +zstd --dict dict file.json + +# Decompress (needs same dictionary!) +zstd -d --dict dict file.json.zst +``` + +**Best for:** +- Small similar files (API responses, configs) +- Known structure (JSON schemas) +- Protocol messages + +## Streaming Compression + +```python +import zstandard as zstd + +# Streaming compress +cctx = zstd.ZstdCompressor() +with open('input.txt', 'rb') as fin: + with open('output.zst', 'wb') as fout: + cctx.copy_stream(fin, fout) + +# Streaming decompress +dctx = zstd.ZstdDecompressor() +with open('input.zst', 'rb') as fin: + with open('output.txt', 'wb') as fout: + dctx.copy_stream(fin, fout) +``` + +```go +// Go with gzip streaming +import ( + "compress/gzip" + "io" + "os" +) + +func compressFile(src, dst string) error { + in, _ := os.Open(src) + defer in.Close() + + out, _ := os.Create(dst) + defer out.Close() + + gz := gzip.NewWriter(out) + defer gz.Close() + + _, err := io.Copy(gz, in) + return err +} +``` + +## Lossy Compression + +### Images + +| Format | Algorithm | Use Case | +|--------|-----------|----------| +| **JPEG** | DCT + Huffman | Photos | +| **WebP** | VP8 | Web images | +| **AVIF** | AV1 | Modern web | +| **HEIC** | HEVC | Apple photos | + +### Audio + +| Format | Algorithm | Bitrate | +|--------|-----------|---------| +| **MP3** | MDCT + Huffman | 128-320 kbps | +| **AAC** | MDCT | 96-256 kbps | +| **Opus** | SILK + CELT | 6-510 kbps | +| **Vorbis** | MDCT | 64-500 kbps | + +### Video + +| Format | Codec | Efficiency | +|--------|-------|------------| +| **H.264/AVC** | Standard | Good | +| **H.265/HEVC** | 2x H.264 | Better | +| **AV1** | 3x H.264 | Best | +| **VP9** | 2x H.264 | Better | + +## Tips + +| Tip | Why | +|-----|-----| +| **Compress before encryption** | Encrypted data doesn't compress | +| **Don't double-compress** | JPEG in gzip = bigger | +| **Match algorithm to data** | Text vs binary vs images | +| **Consider CPU cost** | Fast decompress often matters more | +| **Use dictionaries for small files** | Big wins for JSON APIs | +| **Enable HTTP compression** | Easy 70%+ bandwidth savings | + +## Related + +- [[Big O Notation]] — Algorithm complexity +- [[Networking Fundamentals]] — HTTP compression +- [[Serialization]] — Data formats +- [[Computer Science MOC]] — CS topics diff --git a/Computer Science/Cross-Platform Development Paths.md b/Computer Science/Cross-Platform Development Paths.md new file mode 100644 index 0000000..03ed9e0 --- /dev/null +++ b/Computer Science/Cross-Platform Development Paths.md @@ -0,0 +1,249 @@ +--- +title: Cross-Platform Development Paths +aliases: + - OS Paths Comparison + - Platform Directories + - Config Locations +tags: + - cs + - cross-platform + - comparison +type: comparison +status: complete +created: "2025-12-18" +--- + +# Cross-Platform Development Paths + +Quick reference for where to store files across operating systems. + +## Path Comparison + +### Config Files (User Settings) + +| Purpose | Linux | macOS | Windows | +|---------|-------|-------|---------| +| **Location** | `~/.config/app/` | `~/Library/Application Support/app/` | `%APPDATA%\app\` | +| **XDG Var** | `$XDG_CONFIG_HOME` | - | - | +| **Example** | `~/.config/git/config` | `~/Library/Application Support/Code/` | `C:\Users\name\AppData\Roaming\Code\` | + +### Data Files (App Data) + +| Purpose | Linux | macOS | Windows | +|---------|-------|-------|---------| +| **Location** | `~/.local/share/app/` | `~/Library/Application Support/app/` | `%APPDATA%\app\` | +| **XDG Var** | `$XDG_DATA_HOME` | - | - | +| **Example** | `~/.local/share/nvim/` | `~/Library/Application Support/` | `C:\Users\name\AppData\Roaming\` | + +### Cache Files (Deletable) + +| Purpose | Linux | macOS | Windows | +|---------|-------|-------|---------| +| **Location** | `~/.cache/app/` | `~/Library/Caches/app/` | `%LOCALAPPDATA%\app\cache\` | +| **XDG Var** | `$XDG_CACHE_HOME` | - | - | +| **Example** | `~/.cache/pip/` | `~/Library/Caches/pip/` | `C:\Users\name\AppData\Local\pip\Cache\` | + +### Temp Files (Session) + +| Purpose | Linux | macOS | Windows | +|---------|-------|-------|---------| +| **Location** | `/tmp/` or `$XDG_RUNTIME_DIR` | `$TMPDIR` | `%TEMP%` | +| **Example** | `/tmp/myapp.sock` | `/var/folders/.../T/` | `C:\Users\name\AppData\Local\Temp\` | + +### System-Wide Config + +| Purpose | Linux | macOS | Windows | +|---------|-------|-------|---------| +| **Location** | `/etc/app/` | `/Library/Application Support/` | `%PROGRAMDATA%\app\` | +| **Example** | `/etc/nginx/` | `/Library/Application Support/` | `C:\ProgramData\` | + +### Executables (User) + +| Purpose | Linux | macOS | Windows | +|---------|-------|-------|---------| +| **Location** | `~/.local/bin/` | `/usr/local/bin/` | `%LOCALAPPDATA%\Programs\` | +| **In PATH?** | Usually not | Yes | Usually not | + +### Executables (System) + +| Purpose | Linux | macOS | Windows | +|---------|-------|-------|---------| +| **Location** | `/usr/bin/`, `/usr/local/bin/` | `/usr/local/bin/`, `/opt/homebrew/bin/` | `C:\Program Files\` | + +## Cross-Platform Code + +### Using Libraries (Recommended) + +```python +# Python - platformdirs (best option) +from platformdirs import user_config_dir, user_data_dir, user_cache_dir + +config = user_config_dir("myapp") # Correct path for each OS +data = user_data_dir("myapp") +cache = user_cache_dir("myapp") +``` + +```javascript +// Node.js - env-paths +const envPaths = require('env-paths'); +const paths = envPaths('myapp'); + +console.log(paths.config); // Config directory +console.log(paths.data); // Data directory +console.log(paths.cache); // Cache directory +``` + +```rust +// Rust - dirs crate +use dirs; + +let config = dirs::config_dir().unwrap().join("myapp"); +let data = dirs::data_dir().unwrap().join("myapp"); +let cache = dirs::cache_dir().unwrap().join("myapp"); +``` + +```go +// Go - os.UserConfigDir() (Go 1.13+) +import "os" + +configDir, _ := os.UserConfigDir() +// Linux: ~/.config +// macOS: ~/Library/Application Support +// Windows: %APPDATA% + +cacheDir, _ := os.UserCacheDir() +// Linux: ~/.cache +// macOS: ~/Library/Caches +// Windows: %LOCALAPPDATA% +``` + +### Manual Implementation + +```python +import os +import sys +from pathlib import Path + +def get_config_dir(app_name: str) -> Path: + if sys.platform == 'win32': + base = Path(os.environ.get('APPDATA', Path.home() / 'AppData/Roaming')) + elif sys.platform == 'darwin': + base = Path.home() / 'Library/Application Support' + else: # Linux/BSD + base = Path(os.environ.get('XDG_CONFIG_HOME', Path.home() / '.config')) + return base / app_name + +def get_cache_dir(app_name: str) -> Path: + if sys.platform == 'win32': + base = Path(os.environ.get('LOCALAPPDATA', Path.home() / 'AppData/Local')) + return base / app_name / 'cache' + elif sys.platform == 'darwin': + return Path.home() / 'Library/Caches' / app_name + else: + base = Path(os.environ.get('XDG_CACHE_HOME', Path.home() / '.cache')) + return base / app_name +``` + +## Path Separators + +| OS | Separator | Example | +|----|-----------|---------| +| **Linux/macOS** | `/` | `/home/user/file.txt` | +| **Windows** | `\` | `C:\Users\name\file.txt` | +| **Windows (also works)** | `/` | `C:/Users/name/file.txt` | + +```python +# Always use pathlib or os.path - never hardcode separators +from pathlib import Path + +# Good +config_file = Path.home() / ".config" / "myapp" / "config.json" + +# Bad +config_file = os.environ['HOME'] + "/.config/myapp/config.json" +``` + +## Environment Variables + +| Variable | Linux | macOS | Windows | +|----------|-------|-------|---------| +| **Home** | `$HOME` | `$HOME` | `%USERPROFILE%` | +| **User** | `$USER` | `$USER` | `%USERNAME%` | +| **Temp** | `$TMPDIR` or `/tmp` | `$TMPDIR` | `%TEMP%` | +| **Path** | `$PATH` | `$PATH` | `%PATH%` | +| **Shell** | `$SHELL` | `$SHELL` | N/A | + +```python +# Cross-platform home directory +from pathlib import Path +home = Path.home() # Works everywhere + +# Or +import os +home = os.path.expanduser("~") +``` + +## Shell Config Files + +| Shell | Linux | macOS | Windows | +|-------|-------|-------|---------| +| **bash** | `~/.bashrc` | `~/.bash_profile` | `~/.bashrc` (Git Bash) | +| **zsh** | `~/.zshrc` | `~/.zshrc` | N/A | +| **fish** | `~/.config/fish/config.fish` | Same | Same | +| **PowerShell** | `~/.config/powershell/profile.ps1` | Same | `$PROFILE` | + +## Services/Daemons + +| Concept | Linux | macOS | Windows | +|---------|-------|-------|---------| +| **Service Manager** | systemd | launchd | Services | +| **User Service Dir** | `~/.config/systemd/user/` | `~/Library/LaunchAgents/` | Task Scheduler | +| **System Service Dir** | `/etc/systemd/system/` | `/Library/LaunchDaemons/` | Registry/Services | +| **Start Command** | `systemctl start x` | `launchctl start x` | `Start-Service x` | +| **Auto-start** | `systemctl enable x` | `RunAtLoad` plist key | Service properties | + +## Package Managers + +| OS | Built-in | Popular Third-Party | +|----|----------|---------------------| +| **Linux (Debian)** | apt | snap, flatpak | +| **Linux (Fedora)** | dnf | snap, flatpak | +| **Linux (Arch)** | pacman | yay (AUR) | +| **macOS** | - | Homebrew, MacPorts | +| **Windows** | winget | Chocolatey, Scoop | + +## Decision Guide + +### Where Should My App Store Data? + +``` +Is it user-editable config? +├── Yes → Config dir (XDG_CONFIG_HOME / App Support / APPDATA) +└── No + ├── Is it cacheable/regeneratable? + │ ├── Yes → Cache dir (XDG_CACHE_HOME / Caches / LOCALAPPDATA) + │ └── No + │ ├── Is it important data? + │ │ ├── Yes → Data dir (XDG_DATA_HOME / App Support / APPDATA) + │ │ └── No → State dir (XDG_STATE_HOME / App Support / LOCALAPPDATA) + │ └── Is it temporary (session only)? + │ └── Yes → Temp dir ($TMPDIR / %TEMP%) +``` + +### Best Practices + +| Practice | Why | +|----------|-----| +| **Use libraries** | They handle edge cases | +| **Create dirs on first use** | Don't assume they exist | +| **Respect platform conventions** | Users expect standard locations | +| **Support XDG overrides** | Power users customize paths | +| **Don't pollute home** | Use proper subdirectories | +| **Separate config/data/cache** | Different backup/sync needs | + +## Related + +- [[XDG Base Directory]] — Linux paths in detail +- [[macOS Development Basics]] — macOS paths in detail +- [[Windows Development Basics]] — Windows paths in detail +- [[Computer Science MOC]] — CS topics diff --git a/Computer Science/Git Internals.md b/Computer Science/Git Internals.md new file mode 100644 index 0000000..25cfb5c --- /dev/null +++ b/Computer Science/Git Internals.md @@ -0,0 +1,391 @@ +--- +title: Git Internals +aliases: + - How Git Works + - Git Objects + - Git Architecture +tags: + - cs + - git + - version-control + - internals +type: concept +status: complete +created: "2025-12-16" +--- + +# Git Internals + +Understanding how Git stores and manages version history under the hood. + +## Overview + +Git is fundamentally a content-addressable filesystem with a VCS built on top. + +``` +.git/ +├── HEAD # Current branch pointer +├── config # Repository config +├── index # Staging area +├── objects/ # All content (blobs, trees, commits) +│ ├── ab/ +│ │ └── cdef... # Object files +│ └── pack/ # Packed objects +├── refs/ # Branch and tag pointers +│ ├── heads/ +│ │ └── main +│ └── tags/ +└── logs/ # Reflog history +``` + +## Git Objects + +### Four Object Types + +| Type | Purpose | Content | +|------|---------|---------| +| **Blob** | File content | Raw file data | +| **Tree** | Directory | List of blobs and trees | +| **Commit** | Snapshot | Tree + metadata + parent(s) | +| **Tag** | Named reference | Points to commit | + +### Object Storage + +**All objects stored by SHA-1 hash of content.** + +```bash +# Examine object type +git cat-file -t abc123 + +# View object content +git cat-file -p abc123 + +# Compute hash without storing +echo "hello" | git hash-object --stdin +``` + +### Blob Object + +**Stores file content only (no filename).** + +```bash +# Create a blob +echo "Hello, World!" | git hash-object -w --stdin +# Returns: af5626b4a114abcb82d63db7c8082c3c4756e51b + +# View blob content +git cat-file -p af5626b +# Output: Hello, World! +``` + +### Tree Object + +**Represents a directory.** + +```bash +git cat-file -p HEAD^{tree} + +# Output: +100644 blob abc123... README.md +100644 blob def456... package.json +040000 tree 789abc... src +``` + +| Mode | Type | +|------|------| +| `100644` | Regular file | +| `100755` | Executable file | +| `120000` | Symbolic link | +| `040000` | Directory (tree) | + +### Commit Object + +```bash +git cat-file -p HEAD + +# Output: +tree 4b825dc... +parent a1b2c3d... +author Alice 1704063600 -0500 +committer Alice 1704063600 -0500 + +Add new feature +``` + +| Field | Description | +|-------|-------------| +| `tree` | Root tree SHA | +| `parent` | Parent commit(s) SHA | +| `author` | Who wrote the code | +| `committer` | Who made the commit | +| `message` | Commit message | + +## Object Relationships + +``` +Commit (abc123) + │ + └─► Tree (root) + │ + ├─► Blob (README.md) + ├─► Blob (package.json) + └─► Tree (src/) + │ + ├─► Blob (index.js) + └─► Blob (utils.js) +``` + +## References (Refs) + +### Branch = Pointer to Commit + +```bash +cat .git/refs/heads/main +# Output: abc123def456... (commit SHA) + +# Branches are just files containing SHAs! +``` + +### HEAD = Current Branch + +```bash +cat .git/HEAD +# Output: ref: refs/heads/main + +# Detached HEAD (points directly to commit): +# abc123def456... +``` + +### Reference Types + +| Ref | Location | Purpose | +|-----|----------|---------| +| **Branch** | `refs/heads/` | Movable commit pointer | +| **Tag** | `refs/tags/` | Fixed commit pointer | +| **Remote** | `refs/remotes/` | Remote tracking branches | +| **HEAD** | `.git/HEAD` | Current position | + +## The Index (Staging Area) + +```bash +# View index contents +git ls-files --stage + +# Output: +100644 abc123... 0 README.md +100644 def456... 0 src/index.js + +# Index is a binary file +file .git/index +# Output: Git index, version 2, 15 entries +``` + +### Index States + +``` +Working Directory Index (Stage) Repository + │ │ │ + │ git add file │ │ + ├──────────────────►│ │ + │ │ git commit │ + │ ├───────────────►│ + │ │ │ +``` + +## Content Addressing + +### SHA-1 Hashing + +**Everything in Git is content-addressable:** + +```python +# How Git computes blob hash +import hashlib + +content = b"Hello, World!" +header = f"blob {len(content)}\0".encode() +store = header + content + +sha1 = hashlib.sha1(store).hexdigest() +# af5626b4a114abcb82d63db7c8082c3c4756e51b +``` + +### Why SHA-1? + +| Benefit | Description | +|---------|-------------| +| **Integrity** | Any corruption changes hash | +| **Deduplication** | Same content = same hash | +| **Distributed** | No central authority needed | +| **Immutable** | Can't change without new hash | + +## Pack Files + +### Object Compression + +```bash +# Trigger packing +git gc + +# View pack contents +git verify-pack -v .git/objects/pack/pack-*.idx +``` + +### Delta Compression + +``` +Object A (1000 bytes) ─────────────────────► Stored fully + │ +Object B (1005 bytes) ─► Delta from A (50 bytes) + │ +Object C (1010 bytes) ─► Delta from B (55 bytes) +``` + +Git stores deltas between similar objects to save space. + +## Common Operations Explained + +### git add + +```bash +git add file.txt + +# Internally: +# 1. Hash file content → blob SHA +# 2. Store blob in .git/objects/ +# 3. Update index with (mode, SHA, path) +``` + +### git commit + +```bash +git commit -m "message" + +# Internally: +# 1. Create tree from index +# 2. Create commit pointing to tree +# 3. Update HEAD's branch to new commit +``` + +### git branch + +```bash +git branch feature + +# Internally: +# Create file .git/refs/heads/feature +# Content: current HEAD commit SHA +``` + +### git merge (fast-forward) + +``` +Before: +main: A ─ B ─ C + │ +feature: D ─ E + +After (FF): +main: A ─ B ─ C ─ D ─ E + │ +feature: (same) + +# Just moves main pointer to E +``` + +### git merge (three-way) + +``` +Before: +main: A ─ B ─ C ─ F + │ +feature: D ─ E + +After: +main: A ─ B ─ C ─ F ─ G (merge commit) + │ / +feature: D ─ E + +# G has two parents: F and E +``` + +### git rebase + +``` +Before: +main: A ─ B ─ C + │ +feature: D ─ E + +After: +main: A ─ B ─ C + │ +feature: D' ─ E' + +# D' and E' are NEW commits (different SHA) +# Same changes, different parent +``` + +## Reflog + +**History of HEAD movements.** + +```bash +git reflog + +# Output: +abc123 HEAD@{0}: commit: Add feature +def456 HEAD@{1}: checkout: moving from main to feature +789abc HEAD@{2}: commit: Fix bug + +# Recover "lost" commits +git checkout HEAD@{2} +``` + +## Porcelain vs Plumbing + +| Porcelain (User) | Plumbing (Internal) | +|------------------|---------------------| +| `git add` | `git hash-object`, `git update-index` | +| `git commit` | `git write-tree`, `git commit-tree` | +| `git branch` | `git update-ref` | +| `git log` | `git rev-list`, `git cat-file` | + +### Low-level Example + +```bash +# Create commit manually (plumbing) +echo "Hello" | git hash-object -w --stdin # Create blob +git update-index --add --cacheinfo 100644 hello.txt +git write-tree # Create tree +git commit-tree -m "Manual commit" # Create commit +git update-ref refs/heads/main # Update branch +``` + +## Garbage Collection + +```bash +# Run GC +git gc + +# What it does: +# 1. Packs loose objects +# 2. Removes unreachable objects +# 3. Compresses pack files +# 4. Prunes reflog entries +``` + +### Unreachable Objects + +``` +Reachable: HEAD or any ref can reach it +Unreachable: No ref points to it (orphaned) + +# Keep unreachable objects for 2 weeks (default) +git config gc.pruneExpire "2 weeks ago" +``` + +## Related + +- [[Build Systems]] — Development tooling +- [[Computer Science MOC]] — All CS topics diff --git a/Computer Science/Load Testing.md b/Computer Science/Load Testing.md new file mode 100644 index 0000000..8a03d75 --- /dev/null +++ b/Computer Science/Load Testing.md @@ -0,0 +1,427 @@ +--- +title: Load Testing +aliases: + - Performance Testing + - Stress Testing + - Capacity Testing +tags: + - cs + - testing + - performance + - sre +type: reference +status: complete +created: "2025-12-18" +--- + +# Load Testing + +Testing system behavior under expected and peak load conditions. + +## Overview + +| Aspect | Details | +|--------|---------| +| **Purpose** | Validate performance under load | +| **Metrics** | Response time, throughput, errors | +| **When** | Before releases, capacity planning | +| **Types** | Load, stress, spike, soak, scalability | + +## Types of Performance Tests + +### Test Type Comparison + +| Type | Load Level | Duration | Goal | +|------|------------|----------|------| +| **Load Test** | Expected | Minutes-hours | Validate normal performance | +| **Stress Test** | Above capacity | Until failure | Find breaking point | +| **Spike Test** | Sudden increase | Short bursts | Handle traffic spikes | +| **Soak Test** | Normal-high | Hours-days | Find memory leaks, degradation | +| **Scalability Test** | Incremental | Varies | Measure scaling behavior | + +### Load Patterns + +``` +Load Test: Stress Test: Spike Test: + Users Users Users + │ │ │ + 100├──────────── │ ╱╲ │ ╱╲ + │ │ │ ╱ ╲ │ ╱ ╲ + 50├ │ │ ╱ ╲ │ ╱ ╲ + │ ramp hold │ ╱ ╲ │ ╱ ╲ + │ ╱ │ ╱ ╲ │╱ ╲ + └─────────── └─────────── └─────────── + Time Time Time + +Soak Test: Scalability Test: + Users Users + │ │ + 100├──────────────── │ ╱ + │ │ │ ╱ + │ │ │ ╱ step + │ │ │ ╱ increase + │ sustained │ │ ╱ + └─────────────── └─────────── + Time (hours) Time +``` + +## Key Metrics + +### Response Time + +| Metric | Description | +|--------|-------------| +| **Average** | Mean response time | +| **Median (p50)** | 50th percentile | +| **p90** | 90% of requests faster than this | +| **p95** | 95th percentile | +| **p99** | 99th percentile | +| **Max** | Slowest request | + +``` +Response Time Distribution: + p99 + p95 │ + p90 │ │ + Median │ │ │ +Average │ │ │ │ + │ │ │ │ │ + ▼ ▼ ▼ ▼ ▼ +──────────────────────────────────────► +0ms 100ms 200ms 500ms +``` + +### Throughput + +| Metric | Description | +|--------|-------------| +| **RPS** | Requests per second | +| **TPS** | Transactions per second | +| **Concurrent Users** | Simultaneous active users | + +### Error Rates + +| Metric | Target | +|--------|--------| +| **Error Rate** | < 1% typically | +| **HTTP 5xx** | 0% ideally | +| **Timeouts** | Minimal | + +## Load Testing Tools + +### k6 + +```javascript +// k6 script +import http from 'k6/http'; +import { check, sleep } from 'k6'; + +export const options = { + stages: [ + { duration: '2m', target: 100 }, // Ramp up + { duration: '5m', target: 100 }, // Hold + { duration: '2m', target: 0 }, // Ramp down + ], + thresholds: { + http_req_duration: ['p(95)<500'], // 95% under 500ms + http_req_failed: ['rate<0.01'], // Error rate < 1% + }, +}; + +export default function() { + const res = http.get('https://api.example.com/users'); + + check(res, { + 'status is 200': (r) => r.status === 200, + 'response time < 500ms': (r) => r.timings.duration < 500, + }); + + sleep(1); // Think time +} +``` + +### Locust (Python) + +```python +from locust import HttpUser, task, between + +class WebsiteUser(HttpUser): + wait_time = between(1, 5) # Think time + + @task(3) # Weight 3 + def view_homepage(self): + self.client.get("/") + + @task(1) # Weight 1 + def view_product(self): + self.client.get("/product/123") + + def on_start(self): + """Login on start""" + self.client.post("/login", { + "username": "testuser", + "password": "password" + }) +``` + +### JMeter + +```xml + + + 100 + 60 + true + 300 + + + + api.example.com + /users + GET + +``` + +### Artillery + +```yaml +# artillery.yml +config: + target: "https://api.example.com" + phases: + - duration: 60 + arrivalRate: 5 + name: "Warm up" + - duration: 300 + arrivalRate: 50 + name: "Sustained load" + +scenarios: + - name: "Browse and purchase" + flow: + - get: + url: "/products" + - think: 2 + - get: + url: "/products/{{ $randomNumber(1, 100) }}" + - post: + url: "/cart" + json: + productId: "{{ $randomNumber(1, 100) }}" +``` + +### Tool Comparison + +| Tool | Language | GUI | Cloud | Strengths | +|------|----------|-----|-------|-----------| +| **k6** | JavaScript | No | Yes | Modern, scriptable, Grafana | +| **Locust** | Python | Web UI | Yes | Easy scripting, distributed | +| **JMeter** | Java/XML | Yes | Plugins | Feature-rich, enterprise | +| **Artillery** | YAML/JS | No | Yes | Easy config, serverless | +| **Gatling** | Scala | Yes | Yes | High performance | +| **wrk** | Lua | No | No | Simple, fast | + +## Test Design + +### Workload Modeling + +``` +Real User Flow: +1. Homepage (30% of traffic) +2. Search (25%) +3. Product Page (25%) +4. Add to Cart (15%) +5. Checkout (5%) + +Translate to test weights: +- homepage: weight 6 +- search: weight 5 +- product: weight 5 +- add_to_cart: weight 3 +- checkout: weight 1 +``` + +### Think Time + +```javascript +// Realistic user behavior +export default function() { + http.get('/'); + sleep(randomIntBetween(2, 5)); // Read page + + http.get('/products'); + sleep(randomIntBetween(3, 10)); // Browse + + http.post('/cart', { productId: 1 }); + sleep(randomIntBetween(1, 3)); // Quick action +} +``` + +### Data Parameterization + +```javascript +// k6 with CSV data +import papaparse from 'papaparse'; +import { SharedArray } from 'k6/data'; + +const users = new SharedArray('users', function() { + return papaparse.parse(open('./users.csv'), { header: true }).data; +}); + +export default function() { + const user = users[Math.floor(Math.random() * users.length)]; + http.post('/login', { + username: user.username, + password: user.password, + }); +} +``` + +## Running Tests + +### Pre-Test Checklist + +| Item | Description | +|------|-------------| +| **Baseline** | Measure current performance | +| **Environment** | Production-like, isolated | +| **Data** | Realistic test data | +| **Monitoring** | APM, logs, metrics ready | +| **Stakeholders** | Notify relevant teams | +| **Rollback** | Plan if tests cause issues | + +### During Test + +```bash +# Monitor during test +# CPU, memory, network, disk I/O +top +vmstat 1 +iostat -x 1 +netstat -s + +# Application metrics +# - Response times +# - Error rates +# - Database connections +# - Cache hit rates +# - Queue depths +``` + +### Post-Test Analysis + +| Analyze | Questions | +|---------|-----------| +| **Response times** | Meet SLOs? Degraded over time? | +| **Throughput** | Hit target RPS? | +| **Errors** | What failed? When? | +| **Resources** | CPU/memory bottlenecks? | +| **Database** | Slow queries? Connection pool? | +| **External deps** | Third-party latency? | + +## Results Interpretation + +### Healthy vs Unhealthy + +``` +Healthy Load Test: +Response Time ──────────────────────── + stable, within SLO + time → + +Unhealthy Load Test: +Response Time ╱ + ╱ + ╱ degradation under load + ╱ + ───╱───────────────────── + time → + +Resource Exhaustion: +Response Time │ + │ cliff + ─────────┘ + time → +``` + +### Identifying Bottlenecks + +| Symptom | Possible Cause | +|---------|----------------| +| High CPU | Inefficient code, missing caching | +| High memory | Memory leaks, large caches | +| High DB latency | Missing indexes, N+1 queries | +| Connection errors | Pool exhausted, timeouts | +| Increasing latency | Queue buildup, GC pressure | + +## CI/CD Integration + +```yaml +# GitHub Actions example +name: Performance Tests +on: + push: + branches: [main] + +jobs: + load-test: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - name: Run k6 test + uses: grafana/k6-action@v0.3.0 + with: + filename: tests/load-test.js + + - name: Upload results + uses: actions/upload-artifact@v3 + with: + name: k6-results + path: results.json +``` + +### Performance Budgets + +```javascript +// k6 thresholds as gates +export const options = { + thresholds: { + // Fail if any threshold breached + http_req_duration: ['p(95)<500', 'p(99)<1000'], + http_req_failed: ['rate<0.01'], + http_reqs: ['rate>100'], // Minimum throughput + }, +}; +``` + +## Best Practices + +| Practice | Description | +|----------|-------------| +| **Test in production-like env** | Same infra, data volume | +| **Use realistic data** | Not just "test user" | +| **Include think time** | Real users pause | +| **Ramp gradually** | Don't spike immediately | +| **Monitor everything** | APM, logs, metrics | +| **Test regularly** | Catch regressions | +| **Set baselines** | Compare against known good | +| **Test dependencies** | Don't mock everything | + +## Common Mistakes + +| Mistake | Issue | +|---------|-------| +| **Testing against mocks** | Miss real bottlenecks | +| **No think time** | Unrealistic load pattern | +| **Ignoring ramp-up** | Miss warm-up issues | +| **Single endpoint** | Miss interaction issues | +| **Insufficient duration** | Miss memory leaks | +| **Wrong environment** | Results don't transfer | + +## Related + +- [[SLOs and SLIs]] — Performance targets +- [[Chaos Engineering]] — Failure testing +- [[Testing Frameworks]] — Testing tools +- [[Computer Science MOC]] — CS topics diff --git a/Computer Science/Property-Based Testing.md b/Computer Science/Property-Based Testing.md new file mode 100644 index 0000000..0997ae6 --- /dev/null +++ b/Computer Science/Property-Based Testing.md @@ -0,0 +1,394 @@ +--- +title: Property-Based Testing +aliases: + - PBT + - QuickCheck + - Generative Testing +tags: + - cs + - testing + - practices +type: reference +status: complete +created: "2025-12-18" +--- + +# Property-Based Testing + +Testing technique that verifies properties hold for randomly generated inputs. + +## Overview + +| Aspect | Details | +|--------|---------| +| **Concept** | Define properties, generate test cases | +| **Origin** | QuickCheck (Haskell, 1999) | +| **Approach** | Many random inputs vs few handpicked examples | +| **Benefit** | Finds edge cases humans miss | + +## Example-Based vs Property-Based + +### Example-Based (Traditional) + +```python +def test_reverse(): + assert reverse([1, 2, 3]) == [3, 2, 1] + assert reverse([]) == [] + assert reverse([1]) == [1] + # Limited to cases we think of +``` + +### Property-Based + +```python +from hypothesis import given +from hypothesis import strategies as st + +@given(st.lists(st.integers())) +def test_reverse_twice_is_identity(xs): + assert reverse(reverse(xs)) == xs + +@given(st.lists(st.integers())) +def test_reverse_preserves_length(xs): + assert len(reverse(xs)) == len(xs) + +# Tests run with hundreds of random inputs +``` + +## Key Concepts + +### Properties + +**Invariants that should hold for all valid inputs.** + +| Property Type | Description | Example | +|---------------|-------------|---------| +| **Round-trip** | Operation is reversible | encode(decode(x)) == x | +| **Idempotent** | Applying twice = once | sort(sort(x)) == sort(x) | +| **Commutative** | Order doesn't matter | a + b == b + a | +| **Invariant** | Something preserved | len(sort(x)) == len(x) | +| **Comparison** | Compare with known-good | fastSort(x) == sort(x) | + +### Generators (Strategies) + +```python +from hypothesis import strategies as st + +# Primitive types +st.integers() +st.floats() +st.text() +st.booleans() + +# Bounded +st.integers(min_value=0, max_value=100) +st.text(min_size=1, max_size=10) + +# Collections +st.lists(st.integers()) +st.sets(st.text()) +st.dictionaries(st.text(), st.integers()) + +# Composite +st.tuples(st.integers(), st.text()) +st.one_of(st.integers(), st.text()) + +# Custom +@st.composite +def user_strategy(draw): + name = draw(st.text(min_size=1)) + age = draw(st.integers(min_value=0, max_value=150)) + return User(name=name, age=age) +``` + +### Shrinking + +**When a failing case is found, automatically minimize it.** + +``` +Found failing input: [847, -293, 0, 12, 99, -5, 8] +Shrinking... +Minimal failing input: [0, -1] +``` + +## Common Properties + +### Mathematical Properties + +```python +# Associativity +@given(st.integers(), st.integers(), st.integers()) +def test_addition_associative(a, b, c): + assert (a + b) + c == a + (b + c) + +# Commutativity +@given(st.integers(), st.integers()) +def test_addition_commutative(a, b): + assert a + b == b + a + +# Identity +@given(st.integers()) +def test_addition_identity(a): + assert a + 0 == a +``` + +### Data Structure Properties + +```python +# Sorting +@given(st.lists(st.integers())) +def test_sort_idempotent(xs): + assert sorted(sorted(xs)) == sorted(xs) + +@given(st.lists(st.integers())) +def test_sort_preserves_elements(xs): + assert sorted(xs) == sorted(sorted(xs)) + assert set(sorted(xs)) == set(xs) + +@given(st.lists(st.integers())) +def test_sort_is_ordered(xs): + result = sorted(xs) + for i in range(len(result) - 1): + assert result[i] <= result[i + 1] +``` + +### Encoding/Serialization Properties + +```python +@given(st.dictionaries(st.text(), st.integers())) +def test_json_roundtrip(data): + assert json.loads(json.dumps(data)) == data + +@given(st.binary()) +def test_base64_roundtrip(data): + assert base64.b64decode(base64.b64encode(data)) == data +``` + +### API/Database Properties + +```python +@given(user_strategy()) +def test_user_crud(user): + # Create + user_id = db.create_user(user) + + # Read + retrieved = db.get_user(user_id) + assert retrieved.name == user.name + assert retrieved.age == user.age + + # Update + user.age = user.age + 1 + db.update_user(user_id, user) + assert db.get_user(user_id).age == user.age + + # Delete + db.delete_user(user_id) + assert db.get_user(user_id) is None +``` + +## Property-Based Testing Libraries + +| Language | Library | +|----------|---------| +| **Python** | Hypothesis | +| **JavaScript** | fast-check | +| **Java** | jqwik, junit-quickcheck | +| **Haskell** | QuickCheck | +| **Scala** | ScalaCheck | +| **Rust** | proptest, quickcheck | +| **Go** | gopter | +| **C#** | FsCheck | +| **Elixir** | StreamData | + +## Hypothesis (Python) + +### Basic Usage + +```python +from hypothesis import given, settings, assume +from hypothesis import strategies as st + +@given(st.integers(), st.integers()) +def test_addition_commutative(x, y): + assert x + y == y + x + +# With settings +@settings(max_examples=1000) +@given(st.lists(st.integers())) +def test_with_more_examples(xs): + assert reverse(reverse(xs)) == xs + +# With assumptions (filter invalid inputs) +@given(st.integers(), st.integers()) +def test_division(x, y): + assume(y != 0) # Skip when y is 0 + assert (x * y) / y == x +``` + +### Custom Strategies + +```python +from hypothesis import strategies as st +from dataclasses import dataclass + +@dataclass +class Order: + id: str + items: list + total: float + +# Build strategy from fields +order_strategy = st.builds( + Order, + id=st.uuids().map(str), + items=st.lists(st.text(min_size=1), min_size=1), + total=st.floats(min_value=0, max_value=10000) +) + +@given(order_strategy) +def test_order_processing(order): + result = process_order(order) + assert result.status in ['success', 'pending'] +``` + +## fast-check (JavaScript) + +```javascript +import fc from 'fast-check'; + +// Basic test +test('string reverse', () => { + fc.assert( + fc.property(fc.string(), (s) => { + return reverse(reverse(s)) === s; + }) + ); +}); + +// With complex data +test('user serialization', () => { + const userArb = fc.record({ + name: fc.string({ minLength: 1 }), + email: fc.emailAddress(), + age: fc.integer({ min: 0, max: 150 }) + }); + + fc.assert( + fc.property(userArb, (user) => { + const json = JSON.stringify(user); + const parsed = JSON.parse(json); + return parsed.name === user.name && + parsed.email === user.email && + parsed.age === user.age; + }) + ); +}); + +// With preconditions +test('division', () => { + fc.assert( + fc.property( + fc.integer(), + fc.integer().filter(n => n !== 0), + (x, y) => { + return Math.abs((x * y) / y - x) < 0.0001; + } + ) + ); +}); +``` + +## Common Patterns + +### Oracle Testing + +**Compare your implementation against a reference.** + +```python +@given(st.lists(st.integers())) +def test_quicksort_matches_builtin(xs): + assert quicksort(xs) == sorted(xs) +``` + +### Metamorphic Testing + +**If input changes in known way, output should change predictably.** + +```python +@given(st.lists(st.integers())) +def test_sort_adding_element(xs): + x = 0 + result1 = sorted(xs) + result2 = sorted(xs + [x]) + # Adding element should increase length by 1 + assert len(result2) == len(result1) + 1 +``` + +### Stateful Testing + +**Test sequences of operations.** + +```python +from hypothesis.stateful import RuleBasedStateMachine, rule, invariant + +class SetMachine(RuleBasedStateMachine): + def __init__(self): + super().__init__() + self.model = set() # Reference implementation + self.impl = MySet() # Our implementation + + @rule(x=st.integers()) + def add(self, x): + self.model.add(x) + self.impl.add(x) + + @rule(x=st.integers()) + def remove(self, x): + self.model.discard(x) + self.impl.remove(x) + + @invariant() + def sets_match(self): + assert set(self.impl) == self.model + +TestSet = SetMachine.TestCase +``` + +## Benefits and Challenges + +### Benefits + +| Benefit | Description | +|---------|-------------| +| **Edge case discovery** | Finds inputs you'd never think of | +| **Higher coverage** | Tests thousands of cases | +| **Specification clarity** | Forces you to think about properties | +| **Regression testing** | Shrunk failing cases become examples | +| **Less test maintenance** | Properties stable even as impl changes | + +### Challenges + +| Challenge | Mitigation | +|-----------|------------| +| **Finding good properties** | Start with simple invariants | +| **Slow tests** | Limit examples, optimize generators | +| **Flaky tests** | Use seeds, avoid time-dependent tests | +| **Complex custom types** | Build composable strategies | +| **Understanding failures** | Rely on shrinking | + +## When to Use + +| Good For | Not Ideal For | +|----------|---------------| +| ✅ Pure functions | ❌ UI testing | +| ✅ Parsers/serializers | ❌ Integration tests | +| ✅ Data structures | ❌ Stateful systems (harder) | +| ✅ Mathematical code | ❌ Tests requiring specific setup | +| ✅ Finding edge cases | ❌ When properties unclear | + +## Related + +- [[TDD and BDD]] — Test-driven development +- [[Testing Frameworks]] — Testing tools +- [[Computer Science MOC]] — CS topics diff --git a/Computer Science/REST vs GraphQL vs gRPC.md b/Computer Science/REST vs GraphQL vs gRPC.md new file mode 100644 index 0000000..e396442 --- /dev/null +++ b/Computer Science/REST vs GraphQL vs gRPC.md @@ -0,0 +1,353 @@ +--- +title: REST vs GraphQL vs gRPC +aliases: + - API Styles + - API Comparison +tags: + - cs + - api + - architecture + - comparison +type: comparison +status: complete +created: "2025-12-16" +--- + +# REST vs GraphQL vs gRPC + +Comparison of major API architectural styles for building web services. + +## Overview + +| Aspect | REST | GraphQL | gRPC | +|--------|------|---------|------| +| **Type** | Architectural style | Query language | RPC framework | +| **Protocol** | HTTP | HTTP | HTTP/2 | +| **Data Format** | JSON (typically) | JSON | Protocol Buffers | +| **Schema** | Optional (OpenAPI) | Required (SDL) | Required (Protobuf) | +| **Transport** | Text | Text | Binary | + +## REST + +### Characteristics + +``` +GET /users/123 → Read user +POST /users → Create user +PUT /users/123 → Update user +DELETE /users/123 → Delete user +``` + +| Principle | Description | +|-----------|-------------| +| **Stateless** | Each request contains all needed info | +| **Resource-based** | URLs represent resources | +| **HTTP Verbs** | GET, POST, PUT, PATCH, DELETE | +| **Cacheable** | HTTP caching mechanisms | +| **Uniform Interface** | Standard conventions | + +### Example + +```bash +# Get user +GET /api/users/123 +Accept: application/json + +Response: +{ + "id": 123, + "name": "Alice", + "email": "alice@example.com", + "posts": [ + {"id": 1, "title": "Hello"}, + {"id": 2, "title": "World"} + ] +} +``` + +### Pros & Cons + +| Pros | Cons | +|------|------| +| ✅ Simple, well-understood | ❌ Over-fetching (too much data) | +| ✅ HTTP caching | ❌ Under-fetching (multiple requests) | +| ✅ Stateless, scalable | ❌ No standard query language | +| ✅ Wide tooling support | ❌ Versioning challenges | +| ✅ Browser-friendly | ❌ Multiple round trips | + +## GraphQL + +### Characteristics + +```graphql +# Single endpoint, query what you need +POST /graphql + +query { + user(id: 123) { + name + email + posts(first: 5) { + title + } + } +} +``` + +| Feature | Description | +|---------|-------------| +| **Single Endpoint** | All queries to /graphql | +| **Client-Specified** | Client asks for exact fields | +| **Strongly Typed** | Schema defines all types | +| **Introspection** | Schema is queryable | +| **Real-time** | Subscriptions for live data | + +### Schema Definition + +```graphql +type User { + id: ID! + name: String! + email: String! + posts: [Post!]! +} + +type Post { + id: ID! + title: String! + content: String + author: User! +} + +type Query { + user(id: ID!): User + users(limit: Int): [User!]! +} + +type Mutation { + createUser(name: String!, email: String!): User! + updateUser(id: ID!, name: String): User! +} + +type Subscription { + postCreated: Post! +} +``` + +### Example Query + +```graphql +# Query +query GetUserWithPosts($userId: ID!) { + user(id: $userId) { + name + posts(first: 5) { + title + createdAt + } + } +} + +# Variables +{ "userId": "123" } + +# Response +{ + "data": { + "user": { + "name": "Alice", + "posts": [ + { "title": "Hello", "createdAt": "2024-01-15" }, + { "title": "World", "createdAt": "2024-01-16" } + ] + } + } +} +``` + +### Pros & Cons + +| Pros | Cons | +|------|------| +| ✅ No over/under-fetching | ❌ Complexity (N+1, caching) | +| ✅ Single request | ❌ HTTP caching harder | +| ✅ Strong typing | ❌ File uploads awkward | +| ✅ Great for mobile | ❌ Learning curve | +| ✅ Introspection | ❌ Rate limiting complex | + +## gRPC + +### Characteristics + +```protobuf +// Define service in .proto file +service UserService { + rpc GetUser(GetUserRequest) returns (User); + rpc CreateUser(CreateUserRequest) returns (User); + rpc StreamUsers(StreamRequest) returns (stream User); +} + +message User { + int64 id = 1; + string name = 2; + string email = 3; +} + +message GetUserRequest { + int64 id = 1; +} +``` + +| Feature | Description | +|---------|-------------| +| **Binary Protocol** | Protocol Buffers (efficient) | +| **HTTP/2** | Multiplexing, streaming | +| **Code Generation** | Clients generated from .proto | +| **Streaming** | Client, server, bidirectional | +| **Strongly Typed** | Compiled schema | + +### Example Usage + +```go +// Generated Go client +client := pb.NewUserServiceClient(conn) + +// Unary call +user, err := client.GetUser(ctx, &pb.GetUserRequest{Id: 123}) + +// Server streaming +stream, err := client.StreamUsers(ctx, &pb.StreamRequest{}) +for { + user, err := stream.Recv() + if err == io.EOF { + break + } + fmt.Println(user) +} +``` + +### Streaming Types + +| Type | Description | +|------|-------------| +| **Unary** | Single request, single response | +| **Server Streaming** | Single request, stream of responses | +| **Client Streaming** | Stream of requests, single response | +| **Bidirectional** | Stream both ways | + +### Pros & Cons + +| Pros | Cons | +|------|------| +| ✅ Very fast (binary) | ❌ Not browser-native | +| ✅ Strong typing | ❌ Harder to debug (binary) | +| ✅ Streaming built-in | ❌ Learning curve | +| ✅ Code generation | ❌ Requires HTTP/2 | +| ✅ Bi-directional | ❌ Less tooling than REST | + +## Comparison Matrix + +| Feature | REST | GraphQL | gRPC | +|---------|------|---------|------| +| **Protocol** | HTTP/1.1+ | HTTP/1.1+ | HTTP/2 | +| **Format** | JSON | JSON | Protobuf | +| **Schema** | Optional | Required | Required | +| **Browser Support** | ✅ Native | ✅ Via HTTP | ⚠️ gRPC-web | +| **Caching** | ✅ HTTP caching | ⚠️ Custom | ❌ No standard | +| **Real-time** | ⚠️ WebSocket/SSE | ✅ Subscriptions | ✅ Streaming | +| **File Upload** | ✅ Easy | ⚠️ Workarounds | ⚠️ Chunked | +| **Tooling** | ✅ Excellent | ✅ Good | ⚠️ Growing | +| **Learning Curve** | Low | Medium | Medium-High | + +## Performance + +| Aspect | REST | GraphQL | gRPC | +|--------|------|---------|------| +| **Payload Size** | Medium | Small-Medium | Small | +| **Serialization** | Slow (JSON) | Slow (JSON) | Fast (binary) | +| **Latency** | Higher | Medium | Lowest | +| **Throughput** | Good | Good | Excellent | + +**Typical Latency:** + +``` +REST: ~100ms +GraphQL: ~80ms +gRPC: ~10-30ms +``` + +## Use Case Decision Guide + +### Choose REST When + +| Scenario | Reason | +|----------|--------| +| **Public APIs** | Universal understanding | +| **Simple CRUD** | Resources map well | +| **Caching important** | HTTP caching works | +| **Browser-first** | Native support | +| **Team new to APIs** | Lowest learning curve | + +### Choose GraphQL When + +| Scenario | Reason | +|----------|--------| +| **Mobile apps** | Minimize data transfer | +| **Multiple frontends** | Each gets what it needs | +| **Rapid iteration** | No endpoint changes | +| **Complex data needs** | Nested resources | +| **API aggregation** | Combine multiple sources | + +### Choose gRPC When + +| Scenario | Reason | +|----------|--------| +| **Microservices** | Service-to-service calls | +| **High performance** | Binary protocol | +| **Streaming needed** | Built-in support | +| **Polyglot systems** | Code generation | +| **Internal APIs** | Not browser-facing | + +## Hybrid Approaches + +### Common Patterns + +``` +┌─────────────────────────────────────────────────────┐ +│ Frontend │ +└─────────────────────────┬───────────────────────────┘ + │ + GraphQL / REST + │ + ▼ +┌─────────────────────────────────────────────────────┐ +│ API Gateway │ +└─────────────────────────┬───────────────────────────┘ + │ + gRPC + │ + ┌───────────────────┼───────────────────┐ + ▼ ▼ ▼ +┌──────────┐ ┌──────────┐ ┌──────────┐ +│ Service A│ │ Service B│ │ Service C│ +└──────────┘ └──────────┘ └──────────┘ +``` + +### REST + GraphQL + +- REST for simple resources +- GraphQL for complex queries +- GraphQL wrapping REST endpoints + +### gRPC + REST + +- gRPC for internal microservices +- REST/GraphQL gateway for external APIs +- gRPC-web for browser support + +## Related + +- [[API Design Patterns]] — API best practices +- [[OAuth and OIDC]] — API authentication +- [[Networking Fundamentals]] — HTTP basics +- [[Computer Science MOC]] — All CS topics diff --git a/Computer Science/Regex Reference.md b/Computer Science/Regex Reference.md new file mode 100644 index 0000000..15eb95c --- /dev/null +++ b/Computer Science/Regex Reference.md @@ -0,0 +1,341 @@ +--- +title: Regex Reference +aliases: + - Regular Expressions + - Regex + - RegExp +tags: + - cs + - reference + - patterns + - text-processing +type: reference +status: complete +created: "2025-12-16" +--- + +# Regex Reference + +Comprehensive reference for regular expression syntax and patterns. + +## Basic Syntax + +### Literals + +| Pattern | Matches | +|---------|---------| +| `abc` | Literal "abc" | +| `123` | Literal "123" | + +### Metacharacters + +| Char | Meaning | Example | +|------|---------|---------| +| `.` | Any character (except newline) | `a.c` → "abc", "a1c" | +| `^` | Start of string/line | `^Hello` | +| `$` | End of string/line | `world$` | +| `\|` | Alternation (OR) | `cat\|dog` | +| `\` | Escape metacharacter | `\.` matches "." | + +## Character Classes + +| Pattern | Matches | +|---------|---------| +| `[abc]` | Any of a, b, c | +| `[^abc]` | Not a, b, or c | +| `[a-z]` | Any lowercase letter | +| `[A-Z]` | Any uppercase letter | +| `[0-9]` | Any digit | +| `[a-zA-Z0-9]` | Any alphanumeric | + +### Shorthand Classes + +| Pattern | Equivalent | Meaning | +|---------|------------|---------| +| `\d` | `[0-9]` | Digit | +| `\D` | `[^0-9]` | Non-digit | +| `\w` | `[a-zA-Z0-9_]` | Word character | +| `\W` | `[^a-zA-Z0-9_]` | Non-word character | +| `\s` | `[ \t\n\r\f]` | Whitespace | +| `\S` | `[^ \t\n\r\f]` | Non-whitespace | + +### POSIX Classes + +| Pattern | Meaning | +|---------|---------| +| `[:alpha:]` | Letters | +| `[:digit:]` | Digits | +| `[:alnum:]` | Alphanumeric | +| `[:space:]` | Whitespace | +| `[:upper:]` | Uppercase | +| `[:lower:]` | Lowercase | + +## Quantifiers + +| Pattern | Meaning | Example | +|---------|---------|---------| +| `*` | 0 or more | `ab*c` → "ac", "abc", "abbc" | +| `+` | 1 or more | `ab+c` → "abc", "abbc" (not "ac") | +| `?` | 0 or 1 | `colou?r` → "color", "colour" | +| `{n}` | Exactly n | `\d{4}` → "2024" | +| `{n,}` | n or more | `\d{2,}` → "12", "123", "1234" | +| `{n,m}` | Between n and m | `\d{2,4}` → "12", "123", "1234" | + +### Greedy vs Lazy + +| Greedy | Lazy | Behavior | +|--------|------|----------| +| `*` | `*?` | Match as few as possible | +| `+` | `+?` | Match as few as possible | +| `?` | `??` | Match as few as possible | +| `{n,m}` | `{n,m}?` | Match as few as possible | + +``` +String:
content
+Greedy: <.*> → "
content
" +Lazy: <.*?> → "
" +``` + +## Groups and Capturing + +### Capturing Groups + +```regex +(\d{4})-(\d{2})-(\d{2}) +``` + +| Group | Content | +|-------|---------| +| `$0` or `\0` | Entire match | +| `$1` or `\1` | First group (year) | +| `$2` or `\2` | Second group (month) | +| `$3` or `\3` | Third group (day) | + +### Non-Capturing Groups + +```regex +(?:abc)+ +``` + +Groups without capturing (for quantifiers). + +### Named Groups + +```regex +(?\d{4})-(?\d{2})-(?\d{2}) +``` + +Access by name: `$+{year}`, `\k` + +### Backreferences + +```regex +# Match repeated words +\b(\w+)\s+\1\b +# Matches: "the the", "is is" +``` + +## Anchors + +| Pattern | Meaning | +|---------|---------| +| `^` | Start of string (or line with `m` flag) | +| `$` | End of string (or line with `m` flag) | +| `\b` | Word boundary | +| `\B` | Non-word boundary | +| `\A` | Start of string (absolute) | +| `\Z` | End of string (absolute) | + +### Word Boundary Examples + +```regex +\bcat\b # Matches "cat" not "category" +\bcat # Matches "cat" and "category" +cat\b # Matches "cat" and "tomcat" +``` + +## Lookahead and Lookbehind + +### Lookahead + +| Pattern | Meaning | +|---------|---------| +| `(?=...)` | Positive lookahead | +| `(?!...)` | Negative lookahead | + +```regex +# Match "foo" followed by "bar" +foo(?=bar) # Matches "foo" in "foobar" + +# Match "foo" not followed by "bar" +foo(?!bar) # Matches "foo" in "foobaz" +``` + +### Lookbehind + +| Pattern | Meaning | +|---------|---------| +| `(?<=...)` | Positive lookbehind | +| `(?]*)>(.*?) +``` + +### Slug + +```regex +^[a-z0-9]+(?:-[a-z0-9]+)*$ +``` + +## Language-Specific + +### JavaScript + +```javascript +const regex = /\d{3}-\d{4}/g; +const regex2 = new RegExp('\\d{3}-\\d{4}', 'g'); + +'123-4567'.match(regex); // ['123-4567'] +regex.test('123-4567'); // true +'123-4567'.replace(regex, 'XXX'); // 'XXX' +``` + +### Python + +```python +import re + +pattern = r'\d{3}-\d{4}' +re.search(pattern, '123-4567') # Match object +re.findall(pattern, 'a 123-4567 b') # ['123-4567'] +re.sub(pattern, 'XXX', '123-4567') # 'XXX' + +# Compiled regex +compiled = re.compile(pattern) +compiled.search('123-4567') +``` + +### Go + +```go +import "regexp" + +re := regexp.MustCompile(`\d{3}-\d{4}`) +re.MatchString("123-4567") // true +re.FindString("a 123-4567 b") // "123-4567" +re.ReplaceAllString("123-4567", "X") // "X" +``` + +### Rust + +```rust +use regex::Regex; + +let re = Regex::new(r"\d{3}-\d{4}").unwrap(); +re.is_match("123-4567"); // true +re.find("a 123-4567 b"); // Some(Match) +re.replace("123-4567", "XXX"); // "XXX" +``` + +## Performance Tips + +| Tip | Rationale | +|-----|-----------| +| **Anchor when possible** | `^abc` faster than `abc` | +| **Be specific** | `\d{3}` faster than `\d+` for 3 digits | +| **Avoid catastrophic backtracking** | `(a+)+b` on "aaaaaaaaac" | +| **Use non-capturing groups** | `(?:abc)` when you don't need capture | +| **Compile once, use many** | Don't recreate regex in loops | +| **Use character classes** | `[aeiou]` faster than `a\|e\|i\|o\|u` | + +### Catastrophic Backtracking + +```regex +# Bad - exponential backtracking on failure +(a+)+$ + +# Better - possessive or atomic groups +(?>a+)+$ # Atomic group +a++$ # Possessive quantifier +``` + +## Related + +- [[Testing Frameworks]] — Regex testing +- [[Computer Science MOC]] — All CS topics diff --git a/Computer Science/TDD and BDD.md b/Computer Science/TDD and BDD.md new file mode 100644 index 0000000..e5a98ef --- /dev/null +++ b/Computer Science/TDD and BDD.md @@ -0,0 +1,355 @@ +--- +title: TDD and BDD +aliases: + - Test-Driven Development + - Behavior-Driven Development + - Test First Development +tags: + - cs + - testing + - practices + - methodology +type: reference +status: complete +created: "2025-12-18" +--- + +# TDD and BDD + +Development methodologies that prioritize testing as a design and specification tool. + +## Overview + +| Aspect | TDD | BDD | +|--------|-----|-----| +| **Full Name** | Test-Driven Development | Behavior-Driven Development | +| **Focus** | Code correctness | System behavior | +| **Tests Written By** | Developers | Developers + stakeholders | +| **Language** | Technical assertions | Natural language specs | +| **Granularity** | Unit level | Feature/behavior level | + +## Test-Driven Development (TDD) + +### The TDD Cycle (Red-Green-Refactor) + +``` + ┌─────────────────────┐ + │ │ + │ RED │ + │ Write failing test │ + │ │ + └──────────┬──────────┘ + │ + ▼ + ┌──────────────────────┐ + │ │ + │ GREEN │ + │ Write minimal code │ + │ to pass the test │ + │ │ + └──────────┬───────────┘ + │ + ▼ + ┌──────────────────────┐ + │ │ + │ REFACTOR │ + │ Improve code while │ + │ keeping tests green │ + │ │ + └──────────┬───────────┘ + │ + └───────────────┐ + │ + ▼ + (Repeat cycle) +``` + +### TDD Example + +```python +# Step 1: RED - Write failing test +def test_add_returns_sum(): + result = add(2, 3) + assert result == 5 + +# Running this fails: NameError: name 'add' is not defined + +# Step 2: GREEN - Write minimal code to pass +def add(a, b): + return 5 # Minimal code to pass this specific test + +# Test passes, but implementation is wrong... + +# Step 3: Refactor with more tests +def test_add_returns_sum(): + assert add(2, 3) == 5 + assert add(0, 0) == 0 + assert add(-1, 1) == 0 + +# Now we fix the implementation +def add(a, b): + return a + b +``` + +### TDD Best Practices + +| Practice | Description | +|----------|-------------| +| **One test at a time** | Focus on single behavior | +| **Minimal code** | Only write code to pass test | +| **Run tests frequently** | After every small change | +| **Refactor with confidence** | Tests catch regressions | +| **Test behavior, not implementation** | Focus on what, not how | + +### TDD Anti-Patterns + +| Anti-Pattern | Problem | +|--------------|---------| +| **Writing tests after code** | Loses design benefits | +| **Testing implementation** | Tests break on refactor | +| **Too many tests at once** | Loses focus, harder to debug | +| **Skipping refactor step** | Accumulates technical debt | +| **Testing private methods** | Couples to implementation | + +## Behavior-Driven Development (BDD) + +### BDD Structure (Given-When-Then) + +```gherkin +Feature: User Authentication + As a registered user + I want to log into my account + So that I can access my personal dashboard + + Scenario: Successful login + Given I am on the login page + And I have a registered account with email "user@example.com" + When I enter email "user@example.com" + And I enter password "correctpassword" + And I click the login button + Then I should be redirected to the dashboard + And I should see "Welcome back" message + + Scenario: Failed login with wrong password + Given I am on the login page + When I enter email "user@example.com" + And I enter password "wrongpassword" + And I click the login button + Then I should see "Invalid credentials" error + And I should remain on the login page +``` + +### BDD Flow + +``` +┌─────────────────────────────────────────────────────────────┐ +│ 1. Discovery Workshop │ +│ Stakeholders + Devs + QA discuss feature │ +└──────────────────────────┬──────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────┐ +│ 2. Write Scenarios │ +│ Gherkin/natural language specifications │ +└──────────────────────────┬──────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────┐ +│ 3. Automate Scenarios │ +│ Step definitions that run the application │ +└──────────────────────────┬──────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────┐ +│ 4. Implement Feature │ +│ Make scenarios pass (TDD at implementation level) │ +└─────────────────────────────────────────────────────────────┘ +``` + +### Step Definitions + +```python +# Python with Behave +from behave import given, when, then + +@given('I am on the login page') +def step_on_login_page(context): + context.browser.get('/login') + +@when('I enter email "{email}"') +def step_enter_email(context, email): + context.browser.find_element_by_id('email').send_keys(email) + +@when('I click the login button') +def step_click_login(context): + context.browser.find_element_by_id('login-btn').click() + +@then('I should be redirected to the dashboard') +def step_on_dashboard(context): + assert context.browser.current_url.endswith('/dashboard') +``` + +```javascript +// JavaScript with Cucumber +const { Given, When, Then } = require('@cucumber/cucumber'); + +Given('I am on the login page', async function() { + await this.page.goto('/login'); +}); + +When('I enter email {string}', async function(email) { + await this.page.fill('#email', email); +}); + +Then('I should be redirected to the dashboard', async function() { + expect(this.page.url()).toContain('/dashboard'); +}); +``` + +### BDD Frameworks + +| Language | Framework | +|----------|-----------| +| **JavaScript** | Cucumber.js, Jest with Gherkin | +| **Python** | Behave, pytest-bdd | +| **Ruby** | Cucumber, RSpec | +| **Java** | Cucumber-JVM, JBehave | +| **C#** | SpecFlow | +| **Go** | Godog | + +## TDD vs BDD Comparison + +| Aspect | TDD | BDD | +|--------|-----|-----| +| **Primary Users** | Developers | Whole team | +| **Test Granularity** | Unit tests | Feature/integration | +| **Language** | Code assertions | Natural language | +| **Focus** | Implementation correctness | Business behavior | +| **Documentation** | Code is documentation | Living documentation | +| **Collaboration** | Developer-centric | Cross-functional | + +## Combining TDD and BDD + +``` +┌─────────────────────────────────────────────────────────────┐ +│ BDD Layer │ +│ Feature specs, acceptance tests, business behavior │ +│ (Cucumber/Gherkin scenarios) │ +└──────────────────────────┬──────────────────────────────────┘ + │ + │ Drives + ▼ +┌─────────────────────────────────────────────────────────────┐ +│ TDD Layer │ +│ Unit tests, implementation details, code correctness │ +│ (JUnit, pytest, Jest unit tests) │ +└─────────────────────────────────────────────────────────────┘ +``` + +### Example Workflow + +``` +1. BDD: Write scenario "User can view order history" + +2. Run scenario (fails - no implementation) + +3. TDD: Write unit test for OrderRepository.getByUserId() + - Red: Test fails + - Green: Implement method + - Refactor: Clean up + +4. TDD: Write unit test for OrderService.getUserOrders() + - Red/Green/Refactor cycle + +5. TDD: Write unit test for OrderController endpoint + - Red/Green/Refactor cycle + +6. Run BDD scenario again (now passes) +``` + +## Testing Pyramid with TDD/BDD + +``` + ╱╲ + ╱ ╲ + ╱ E2E╲ BDD Scenarios + ╱──────╲ (Few, slow) + ╱ ╲ + ╱Integration╲ BDD + TDD + ╱────────────╲ (Some, medium) + ╱ ╲ + ╱ Unit Tests ╲ TDD + ╱──────────────────╲ (Many, fast) + ╱ ╲ +``` + +## Benefits + +### TDD Benefits + +| Benefit | Description | +|---------|-------------| +| **Design feedback** | Tests reveal poor design early | +| **Regression safety** | Changes don't break existing code | +| **Documentation** | Tests document expected behavior | +| **Confidence** | Refactor without fear | +| **Focus** | Work on one thing at a time | + +### BDD Benefits + +| Benefit | Description | +|---------|-------------| +| **Shared understanding** | Team aligned on behavior | +| **Living documentation** | Specs always up to date | +| **Business alignment** | Tests reflect requirements | +| **Reduced rework** | Catch misunderstandings early | +| **Traceability** | Link tests to requirements | + +## Common Patterns + +### Arrange-Act-Assert (AAA) + +```python +def test_deposit_increases_balance(): + # Arrange + account = BankAccount(initial_balance=100) + + # Act + account.deposit(50) + + # Assert + assert account.balance == 150 +``` + +### Test Doubles + +| Type | Purpose | Example | +|------|---------|---------| +| **Stub** | Provide canned answers | Return fixed value | +| **Mock** | Verify interactions | Check method called | +| **Fake** | Working implementation | In-memory database | +| **Spy** | Record calls | Log method invocations | + +```python +# Mock example +def test_sends_welcome_email(): + # Arrange + email_service = Mock() + user_service = UserService(email_service) + + # Act + user_service.register("user@example.com") + + # Assert + email_service.send.assert_called_once_with( + to="user@example.com", + template="welcome" + ) +``` + +## Related + +- [[Testing Frameworks]] — Testing tools +- [[Property-Based Testing]] — Generative testing +- [[Code Review]] — Quality practices +- [[Computer Science MOC]] — CS topics diff --git a/Computer Science/Technical Writing.md b/Computer Science/Technical Writing.md new file mode 100644 index 0000000..0d21a95 --- /dev/null +++ b/Computer Science/Technical Writing.md @@ -0,0 +1,439 @@ +--- +title: Technical Writing +aliases: + - Technical Documentation + - Developer Documentation + - Tech Writing +tags: + - cs + - practices + - documentation + - communication +type: reference +status: complete +created: "2025-12-18" +--- + +# Technical Writing + +Creating clear, accurate documentation for technical audiences. + +## Overview + +| Aspect | Details | +|--------|---------| +| **Purpose** | Enable users to accomplish tasks | +| **Audience** | Developers, users, operators | +| **Formats** | Docs, READMEs, API refs, tutorials | +| **Key Principle** | Clarity over cleverness | + +## Documentation Types + +| Type | Purpose | Audience | +|------|---------|----------| +| **README** | Project overview, quick start | New users | +| **Tutorial** | Learning-oriented, step-by-step | Beginners | +| **How-to Guide** | Task-oriented, solve problems | Practitioners | +| **Reference** | Information-oriented, complete | Experienced users | +| **Explanation** | Understanding-oriented, concepts | All levels | +| **API Reference** | Endpoint/function details | Developers | +| **Architecture** | System design decisions | Engineers | + +## The Four Quadrants (Diátaxis) + +``` + PRACTICAL + │ + ┌───────────────────┼───────────────────┐ + │ │ │ + │ TUTORIALS │ HOW-TO GUIDES │ + │ (Learning) │ (Goals) │ + │ │ │ + │ "Follow along │ "How do I...?" │ +STUDY │ to learn" │ │ WORK + │ │ │ + ├───────────────────┼───────────────────┤ + │ │ │ + │ EXPLANATION │ REFERENCE │ + │ (Understanding) │ (Information) │ + │ │ │ + │ "Why does this │ "What are the │ + │ work this way?" │ options?" │ + └───────────────────┼───────────────────┘ + │ + THEORETICAL +``` + +## Writing Principles + +### Clarity + +| Do | Don't | +|----|-------| +| Use simple words | Use jargon unnecessarily | +| Short sentences | Run-on sentences | +| Active voice | Passive voice | +| One idea per sentence | Pack multiple concepts | +| Define acronyms first | Assume knowledge | + +### Examples + +```markdown +# Bad +The API endpoint, when invoked with the appropriate parameters +that have been correctly formatted according to the specification, +will return a response object containing the requested data. + +# Good +Call the `/users` endpoint with a user ID. +The API returns the user's profile data as JSON. +``` + +### Voice and Tone + +| Use | Avoid | +|-----|-------| +| Direct address ("you") | Third person ("the user") | +| Present tense | Future tense | +| Imperative mood | Suggestions | +| Confident | Hedging | + +```markdown +# Bad +The user might want to click the button, which will probably +start the process. + +# Good +Click **Start** to begin the process. +``` + +## README Structure + +```markdown +# Project Name + +One-line description of what this project does. + +## Features + +- Feature 1 +- Feature 2 + +## Quick Start + +\`\`\`bash +npm install myproject +myproject init +\`\`\` + +## Installation + +### Prerequisites +- Node.js 18+ +- npm or yarn + +### Steps +1. Clone the repository +2. Install dependencies +3. Configure environment + +## Usage + +Basic usage example with code. + +## Configuration + +| Option | Default | Description | +|--------|---------|-------------| +| `port` | 3000 | Server port | + +## API + +Brief API overview or link to full docs. + +## Contributing + +How to contribute to the project. + +## License + +MIT License +``` + +## API Documentation + +### Endpoint Documentation + +```markdown +## Create User + +Creates a new user account. + +### Request + +\`\`\` +POST /api/v1/users +\`\`\` + +**Headers** +| Header | Required | Description | +|--------|----------|-------------| +| Authorization | Yes | Bearer token | +| Content-Type | Yes | application/json | + +**Body** +\`\`\`json +{ + "email": "user@example.com", + "name": "Alice Smith", + "role": "member" +} +\`\`\` + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| email | string | Yes | Valid email address | +| name | string | Yes | Display name | +| role | string | No | User role (default: member) | + +### Response + +**Success (201)** +\`\`\`json +{ + "id": "user_123", + "email": "user@example.com", + "name": "Alice Smith", + "createdAt": "2024-01-15T10:30:00Z" +} +\`\`\` + +**Error (400)** +\`\`\`json +{ + "error": "validation_error", + "message": "Email already exists" +} +\`\`\` + +### Example + +\`\`\`bash +curl -X POST https://api.example.com/api/v1/users \ + -H "Authorization: Bearer $TOKEN" \ + -H "Content-Type: application/json" \ + -d '{"email": "user@example.com", "name": "Alice"}' +\`\`\` +``` + +## Code Examples + +### Good Code Examples + +```markdown +# Good: Complete, runnable example +\`\`\`python +from mylib import Client + +# Initialize the client +client = Client(api_key="your-api-key") + +# Fetch user data +user = client.get_user(user_id=123) +print(f"Hello, {user.name}") +\`\`\` + +# Bad: Incomplete, assumes context +\`\`\`python +user = client.get_user(123) +\`\`\` +``` + +### Progressive Complexity + +```markdown +## Basic Usage + +\`\`\`python +# Simple example - most common use case +result = search("query") +\`\`\` + +## Advanced Usage + +\`\`\`python +# With all options +result = search( + query="query", + filters={"date": "2024-01"}, + sort="relevance", + limit=10, + include_metadata=True +) +\`\`\` +``` + +## Formatting Best Practices + +### Use Tables for Options + +```markdown +| Option | Type | Default | Description | +|--------|------|---------|-------------| +| `timeout` | number | 30 | Request timeout in seconds | +| `retries` | number | 3 | Number of retry attempts | +| `debug` | boolean | false | Enable debug logging | +``` + +### Use Admonitions + +```markdown +> **Note**: This feature requires version 2.0+ + +> **Warning**: This action cannot be undone. + +> **Tip**: Use tab completion for faster input. + +> **Important**: Back up your data before proceeding. +``` + +### Use Lists for Steps + +```markdown +## Installation + +1. Download the installer +2. Run the setup wizard +3. Accept the license agreement +4. Choose installation directory +5. Click **Install** +``` + +## Error Messages + +### Document Error Codes + +```markdown +## Error Codes + +| Code | Message | Solution | +|------|---------|----------| +| E001 | Invalid API key | Check your key in Settings | +| E002 | Rate limit exceeded | Wait 60s and retry | +| E003 | Resource not found | Verify the ID exists | +``` + +### Troubleshooting Sections + +```markdown +## Troubleshooting + +### Connection refused + +**Symptom**: `Error: ECONNREFUSED` + +**Cause**: The server isn't running or is unreachable. + +**Solution**: +1. Verify the server is running: `systemctl status myapp` +2. Check the port is correct in config +3. Ensure firewall allows the port +``` + +## Versioning Documentation + +### Version Notices + +```markdown +> **Version**: This feature is available in v2.0+ + +> **Deprecated**: This method is deprecated. +> Use `newMethod()` instead. + +> **Breaking Change** (v3.0): The `oldParam` parameter +> has been renamed to `newParam`. +``` + +### Changelog Format + +```markdown +## Changelog + +### [2.1.0] - 2024-01-15 + +#### Added +- New export feature (#123) +- Support for custom themes + +#### Changed +- Improved search performance + +#### Fixed +- Bug where dates displayed incorrectly (#456) + +#### Deprecated +- `oldMethod()` - use `newMethod()` instead +``` + +## Tools + +| Tool | Purpose | +|------|---------| +| **Docusaurus** | Documentation sites (React) | +| **MkDocs** | Documentation sites (Python) | +| **Sphinx** | Python documentation | +| **GitBook** | Hosted documentation | +| **ReadTheDocs** | Documentation hosting | +| **Swagger/OpenAPI** | API documentation | +| **JSDoc/TSDoc** | JS/TS code documentation | +| **Vale** | Prose linting | + +## Documentation as Code + +```yaml +# .github/workflows/docs.yml +name: Build Docs +on: + push: + paths: + - 'docs/**' + +jobs: + build: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Build docs + run: npm run docs:build + - name: Deploy + uses: peaceiris/actions-gh-pages@v3 + with: + github_token: ${{ secrets.GITHUB_TOKEN }} + publish_dir: ./docs/build +``` + +## Measuring Documentation + +| Metric | How to Measure | +|--------|----------------| +| **Completeness** | Coverage of features | +| **Freshness** | Last updated date | +| **Findability** | Search analytics | +| **Usefulness** | User feedback, ratings | +| **Accuracy** | Bug reports, complaints | + +## Common Mistakes + +| Mistake | Better Approach | +|---------|-----------------| +| Assuming knowledge | Define terms, link to prerequisites | +| Wall of text | Use headings, lists, whitespace | +| Outdated examples | Test examples in CI | +| No examples | Every concept needs an example | +| Only happy path | Document errors and edge cases | +| Too much detail upfront | Progressive disclosure | + +## Related + +- [[Code Review]] — Review documentation too +- [[Computer Science MOC]] — CS topics diff --git a/Computer Science/Unix History.md b/Computer Science/Unix History.md new file mode 100644 index 0000000..39557d1 --- /dev/null +++ b/Computer Science/Unix History.md @@ -0,0 +1,358 @@ +--- +title: Unix History +aliases: + - History of Unix + - Unix Timeline + - Unix Family Tree +tags: + - cs + - history + - unix + - operating-systems +type: concept +status: complete +created: "2025-12-16" +--- + +# Unix History + +The evolution of Unix from Bell Labs research project to the foundation of modern computing. + +## Timeline + +``` +1969 ┌─────────────────────────────────────────────────────┐ + │ Unix born at Bell Labs (Ken Thompson, Dennis Ritchie) +1971 │ First Edition Unix +1973 │ Unix rewritten in C + │ +1975 │ Version 6 (V6) - First widely distributed + │ ↓ +1977 │ ┌────┴────┐ + │ ↓ ↓ + │ BSD System III +1979 │ │ │ + │ 3BSD System V +1983 │ │ │ + │ 4.2BSD SVR2 +1986 │ │ │ + │ 4.3BSD SVR3 → Solaris, HP-UX, AIX +1989 │ │ + │ 4.4BSD → FreeBSD, NetBSD, OpenBSD +1991 │ + │ Linux (Linus Torvalds) - Not Unix, Unix-like +1994 │ + │ Single Unix Specification (POSIX) +2000 │ + │ macOS (Darwin/BSD kernel) + └─────────────────────────────────────────────────────┘ +``` + +## Origins (1969-1975) + +### Bell Labs + +| Year | Event | +|------|-------| +| **1964** | Multics project begins (MIT, GE, Bell Labs) | +| **1969** | Bell Labs withdraws from Multics | +| **1969** | Ken Thompson writes Unix on PDP-7 | +| **1970** | Unix named (play on Multics) | +| **1971** | First Edition, PDP-11 | +| **1972** | Pipes introduced | +| **1973** | Rewritten in C (Dennis Ritchie) | + +### Key Innovations + +| Innovation | Impact | +|------------|--------| +| **Written in C** | Portable across hardware | +| **Hierarchical filesystem** | /usr/bin/program | +| **Pipes** | cmd1 \| cmd2 \| cmd3 | +| **Everything is a file** | Unified I/O model | +| **Shell** | Programmable command interpreter | +| **Small tools** | Do one thing well | + +### The Unix Philosophy + +1. Write programs that do one thing well +2. Write programs to work together +3. Write programs to handle text streams (universal interface) +4. Small is beautiful +5. Build a prototype as soon as possible +6. Choose portability over efficiency +7. Store data in flat text files +8. Use software leverage to your advantage +9. Avoid captive user interfaces +10. Make every program a filter + +## The Great Schism (1977-1994) + +### BSD Line (Berkeley) + +| Version | Year | Key Features | +|---------|------|--------------| +| **1BSD** | 1977 | Ex editor, Pascal compiler | +| **2BSD** | 1978 | Vi editor, C shell | +| **3BSD** | 1979 | Virtual memory | +| **4.2BSD** | 1983 | TCP/IP networking, sockets | +| **4.3BSD** | 1986 | Performance improvements | +| **4.4BSD** | 1993 | Final Berkeley release | + +**BSD Legacy:** + +- TCP/IP implementation (the internet runs on it) +- Sockets API +- Vi editor +- C shell (csh) +- Virtual memory + +### System V Line (AT&T) + +| Version | Year | Key Features | +|---------|------|--------------| +| **System III** | 1981 | First commercial Unix | +| **System V** | 1983 | Init system, curses | +| **SVR2** | 1984 | Shell functions | +| **SVR3** | 1986 | STREAMS, shared libraries | +| **SVR4** | 1989 | Merged BSD features | + +**System V Legacy:** + +- Init scripts (/etc/init.d) +- System V IPC (shared memory, semaphores) +- STREAMS I/O framework +- terminfo/curses + +### The Unix Wars + +``` + AT&T (System V) Berkeley (BSD) + ↓ ↓ + ┌─────────┼─────────┐ ┌───────┼───────┐ + ↓ ↓ ↓ ↓ ↓ ↓ + HP-UX Solaris AIX SunOS FreeBSD macOS + (HP) (Sun/Oracle) (IBM) ↓ + NetBSD + ↓ + OpenBSD +``` + +## BSD Descendants + +### FreeBSD (1993) + +**Focus:** Performance, features, servers. + +- PlayStation 4/5 OS based on FreeBSD +- Netflix CDN runs on FreeBSD +- WhatsApp infrastructure +- ZFS, jails, bhyve + +### NetBSD (1993) + +**Focus:** Portability. + +- "Of course it runs NetBSD" +- Supports 50+ hardware platforms +- Clean, portable codebase + +### OpenBSD (1995) + +**Focus:** Security, correctness. + +- Created by Theo de Raadt (forked from NetBSD) +- "Only two remote holes in the default install" +- OpenSSH, LibreSSL, PF firewall +- Proactive security auditing + +### Darwin/macOS (2000) + +**Foundation of Apple's OS.** + +- NeXTSTEP (Mach microkernel + BSD) +- Acquired by Apple (1997) +- macOS, iOS, iPadOS, watchOS, tvOS + +## System V Descendants + +### Solaris (Sun Microsystems, 1992) + +- SunOS → Solaris transition +- ZFS, DTrace, Zones, SMF +- Oracle Solaris (2010+) +- OpenSolaris → illumos (open source) + +### HP-UX (Hewlett-Packard, 1984) + +- PA-RISC, then Itanium +- Enterprise Unix for HP hardware + +### AIX (IBM, 1986) + +- IBM POWER systems +- Enterprise features, LPARs +- Still actively developed + +## Linux: The Unix-Like (1991) + +### Not Unix, But + +**Linux is Unix-like, not Unix.** + +- Written from scratch by Linus Torvalds +- POSIX-compatible (mostly) +- GNU userland (GNU's Not Unix) +- Free and open source (GPL) + +### Linux Timeline + +| Year | Event | +|------|-------| +| **1991** | Linux 0.01 announced | +| **1992** | Linux goes GPL | +| **1993** | Slackware, Debian | +| **1994** | Linux 1.0, Red Hat | +| **1996** | Linux 2.0 (SMP) | +| **2003** | Linux 2.6 | +| **2011** | Linux 3.0 | +| **2015** | Linux 4.0 | +| **2019** | Linux 5.0 | +| **2022** | Linux 6.0 | + +### Linux Distributions + +``` + Debian (1993) + ↓ + ┌───────────┼───────────┐ + ↓ ↓ ↓ + Ubuntu Linux Mint Raspbian + (2004) + + Red Hat (1994) + ↓ + ┌───────────┼───────────┐ + ↓ ↓ ↓ + RHEL Fedora CentOS→Rocky + + Slackware (1993) + ↓ + Arch (2002) + ↓ + Manjaro +``` + +## Standards and POSIX + +### POSIX (1988+) + +**Portable Operating System Interface.** + +- IEEE standard for Unix compatibility +- Defines APIs, shell, utilities +- Allows code portability + +### Single Unix Specification (SUS) + +**Official "Unix" certification.** + +- The Open Group owns "UNIX" trademark +- Only certified systems can use the name +- macOS, Solaris, AIX are certified Unix +- Linux is Unix-like, not certified + +### Certified UNIX Systems + +| System | Version | +|--------|---------| +| macOS | Current | +| AIX | 7.x | +| HP-UX | 11i | +| Solaris | 11 | +| z/OS Unix System Services | Current | + +## Unix Family Tree + +``` +Multics (1964) + │ + ↓ +UNICS/Unix (1969) ─────────────────────────────────────┐ + │ │ + ├─→ BSD (1977) ──────────────────────┐ │ + │ │ │ │ + │ ├─→ SunOS │ │ + │ │ │ │ + │ ├─→ FreeBSD (1993) │ │ + │ │ │ │ + │ ├─→ NetBSD (1993) │ │ + │ │ │ │ │ + │ │ └─→ OpenBSD (1995) │ │ + │ │ │ │ + │ └─→ NeXTSTEP → Darwin/macOS │ │ + │ │ │ + └─→ System V (1983) ─────────────────┤ │ + │ │ │ + ├─→ Solaris │ │ + │ │ │ │ + │ └─→ illumos │ │ + │ │ │ + ├─→ HP-UX │ │ + │ │ │ + └─→ AIX │ │ + │ │ + Linux (1991) ─────────────────────────┘ │ + │ (Unix-like, not derived) │ + │ │ + ├─→ Debian → Ubuntu, Mint │ + ├─→ Red Hat → RHEL, Fedora │ + ├─→ Slackware → Arch │ + └─→ Android (Linux kernel) │ + │ + Minix (1987) ───────────────────────────────────────┘ + (Inspired Linux, now microkernel research) +``` + +## Key People + +| Person | Contribution | +|--------|--------------| +| **Ken Thompson** | Original Unix, B language, Go | +| **Dennis Ritchie** | C language, Unix | +| **Brian Kernighan** | "K&R C", Unix documentation | +| **Bill Joy** | BSD, vi, csh, Sun co-founder | +| **Richard Stallman** | GNU project, FSF, GPL | +| **Linus Torvalds** | Linux kernel | +| **Andrew Tanenbaum** | MINIX, inspired Linux | +| **Theo de Raadt** | OpenBSD, OpenSSH | + +## Legacy and Influence + +### Concepts From Unix + +| Concept | Modern Manifestation | +|---------|---------------------| +| Pipes | Unix shells, PowerShell | +| Everything is a file | /proc, /sys, Plan 9 | +| Plain text config | YAML, JSON, TOML | +| Small tools | Unix philosophy, microservices | +| Scripting | Shell, Python, automation | +| Hierarchical filesystem | All modern OS | + +### Running Unix Today + +| Want... | Use... | +|---------|--------| +| BSD experience | FreeBSD, OpenBSD | +| macOS | Certified Unix with GUI | +| Unix-like (most common) | Linux (Ubuntu, RHEL) | +| Enterprise Unix | AIX, Solaris | +| Learning | FreeBSD, OpenBSD | + +## Related + +- [[Linux Distributions]] — Linux flavors +- [[Shells]] — Unix command interpreters +- [[C]] — Language Unix was written in +- [[Computer Science MOC]] — All CS topics diff --git a/Computer Science/Windows Development Basics.md b/Computer Science/Windows Development Basics.md new file mode 100644 index 0000000..57855b7 --- /dev/null +++ b/Computer Science/Windows Development Basics.md @@ -0,0 +1,392 @@ +--- +title: Windows Development Basics +aliases: + - Windows Dev + - Windows Paths + - AppData +tags: + - cs + - windows + - configuration +type: reference +status: complete +created: "2025-12-18" +--- + +# Windows Development Basics + +Essential Windows concepts for developers from other platforms. + +## Overview + +| Aspect | Details | +|--------|---------| +| **Config Location** | `%APPDATA%`, `%LOCALAPPDATA%` | +| **System Config** | Registry | +| **Package Manager** | winget, Chocolatey, Scoop | +| **Shell** | PowerShell, CMD, WSL | + +## Important Directories + +### Environment Variables + +| Variable | Typical Path | Purpose | +|----------|--------------|---------| +| `%USERPROFILE%` | `C:\Users\username` | Home directory | +| `%APPDATA%` | `C:\Users\username\AppData\Roaming` | Roaming app data | +| `%LOCALAPPDATA%` | `C:\Users\username\AppData\Local` | Local app data | +| `%PROGRAMDATA%` | `C:\ProgramData` | System-wide app data | +| `%PROGRAMFILES%` | `C:\Program Files` | 64-bit programs | +| `%PROGRAMFILES(X86)%` | `C:\Program Files (x86)` | 32-bit programs | +| `%TEMP%` / `%TMP%` | `C:\Users\username\AppData\Local\Temp` | Temporary files | +| `%SYSTEMROOT%` | `C:\Windows` | Windows directory | + +### AppData Structure + +``` +C:\Users\username\AppData\ +├── Local\ # Machine-specific data +│ ├── Programs\ # User-installed programs +│ ├── Microsoft\ +│ │ └── WindowsApps\ # Store apps +│ ├── Temp\ # Temp files +│ ├── npm-cache\ # npm cache +│ └── myapp\ # Your app (cache, large data) +│ +├── LocalLow\ # Low-integrity apps (sandboxed) +│ └── (rarely used by devs) +│ +└── Roaming\ # Synced across domain machines + ├── npm\ # npm global packages + ├── Code\ # VS Code settings + ├── .config\ # Some apps use XDG-style + └── myapp\ # Your app config +``` + +### Local vs Roaming + +| Location | Use For | Synced? | +|----------|---------|---------| +| **Roaming** | Settings, small config | Yes (domain) | +| **Local** | Cache, large files, machine-specific | No | +| **LocalLow** | Sandboxed/low-trust apps | No | + +### Mapping to XDG + +| XDG (Linux) | Windows Equivalent | +|-------------|-------------------| +| `XDG_CONFIG_HOME` | `%APPDATA%` | +| `XDG_DATA_HOME` | `%APPDATA%` | +| `XDG_CACHE_HOME` | `%LOCALAPPDATA%` | +| `XDG_STATE_HOME` | `%LOCALAPPDATA%` | +| `XDG_RUNTIME_DIR` | `%TEMP%` | + +## Registry + +### What Is It? + +Hierarchical database storing system and application settings. + +``` +HKEY_CURRENT_USER (HKCU) # Current user settings +├── Software\ +│ ├── Microsoft\ +│ └── MyCompany\ +│ └── MyApp\ +│ ├── Settings +│ └── WindowPosition +│ +HKEY_LOCAL_MACHINE (HKLM) # System-wide settings +├── SOFTWARE\ +│ ├── Microsoft\ +│ └── MyCompany\ +│ +HKEY_CLASSES_ROOT (HKCR) # File associations +HKEY_USERS (HKU) # All user profiles +HKEY_CURRENT_CONFIG (HKCC) # Hardware config +``` + +### Registry vs Files + +| Use Registry For | Use Files For | +|------------------|---------------| +| Simple settings | Complex config | +| Windows integration | Portability | +| File associations | Cross-platform | +| Admin-controlled settings | User-editable config | + +### Registry Operations + +```powershell +# PowerShell - Read +Get-ItemProperty -Path "HKCU:\Software\MyApp" -Name "Setting" + +# PowerShell - Write +Set-ItemProperty -Path "HKCU:\Software\MyApp" -Name "Setting" -Value "NewValue" + +# PowerShell - Create key +New-Item -Path "HKCU:\Software\MyApp" -Force + +# PowerShell - Delete +Remove-ItemProperty -Path "HKCU:\Software\MyApp" -Name "Setting" +Remove-Item -Path "HKCU:\Software\MyApp" -Recurse +``` + +```csharp +// C# - Microsoft.Win32 +using Microsoft.Win32; + +// Read +using var key = Registry.CurrentUser.OpenSubKey(@"Software\MyApp"); +var value = key?.GetValue("Setting") as string; + +// Write +using var key = Registry.CurrentUser.CreateSubKey(@"Software\MyApp"); +key.SetValue("Setting", "NewValue"); +``` + +```python +# Python - winreg +import winreg + +# Read +key = winreg.OpenKey(winreg.HKEY_CURRENT_USER, r"Software\MyApp") +value, _ = winreg.QueryValueEx(key, "Setting") +winreg.CloseKey(key) + +# Write +key = winreg.CreateKey(winreg.HKEY_CURRENT_USER, r"Software\MyApp") +winreg.SetValueEx(key, "Setting", 0, winreg.REG_SZ, "NewValue") +winreg.CloseKey(key) +``` + +## Path Handling + +### Path Separator + +```python +# Wrong - hardcoded +path = "C:\\Users\\name\\file.txt" # Windows only +path = "/home/name/file.txt" # Unix only + +# Right - use os.path or pathlib +from pathlib import Path +path = Path.home() / "Documents" / "file.txt" # Works everywhere +``` + +### Drive Letters + +```python +# Windows paths have drive letters +from pathlib import Path + +p = Path("C:/Users/name/file.txt") +print(p.drive) # 'C:' +print(p.root) # '\\' +print(p.anchor) # 'C:\\' + +# UNC paths (network shares) +p = Path("//server/share/folder") +print(p.drive) # '\\\\server\\share' +``` + +### Common Gotchas + +```python +# Case insensitive (usually) +Path("C:/Users") == Path("c:/users") # True on Windows + +# But not always! WSL, some tools are case-sensitive + +# Max path length (historically 260 chars) +# Modern Windows supports long paths if enabled + +# Reserved names - can't create files named: +# CON, PRN, AUX, NUL, COM1-9, LPT1-9 +``` + +## Services + +### Windows Services + +Long-running background processes (like systemd units on Linux). + +```powershell +# List services +Get-Service + +# Start/stop +Start-Service -Name "MyService" +Stop-Service -Name "MyService" +Restart-Service -Name "MyService" + +# Status +Get-Service -Name "MyService" | Select-Object Status, StartType + +# Create service (from elevated prompt) +New-Service -Name "MyService" -BinaryPathName "C:\path\to\app.exe" +``` + +### Task Scheduler + +For scheduled/periodic tasks (like cron on Linux). + +```powershell +# List tasks +Get-ScheduledTask + +# Create scheduled task +$action = New-ScheduledTaskAction -Execute "C:\path\to\script.ps1" +$trigger = New-ScheduledTaskTrigger -Daily -At "3:00AM" +Register-ScheduledTask -TaskName "MyTask" -Action $action -Trigger $trigger +``` + +## Environment Variables + +### Setting Variables + +```powershell +# Session only +$env:MY_VAR = "value" + +# User permanent (survives reboot) +[Environment]::SetEnvironmentVariable("MY_VAR", "value", "User") + +# System permanent (requires admin) +[Environment]::SetEnvironmentVariable("MY_VAR", "value", "Machine") + +# Add to PATH (user) +$path = [Environment]::GetEnvironmentVariable("PATH", "User") +[Environment]::SetEnvironmentVariable("PATH", "$path;C:\new\path", "User") +``` + +### Common Variables + +| Variable | Example | +|----------|---------| +| `PATH` | Executable search path | +| `PATHEXT` | `.COM;.EXE;.BAT;.CMD;.PS1` | +| `COMPUTERNAME` | Machine name | +| `USERNAME` | Current user | +| `USERDOMAIN` | Domain name | + +## Package Managers + +### winget (Built-in) + +```powershell +# Search +winget search vscode + +# Install +winget install Microsoft.VisualStudioCode + +# Upgrade all +winget upgrade --all + +# List installed +winget list +``` + +### Chocolatey + +```powershell +# Install Chocolatey (admin PowerShell) +Set-ExecutionPolicy Bypass -Scope Process -Force +iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1')) + +# Install packages +choco install git nodejs python -y + +# Upgrade +choco upgrade all -y +``` + +### Scoop (User-level) + +```powershell +# Install Scoop (no admin needed) +irm get.scoop.sh | iex + +# Install packages +scoop install git nodejs python + +# Apps install to ~/scoop/apps/ +``` + +## PowerShell Basics + +### Essential Commands + +| PowerShell | Bash Equivalent | +|------------|-----------------| +| `Get-ChildItem` / `ls` | `ls` | +| `Set-Location` / `cd` | `cd` | +| `Get-Content` / `cat` | `cat` | +| `Copy-Item` / `cp` | `cp` | +| `Move-Item` / `mv` | `mv` | +| `Remove-Item` / `rm` | `rm` | +| `Get-Process` | `ps` | +| `Stop-Process` | `kill` | +| `$env:VAR` | `$VAR` | + +### PowerShell Profile + +```powershell +# Location: $PROFILE +# Usually: C:\Users\name\Documents\PowerShell\Microsoft.PowerShell_profile.ps1 + +# Edit profile +notepad $PROFILE + +# Example profile content +Set-Alias g git +Set-Alias k kubectl + +function dev { Set-Location C:\dev } + +# Oh-My-Posh prompt +oh-my-posh init pwsh | Invoke-Expression +``` + +## File Associations + +### Query Association + +```powershell +# What opens .txt files? +cmd /c assoc .txt +# .txt=txtfile + +cmd /c ftype txtfile +# txtfile=%SystemRoot%\system32\NOTEPAD.EXE %1 +``` + +### Register Application + +```powershell +# Register file type (requires admin) +New-Item -Path "HKCR:\.myext" -Value "MyApp.Document" -Force +New-Item -Path "HKCR:\MyApp.Document\shell\open\command" ` + -Value '"C:\path\to\myapp.exe" "%1"' -Force +``` + +## Common Locations + +| What | Location | +|------|----------| +| **User home** | `C:\Users\username` | +| **Desktop** | `%USERPROFILE%\Desktop` | +| **Documents** | `%USERPROFILE%\Documents` | +| **Downloads** | `%USERPROFILE%\Downloads` | +| **Start Menu** | `%APPDATA%\Microsoft\Windows\Start Menu` | +| **Startup** | `%APPDATA%\Microsoft\Windows\Start Menu\Programs\Startup` | +| **Fonts** | `C:\Windows\Fonts` or `%LOCALAPPDATA%\Microsoft\Windows\Fonts` | +| **Hosts file** | `C:\Windows\System32\drivers\etc\hosts` | + +## Related + +- [[WSL]] — Windows Subsystem for Linux +- [[XDG Base Directory]] — Linux equivalent +- [[macOS Development Basics]] — macOS equivalent +- [[Computer Science MOC]] — CS topics diff --git a/Computer Science/XDG Base Directory.md b/Computer Science/XDG Base Directory.md new file mode 100644 index 0000000..b59b071 --- /dev/null +++ b/Computer Science/XDG Base Directory.md @@ -0,0 +1,417 @@ +--- +title: XDG Base Directory +aliases: + - XDG + - XDG Specification + - freedesktop.org Directories +tags: + - cs + - linux + - standards + - configuration +type: reference +status: complete +created: "2025-12-18" +--- + +# XDG Base Directory + +Linux/Unix standard for where applications should store files. + +## Overview + +| Aspect | Details | +|--------|---------| +| **Full Name** | XDG Base Directory Specification | +| **Maintained By** | freedesktop.org | +| **Purpose** | Standardize config/data/cache locations | +| **Adopted By** | Most modern Linux applications | + +## Why XDG Matters + +### The Problem (Before XDG) + +``` +~/ (home directory) +├── .app1rc +├── .app2config +├── .app3/ +│ ├── config +│ ├── cache +│ └── data +├── .randomapp +├── .another-app-settings.json +└── ... (dozens of dotfiles/directories) +``` + +**Issues:** + +- Home directory cluttered with dotfiles +- No separation of config vs data vs cache +- Hard to back up just configs +- Hard to clear caches without losing data + +### The Solution (With XDG) + +``` +~/ +├── .config/ # XDG_CONFIG_HOME +│ ├── app1/ +│ ├── app2/ +│ └── app3/ +├── .local/ +│ ├── share/ # XDG_DATA_HOME +│ │ ├── app1/ +│ │ └── app2/ +│ └── state/ # XDG_STATE_HOME +│ └── app1/ +└── .cache/ # XDG_CACHE_HOME + ├── app1/ + └── app2/ +``` + +## Environment Variables + +### Core Directories + +| Variable | Default | Purpose | +|----------|---------|---------| +| `XDG_CONFIG_HOME` | `~/.config` | User config files | +| `XDG_DATA_HOME` | `~/.local/share` | User data files | +| `XDG_STATE_HOME` | `~/.local/state` | User state files | +| `XDG_CACHE_HOME` | `~/.cache` | Non-essential cached data | +| `XDG_RUNTIME_DIR` | `/run/user/$UID` | Runtime files (sockets, etc.) | + +### Search Paths + +| Variable | Default | Purpose | +|----------|---------|---------| +| `XDG_CONFIG_DIRS` | `/etc/xdg` | System-wide config search path | +| `XDG_DATA_DIRS` | `/usr/local/share:/usr/share` | System-wide data search path | + +## What Goes Where + +### Config (`XDG_CONFIG_HOME`) + +**User-editable settings that persist.** + +``` +~/.config/ +├── git/ +│ └── config # Git configuration +├── nvim/ +│ └── init.lua # Neovim config +├── alacritty/ +│ └── alacritty.toml # Terminal config +├── fish/ +│ └── config.fish # Shell config +└── myapp/ + └── settings.json # Your app's config +``` + +**Characteristics:** + +- User might edit manually +- Should be version-controlled/backed up +- Portable between machines +- Small files, text-based preferred + +### Data (`XDG_DATA_HOME`) + +**Application data that should persist but isn't config.** + +``` +~/.local/share/ +├── nvim/ +│ ├── shada/ # Shared data (history, marks) +│ └── site/ # Plugins +├── fish/ +│ └── fish_history # Command history +├── applications/ # .desktop files +├── fonts/ # User fonts +└── myapp/ + ├── database.sqlite # App database + └── templates/ # User templates +``` + +**Characteristics:** + +- Generated by app, not user-edited +- Should be backed up +- May be large +- Loss = loss of work/history + +### State (`XDG_STATE_HOME`) + +**State that persists but is less important than data.** + +``` +~/.local/state/ +├── nvim/ +│ └── undo/ # Undo history +├── bash/ +│ └── history # Bash history +└── myapp/ + ├── logs/ # Application logs + └── recent-files # Recently opened +``` + +**Characteristics:** + +- Persists between runs +- OK to lose (inconvenient, not catastrophic) +- Logs, history, recent files +- Lower backup priority than data + +### Cache (`XDG_CACHE_HOME`) + +**Expendable cached data.** + +``` +~/.cache/ +├── pip/ # Python package cache +├── npm/ # npm cache +├── thumbnails/ # Image thumbnails +├── fontconfig/ # Font cache +└── myapp/ + ├── http-cache/ # API response cache + └── compiled/ # Compiled assets +``` + +**Characteristics:** + +- Can be deleted anytime +- Regenerated automatically +- Improves performance +- Often large, don't back up + +### Runtime (`XDG_RUNTIME_DIR`) + +**Temporary files for current session only.** + +``` +/run/user/1000/ # $UID = 1000 +├── pulse/ # PulseAudio socket +├── dbus/ # D-Bus socket +├── gnupg/ # GPG agent +└── myapp.sock # Your app's socket +``` + +**Characteristics:** + +- Must exist, be owned by user +- Cleared on logout/reboot +- tmpfs (RAM-based) +- Sockets, locks, PIDs +- Mode 0700 (user-only) + +## Implementation + +### Reading XDG Paths + +```bash +# Bash - with fallbacks +config_dir="${XDG_CONFIG_HOME:-$HOME/.config}/myapp" +data_dir="${XDG_DATA_HOME:-$HOME/.local/share}/myapp" +cache_dir="${XDG_CACHE_HOME:-$HOME/.cache}/myapp" +state_dir="${XDG_STATE_HOME:-$HOME/.local/state}/myapp" +``` + +```python +# Python +import os +from pathlib import Path + +def get_config_dir(app_name: str) -> Path: + base = os.environ.get('XDG_CONFIG_HOME', Path.home() / '.config') + return Path(base) / app_name + +def get_data_dir(app_name: str) -> Path: + base = os.environ.get('XDG_DATA_HOME', Path.home() / '.local/share') + return Path(base) / app_name + +def get_cache_dir(app_name: str) -> Path: + base = os.environ.get('XDG_CACHE_HOME', Path.home() / '.cache') + return Path(base) / app_name + +# Usage +config = get_config_dir('myapp') +config.mkdir(parents=True, exist_ok=True) +``` + +```javascript +// Node.js +const os = require('os'); +const path = require('path'); + +function getConfigDir(appName) { + const base = process.env.XDG_CONFIG_HOME || + path.join(os.homedir(), '.config'); + return path.join(base, appName); +} + +function getDataDir(appName) { + const base = process.env.XDG_DATA_HOME || + path.join(os.homedir(), '.local', 'share'); + return path.join(base, appName); +} + +function getCacheDir(appName) { + const base = process.env.XDG_CACHE_HOME || + path.join(os.homedir(), '.cache'); + return path.join(base, appName); +} +``` + +```go +// Go +package main + +import ( + "os" + "path/filepath" +) + +func getConfigDir(appName string) string { + if dir := os.Getenv("XDG_CONFIG_HOME"); dir != "" { + return filepath.Join(dir, appName) + } + home, _ := os.UserHomeDir() + return filepath.Join(home, ".config", appName) +} + +func getDataDir(appName string) string { + if dir := os.Getenv("XDG_DATA_HOME"); dir != "" { + return filepath.Join(dir, appName) + } + home, _ := os.UserHomeDir() + return filepath.Join(home, ".local", "share", appName) +} +``` + +```rust +// Rust - use the `dirs` crate +use dirs; + +fn get_config_dir(app_name: &str) -> Option { + dirs::config_dir().map(|p| p.join(app_name)) +} + +fn get_data_dir(app_name: &str) -> Option { + dirs::data_dir().map(|p| p.join(app_name)) +} + +fn get_cache_dir(app_name: &str) -> Option { + dirs::cache_dir().map(|p| p.join(app_name)) +} +``` + +### Libraries + +| Language | Library | +|----------|---------| +| **Python** | `platformdirs`, `appdirs` | +| **Rust** | `dirs`, `directories` | +| **Go** | `os.UserConfigDir()`, `adrg/xdg` | +| **Node.js** | `env-paths`, `xdg-basedir` | +| **C/C++** | `glib` (g_get_user_config_dir) | + +```python +# Python with platformdirs (recommended) +from platformdirs import user_config_dir, user_data_dir, user_cache_dir + +config = user_config_dir("myapp") # Cross-platform! +data = user_data_dir("myapp") +cache = user_cache_dir("myapp") +``` + +## Search Path Resolution + +### Config File Lookup + +```python +# Search order: user config, then system configs +import os +from pathlib import Path + +def find_config(app_name: str, filename: str) -> Path | None: + # User config first + user_config = Path(os.environ.get('XDG_CONFIG_HOME', + Path.home() / '.config')) / app_name / filename + if user_config.exists(): + return user_config + + # Then system configs + system_dirs = os.environ.get('XDG_CONFIG_DIRS', '/etc/xdg').split(':') + for dir in system_dirs: + system_config = Path(dir) / app_name / filename + if system_config.exists(): + return system_config + + return None +``` + +## Common Applications + +| Application | Config | Data | Cache | +|-------------|--------|------|-------| +| **Git** | `~/.config/git/` | - | - | +| **Neovim** | `~/.config/nvim/` | `~/.local/share/nvim/` | `~/.cache/nvim/` | +| **Fish** | `~/.config/fish/` | `~/.local/share/fish/` | - | +| **VS Code** | `~/.config/Code/` | `~/.config/Code/` | `~/.cache/Code/` | +| **npm** | `~/.npmrc` | `~/.local/share/npm/` | `~/.cache/npm/` | +| **pip** | `~/.config/pip/` | - | `~/.cache/pip/` | + +## Migration Tips + +### Moving Legacy Dotfiles + +```bash +# Example: migrate .myapprc to XDG +old_config="$HOME/.myapprc" +new_config="${XDG_CONFIG_HOME:-$HOME/.config}/myapp/config" + +if [ -f "$old_config" ] && [ ! -f "$new_config" ]; then + mkdir -p "$(dirname "$new_config")" + mv "$old_config" "$new_config" + echo "Migrated config to $new_config" +fi +``` + +### Backward Compatibility + +```python +# Support both old and new locations +def get_config_path(app_name: str) -> Path: + # Try XDG first + xdg_config = Path(os.environ.get('XDG_CONFIG_HOME', + Path.home() / '.config')) / app_name / 'config' + if xdg_config.exists(): + return xdg_config + + # Fall back to legacy location + legacy = Path.home() / f'.{app_name}rc' + if legacy.exists(): + return legacy + + # Default to XDG for new installs + return xdg_config +``` + +## Best Practices + +| Practice | Rationale | +|----------|-----------| +| **Always use XDG** | Don't clutter home directory | +| **Respect env vars** | User may have custom locations | +| **Create dirs as needed** | Don't assume they exist | +| **Use libraries** | Cross-platform, handles edge cases | +| **Separate concerns** | Config vs data vs cache | +| **Document locations** | Tell users where files are | + +## Related + +- [[macOS Development Basics]] — macOS equivalent paths +- [[Windows Development Basics]] — Windows equivalent paths +- [[Linux Distributions]] — Linux overview +- [[Computer Science MOC]] — CS topics diff --git a/Computer Science/macOS Development Basics.md b/Computer Science/macOS Development Basics.md new file mode 100644 index 0000000..3f0078b --- /dev/null +++ b/Computer Science/macOS Development Basics.md @@ -0,0 +1,494 @@ +--- +title: macOS Development Basics +aliases: + - macOS Dev + - macOS Paths + - Library Folder +tags: + - cs + - macos + - configuration +type: reference +status: complete +created: "2025-12-18" +--- + +# macOS Development Basics + +Essential macOS concepts for developers from other platforms. + +## Overview + +| Aspect | Details | +|--------|---------| +| **Config Location** | `~/Library`, `~/.config` | +| **System Config** | `/Library`, `/System` | +| **Package Manager** | Homebrew, MacPorts | +| **Shell** | zsh (default), bash, fish | +| **Based On** | BSD Unix (Darwin) | + +## Important Directories + +### Library Folders + +``` +/ # Root +├── Library/ # System-wide (all users) +│ ├── Application Support/ # App data +│ ├── Preferences/ # System preferences +│ └── LaunchDaemons/ # System services +│ +├── System/Library/ # Apple only (SIP protected) +│ └── (don't touch) +│ +└── Users/username/ + └── Library/ # User-specific (hidden by default) + ├── Application Support/ # App data + ├── Preferences/ # .plist files + ├── Caches/ # App caches + ├── Logs/ # App logs + ├── LaunchAgents/ # User services + └── Saved Application State/ # App state +``` + +### Mapping to XDG + +| XDG (Linux) | macOS Equivalent | +|-------------|------------------| +| `XDG_CONFIG_HOME` | `~/Library/Application Support` or `~/.config` | +| `XDG_DATA_HOME` | `~/Library/Application Support` | +| `XDG_CACHE_HOME` | `~/Library/Caches` | +| `XDG_STATE_HOME` | `~/Library/Application Support` | +| `XDG_RUNTIME_DIR` | `$TMPDIR` (per-user temp) | + +### Common Paths + +| What | Location | +|------|----------| +| **Home** | `/Users/username` or `~` | +| **Applications** | `/Applications` or `~/Applications` | +| **Desktop** | `~/Desktop` | +| **Documents** | `~/Documents` | +| **Downloads** | `~/Downloads` | +| **Temp** | `$TMPDIR` (e.g., `/var/folders/...`) | +| **Homebrew (Intel)** | `/usr/local` | +| **Homebrew (Apple Silicon)** | `/opt/homebrew` | + +## The Library Folder + +### Accessing Library + +```bash +# Library is hidden by default in Finder +# Option 1: Go menu → hold Option → Library appears + +# Option 2: Terminal +open ~/Library + +# Option 3: Show permanently +chflags nohidden ~/Library +``` + +### Application Support + +**Where apps store data that isn't preferences.** + +``` +~/Library/Application Support/ +├── Code/ # VS Code +│ └── User/ +│ ├── settings.json +│ └── keybindings.json +├── Firefox/ # Firefox profiles +├── Slack/ # Slack data +└── MyApp/ # Your app + ├── data.db + └── plugins/ +``` + +### Preferences (plist files) + +**Property list files storing app settings.** + +``` +~/Library/Preferences/ +├── com.apple.Terminal.plist +├── com.microsoft.VSCode.plist +├── com.googlecode.iterm2.plist +└── com.mycompany.myapp.plist +``` + +### Working with plists + +```bash +# Read plist +defaults read com.apple.Terminal + +# Read specific key +defaults read com.apple.finder ShowPathbar + +# Write value +defaults write com.apple.finder ShowPathbar -bool true + +# Delete key +defaults delete com.apple.finder ShowPathbar + +# Convert plist to XML (readable) +plutil -convert xml1 ~/Library/Preferences/com.myapp.plist + +# Convert to JSON +plutil -convert json ~/Library/Preferences/com.myapp.plist -o - | jq +``` + +```swift +// Swift - UserDefaults +let defaults = UserDefaults.standard + +// Read +let value = defaults.string(forKey: "setting") + +// Write +defaults.set("value", forKey: "setting") +defaults.synchronize() +``` + +```python +# Python - plistlib +import plistlib +from pathlib import Path + +# Read +plist_path = Path.home() / "Library/Preferences/com.myapp.plist" +with open(plist_path, 'rb') as f: + data = plistlib.load(f) + +# Write +with open(plist_path, 'wb') as f: + plistlib.dump(data, f) +``` + +### Caches + +``` +~/Library/Caches/ +├── com.apple.Safari/ +├── Homebrew/ +├── pip/ +└── com.mycompany.myapp/ +``` + +```bash +# Clear specific app cache +rm -rf ~/Library/Caches/com.myapp + +# Clear all caches (use with caution) +rm -rf ~/Library/Caches/* +``` + +## Launch Services + +### LaunchAgents vs LaunchDaemons + +| Type | Location | Runs As | When | +|------|----------|---------|------| +| **User Agent** | `~/Library/LaunchAgents/` | User | User login | +| **Global Agent** | `/Library/LaunchAgents/` | User | Any user login | +| **Daemon** | `/Library/LaunchDaemons/` | root | System boot | + +### Creating a LaunchAgent + +```xml + + + + + + Label + com.mycompany.myapp + + ProgramArguments + + /usr/local/bin/myapp + --daemon + + + RunAtLoad + + + KeepAlive + + + StandardOutPath + /tmp/myapp.log + + StandardErrorPath + /tmp/myapp.error.log + + +``` + +### Managing Services + +```bash +# Load (start at login) +launchctl load ~/Library/LaunchAgents/com.myapp.plist + +# Unload (stop and disable) +launchctl unload ~/Library/LaunchAgents/com.myapp.plist + +# Start now +launchctl start com.myapp + +# Stop +launchctl stop com.myapp + +# List loaded +launchctl list | grep myapp + +# Modern syntax (macOS 10.10+) +launchctl bootstrap gui/$UID ~/Library/LaunchAgents/com.myapp.plist +launchctl bootout gui/$UID ~/Library/LaunchAgents/com.myapp.plist +``` + +### Homebrew Services + +```bash +# Easier than raw launchctl +brew services list +brew services start postgresql +brew services stop postgresql +brew services restart postgresql +``` + +## Homebrew + +### Installation + +```bash +/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" + +# Apple Silicon: Add to PATH +echo 'eval "$(/opt/homebrew/bin/brew shellenv)"' >> ~/.zprofile +``` + +### Basic Usage + +```bash +# Search +brew search node + +# Install +brew install node + +# Install GUI app (cask) +brew install --cask visual-studio-code + +# Upgrade +brew upgrade + +# Cleanup old versions +brew cleanup + +# List installed +brew list +brew list --cask + +# Info +brew info node +``` + +### Brewfile (like package.json for system) + +```ruby +# Brewfile +tap "homebrew/bundle" +tap "homebrew/cask-fonts" + +# CLI tools +brew "git" +brew "node" +brew "python" +brew "ripgrep" +brew "fzf" + +# Applications +cask "visual-studio-code" +cask "iterm2" +cask "docker" + +# Fonts +cask "font-fira-code-nerd-font" + +# App Store apps (requires mas) +mas "Xcode", id: 497799835 +``` + +```bash +# Install from Brewfile +brew bundle + +# Generate Brewfile from installed +brew bundle dump +``` + +## Security & Permissions + +### System Integrity Protection (SIP) + +```bash +# Check status +csrutil status + +# Can't modify: +# /System, /usr (except /usr/local), /bin, /sbin +# Signed Apple apps +``` + +### Gatekeeper + +```bash +# Allow app from unidentified developer +xattr -d com.apple.quarantine /Applications/MyApp.app + +# Or: System Preferences → Security → Open Anyway +``` + +### Privacy Permissions + +| Permission | Controls | +|------------|----------| +| **Full Disk Access** | Access all files | +| **Files and Folders** | Specific folder access | +| **Accessibility** | Control other apps | +| **Screen Recording** | Capture screen | +| **Automation** | AppleScript/control apps | +| **Developer Tools** | Debugging | + +```bash +# Open privacy settings +open "x-apple.systempreferences:com.apple.preference.security?Privacy" +``` + +## Environment Variables + +### Shell Config + +```bash +# zsh (default) +~/.zshrc # Interactive shells +~/.zprofile # Login shells + +# bash +~/.bashrc +~/.bash_profile + +# Both +~/.profile # Fallback +``` + +### Setting Variables + +```bash +# ~/.zprofile or ~/.zshrc +export MY_VAR="value" +export PATH="/opt/homebrew/bin:$PATH" + +# Or use launchctl for GUI apps +launchctl setenv MY_VAR "value" +``` + +### PATH on macOS + +```bash +# System PATH construction: +# 1. /etc/paths (base paths) +# 2. /etc/paths.d/* (additional paths) +# 3. Shell profile additions + +cat /etc/paths +# /usr/local/bin +# /usr/bin +# /bin +# /usr/sbin +# /sbin +``` + +## Common Developer Tools + +### Xcode Command Line Tools + +```bash +# Install (required for many dev tools) +xcode-select --install + +# Location +xcode-select -p +# /Library/Developer/CommandLineTools + +# Reset +sudo xcode-select --reset +``` + +### Developer Directory + +``` +~/Library/Developer/ +├── CoreSimulator/ # iOS Simulators +├── Xcode/ +│ └── DerivedData/ # Build artifacts (safe to delete) +└── CommandLineTools/ # If no full Xcode +``` + +## Keychain + +```bash +# Store secret +security add-generic-password -a "$USER" -s "myapp" -w "secret123" + +# Retrieve secret +security find-generic-password -a "$USER" -s "myapp" -w + +# Delete +security delete-generic-password -a "$USER" -s "myapp" +``` + +```python +# Python with keyring +import keyring + +keyring.set_password("myapp", "api_key", "secret123") +password = keyring.get_password("myapp", "api_key") +``` + +## App Bundles + +### Structure + +``` +MyApp.app/ +└── Contents/ + ├── Info.plist # App metadata + ├── MacOS/ + │ └── MyApp # Main executable + ├── Resources/ # Assets, localizations + │ ├── en.lproj/ + │ └── icon.icns + └── Frameworks/ # Bundled libraries +``` + +### Info.plist Keys + +| Key | Description | +|-----|-------------| +| `CFBundleIdentifier` | Reverse-DNS ID (com.mycompany.myapp) | +| `CFBundleName` | Display name | +| `CFBundleVersion` | Build number | +| `CFBundleShortVersionString` | Version string | +| `LSMinimumSystemVersion` | Minimum macOS version | + +## Related + +- [[XDG Base Directory]] — Linux equivalent +- [[Windows Development Basics]] — Windows equivalent +- [[Computer Science MOC]] — CS topics diff --git a/Hardware/CPU Architectures.md b/Hardware/CPU Architectures.md new file mode 100644 index 0000000..d74e21f --- /dev/null +++ b/Hardware/CPU Architectures.md @@ -0,0 +1,319 @@ +--- +title: CPU Architectures +aliases: + - Processor Architectures + - ISA + - Instruction Set Architecture +tags: + - hardware + - cpu + - architecture + - systems +type: reference +status: complete +created: "2025-12-16" +--- + +# CPU Architectures + +Instruction Set Architectures (ISAs) defining how software interfaces with processors, from desktop to embedded systems. + +## Overview + +| Architecture | Type | Primary Use Cases | +|--------------|------|-------------------| +| **x86-64** | CISC | Desktop, server, workstation | +| **ARM** | RISC | Mobile, embedded, servers, Apple Silicon | +| **RISC-V** | RISC | Embedded, emerging server/desktop | +| **z/Architecture** | CISC | IBM mainframes | +| **POWER** | RISC | IBM servers, HPC | +| **MIPS** | RISC | Embedded, networking (legacy) | +| **SPARC** | RISC | Oracle servers (legacy) | + +## x86-64 (AMD64) + +### Overview + +| Aspect | Details | +|--------|---------| +| **Origin** | AMD (2003), backward-compatible with Intel x86 | +| **Type** | CISC (Complex Instruction Set Computer) | +| **Register Size** | 64-bit | +| **Vendors** | Intel, AMD | +| **Market** | 95%+ of servers and desktops | + +### Key Features + +| Feature | Description | +|---------|-------------| +| **Backward Compatibility** | Runs 16-bit, 32-bit, and 64-bit code | +| **Virtual Memory** | 48-bit virtual (256 TB), 52-bit physical | +| **Registers** | 16 general-purpose (RAX-R15) | +| **Extensions** | SSE, AVX, AVX-512, AMX | + +### Intel vs AMD + +| Aspect | Intel | AMD | +|--------|-------|-----| +| **Current Desktop** | Core Ultra (Meteor Lake) | Ryzen 9000 (Zen 5) | +| **Current Server** | Xeon (Sapphire Rapids) | EPYC (Genoa/Bergamo) | +| **Manufacturing** | Intel 4/3 | TSMC 4/5nm | +| **Chiplet Design** | E-cores + P-cores | Multi-CCD architecture | +| **Memory** | DDR5, HBM | DDR5, HBM | + +### Extensions + +| Extension | Purpose | Operations | +|-----------|---------|------------| +| **SSE** | SIMD (128-bit) | Floating-point | +| **AVX2** | SIMD (256-bit) | FP, integer | +| **AVX-512** | SIMD (512-bit) | AI, HPC | +| **AMX** | Matrix operations | AI acceleration | +| **AES-NI** | Encryption | Hardware AES | + +## ARM + +### Overview + +| Aspect | Details | +|--------|---------| +| **Origin** | Acorn (1985), ARM Holdings (1990) | +| **Type** | RISC (Reduced Instruction Set Computer) | +| **Licensing** | Architecture license, core license | +| **Market** | 99% of mobile, growing in servers | + +### Architecture Versions + +| Version | Year | Key Features | +|---------|------|--------------| +| **ARMv7** | 2005 | 32-bit, Cortex-A series | +| **ARMv8** | 2011 | 64-bit (AArch64), Cortex-A53/A57 | +| **ARMv9** | 2021 | SVE2, security improvements | + +### Core Families + +| Family | Target | Examples | +|--------|--------|----------| +| **Cortex-A** | Application (phones, servers) | A78, X4, Neoverse | +| **Cortex-R** | Real-time | Automotive, storage | +| **Cortex-M** | Microcontroller | IoT, embedded | +| **Neoverse** | Infrastructure | N2, V2 (servers) | + +### Apple Silicon + +**ARM-based chips for Mac, iPad, iPhone.** + +| Chip | Year | Cores | Target | +|------|------|-------|--------| +| **M1** | 2020 | 8 (4P+4E) | MacBook Air, Mac mini | +| **M2** | 2022 | 8 (4P+4E) | MacBook Air/Pro | +| **M3** | 2023 | 8 (4P+4E) | MacBook Pro, iMac | +| **M4** | 2024 | Up to 10 | iPad Pro, Mac | +| **M1/M2/M3 Ultra** | — | Up to 24 | Mac Studio, Mac Pro | + +**Advantages:** + +- Unified memory architecture +- Excellent performance per watt +- Tight hardware/software integration +- Neural Engine for ML + +### ARM in Servers + +| Product | Vendor | Notes | +|---------|--------|-------| +| **Graviton4** | AWS | Neoverse V2-based | +| **Ampere Altra** | Ampere | Up to 128 cores | +| **Grace** | NVIDIA | CPU for AI workloads | +| **Cobalt 100** | Microsoft | Azure ARM server | + +### ARM vs x86 Trade-offs + +| Aspect | ARM | x86 | +|--------|-----|-----| +| **Power Efficiency** | ✅ Excellent | ⚠️ Improving | +| **Single-Thread** | ⚠️ Catching up | ✅ Historically stronger | +| **Software Ecosystem** | ⚠️ Growing | ✅ Mature | +| **Virtualization** | ✅ Mature (ARMv8) | ✅ Mature (VT-x) | +| **Price/Performance** | ✅ Often better | ⚠️ Varies | + +## RISC-V + +### Overview + +| Aspect | Details | +|--------|---------| +| **Origin** | UC Berkeley (2010) | +| **Type** | RISC, open-source ISA | +| **Licensing** | Free, no royalties | +| **Status** | Emerging in embedded, growing elsewhere | + +### Key Characteristics + +| Feature | Description | +|---------|-------------| +| **Open Standard** | Free to implement | +| **Modular** | Base ISA + optional extensions | +| **Customizable** | Custom instructions allowed | +| **No Royalties** | Unlike ARM | + +### Extensions + +| Extension | Description | +|-----------|-------------| +| **I** | Integer base (mandatory) | +| **M** | Integer multiply/divide | +| **A** | Atomic operations | +| **F** | Single-precision FP | +| **D** | Double-precision FP | +| **C** | Compressed instructions | +| **V** | Vector operations | + +**Common Profiles:** + +- **RV32IMAC** — 32-bit embedded +- **RV64GC** — 64-bit general-purpose (G = IMAFD) + +### Current Products + +| Product | Vendor | Target | +|---------|--------|--------| +| **SiFive U74** | SiFive | Linux-capable SoC | +| **ESP32-C3** | Espressif | IoT (replacing Xtensa) | +| **StarFive JH7110** | StarFive | Single-board computers | +| **Kendryte K210** | Canaan | AI edge | + +### RISC-V Adoption + +| Domain | Status | +|--------|--------| +| **Embedded/IoT** | ✅ Growing rapidly | +| **Desktop** | ⚠️ Very early (VisionFive) | +| **Server** | ⚠️ In development | +| **HPC** | ⚠️ European initiatives | + +## Other Architectures + +### z/Architecture (IBM Mainframe) + +| Aspect | Details | +|--------|---------| +| **Type** | CISC | +| **Register Size** | 64-bit | +| **Current** | z16 (Telum processor) | +| **Features** | On-chip AI, extreme reliability | + +See [[Mainframes]] for details. + +### IBM POWER + +| Aspect | Details | +|--------|---------| +| **Type** | RISC | +| **Current** | POWER10 | +| **Target** | Enterprise servers, HPC | +| **Features** | SMT8, high memory bandwidth | + +### MIPS (Legacy) + +| Aspect | Details | +|--------|---------| +| **Status** | Largely replaced by ARM/RISC-V | +| **Historical Use** | Networking, embedded, consoles | +| **Modern** | Open-sourced, limited new development | + +## Architecture Comparison + +### Instruction Set Philosophy + +| Aspect | CISC (x86) | RISC (ARM, RISC-V) | +|--------|------------|---------------------| +| **Instructions** | Many, complex | Fewer, simpler | +| **Instruction Length** | Variable | Fixed (mostly) | +| **Decode Complexity** | Higher | Lower | +| **Memory Access** | In most instructions | Load/store only | +| **Code Density** | Good | Requires more instructions | + +### Modern Reality + +The CISC/RISC distinction is blurred: + +- x86 CPUs internally decode to RISC-like micro-ops +- ARM has grown more complex over time +- Performance depends more on implementation than ISA + +### Performance Comparison + +| Workload | x86 | ARM | RISC-V | +|----------|-----|-----|--------| +| **Single-Thread** | ✅ Strong | ✅ Apple leads | ⚠️ Developing | +| **Multi-Thread** | ✅ Strong | ✅ High core counts | ⚠️ Developing | +| **Power Efficiency** | ⚠️ Improving | ✅ Excellent | ✅ Good | +| **Embedded** | ❌ Too power-hungry | ✅ Dominant | ✅ Growing | +| **Server** | ✅ Dominant | ✅ Growing (AWS, Azure) | ⚠️ Emerging | + +## Memory and Virtualization + +### Memory Models + +| Feature | x86-64 | ARM | RISC-V | +|---------|--------|-----|--------| +| **Ordering** | Strong (TSO) | Relaxed | Relaxed (RVWMO) | +| **Virtual Memory** | 48-bit+ | 48-bit (ARMv8.2: 52-bit) | Sv39/Sv48/Sv57 | +| **Page Sizes** | 4K, 2M, 1G | 4K, 16K, 64K, + huge | 4K, 2M, 1G | + +### Virtualization Support + +| Feature | x86 (Intel) | x86 (AMD) | ARM | RISC-V | +|---------|-------------|-----------|-----|--------| +| **Technology** | VT-x, VT-d | AMD-V, IOMMU | EL2/VHE | H extension | +| **Maturity** | ✅ Mature | ✅ Mature | ✅ Mature | ⚠️ Developing | +| **Nested** | ✅ Yes | ✅ Yes | ✅ Yes | ⚠️ Planned | + +## Development Considerations + +### Cross-Compilation + +| Host | Target | Toolchain | +|------|--------|-----------| +| x86 | ARM | `aarch64-linux-gnu-gcc` | +| x86 | RISC-V | `riscv64-unknown-linux-gnu-gcc` | +| ARM | x86 | `x86_64-linux-gnu-gcc` | + +### Emulation + +| Tool | Purpose | +|------|---------| +| **QEMU** | Full system/user-mode emulation | +| **Rosetta 2** | x86 on Apple Silicon | +| **Box64** | x86 on ARM Linux | +| **FEX-Emu** | x86 on ARM Linux | + +### Containerization + +| Platform | Multi-arch Support | +|----------|-------------------| +| **Docker** | `--platform linux/amd64,linux/arm64` | +| **Podman** | QEMU-based cross-arch | +| **Kubernetes** | Node selectors, taints | + +## When to Choose + +| Scenario | Recommendation | +|----------|----------------| +| **Desktop/Laptop** | x86-64 (most software) or Apple Silicon (Mac) | +| **Cloud Server** | x86-64 or ARM (Graviton for cost) | +| **Mobile** | ARM (no real alternative) | +| **Embedded (low power)** | ARM Cortex-M or RISC-V | +| **Embedded (Linux)** | ARM Cortex-A or RISC-V | +| **HPC** | x86-64 (AMD EPYC) or ARM (emerging) | +| **Mainframe** | z/Architecture (IBM only) | +| **Custom chip (no royalties)** | RISC-V | + +## Related + +- [[Mainframes]] — IBM z/Architecture details +- [[GPUs]] — Parallel compute accelerators +- [[Domains/Embedded & Systems]] — Low-level development +- [[Languages/C|C]] — Systems programming language +- [[Rust]] — Modern systems language diff --git a/Hardware/Firewalls.md b/Hardware/Firewalls.md new file mode 100644 index 0000000..ec3da4d --- /dev/null +++ b/Hardware/Firewalls.md @@ -0,0 +1,387 @@ +--- +title: Firewalls +aliases: + - Network Firewall + - Firewall Security +tags: + - hardware + - networking + - security +type: reference +status: complete +created: "2025-12-18" +--- + +# Firewalls + +Network security devices that monitor and control traffic based on security rules. + +## Overview + +| Aspect | Details | +|--------|---------| +| **Purpose** | Control network traffic flow | +| **Function** | Allow/deny based on rules | +| **Placement** | Network perimeter, segments | +| **Types** | Hardware, software, cloud | + +## Firewall Types + +### By Implementation + +| Type | Description | Use Case | +|------|-------------|----------| +| **Hardware Firewall** | Dedicated appliance | Enterprise perimeter | +| **Software Firewall** | OS-based (iptables, Windows Firewall) | Host protection | +| **Virtual Firewall** | VM-based appliance | Cloud, virtualization | +| **Cloud Firewall** | Provider-managed (AWS SG, Azure NSG) | Cloud workloads | + +### By Generation + +| Generation | Technology | Capabilities | +|------------|------------|--------------| +| **Packet Filter** | L3-L4 inspection | IP, port filtering | +| **Stateful** | Connection tracking | Session awareness | +| **Application (WAF)** | L7 inspection | HTTP, SQL injection | +| **NGFW** | Deep inspection | App awareness, IPS, SSL | + +## How Firewalls Work + +### Packet Filtering + +``` +┌─────────────────────────────────────────────────────┐ +│ Packet arrives │ +│ ┌─────────────────────────────────────────────────┐ │ +│ │ Source IP: 192.168.1.100 │ │ +│ │ Dest IP: 10.0.0.50 │ │ +│ │ Protocol: TCP │ │ +│ │ Source Port: 52431 │ │ +│ │ Dest Port: 443 │ │ +│ └─────────────────────────────────────────────────┘ │ +└─────────────────────────────────────────────────────┘ + │ + ▼ + ┌──────────────────────┐ + │ Check against rules │ + │ │ + │ Rule 1: ALLOW │ + │ TCP to port 443 │ + │ │ + │ Rule 2: DENY │ + │ TCP to port 23 │ + │ │ + │ Default: DENY ALL │ + └──────────────────────┘ + │ + ▼ + ┌────────────┐ + │ ALLOWED │ + └────────────┘ +``` + +### Stateful Inspection + +``` +┌───────────────────────────────────────────────────────┐ +│ State Table │ +├───────────────────────────────────────────────────────┤ +│ Src IP │ Dst IP │ Src Port │ Dst Port │ State │ +│ 192.168.1.100 │ 10.0.0.50 │ 52431 │ 443 │ ESTABLISHED│ +│ 192.168.1.101 │ 8.8.8.8 │ 43211 │ 53 │ NEW │ +└───────────────────────────────────────────────────────┘ + +• Tracks connection state (NEW, ESTABLISHED, RELATED) +• Return traffic automatically allowed for established connections +• More secure than stateless packet filtering +``` + +## Rule Structure + +### Basic Components + +| Component | Description | Example | +|-----------|-------------|---------| +| **Action** | Allow or deny | ACCEPT, DROP, REJECT | +| **Source** | Origin IP/network | 192.168.1.0/24 | +| **Destination** | Target IP/network | 10.0.0.0/8 | +| **Protocol** | L4 protocol | TCP, UDP, ICMP | +| **Port** | Service port | 80, 443, 22 | +| **Direction** | Inbound/outbound | IN, OUT | + +### Rule Processing + +``` +┌─────────────────────────────────────┐ +│ Rules processed TOP to BOTTOM │ +├─────────────────────────────────────┤ +│ Rule 1: Allow HTTPS (443) │ ← First match wins +│ Rule 2: Allow SSH from admin │ +│ Rule 3: Allow DNS │ +│ Rule 4: Deny all from blacklist │ +│ ... │ +│ Default: DENY ALL │ ← Implicit deny +└─────────────────────────────────────┘ +``` + +## Linux iptables + +### Basic Commands + +```bash +# View rules +iptables -L -v -n +iptables -L INPUT -v -n --line-numbers + +# Allow incoming SSH +iptables -A INPUT -p tcp --dport 22 -j ACCEPT + +# Allow established connections +iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT + +# Allow from specific IP +iptables -A INPUT -s 192.168.1.100 -j ACCEPT + +# Block IP +iptables -A INPUT -s 10.0.0.50 -j DROP + +# Allow HTTP/HTTPS +iptables -A INPUT -p tcp --dport 80 -j ACCEPT +iptables -A INPUT -p tcp --dport 443 -j ACCEPT + +# Default deny +iptables -P INPUT DROP +iptables -P FORWARD DROP +iptables -P OUTPUT ACCEPT + +# Save rules (Debian/Ubuntu) +iptables-save > /etc/iptables/rules.v4 + +# Delete rule by number +iptables -D INPUT 3 +``` + +### iptables Chains + +| Chain | Purpose | +|-------|---------| +| **INPUT** | Packets destined for local system | +| **OUTPUT** | Packets originating from local system | +| **FORWARD** | Packets routed through system | +| **PREROUTING** | Before routing decision (NAT) | +| **POSTROUTING** | After routing decision (NAT) | + +## nftables (Modern Linux) + +```bash +# Basic nftables rules +nft add table inet filter +nft add chain inet filter input { type filter hook input priority 0 \; policy drop \; } + +# Allow SSH +nft add rule inet filter input tcp dport 22 accept + +# Allow established +nft add rule inet filter input ct state established,related accept + +# View rules +nft list ruleset +``` + +## Windows Firewall + +### PowerShell Commands + +```powershell +# View rules +Get-NetFirewallRule | Format-Table Name, Enabled, Direction, Action + +# Create inbound rule +New-NetFirewallRule -DisplayName "Allow SSH" ` + -Direction Inbound -Protocol TCP -LocalPort 22 -Action Allow + +# Block IP +New-NetFirewallRule -DisplayName "Block Bad IP" ` + -Direction Inbound -RemoteAddress 10.0.0.50 -Action Block + +# Enable/disable rule +Enable-NetFirewallRule -DisplayName "Allow SSH" +Disable-NetFirewallRule -DisplayName "Allow SSH" + +# Remove rule +Remove-NetFirewallRule -DisplayName "Allow SSH" +``` + +## Cloud Security Groups + +### AWS Security Groups + +```hcl +# Terraform example +resource "aws_security_group" "web" { + name = "web-sg" + description = "Web server security group" + vpc_id = aws_vpc.main.id + + # Inbound rules + ingress { + from_port = 443 + to_port = 443 + protocol = "tcp" + cidr_blocks = ["0.0.0.0/0"] + } + + ingress { + from_port = 22 + to_port = 22 + protocol = "tcp" + cidr_blocks = ["10.0.0.0/8"] # Internal only + } + + # Outbound rules + egress { + from_port = 0 + to_port = 0 + protocol = "-1" + cidr_blocks = ["0.0.0.0/0"] + } +} +``` + +### Key Differences + +| Feature | Security Group | NACL | +|---------|----------------|------| +| **Level** | Instance | Subnet | +| **State** | Stateful | Stateless | +| **Rules** | Allow only | Allow and deny | +| **Evaluation** | All rules | Numbered order | + +## Next-Generation Firewall (NGFW) + +### Features + +| Feature | Description | +|---------|-------------| +| **Application Awareness** | Identify apps regardless of port | +| **User Identity** | Rules based on user, not just IP | +| **IPS/IDS** | Intrusion detection/prevention | +| **SSL Inspection** | Decrypt and inspect HTTPS | +| **URL Filtering** | Block categories of websites | +| **Malware Protection** | Sandboxing, threat intelligence | +| **Cloud Integration** | Sync with cloud workloads | + +### Example: Palo Alto + +``` +# Application-based rule +rule "Allow-Web-Apps" { + source-zone: trust + destination-zone: untrust + application: [ ssl, web-browsing, google-base ] + user: [ domain\employees ] + action: allow + profile: { + url-filtering: corporate-filter + threat-prevention: strict + } +} +``` + +## Firewall Architecture + +### DMZ Design + +``` + Internet + │ + ┌───────▼───────┐ + │ External FW │ + └───────┬───────┘ + │ + ┌─────────────┼─────────────┐ + │ │ │ + ┌────▼────┐ ┌─────▼─────┐ ┌────▼────┐ + │ Web │ │ Mail │ │ DNS │ + │ Server │ │ Server │ │ Server │ + └─────────┘ └───────────┘ └─────────┘ + DMZ Network + │ + ┌───────▼───────┐ + │ Internal FW │ + └───────┬───────┘ + │ + ┌─────────────┼─────────────┐ + │ │ │ + ┌────▼────┐ ┌─────▼─────┐ ┌────▼────┐ + │Database │ │App │ │Internal │ + │Servers │ │Servers │ │Users │ + └─────────┘ └───────────┘ └─────────┘ + Internal Network +``` + +### Zero Trust Model + +``` +┌─────────────────────────────────────────────────────┐ +│ Traditional: Trust internal, verify external │ +│ Zero Trust: Never trust, always verify │ +└─────────────────────────────────────────────────────┘ + +• Micro-segmentation +• Identity-based access +• Least privilege +• Continuous verification +• Assume breach +``` + +## Common Vendors + +| Vendor | Products | Strengths | +|--------|----------|-----------| +| **Palo Alto** | PA Series | App-ID, threat prevention | +| **Fortinet** | FortiGate | Price/performance, UTM | +| **Cisco** | Firepower, ASA | Integration, enterprise | +| **Check Point** | Quantum | Management, security | +| **Sophos** | XG Firewall | Ease of use, SMB | +| **pfSense** | pfSense | Open source, flexible | + +## Best Practices + +| Practice | Description | +|----------|-------------| +| **Default deny** | Block all, allow specific | +| **Least privilege** | Minimum necessary access | +| **Document rules** | Comments, change tracking | +| **Regular audits** | Review and clean up rules | +| **Log everything** | Enable logging for analysis | +| **Segment networks** | Isolate sensitive systems | +| **Update signatures** | Keep IPS/threat DB current | +| **Test changes** | Validate before production | + +## Troubleshooting + +```bash +# Linux - check if traffic is being dropped +iptables -L -v -n | grep DROP + +# Watch live traffic +tcpdump -i eth0 host 192.168.1.100 + +# Check connection tracking +conntrack -L + +# Windows - check firewall logs +Get-WinEvent -LogName "Microsoft-Windows-Windows Firewall With Advanced Security/Firewall" + +# Test connectivity +nc -zv hostname 443 +telnet hostname 443 +``` + +## Related + +- [[Routers and Switches]] — Network infrastructure +- [[Network Topologies]] — Network design +- [[Networking Fundamentals]] — Protocols and layers +- [[Security MOC]] — Security topics diff --git a/Hardware/GPUs.md b/Hardware/GPUs.md new file mode 100644 index 0000000..84fa5b5 --- /dev/null +++ b/Hardware/GPUs.md @@ -0,0 +1,344 @@ +--- +title: GPUs +aliases: + - Graphics Processing Unit + - Graphics Cards + - Accelerators + - GPU Computing +tags: + - hardware + - gpu + - compute + - ml + - gaming +type: reference +status: complete +created: "2025-12-16" +--- + +# GPUs + +Massively parallel processors originally designed for graphics, now essential for machine learning, scientific computing, and high-performance workloads. + +## Overview + +| Aspect | Details | +|--------|---------| +| **Primary Vendors** | NVIDIA, AMD, Intel | +| **Core Architecture** | Thousands of small cores (SIMT) | +| **Primary Use Cases** | Graphics, ML/AI, HPC, crypto | +| **Memory Type** | GDDR6, HBM2e, HBM3 | +| **Key Advantage** | Parallel computation | + +## Architecture + +### GPU vs CPU + +| Aspect | CPU | GPU | +|--------|-----|-----| +| **Cores** | Few (4-64) powerful cores | Thousands of simple cores | +| **Optimization** | Low latency | High throughput | +| **Task Type** | Complex, sequential | Simple, parallel | +| **Cache** | Large | Smaller | +| **Control** | Complex branch prediction | Simpler control flow | + +### Execution Model + +**SIMT (Single Instruction, Multiple Threads):** + +``` +┌────────────────────────────────────────┐ +│ Streaming Multiprocessor │ +│ ┌────┐ ┌────┐ ┌────┐ ┌────┐ ┌────┐ │ +│ │Core│ │Core│ │Core│ │Core│ │Core│ │ +│ └────┘ └────┘ └────┘ └────┘ └────┘ │ +│ ┌────┐ ┌────┐ ┌────┐ ┌────┐ ┌────┐ │ +│ │Core│ │Core│ │Core│ │Core│ │Core│ │ +│ └────┘ └────┘ └────┘ └────┘ └────┘ │ +│ ... (32-128 cores) │ +│ ┌─────────────────────────────────┐ │ +│ │ Shared Memory/L1 │ │ +│ └─────────────────────────────────┘ │ +└────────────────────────────────────────┘ + × 80-144 SMs per GPU +``` + +### Memory Hierarchy + +| Level | Size | Latency | +|-------|------|---------| +| **Registers** | Per-thread | 1 cycle | +| **Shared Memory** | Per SM (64-228 KB) | ~20 cycles | +| **L2 Cache** | Shared (6-96 MB) | ~200 cycles | +| **Global Memory (VRAM)** | 8-80 GB | ~400 cycles | + +## NVIDIA Ecosystem + +### Current Architectures + +| Architecture | Year | Key Features | +|--------------|------|--------------| +| **Hopper** | 2022 | H100, Transformer Engine, HBM3 | +| **Ada Lovelace** | 2022 | RTX 40 series, DLSS 3 | +| **Ampere** | 2020 | A100, 3rd gen Tensor Cores | +| **Blackwell** | 2024 | B100/B200, next-gen AI | + +### Product Lines + +| Line | Target | Examples | +|------|--------|----------| +| **GeForce** | Gaming/Consumer | RTX 4090, 4080, 4070 | +| **RTX Professional** | Workstation | RTX 6000, RTX 5000 | +| **Data Center** | AI/HPC | H100, A100, H200 | +| **Jetson** | Edge AI | Orin, Xavier | + +### Key Technologies + +| Technology | Purpose | +|------------|---------| +| **CUDA Cores** | General-purpose compute | +| **Tensor Cores** | Matrix operations (AI) | +| **RT Cores** | Ray tracing acceleration | +| **NVLink** | GPU-to-GPU interconnect | +| **NVSwitch** | Multi-GPU fabric | + +### CUDA Programming + +**Dominant GPU computing platform.** + +```cuda +__global__ void vectorAdd(float *a, float *b, float *c, int n) { + int i = blockIdx.x * blockDim.x + threadIdx.x; + if (i < n) { + c[i] = a[i] + b[i]; + } +} + +// Launch kernel +vectorAdd<<>>(a, b, c, n); +``` + +**CUDA Concepts:** + +| Concept | Description | +|---------|-------------| +| **Kernel** | Function running on GPU | +| **Thread** | Single execution unit | +| **Block** | Group of threads (share memory) | +| **Grid** | Collection of blocks | +| **Warp** | 32 threads executing together | + +## AMD Ecosystem + +### Current Architectures + +| Architecture | Year | Key Features | +|--------------|------|--------------| +| **CDNA 3** | 2023 | MI300X, unified memory | +| **RDNA 3** | 2022 | RX 7000 series, chiplets | +| **CDNA 2** | 2021 | MI250X, exascale HPC | + +### Product Lines + +| Line | Target | Examples | +|------|--------|----------| +| **Radeon RX** | Gaming | RX 7900 XTX, 7800 XT | +| **Radeon PRO** | Workstation | PRO W7900, W7800 | +| **Instinct** | Data Center | MI300X, MI300A | + +### ROCm Platform + +**AMD's open-source GPU compute stack.** + +```cpp +// HIP (CUDA-like API) +__global__ void vectorAdd(float *a, float *b, float *c, int n) { + int i = hipBlockIdx_x * hipBlockDim_x + hipThreadIdx_x; + if (i < n) { + c[i] = a[i] + b[i]; + } +} + +hipLaunchKernelGGL(vectorAdd, numBlocks, threadsPerBlock, 0, 0, a, b, c, n); +``` + +| Component | Purpose | +|-----------|---------| +| **HIP** | CUDA-like programming API | +| **rocBLAS** | Linear algebra | +| **MIOpen** | Deep learning primitives | +| **rocFFT** | FFT library | + +## Intel GPUs + +### Arc Graphics + +**Discrete gaming GPUs:** + +| Architecture | Year | Products | +|--------------|------|----------| +| **Battlemage** | 2024 | Arc B-series | +| **Alchemist** | 2022 | Arc A770, A750, A380 | + +### Data Center GPUs + +| Product | Target | +|---------|--------| +| **Max Series (Ponte Vecchio)** | HPC/AI (Aurora supercomputer) | +| **Flex Series** | Cloud gaming, media | + +### oneAPI + +**Cross-architecture programming model.** + +```cpp +// SYCL (oneAPI) +q.submit([&](handler &h) { + h.parallel_for(range<1>(n), [=](id<1> i) { + c[i] = a[i] + b[i]; + }); +}); +``` + +## Memory Technologies + +| Type | Bandwidth | Capacity | Use Case | +|------|-----------|----------|----------| +| **GDDR6** | ~900 GB/s | 8-24 GB | Gaming | +| **GDDR6X** | ~1 TB/s | 16-24 GB | High-end gaming | +| **HBM2e** | ~2 TB/s | 40-80 GB | Data center | +| **HBM3** | ~3+ TB/s | 80+ GB | AI accelerators | + +### Memory Bandwidth Importance + +**AI/ML is memory-bound:** + +- Large model weights +- Batch processing +- Tensor operations + +Higher bandwidth = faster training/inference. + +## AI/ML Hardware + +### Training GPUs + +| GPU | Memory | FP16 TFLOPs | Use Case | +|-----|--------|-------------|----------| +| **H100 SXM** | 80 GB HBM3 | 1,979 | LLM training | +| **H200** | 141 GB HBM3e | 1,979 | Large models | +| **MI300X** | 192 GB HBM3 | 1,307 | AMD alternative | +| **A100** | 40/80 GB HBM2e | 312 | Previous gen | + +### Inference Accelerators + +| Accelerator | Vendor | Notes | +|-------------|--------|-------| +| **L4** | NVIDIA | Efficient inference | +| **L40S** | NVIDIA | Generative AI | +| **Inferentia** | AWS | Custom ASIC | +| **TPU** | Google | Tensor Processing Unit | +| **Trainium** | AWS | Training ASIC | + +### Multi-GPU Systems + +| Configuration | Description | +|---------------|-------------| +| **NVLink** | GPU-to-GPU (900 GB/s on H100) | +| **NVSwitch** | All-to-all GPU fabric | +| **DGX H100** | 8x H100 system | +| **HGX** | Reference multi-GPU platform | + +## Software Ecosystem + +### Frameworks + +| Framework | GPU Support | +|-----------|-------------| +| **PyTorch** | CUDA, ROCm, MPS | +| **TensorFlow** | CUDA, ROCm | +| **JAX** | CUDA, TPU | +| **ONNX Runtime** | CUDA, DirectML, ROCm | + +### Libraries + +| Library | Purpose | +|---------|---------| +| **cuDNN** | Deep learning primitives | +| **cuBLAS** | Linear algebra | +| **TensorRT** | Inference optimization | +| **Triton** | Inference serving | +| **FlashAttention** | Efficient attention | + +### Containers + +| Tool | Purpose | +|------|---------| +| **NGC** | NVIDIA GPU Cloud containers | +| **nvidia-docker** | Docker GPU support | +| **NVIDIA Driver Container** | Driver in container | + +## Comparison + +### Gaming vs Compute GPUs + +| Aspect | Gaming (GeForce) | Compute (A100/H100) | +|--------|------------------|---------------------| +| **Memory** | GDDR6/6X (up to 24 GB) | HBM (40-80 GB) | +| **ECC** | No | Yes | +| **FP64** | Limited | Full speed | +| **NVLink** | No | Yes | +| **MIG** | No | Yes (Multi-Instance GPU) | +| **Price** | $500-$2000 | $10,000-$40,000 | + +### NVIDIA vs AMD vs Intel + +| Aspect | NVIDIA | AMD | Intel | +|--------|--------|-----|-------| +| **Market Share** | ✅ Dominant | ⚠️ Growing | ⚠️ New entrant | +| **Software (AI)** | ✅ CUDA ecosystem | ⚠️ ROCm catching up | ⚠️ oneAPI developing | +| **Gaming** | ✅ Strong | ✅ Strong | ⚠️ Improving | +| **Ray Tracing** | ✅ Mature | ⚠️ Capable | ⚠️ Present | +| **Open Source** | ❌ Proprietary CUDA | ✅ ROCm is open | ✅ oneAPI is open | +| **Data Center** | ✅ Dominant | ✅ Growing (MI300X) | ⚠️ Limited | + +## When to Use GPUs + +### Strengths + +| Strength | Rationale | +|----------|-----------| +| **Parallel Workloads** | Matrix operations, simulations | +| **ML Training** | Essential for deep learning | +| **Graphics Rendering** | Original purpose | +| **Scientific Computing** | Molecular dynamics, CFD | + +### Considerations + +| Consideration | Impact | +|---------------|--------| +| **Memory Limits** | Model size constrained by VRAM | +| **Power Consumption** | 300-700W per GPU | +| **Programming Model** | CUDA lock-in concerns | +| **Cost** | High-end AI GPUs $10K+ | + +### Best For + +- **Deep Learning** — Training and inference +- **Scientific Simulation** — Physics, chemistry, weather +- **Graphics/Video** — Rendering, encoding +- **Cryptocurrency** — Mining (Ethereum PoS changed this) + +### Not Ideal For + +- Sequential algorithms +- Low-latency, small-batch inference +- Memory-bound tasks exceeding VRAM +- Cost-sensitive applications + +## Related + +- [[CPU Architectures]] — Comparison with CPUs +- [[Machine Learning MOC]] — ML/AI concepts +- [[Neural Networks]] — GPU-accelerated models +- [[Mainframes]] — Enterprise computing alternative diff --git a/Hardware/Hypervisors.md b/Hardware/Hypervisors.md new file mode 100644 index 0000000..a8bac78 --- /dev/null +++ b/Hardware/Hypervisors.md @@ -0,0 +1,367 @@ +--- +title: Hypervisors +aliases: + - Virtual Machine Monitor + - VMM + - Virtualization +tags: + - hardware + - virtualization + - infrastructure +type: reference +status: complete +created: "2025-12-18" +--- + +# Hypervisors + +Software layer that creates and manages virtual machines by abstracting physical hardware. + +## Overview + +| Aspect | Details | +|--------|---------| +| **Purpose** | Run multiple OS instances on single hardware | +| **Types** | Type 1 (bare-metal), Type 2 (hosted) | +| **Use Cases** | Server consolidation, dev/test, cloud | +| **Key Benefit** | Hardware utilization, isolation | + +## Types of Hypervisors + +### Type 1 (Bare-Metal) + +``` +┌─────────┐ ┌─────────┐ ┌─────────┐ +│ VM 1 │ │ VM 2 │ │ VM 3 │ +│ (Linux) │ │(Windows)│ │ (Linux) │ +└────┬────┘ └────┬────┘ └────┬────┘ + │ │ │ +┌────┴───────────┴───────────┴────┐ +│ HYPERVISOR │ +│ (VMware ESXi, Hyper-V) │ +└────────────────┬────────────────┘ + │ +┌────────────────┴────────────────┐ +│ HARDWARE │ +│ (CPU, RAM, Storage, NIC) │ +└─────────────────────────────────┘ +``` + +| Product | Vendor | Use Case | +|---------|--------|----------| +| **VMware ESXi** | VMware | Enterprise | +| **Microsoft Hyper-V** | Microsoft | Windows environments | +| **Xen** | Linux Foundation | Cloud (AWS original) | +| **KVM** | Open source | Linux, OpenStack | +| **Proxmox VE** | Proxmox | Open source enterprise | + +### Type 2 (Hosted) + +``` +┌─────────┐ ┌─────────┐ ┌─────────┐ +│ VM 1 │ │ VM 2 │ │ VM 3 │ +└────┬────┘ └────┬────┘ └────┬────┘ + │ │ │ +┌────┴───────────┴───────────┴────┐ +│ HYPERVISOR │ +│ (VirtualBox, VMware Workstation)│ +└────────────────┬────────────────┘ + │ +┌────────────────┴────────────────┐ +│ HOST OPERATING SYSTEM │ +│ (Windows, macOS, Linux) │ +└────────────────┬────────────────┘ + │ +┌────────────────┴────────────────┐ +│ HARDWARE │ +└─────────────────────────────────┘ +``` + +| Product | Vendor | Use Case | +|---------|--------|----------| +| **VirtualBox** | Oracle | Development, testing | +| **VMware Workstation** | VMware | Professional desktop | +| **VMware Fusion** | VMware | macOS | +| **Parallels** | Parallels | macOS | +| **QEMU** | Open source | Emulation, development | + +## Type 1 vs Type 2 + +| Aspect | Type 1 | Type 2 | +|--------|--------|--------| +| **Performance** | Near-native | Some overhead | +| **Use Case** | Production servers | Desktop/dev | +| **Hardware Access** | Direct | Through host OS | +| **Management** | Enterprise tools | Desktop apps | +| **Cost** | Often licensed | Often free | + +## How Virtualization Works + +### CPU Virtualization + +``` +┌─────────────────────────────────────────────────────┐ +│ Physical CPU │ +│ ┌──────────────────────────────────────────────┐ │ +│ │ Ring 0 (Kernel) │ │ +│ │ ┌────────────────────────────────────────┐ │ │ +│ │ │ Ring -1 (Hypervisor) │ │ │ +│ │ │ VMX root (host mode) │ │ │ +│ │ └────────────────────────────────────────┘ │ │ +│ │ │ │ │ +│ │ VM Entry/Exit │ │ +│ │ │ │ │ +│ │ ┌────────────────────────────────────────┐ │ │ +│ │ │ VMX non-root (guest mode) │ │ │ +│ │ │ VM Ring 0 (Guest Kernel) │ │ │ +│ │ │ VM Ring 3 (Guest Apps) │ │ │ +│ │ └────────────────────────────────────────┘ │ │ +│ └──────────────────────────────────────────────┘ │ +└─────────────────────────────────────────────────────┘ +``` + +### Hardware Assist + +| Technology | Vendor | Function | +|------------|--------|----------| +| **VT-x** | Intel | CPU virtualization | +| **AMD-V** | AMD | CPU virtualization | +| **VT-d / AMD-Vi** | Intel/AMD | I/O virtualization (IOMMU) | +| **EPT / NPT** | Intel/AMD | Memory virtualization | +| **SR-IOV** | PCIe spec | Network virtualization | + +### Memory Virtualization + +``` +┌─────────────────────────────────────────────────────┐ +│ Physical Memory │ +│ ┌──────────────────────────────────────────────┐ │ +│ │ │ │ +│ │ VM1 Physical → Host Physical (EPT/NPT) │ │ +│ │ │ │ +│ │ ┌────────┐ ┌────────┐ ┌────────┐ │ │ +│ │ │ VM1 │────▶│ Shadow │────▶│Physical│ │ │ +│ │ │Page Tbl│ │Page Tbl│ │ Memory │ │ │ +│ │ └────────┘ └────────┘ └────────┘ │ │ +│ │ │ │ +│ └──────────────────────────────────────────────┘ │ +└─────────────────────────────────────────────────────┘ +``` + +## KVM (Kernel-based Virtual Machine) + +### Architecture + +``` +┌─────────┐ ┌─────────┐ +│ VM 1 │ │ VM 2 │ (QEMU processes) +└────┬────┘ └────┬────┘ + │ │ +┌────┴───────────┴────┐ +│ QEMU (Device │ +│ Emulation) │ +└──────────┬──────────┘ + │ +┌──────────┴──────────┐ +│ Linux Kernel │ +│ ┌────────────────┐ │ +│ │ KVM Module │ │ +│ └────────────────┘ │ +└──────────┬──────────┘ + │ +┌──────────┴──────────┐ +│ Hardware │ +│ (VT-x/AMD-V) │ +└─────────────────────┘ +``` + +### KVM Commands + +```bash +# Check if KVM is available +lscpu | grep Virtualization +cat /proc/cpuinfo | grep vmx # Intel +cat /proc/cpuinfo | grep svm # AMD + +# Load KVM module +modprobe kvm +modprobe kvm_intel # or kvm_amd + +# Create VM with virt-install +virt-install \ + --name myvm \ + --ram 2048 \ + --vcpus 2 \ + --disk size=20 \ + --os-variant ubuntu22.04 \ + --cdrom ubuntu.iso + +# List VMs +virsh list --all + +# Start/stop VM +virsh start myvm +virsh shutdown myvm +virsh destroy myvm # Force stop + +# Connect to console +virsh console myvm +``` + +## VMware ESXi + +### Architecture + +``` +┌─────────────────────────────────────────────────────┐ +│ vSphere Client │ +└────────────────────────┬────────────────────────────┘ + │ +┌────────────────────────┴────────────────────────────┐ +│ vCenter Server │ +│ (Management, vMotion, HA, DRS) │ +└────────────────────────┬────────────────────────────┘ + │ + ┌────────────────────┼────────────────────────────┐ + │ │ │ +┌───┴────┐ ┌────┴───┐ ┌────────────┐ +│ ESXi │ │ ESXi │ │ ESXi │ +│ Host 1 │ │ Host 2 │ │ Host 3 │ +└────────┘ └────────┘ └────────────┘ +``` + +### Key Features + +| Feature | Description | +|---------|-------------| +| **vMotion** | Live migrate VMs between hosts | +| **Storage vMotion** | Migrate VM storage without downtime | +| **HA** | Restart VMs on another host if one fails | +| **DRS** | Automatic load balancing across hosts | +| **Fault Tolerance** | Shadow VM for zero-downtime failover | +| **VSAN** | Software-defined storage | + +## Hyper-V + +### Architecture + +``` +┌─────────┐ ┌─────────┐ ┌─────────┐ +│ Child │ │ Child │ │ Root │ +│Partition│ │Partition│ │Partition│ +│ (VM 1) │ │ (VM 2) │ │(Windows)│ +└────┬────┘ └────┬────┘ └────┬────┘ + │ │ │ +┌────┴───────────┴───────────┴────┐ +│ Hyper-V Hypervisor │ +└────────────────┬────────────────┘ + │ +┌────────────────┴────────────────┐ +│ Hardware │ +└─────────────────────────────────┘ +``` + +### PowerShell Management + +```powershell +# Create VM +New-VM -Name "MyVM" -MemoryStartupBytes 2GB -NewVHDPath "C:\VMs\MyVM.vhdx" -NewVHDSizeBytes 40GB + +# Start/Stop +Start-VM -Name "MyVM" +Stop-VM -Name "MyVM" + +# List VMs +Get-VM + +# Configure VM +Set-VM -Name "MyVM" -ProcessorCount 4 + +# Enable nested virtualization +Set-VMProcessor -VMName "MyVM" -ExposeVirtualizationExtensions $true + +# Create checkpoint (snapshot) +Checkpoint-VM -Name "MyVM" -SnapshotName "BeforeUpdate" +``` + +## Resource Management + +### CPU Allocation + +| Method | Description | +|--------|-------------| +| **Reservation** | Guaranteed minimum CPU | +| **Limit** | Maximum CPU allowed | +| **Shares** | Relative priority | + +### Memory Techniques + +| Technique | Description | +|-----------|-------------| +| **Memory Overcommit** | Allocate more than physical | +| **Ballooning** | Reclaim unused guest memory | +| **Page Sharing** | Deduplicate identical pages | +| **Memory Compression** | Compress less-used pages | +| **Swap to Disk** | Last resort overflow | + +### Storage + +| Type | Description | +|------|-------------| +| **Thick Provisioned** | Full space allocated upfront | +| **Thin Provisioned** | Grows as needed | +| **Eager Zeroed** | Pre-zeroed (better performance) | +| **Lazy Zeroed** | Zeroed on first write | + +## Containers vs VMs + +``` +Virtual Machines: Containers: +┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ +│ App A │ │ App B │ │ App A │ │ App B │ +├─────────┤ ├─────────┤ ├─────────┤ ├─────────┤ +│ Bins │ │ Bins │ │ Bins │ │ Bins │ +├─────────┤ ├─────────┤ └────┬────┘ └────┬────┘ +│Guest OS │ │Guest OS │ │ │ +└────┬────┘ └────┬────┘ ┌────┴───────────┴────┐ + │ │ │ Container Runtime │ +┌────┴───────────┴────┐ │ (Docker) │ +│ Hypervisor │ └──────────┬──────────┘ +└──────────┬──────────┘ │ + │ ┌──────────┴──────────┐ +┌──────────┴──────────┐ │ Host OS Kernel │ +│ Host OS │ └──────────┬──────────┘ +└──────────┬──────────┘ │ + │ ┌──────────┴──────────┐ +┌──────────┴──────────┐ │ Hardware │ +│ Hardware │ └─────────────────────┘ +└─────────────────────┘ +``` + +| Aspect | VMs | Containers | +|--------|-----|------------| +| **Isolation** | Full (separate kernel) | Process-level | +| **Boot Time** | Minutes | Seconds | +| **Size** | GBs | MBs | +| **Density** | 10s per host | 100s per host | +| **Overhead** | Significant | Minimal | +| **Use Case** | Different OS, strong isolation | Microservices, CI/CD | + +## Best Practices + +| Practice | Rationale | +|----------|-----------| +| **Right-size VMs** | Don't over-provision | +| **Use templates** | Consistent deployments | +| **Regular snapshots** | Before changes | +| **Monitor utilization** | Identify contention | +| **Update hypervisor** | Security patches | +| **Separate workloads** | Performance isolation | +| **Plan for capacity** | Growth projections | + +## Related + +- [[Virtual Networking]] — VM networking +- [[CPU Architectures]] — x86, ARM virtualization +- [[Server Hardware]] — Physical infrastructure +- [[Hardware MOC]] — Hardware overview diff --git a/Hardware/Mainframes.md b/Hardware/Mainframes.md new file mode 100644 index 0000000..54dcc68 --- /dev/null +++ b/Hardware/Mainframes.md @@ -0,0 +1,376 @@ +--- +title: Mainframes +aliases: + - IBM Mainframe + - z/Series + - Big Iron + - Enterprise Servers +tags: + - hardware + - enterprise + - legacy + - mainframe +type: reference +status: complete +created: "2025-12-16" +--- + +# Mainframes + +High-reliability enterprise computing systems designed for massive transaction volumes, continuous availability, and backward compatibility spanning decades. + +## Overview + +| Aspect | Details | +|--------|---------| +| **Primary Vendor** | IBM (90%+ market share) | +| **Current Platform** | IBM z16 (2022), z17 (expected) | +| **Architecture** | z/Architecture (64-bit, big-endian) | +| **Operating Systems** | z/OS, z/VM, z/VSE, Linux on Z | +| **Key Workloads** | Banking, insurance, airlines, government | +| **Reliability** | 99.999%+ uptime (5+ nines) | +| **Daily Transactions** | Trillions globally | + +## Architecture + +### z/Architecture + +**Successor to S/360 (1964)** — Continuous binary compatibility for 60+ years. + +| Feature | Description | +|---------|-------------| +| **Registers** | 16 general-purpose (64-bit) | +| **Addressing** | 64-bit virtual, up to 16 EB addressable | +| **Instruction Set** | CISC, thousands of instructions | +| **Endianness** | Big-endian | +| **I/O** | Channel-based (parallel I/O processors) | + +### Hardware Components + +``` +┌─────────────────────────────────────────────┐ +│ Central Processor │ +│ ┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐ │ +│ │ CP │ │ CP │ │ CP │ │ zIIP│ │ zIAP│ │ +│ └─────┘ └─────┘ └─────┘ └─────┘ └─────┘ │ +├─────────────────────────────────────────────┤ +│ Memory (up to 40 TB) │ +├─────────────────────────────────────────────┤ +│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ +│ │ Channel │ │ Channel │ │ Channel │ │ +│ └────┬─────┘ └────┬─────┘ └────┬─────┘ │ +│ │ │ │ │ +│ ┌────▼─────┐ ┌────▼─────┐ ┌────▼─────┐ │ +│ │ DASD │ │ Tape │ │ Network │ │ +│ └──────────┘ └──────────┘ └──────────┘ │ +└─────────────────────────────────────────────┘ +``` + +### Processor Types + +| Type | Purpose | Billing | +|------|---------|---------| +| **CP** | General Purpose (GP) | Full cost | +| **zIIP** | Java, XML, DB2 workloads | Reduced cost | +| **zAAP** | Java workloads (deprecated) | Reduced cost | +| **IFL** | Linux-only processor | Reduced cost | +| **ICF** | Coupling Facility | Clustering | +| **SAP** | System Assist Processor | I/O management | + +### I/O Subsystem + +**Channel Architecture** — Dedicated I/O processors. + +| Component | Function | +|-----------|----------| +| **Channel** | I/O processor handling device communication | +| **Control Unit** | Device controller (DASD, tape) | +| **FICON** | Fiber channel connection (up to 16 Gbps) | +| **ESCON** | Earlier serial channel (obsolete) | + +## Operating Systems + +### z/OS + +**Primary mainframe OS** — Evolved from OS/360 (1966). + +| Component | Purpose | +|-----------|---------| +| **JES2/JES3** | Job Entry Subsystem (batch scheduling) | +| **RACF** | Security (authentication, authorization) | +| **SMS** | Storage Management Subsystem | +| **WLM** | Workload Manager (resource allocation) | +| **VTAM** | Network communications | + +**Key Features:** + +- Runs [[COBOL]], [[JCL]], Assembler, PL/I +- DB2 for z/OS (relational database) +- CICS (online transaction processing) +- IMS (hierarchical database) +- MQ Series (messaging) + +### z/VM + +**Virtualization hypervisor** — Runs thousands of virtual machines. + +- Type 1 hypervisor (bare metal) +- Each VM is isolated +- Hosts Linux guests efficiently +- CMS (Conversational Monitor System) for interactive use + +### Linux on Z + +**Enterprise Linux on mainframe hardware.** + +| Distribution | Vendor | +|--------------|--------| +| RHEL for IBM Z | Red Hat | +| SUSE Linux Enterprise | SUSE | +| Ubuntu for IBM Z | Canonical | + +**Benefits:** + +- Consolidation (thousands of Linux VMs) +- Mainframe reliability +- Existing Linux skills transfer +- Open-source ecosystem + +## Data Storage + +### DASD (Direct Access Storage Device) + +**Disk storage with mainframe-specific features.** + +| Technology | Type | +|------------|------| +| **DS8900F** | High-end enterprise storage | +| **3390** | Logical disk geometry (legacy) | +| **ECKD** | Extended Count Key Data format | + +### Tape Storage + +**Still critical for mainframe backup and archive.** + +| Technology | Capacity | +|------------|----------| +| **TS7770** | Virtual tape (disk cache) | +| **3592** | Physical tape drives | +| **LTO** | Linear Tape-Open | + +### VSAM (Virtual Storage Access Method) + +**Native file access method:** + +| Type | Description | +|------|-------------| +| **KSDS** | Key-Sequenced Data Set | +| **ESDS** | Entry-Sequenced Data Set | +| **RRDS** | Relative Record Data Set | +| **LDS** | Linear Data Set | + +## Transaction Processing + +### CICS (Customer Information Control System) + +**Online transaction processing (OLTP).** + +| Feature | Description | +|---------|-------------| +| **Transactions/sec** | Thousands per second | +| **ACID** | Full transaction support | +| **API** | EXEC CICS commands in COBOL | +| **Regions** | Multiple isolated environments | + +### IMS (Information Management System) + +**Hierarchical database and transaction manager.** + +- Pre-dates relational databases +- Still used for high-volume banking +- Fastest transaction throughput + +### DB2 for z/OS + +**Enterprise relational database.** + +- SQL support +- Tight z/OS integration +- Handles massive data volumes +- Data sharing across sysplexes + +## High Availability + +### Parallel Sysplex + +**Cluster of mainframes sharing data.** + +``` +┌──────────────┐ ┌──────────────┐ +│ z/OS LPAR │ │ z/OS LPAR │ +│ System A │ │ System B │ +└──────┬───────┘ └───────┬──────┘ + │ │ + └──────────┬──────────┘ + │ + ┌────────▼────────┐ + │ Coupling Facility│ + │ (Shared State) │ + └─────────────────┘ +``` + +| Feature | Benefit | +|---------|---------| +| **Data Sharing** | Multiple systems access same DB2 | +| **Workload Balancing** | Automatic distribution | +| **Rolling Upgrades** | Zero-downtime updates | +| **Disaster Recovery** | Geographic distribution | + +### GDPS (Geographically Dispersed Parallel Sysplex) + +**Multi-site disaster recovery.** + +- Synchronous replication (near-zero data loss) +- Automatic failover +- Continuous availability + +## Modernization + +### API Enablement + +**Exposing mainframe functions via REST/GraphQL:** + +| Tool | Purpose | +|------|---------| +| **z/OS Connect** | REST API gateway | +| **CICS TS** | Web services support | +| **IBM API Connect** | API management | + +### DevOps on Mainframe + +| Tool | Purpose | +|------|---------| +| **Wazi** | VS Code for mainframe | +| **DBB** | Dependency-based build | +| **Git for z/OS** | Source control | +| **Jenkins/Tekton** | CI/CD pipelines | + +### Containerization + +- **zCX** — Run Docker containers on z/OS +- **OpenShift on Z** — Kubernetes on Linux on Z +- **Hybrid workloads** — Mix containers with COBOL + +## Comparison with Distributed Systems + +| Aspect | Mainframe | x86 Cluster | +|--------|-----------|-------------| +| **Reliability** | ✅ 99.999%+ | ⚠️ Requires HA design | +| **Transaction Volume** | ✅ Billions/day | ⚠️ Scaling complexity | +| **Security** | ✅ Hardware crypto, RACF | ⚠️ Software-based | +| **Virtualization** | ✅ Native, efficient | ✅ Mature (VMware, KVM) | +| **Cost Model** | ❌ High MIPS-based | ✅ Commodity pricing | +| **Talent Pool** | ❌ Shrinking | ✅ Large | +| **Agility** | ⚠️ Slower change cycles | ✅ Fast iteration | +| **Legacy Support** | ✅ 60 years of code runs | ❌ Migration needed | + +## Economics + +### MIPS-Based Pricing + +**Software licensing tied to CPU capacity.** + +| Term | Meaning | +|------|---------| +| **MIPS** | Millions of Instructions Per Second | +| **MSU** | Million Service Units | +| **MLC** | Monthly License Charge | +| **OTC** | One-Time Charge | + +**Cost Optimization:** + +- Shift workloads to zIIP (cheaper) +- Use Linux on Z (IFL pricing) +- Right-size LPAR capacity +- Container consolidation + +### Total Cost of Ownership + +| Factor | Mainframe | Distributed | +|--------|-----------|-------------| +| **Hardware** | High | Lower | +| **Software Licensing** | Very High | Varies | +| **Operations Staff** | Lower (consolidated) | Higher (many servers) | +| **Floor Space/Power** | Lower | Higher | +| **Reliability Cost** | Built-in | Additional infrastructure | + +## Industry Usage + +| Industry | Workloads | +|----------|-----------| +| **Banking** | Core banking, ATM networks, payments | +| **Insurance** | Policy administration, claims | +| **Airlines** | Reservations, scheduling | +| **Retail** | Inventory, POS transactions | +| **Government** | Tax processing, social security | +| **Healthcare** | Claims processing | + +### By the Numbers + +| Metric | Value | +|--------|-------| +| **Fortune 100** | 71% use mainframes | +| **Global Banking** | 96 of top 100 banks | +| **Daily Transactions** | 30+ billion | +| **COBOL Lines** | 220+ billion in production | + +## When to Use Mainframes + +### Strengths + +| Strength | Rationale | +|----------|-----------| +| **Extreme Reliability** | Hardware redundancy, proven uptime | +| **Transaction Volume** | Designed for billions of daily transactions | +| **Data Integrity** | Hardware-level protection | +| **Security** | Cryptographic processors, RACF | +| **Backward Compatibility** | Run 1960s code unmodified | + +### Considerations + +| Consideration | Impact | +|---------------|--------| +| **Cost** | High licensing fees (MIPS-based) | +| **Skills Gap** | Aging workforce | +| **Vendor Lock-in** | IBM dominance | +| **Perception** | "Legacy" stigma affects hiring | +| **Agility** | Slower development cycles | + +### Best For + +- **High-volume OLTP** — Banking, payments, reservations +- **Mission-critical** — Systems where downtime costs millions/hour +- **Regulatory compliance** — Audit trails, data integrity +- **Existing investment** — Large COBOL codebases + +### Migration Considerations + +**Not all workloads should migrate away:** + +- Transaction-heavy batch processing +- Systems with regulatory data requirements +- Stable, rarely-changed business logic + +**Candidates for migration:** + +- Web-facing applications +- Analytics and reporting +- New development initiatives + +## Related + +- [[COBOL]] — Primary mainframe programming language +- [[JCL]] — Job Control Language for batch processing +- [[Fortran]] — Scientific computing on mainframes +- [[CPU Architectures]] — Comparison with x86, ARM +- [[Languages MOC]] — Overview of all languages diff --git a/Hardware/Network Topologies.md b/Hardware/Network Topologies.md new file mode 100644 index 0000000..7df6e00 --- /dev/null +++ b/Hardware/Network Topologies.md @@ -0,0 +1,422 @@ +--- +title: Network Topologies +aliases: + - Network Design + - Network Architecture + - LAN Topologies +tags: + - hardware + - networking + - architecture +type: reference +status: complete +created: "2025-12-18" +--- + +# Network Topologies + +Physical and logical arrangements of network devices and connections. + +## Overview + +| Aspect | Physical Topology | Logical Topology | +|--------|-------------------|------------------| +| **Definition** | Actual cable/device layout | How data flows | +| **Example** | Star wiring to switch | Ethernet broadcast | +| **Visibility** | What you see | What you configure | + +## Physical Topologies + +### Bus Topology + +``` +┌──────┐ ┌──────┐ ┌──────┐ ┌──────┐ +│ PC 1 │ │ PC 2 │ │ PC 3 │ │ PC 4 │ +└──┬───┘ └──┬───┘ └──┬───┘ └──┬───┘ + │ │ │ │ +═══╧═══════════╧═══════════╧═══════════╧═══ + Shared Backbone Cable +``` + +| Aspect | Details | +|--------|---------| +| **Pros** | Simple, inexpensive | +| **Cons** | Single point of failure, collisions | +| **Legacy** | 10BASE2, 10BASE5 Ethernet | +| **Status** | Obsolete for LANs | + +### Star Topology + +``` + ┌──────┐ + │ PC 1 │ + └──┬───┘ + │ + ┌──────┐ ┌──▼───┐ ┌──────┐ + │ PC 2 ├───┤Switch├───┤ PC 4 │ + └──────┘ └──┬───┘ └──────┘ + │ + ┌──▼───┐ + │ PC 3 │ + └──────┘ +``` + +| Aspect | Details | +|--------|---------| +| **Pros** | Easy management, fault isolation | +| **Cons** | Switch is single point of failure | +| **Use** | Most common LAN topology | +| **Standard** | Modern Ethernet | + +### Ring Topology + +``` + ┌──────┐ + │ PC 1 │ + └──┬───┘ + │ + ┌────▼────┐ + │ │ +┌───┴──┐ ┌───┴──┐ +│ PC 4 │ │ PC 2 │ +└───┬──┘ └───┬──┘ + │ │ + └────┬────┘ + │ + ┌──▼───┐ + │ PC 3 │ + └──────┘ +``` + +| Aspect | Details | +|--------|---------| +| **Pros** | Predictable performance, no collisions | +| **Cons** | Single break affects all | +| **Use** | Token Ring (legacy), FDDI, SONET | +| **Variant** | Dual ring for redundancy | + +### Mesh Topology + +``` +Full Mesh: Partial Mesh: +┌──────┐────────┌──────┐ ┌──────┐────────┌──────┐ +│ A │────┐ │ B │ │ A │ │ B │ +└──┬───┘ │ └──┬───┘ └──┬───┘ └──┬───┘ + │ \ │ / │ │ │ + │ \ │ / │ │ │ + │ \ │ / │ └──────┬───────┘ + │ \ │/ │ │ +┌──┴───┐───────┌───┴──┐ ┌──────┐───┴───┌──────┐ +│ C │────────│ D │ │ C │───────│ D │ +└──────┘ └──────┘ └──────┘ └──────┘ +``` + +| Aspect | Full Mesh | Partial Mesh | +|--------|-----------|--------------| +| **Redundancy** | Maximum | Selective | +| **Cost** | Very high | Moderate | +| **Use** | Core networks | WAN, backbone | +| **Connections** | n(n-1)/2 | As needed | + +### Hybrid Topology + +``` + Core (Mesh) + ┌─────────────────────┐ + │ │ + ┌────┴────┐ ┌─────┴───┐ + │Router A │──────────│Router B │ + └────┬────┘ └────┬────┘ + │ │ + ┌────────┴────────┐ │ + │ │ │ +┌───┴───┐ ┌────┴──┐ ┌────┴──┐ +│Switch1│ │Switch2│ │Switch3│ +└───┬───┘ └───┬───┘ └───┬───┘ + │ │ │ + Star Star Star + Network Network Network +``` + +## Enterprise Network Design + +### Three-Tier Architecture + +``` + ┌─────────────────────┐ + │ CORE │ + │ (High-speed mesh) │ + │ Core Switches │ + └──────────┬──────────┘ + │ + ┌────────────────────┼────────────────────┐ + │ │ │ + ┌────────▼────────┐ ┌───────▼────────┐ ┌───────▼────────┐ + │ DISTRIBUTION │ │ DISTRIBUTION │ │ DISTRIBUTION │ + │ (Routing, │ │ (Routing, │ │ (Routing, │ + │ VLANs) │ │ VLANs) │ │ VLANs) │ + └────────┬────────┘ └───────┬────────┘ └───────┬────────┘ + │ │ │ + ┌────────┴────────┐ │ ┌───────┴────────┐ + │ │ │ │ │ +┌────▼────┐ ┌─────▼──┐ ┌─────▼──┐ ┌─────▼──┐ ┌─────▼──┐ +│ ACCESS │ │ ACCESS │ │ ACCESS │ │ ACCESS │ │ ACCESS │ +│ Switch │ │ Switch │ │ Switch │ │ Switch │ │ Switch │ +└────┬────┘ └────┬───┘ └────┬───┘ └────┬───┘ └────┬───┘ + │ │ │ │ │ + Users Users Users Users Users +``` + +| Tier | Function | Equipment | +|------|----------|-----------| +| **Core** | High-speed backbone | L3 switches, routers | +| **Distribution** | Routing, filtering, VLANs | L3 switches | +| **Access** | End-user connectivity | L2 switches | + +### Collapsed Core (Two-Tier) + +``` + ┌──────────────────────────┐ + │ CORE + DISTRIBUTION │ + │ (Combined Layer) │ + │ L3 Switches │ + └────────────┬─────────────┘ + │ + ┌───────────────────────┼───────────────────────┐ + │ │ │ + ┌────▼────┐ ┌─────▼────┐ ┌─────▼────┐ + │ ACCESS │ │ ACCESS │ │ ACCESS │ + │ Switch │ │ Switch │ │ Switch │ + └────┬────┘ └────┬─────┘ └────┬─────┘ + │ │ │ + Users Users Users +``` + +| Use Case | Benefits | +|----------|----------| +| Small-medium business | Cost savings, simpler | +| Single building | Fewer devices | +| < 500 users | Adequate performance | + +### Spine-Leaf Architecture + +``` + SPINE LAYER (East-West Traffic) + ┌─────────┐ ┌─────────┐ ┌─────────┐ + │ Spine 1 │ │ Spine 2 │ │ Spine 3 │ + └────┬────┘ └────┬────┘ └────┬────┘ + │ │ │ + ┌────┼──────────────┼──────────────┼────┐ + │ │ │ │ │ + │ ┌──┴──────────────┴──────────────┴──┐ │ + │ │ Full Mesh Connections │ │ + │ └──┬──────────────┬──────────────┬──┘ │ + │ │ │ │ │ + └────┼──────────────┼──────────────┼────┘ + │ │ │ + ┌────▼────┐ ┌────▼────┐ ┌────▼────┐ + │ Leaf 1 │ │ Leaf 2 │ │ Leaf 3 │ + └────┬────┘ └────┬────┘ └────┬────┘ + │ │ │ + Servers Servers Servers + +Every leaf connects to every spine = predictable latency +``` + +| Aspect | Details | +|--------|---------| +| **Use** | Data centers, cloud | +| **Benefits** | Predictable latency, scalable | +| **Routing** | BGP or OSPF between layers | +| **Protocols** | VXLAN, EVPN for overlay | + +## Data Center Design + +### Pod Design + +``` +┌───────────────────────────────────────────────────────┐ +│ POD 1 │ +│ ┌─────────┐ ┌─────────┐ │ +│ │ ToR Sw │ │ ToR Sw │ Top of Rack │ +│ └────┬────┘ └────┬────┘ │ +│ │ │ │ +│ ┌────▼──────────────▼────┐ │ +│ │ Aggregation Sw │ │ +│ └────────────┬───────────┘ │ +│ │ │ +│ ┌────────────┴────────────┐ │ +│ │ Racks │ │ +│ │ [Srv][Srv][Srv][Srv] │ │ +│ └─────────────────────────┘ │ +└───────────────────────────────────────────────────────┘ +``` + +### Clos Network + +``` + Tier 3 (Spine) + ┌───┐ ┌───┐ ┌───┐ + │ S │ │ S │ │ S │ + └─┬─┘ └─┬─┘ └─┬─┘ + │ │ │ + └────────┼────────┘ + │ + ┌──────────┼──────────┐ + │ │ │ + ┌─┴─┐ ┌─┴─┐ ┌─┴─┐ + │ A │ │ A │ │ A │ Tier 2 (Aggregation) + └─┬─┘ └─┬─┘ └─┬─┘ + │ │ │ + └──────────┼──────────┘ + │ + ┌──────────┼──────────┐ + │ │ │ + ┌─┴─┐ ┌─┴─┐ ┌─┴─┐ + │ L │ │ L │ │ L │ Tier 1 (Leaf/ToR) + └─┬─┘ └─┬─┘ └─┬─┘ + │ │ │ + Servers Servers Servers +``` + +## WAN Topologies + +### Hub and Spoke + +``` + ┌──────────┐ + ┌─────┤ HQ ├─────┐ + │ │ (Hub) │ │ + │ └────┬─────┘ │ + │ │ │ + ┌────▼───┐ ┌────▼───┐ ┌─────▼──┐ + │Branch 1│ │Branch 2│ │Branch 3│ + │(Spoke) │ │(Spoke) │ │(Spoke) │ + └────────┘ └────────┘ └────────┘ +``` + +| Aspect | Details | +|--------|---------| +| **Pros** | Centralized control, cost-effective | +| **Cons** | Hub is single point of failure | +| **Use** | Enterprise WAN, retail | + +### Full/Partial Mesh WAN + +``` + ┌────────────┐ + ┌────┤ HQ ├────┐ + │ └──────┬─────┘ │ + │ │ │ + │ │ │ +┌─────▼────┐ │ ┌─────▼────┐ +│ Region A │◄─────┼────► Region B │ +└─────┬────┘ │ └─────┬────┘ + │ │ │ + │ ┌────▼───┐ │ + └──────► Region C◄──────┘ + └────────┘ +``` + +## Redundancy Patterns + +### Dual-Homed + +``` + ┌─────────────────┐ + │ Internet │ + └────┬───────┬────┘ + │ │ + ┌────▼─┐ ┌───▼────┐ + │ ISP1 │ │ ISP2 │ + └────┬─┘ └───┬────┘ + │ │ + ┌────▼───────▼────┐ + │ Router │ + │ (Multi-homing) │ + └─────────────────┘ +``` + +### High Availability Pair + +``` + ┌─────────────┐ + │ Uplink │ + └──────┬──────┘ + │ + ┌──────────┴──────────┐ + │ │ +┌───▼───┐ VRRP/ ┌────▼──┐ +│ FW 1 │ HSRP │ FW 2 │ +│Active │◄──────────►│Standby│ +└───┬───┘ └───┬───┘ + │ │ + └─────────┬──────────┘ + │ + ┌─────▼─────┐ + │ Network │ + └───────────┘ +``` + +## Network Segmentation + +### VLAN Segmentation + +``` +┌─────────────────────────────────────────────┐ +│ Physical Switch │ +│ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ +│ │ VLAN 10 │ │ VLAN 20 │ │ VLAN 30 │ │ +│ │ Users │ │ Servers │ │ Guest │ │ +│ └─────────┘ └─────────┘ └─────────┘ │ +└─────────────────────────────────────────────┘ + +• Logical separation on shared hardware +• Inter-VLAN routing required for communication +• Broadcast domain isolation +``` + +### Micro-segmentation + +``` +┌───────────────────────────────────────────────────┐ +│ Data Center │ +│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ +│ │ Web Tier │ │ App Tier │ │ DB Tier │ │ +│ │ ↓ │→→│ ↓ │→→│ ↓ │ │ +│ │ Firewall │ │ Firewall │ │ Firewall │ │ +│ └──────────┘ └──────────┘ └──────────┘ │ +│ Fine-grained east-west security controls │ +└───────────────────────────────────────────────────┘ +``` + +## Design Considerations + +| Factor | Considerations | +|--------|----------------| +| **Scale** | Number of devices, growth | +| **Redundancy** | Fault tolerance requirements | +| **Performance** | Bandwidth, latency needs | +| **Security** | Segmentation, filtering | +| **Management** | Complexity, automation | +| **Cost** | CapEx, OpEx | +| **Standards** | Compliance requirements | + +## Common Protocols + +| Protocol | Purpose | +|----------|---------| +| **STP/RSTP** | Loop prevention | +| **LACP** | Link aggregation | +| **VRRP/HSRP** | Gateway redundancy | +| **BGP** | WAN/Internet routing | +| **OSPF** | Internal routing | +| **VXLAN** | Overlay networking | +| **EVPN** | Multi-tenant networking | + +## Related + +- [[Routers and Switches]] — Network devices +- [[Firewalls]] — Security placement +- [[Networking Fundamentals]] — Protocols +- [[Hardware MOC]] — Hardware overview diff --git a/Hardware/Routers and Switches.md b/Hardware/Routers and Switches.md new file mode 100644 index 0000000..8c4ed63 --- /dev/null +++ b/Hardware/Routers and Switches.md @@ -0,0 +1,340 @@ +--- +title: Routers and Switches +aliases: + - Network Switches + - Network Routers + - L2 and L3 Devices +tags: + - hardware + - networking + - infrastructure +type: reference +status: complete +created: "2025-12-18" +--- + +# Routers and Switches + +Fundamental network devices for connecting and routing traffic. + +## Overview + +| Device | OSI Layer | Function | +|--------|-----------|----------| +| **Hub** | Layer 1 (Physical) | Broadcast to all ports | +| **Switch** | Layer 2 (Data Link) | Forward by MAC address | +| **Router** | Layer 3 (Network) | Forward by IP address | +| **L3 Switch** | Layer 2-3 | Switch + routing | + +## Switches + +### How Switches Work + +``` +Device A ────┐ + │ +Device B ────┼───[ SWITCH ]───┬─── Device D + │ │ +Device C ────┘ └─── Device E + +1. Device A sends frame to Device D +2. Switch reads destination MAC address +3. Switch checks MAC address table +4. Switch forwards ONLY to port with Device D +``` + +### MAC Address Table + +| Port | MAC Address | VLAN | +|------|-------------|------| +| 1 | AA:BB:CC:DD:EE:01 | 10 | +| 2 | AA:BB:CC:DD:EE:02 | 10 | +| 3 | AA:BB:CC:DD:EE:03 | 20 | + +```bash +# View MAC table (Cisco) +show mac address-table + +# Clear MAC table +clear mac address-table dynamic +``` + +### Switch Types + +| Type | Description | Use Case | +|------|-------------|----------| +| **Unmanaged** | Plug-and-play, no config | Home, small office | +| **Smart/Web-managed** | Basic GUI management | Small business | +| **Managed** | Full CLI/GUI, advanced features | Enterprise | +| **PoE Switch** | Power over Ethernet | IP phones, cameras, APs | +| **Stackable** | Multiple units as one logical switch | Data centers | + +### Key Features + +| Feature | Description | +|---------|-------------| +| **VLANs** | Virtual LANs for segmentation | +| **STP** | Spanning Tree prevents loops | +| **Port Security** | Limit MACs per port | +| **Link Aggregation** | Combine ports for bandwidth | +| **QoS** | Traffic prioritization | +| **Port Mirroring** | Copy traffic for monitoring | + +### VLAN Configuration + +```bash +# Cisco IOS +# Create VLAN +vlan 10 + name Engineering + +# Assign port to VLAN +interface GigabitEthernet0/1 + switchport mode access + switchport access vlan 10 + +# Trunk port (carries multiple VLANs) +interface GigabitEthernet0/24 + switchport mode trunk + switchport trunk allowed vlan 10,20,30 +``` + +### Spanning Tree Protocol (STP) + +``` + [Root Bridge] + / \ + / \ + [Switch A] [Switch B] + \ / + \ / + [Switch C] + X (Blocked port - prevents loop) +``` + +| Protocol | Convergence | Description | +|----------|-------------|-------------| +| **STP (802.1D)** | 30-50 sec | Original standard | +| **RSTP (802.1w)** | 1-3 sec | Rapid convergence | +| **MSTP (802.1s)** | 1-3 sec | Multiple spanning trees | + +## Routers + +### How Routers Work + +``` +Network A Network B +10.0.1.0/24 10.0.2.0/24 + │ │ + └────[ ROUTER ]───────┘ + 10.0.1.1 10.0.2.1 + +1. Packet arrives from Network A destined for Network B +2. Router checks routing table +3. Router decrements TTL +4. Router forwards to appropriate interface +``` + +### Routing Table + +```bash +# View routing table +show ip route + +# Example output +C 10.0.1.0/24 is directly connected, GigabitEthernet0/0 +C 10.0.2.0/24 is directly connected, GigabitEthernet0/1 +S 192.168.1.0/24 [1/0] via 10.0.1.254 +O 172.16.0.0/16 [110/20] via 10.0.2.254 +``` + +| Code | Meaning | +|------|---------| +| C | Directly connected | +| S | Static route | +| O | OSPF | +| B | BGP | +| R | RIP | +| * | Default route | + +### Static vs Dynamic Routing + +| Aspect | Static | Dynamic | +|--------|--------|---------| +| **Configuration** | Manual | Automatic | +| **Scalability** | Poor | Good | +| **Bandwidth** | None | Protocol overhead | +| **Failover** | Manual | Automatic | +| **Use Case** | Small networks, stubs | Large, complex networks | + +### Routing Protocols + +| Protocol | Type | Metric | Use Case | +|----------|------|--------|----------| +| **RIP** | Distance Vector | Hop count | Legacy, small networks | +| **OSPF** | Link State | Cost | Enterprise internal | +| **EIGRP** | Hybrid | Composite | Cisco networks | +| **BGP** | Path Vector | AS Path | Internet, between ISPs | +| **IS-IS** | Link State | Cost | Large ISPs | + +### OSPF Configuration + +```bash +# Enable OSPF +router ospf 1 + router-id 1.1.1.1 + network 10.0.1.0 0.0.0.255 area 0 + network 10.0.2.0 0.0.0.255 area 0 + +# Verify +show ip ospf neighbor +show ip ospf interface +``` + +### BGP Configuration + +```bash +# Configure BGP +router bgp 65001 + neighbor 203.0.113.1 remote-as 65002 + network 198.51.100.0 mask 255.255.255.0 + +# Verify +show ip bgp summary +show ip bgp neighbors +``` + +## Layer 3 Switches + +**Combines switching and routing in hardware.** + +``` + ┌──────────────────┐ + │ L3 Switch │ + │ ┌──────────────┐ │ + │ │ Routing │ │ + │ │ (Hardware) │ │ + │ └──────────────┘ │ + │ ┌──────────────┐ │ + │ │ Switching │ │ + │ │ (Hardware) │ │ + │ └──────────────┘ │ + └──────────────────┘ +``` + +### Inter-VLAN Routing + +```bash +# Create SVIs (Switch Virtual Interfaces) +interface vlan 10 + ip address 10.0.10.1 255.255.255.0 + no shutdown + +interface vlan 20 + ip address 10.0.20.1 255.255.255.0 + no shutdown + +# Enable IP routing +ip routing +``` + +## Router vs L3 Switch + +| Aspect | Router | L3 Switch | +|--------|--------|-----------| +| **Routing Speed** | Software/hardware | Wire-speed (hardware) | +| **Port Density** | Low (few ports) | High (many ports) | +| **WAN Features** | Full (NAT, VPN, etc.) | Limited | +| **Cost per Port** | High | Low | +| **Use Case** | WAN edge, internet | Campus core, distribution | + +## Network Address Translation (NAT) + +``` +Private Network Router Internet +192.168.1.x ───────[ NAT Router ]─────── 203.0.113.1 + (Translation) +``` + +### NAT Types + +| Type | Description | +|------|-------------| +| **Static NAT** | 1:1 private to public mapping | +| **Dynamic NAT** | Pool of public IPs | +| **PAT (Overload)** | Many private to one public + ports | + +```bash +# PAT Configuration (Cisco) +ip nat inside source list 1 interface GigabitEthernet0/1 overload + +access-list 1 permit 192.168.1.0 0.0.0.255 + +interface GigabitEthernet0/0 + ip nat inside + +interface GigabitEthernet0/1 + ip nat outside +``` + +## Access Control Lists (ACLs) + +```bash +# Standard ACL (source only) +access-list 10 permit 192.168.1.0 0.0.0.255 +access-list 10 deny any + +# Extended ACL (source, dest, protocol, port) +access-list 100 permit tcp 192.168.1.0 0.0.0.255 any eq 80 +access-list 100 permit tcp 192.168.1.0 0.0.0.255 any eq 443 +access-list 100 deny ip any any + +# Apply to interface +interface GigabitEthernet0/0 + ip access-group 100 in +``` + +## Common Vendors + +| Vendor | Products | Strengths | +|--------|----------|-----------| +| **Cisco** | Catalyst, Nexus, ISR | Industry standard, enterprise | +| **Juniper** | EX, QFX, MX | High performance, service providers | +| **Arista** | 7000 series | Data center, low latency | +| **HPE/Aruba** | ProCurve, CX | Cost-effective enterprise | +| **Ubiquiti** | UniFi, EdgeSwitch | SMB, prosumer | + +## Troubleshooting Commands + +```bash +# Cisco IOS +show interfaces status # Port status +show interfaces trunk # Trunk ports +show vlan brief # VLAN summary +show mac address-table # MAC table +show ip route # Routing table +show ip ospf neighbor # OSPF neighbors +show spanning-tree # STP status +show cdp neighbors # Connected Cisco devices +ping 10.0.1.1 # Basic connectivity +traceroute 10.0.1.1 # Path to destination +``` + +## Best Practices + +| Practice | Rationale | +|----------|-----------| +| **Document everything** | Diagrams, IP assignments | +| **Use VLANs** | Segment traffic, security | +| **Enable STP** | Prevent loops | +| **Secure management** | SSH, no telnet | +| **Disable unused ports** | Security | +| **Regular backups** | Config backup | +| **Monitor** | SNMP, syslog | + +## Related + +- [[Networking Fundamentals]] — OSI model, protocols +- [[Firewalls]] — Network security devices +- [[Network Topologies]] — Network designs +- [[Hardware MOC]] — Hardware overview diff --git a/Hardware/Server Hardware.md b/Hardware/Server Hardware.md new file mode 100644 index 0000000..524611f --- /dev/null +++ b/Hardware/Server Hardware.md @@ -0,0 +1,358 @@ +--- +title: Server Hardware +aliases: + - Servers + - Rack Servers + - Data Center Hardware +tags: + - hardware + - servers + - infrastructure + - enterprise +type: reference +status: complete +created: "2025-12-16" +--- + +# Server Hardware + +Enterprise computing systems designed for reliability, scalability, and remote management in data centers. + +## Overview + +| Category | Description | Examples | +|----------|-------------|----------| +| **Rack Servers** | Standard 19" rack mount | Dell PowerEdge, HPE ProLiant | +| **Blade Servers** | High-density, shared chassis | HPE BladeSystem, Dell M-series | +| **Tower Servers** | Standalone, SMB/SOHO | Dell PowerEdge Tower | +| **Mainframes** | Enterprise transaction systems | IBM z16 | +| **Hyperconverged** | Compute + Storage integrated | Nutanix, VMware vSAN | + +## Form Factors + +### Rack Units (U) + +| Height | Use Case | +|--------|----------| +| **1U** | Dense compute, web servers | +| **2U** | General purpose, storage | +| **4U** | GPU servers, high storage | +| **8U+** | Specialized systems | + +**Standard Rack:** 42U height, 19" width. + +### Server Types + +``` +┌─────────────────────────────────────┐ +│ Tower Server │ +│ (Standalone, quiet, SMB) │ +└─────────────────────────────────────┘ + +┌─────────────────────────────────────┐ ─┐ +│ 1U Rack Server │ │ +├─────────────────────────────────────┤ │ +│ 1U Rack Server │ │ 42U +├─────────────────────────────────────┤ │ Rack +│ 2U Rack Server │ │ +│ │ │ +├─────────────────────────────────────┤ │ +│ 4U GPU Server │ │ +│ │ │ +│ │ │ +│ │ │ +└─────────────────────────────────────┘ ─┘ + +┌─────────────────────────────────────┐ +│ Blade Chassis (10U+) │ +│ ┌─────┐┌─────┐┌─────┐┌─────┐ │ +│ │Blade││Blade││Blade││Blade│ │ +│ └─────┘└─────┘└─────┘└─────┘ │ +│ (Shared power, networking, mgmt) │ +└─────────────────────────────────────┘ +``` + +## Components + +### Processors + +| Vendor | Product Line | Target | +|--------|--------------|--------| +| **Intel** | Xeon Scalable (Sapphire Rapids) | General, HPC | +| **AMD** | EPYC (Genoa, Bergamo) | High core count | +| **Arm** | Ampere Altra, AWS Graviton | Cloud, efficiency | +| **IBM** | POWER10 | Enterprise, AIX | + +**Socket Configurations:** + +| Config | Use Case | +|--------|----------| +| **1P** | Web, small workloads | +| **2P** | General enterprise | +| **4P/8P** | Large databases, in-memory | + +### Memory + +| Type | Speed | Use Case | +|------|-------|----------| +| **DDR4** | 3200 MT/s | Current generation | +| **DDR5** | 4800-6400 MT/s | New platforms | +| **RDIMM** | Standard | General | +| **LRDIMM** | High capacity | Large memory | +| **NVDIMM** | Persistent | Database caching | + +**Capacity:** Up to 8 TB+ per server (8-socket systems). + +### Storage Controllers + +| Type | Description | +|------|-------------| +| **RAID Controller** | Hardware RAID, battery-backed | +| **HBA** | Pass-through, software RAID | +| **NVMe** | Direct PCIe, no controller | + +### Networking + +| Interface | Speed | Use Case | +|-----------|-------|----------| +| **1GbE** | 1 Gbps | Management, legacy | +| **10GbE** | 10 Gbps | Standard data center | +| **25GbE** | 25 Gbps | Modern standard | +| **100GbE** | 100 Gbps | Storage, HPC | +| **400GbE** | 400 Gbps | Spine, AI clusters | + +**Technologies:** + +| Technology | Purpose | +|------------|---------| +| **RDMA** | Low-latency networking | +| **RoCE** | RDMA over Converged Ethernet | +| **InfiniBand** | HPC interconnect | +| **SR-IOV** | Virtual NIC hardware | + +### Power and Cooling + +| Component | Description | +|-----------|-------------| +| **Redundant PSU** | N+1 or 2N power | +| **Hot-swap** | Replace without downtime | +| **240V/208V** | Data center power | +| **PDU** | Power Distribution Unit | + +**Cooling:** + +- Air cooling (most common) +- Rear-door heat exchangers +- Liquid cooling (GPUs, HPC) +- Immersion cooling (emerging) + +## Management + +### Out-of-Band Management + +| Vendor | Technology | +|--------|------------| +| **Dell** | iDRAC | +| **HPE** | iLO | +| **Lenovo** | XClarity | +| **Supermicro** | BMC/IPMI | +| **Intel** | BMC (standard) | + +**Features:** + +- Remote power control +- Virtual console (KVM over IP) +- Virtual media (ISO mounting) +- Hardware monitoring +- Firmware updates + +### IPMI (Intelligent Platform Management Interface) + +**Standard for server management.** + +| Feature | Description | +|---------|-------------| +| **SOL** | Serial Over LAN | +| **Sensor Data** | Temperature, voltage, fans | +| **Event Logging** | Hardware events | +| **Power Control** | On/off/reset | + +### Redfish + +**Modern REST API for hardware management.** + +```bash +# Get system info via Redfish +curl -k https://bmc-ip/redfish/v1/Systems/1 \ + -u admin:password +``` + +**Advantages over IPMI:** + +- RESTful (JSON) +- HTTPS security +- Scalable +- Schema-driven + +## Vendors + +### Enterprise Vendors + +| Vendor | Products | Strengths | +|--------|----------|-----------| +| **Dell** | PowerEdge | Broad portfolio, iDRAC | +| **HPE** | ProLiant, Synergy | Enterprise support, iLO | +| **Lenovo** | ThinkSystem | Acquired IBM x86 | +| **Cisco** | UCS | Unified fabric | + +### ODM/Whitebox + +| Vendor | Notes | +|--------|-------| +| **Supermicro** | Customizable, fast | +| **Inspur** | Large Chinese vendor | +| **Quanta** | Cloud provider supplier | +| **Wiwynn** | Facebook/Microsoft supplier | + +### Cloud Hardware + +| Provider | Custom Hardware | +|----------|-----------------| +| **AWS** | Nitro, Graviton | +| **Google** | Custom TPU, servers | +| **Microsoft** | Cobalt, custom racks | +| **Meta** | Open Compute Project | + +## Server Categories + +### General Purpose + +**Balanced compute, memory, I/O.** + +| Vendor | Example | +|--------|---------| +| **Dell** | PowerEdge R760 | +| **HPE** | ProLiant DL380 Gen11 | +| **Lenovo** | ThinkSystem SR650 V3 | + +### High Memory + +**In-memory databases, SAP HANA.** + +- 4-8 sockets +- 6-24 TB RAM +- Intel Xeon or POWER + +### GPU Servers + +**AI training, HPC, rendering.** + +| Config | GPUs | Use Case | +|--------|------|----------| +| **4U, 4-8 GPUs** | A100/H100 | AI training | +| **DGX H100** | 8x H100 | LLM training | +| **OVX** | NVIDIA OVX | Digital twins | + +### Storage Servers + +**High-density disk/SSD platforms.** + +- 2U with 24 NVMe bays +- 4U with 60-90 HDDs +- JBOD expansion shelves + +## Reliability Features + +### Hardware Redundancy + +| Component | Redundancy | +|-----------|------------| +| **Power Supplies** | N+1, 2N | +| **Fans** | N+1 | +| **NICs** | Bonding/teaming | +| **Disks** | RAID, erasure coding | + +### Error Correction + +| Feature | Description | +|---------|-------------| +| **ECC Memory** | Single-bit correction, multi-bit detection | +| **SDDC** | Single Device Data Correction | +| **Memory Mirroring** | Full redundancy (50% capacity) | +| **Memory Sparing** | Hot spare DIMMs | + +### Predictive Failure + +- **SMART** for drives +- **DIMM error logging** +- **PCIe error monitoring** +- **ML-based predictions** + +## Comparison + +### Rack vs Blade vs HCI + +| Aspect | Rack | Blade | HCI | +|--------|------|-------|-----| +| **Density** | Medium | High | Medium | +| **Flexibility** | ✅ High | ⚠️ Chassis-locked | ⚠️ Software-locked | +| **Management** | Per-server | Unified | Software-defined | +| **Networking** | Standard | Integrated | Converged | +| **Cost** | Lower entry | Higher (chassis) | Premium | +| **Use Case** | General | Very high density | Simplified ops | + +### Bare Metal vs Cloud + +| Aspect | Bare Metal | Cloud | +|--------|------------|-------| +| **Cost** | CapEx, lower TCO at scale | OpEx, pay-per-use | +| **Control** | ✅ Full | ⚠️ Limited | +| **Provisioning** | Hours/days | Minutes | +| **Performance** | ✅ Consistent | ⚠️ Variable (shared) | +| **Compliance** | ✅ Full control | ⚠️ Shared responsibility | + +## Data Center Considerations + +### Power + +| Metric | Description | +|--------|-------------| +| **PUE** | Power Usage Effectiveness (1.0 = perfect) | +| **TDP** | Thermal Design Power per server | +| **kW/Rack** | Typical 10-30 kW | + +### Networking + +| Topology | Description | +|----------|-------------| +| **Spine-Leaf** | Standard data center | +| **Fat Tree** | HPC, AI clusters | +| **Dragonfly** | Large-scale HPC | + +### Standards + +| Standard | Purpose | +|----------|---------| +| **Open Compute Project (OCP)** | Open hardware designs | +| **Redfish** | Management API | +| **CXL** | Compute Express Link (memory pooling) | + +## When to Choose + +| Workload | Recommendation | +|----------|----------------| +| **Web/App servers** | 1U-2U rack servers | +| **Databases** | 2U with NVMe, high memory | +| **AI Training** | 4U GPU servers | +| **Virtualization** | 2U balanced servers | +| **HPC** | Blade or dense rack | +| **SMB/Branch** | Tower or 1U | +| **Edge** | Ruggedized, low power | + +## Related + +- [[CPU Architectures]] — Processor choices +- [[GPUs]] — Accelerator hardware +- [[Storage Systems]] — Enterprise storage +- [[Mainframes]] — IBM enterprise systems +- [[Kubernetes]] — Container orchestration on servers diff --git a/Hardware/Storage Systems.md b/Hardware/Storage Systems.md new file mode 100644 index 0000000..32534e8 --- /dev/null +++ b/Hardware/Storage Systems.md @@ -0,0 +1,358 @@ +--- +title: Storage Systems +aliases: + - Enterprise Storage + - Storage Hardware + - Data Storage +tags: + - hardware + - storage + - enterprise + - infrastructure +type: reference +status: complete +created: "2025-12-16" +--- + +# Storage Systems + +Hardware and architectures for persisting data, from local disks to distributed storage arrays. + +## Overview + +| Category | Examples | Use Case | +|----------|----------|----------| +| **Local Storage** | SSD, HDD, NVMe | Single server | +| **Network Storage** | NAS, SAN | Shared enterprise | +| **Object Storage** | S3, MinIO, Ceph | Cloud, unstructured data | +| **Distributed** | HDFS, Ceph, GlusterFS | Big data, scale-out | + +## Storage Media + +### Solid State Drives (SSD) + +| Interface | Speed | Use Case | +|-----------|-------|----------| +| **SATA SSD** | ~550 MB/s | Budget, capacity | +| **NVMe (PCIe 3.0)** | ~3.5 GB/s | Consumer, workstation | +| **NVMe (PCIe 4.0)** | ~7 GB/s | High performance | +| **NVMe (PCIe 5.0)** | ~14 GB/s | Data center | + +**NAND Types:** + +| Type | Bits/Cell | Endurance | Cost | +|------|-----------|-----------|------| +| **SLC** | 1 | Highest | Expensive | +| **MLC** | 2 | High | Moderate | +| **TLC** | 3 | Moderate | Common | +| **QLC** | 4 | Lower | Cheapest | + +### Hard Disk Drives (HDD) + +| Type | RPM | Use Case | +|------|-----|----------| +| **Enterprise** | 7200-15000 | Servers, NAS | +| **Desktop** | 7200 | Workstations | +| **SMR (Shingled)** | 5400-7200 | Archive, backup | +| **CMR** | Varies | Random I/O | + +**Capacity:** Up to 32 TB per drive. + +**Still relevant for:** + +- Cold storage / archive +- Cost per TB (3x cheaper than SSD) +- Large sequential workloads + +### NVMe Form Factors + +| Form Factor | Description | +|-------------|-------------| +| **M.2** | Small, consumer/workstation | +| **U.2** | 2.5" enterprise | +| **U.3** | Universal Bay (SAS/SATA/NVMe) | +| **EDSFF E1.S/E3.S** | Next-gen data center | +| **Add-in Card** | PCIe slot mount | + +## Storage Architectures + +### Direct-Attached Storage (DAS) + +``` +┌─────────────┐ +│ Server │ +│ ┌───────┐ │ +│ │ Disk │ │ +│ │ Disk │ │ +│ │ Disk │ │ +│ └───────┘ │ +└─────────────┘ +``` + +**Characteristics:** + +- Lowest latency +- No network overhead +- Limited sharing +- Simple + +### Network-Attached Storage (NAS) + +``` +┌────────┐ ┌─────────────────┐ +│ Server │────▶│ │ +└────────┘ │ NAS Device │ +┌────────┐ │ ┌───────────┐ │ +│ Server │────▶│ │ Storage │ │ +└────────┘ │ └───────────┘ │ +┌────────┐ │ │ +│ Laptop │────▶│ (NFS/SMB) │ +└────────┘ └─────────────────┘ +``` + +**Protocols:** + +| Protocol | Platform | Use Case | +|----------|----------|----------| +| **NFS** | Unix/Linux | File sharing | +| **SMB/CIFS** | Windows | File sharing | +| **AFP** | macOS (legacy) | Apple file sharing | + +**Vendors:** Synology, QNAP, NetApp, Dell EMC Isilon. + +### Storage Area Network (SAN) + +``` +┌────────┐ ┌──────────────┐ ┌──────────────┐ +│ Server │────▶│ │ │ │ +└────────┘ │ Fibre │────▶│ Storage │ +┌────────┐ │ Channel │ │ Array │ +│ Server │────▶│ Switch │────▶│ │ +└────────┘ │ │ │ (Block I/O) │ +└────────┘ └──────────────┘ └──────────────┘ +``` + +**Characteristics:** + +- Block-level storage +- High performance +- Expensive +- Complex + +**Protocols:** + +| Protocol | Speed | Description | +|----------|-------|-------------| +| **Fibre Channel** | 16/32/64 Gbps | Traditional SAN | +| **iSCSI** | 1-100 Gbps | Block over Ethernet | +| **NVMe-oF** | 100+ Gbps | NVMe over fabric | +| **FCoE** | 10-100 Gbps | FC over Ethernet | + +**Vendors:** Dell EMC (PowerStore), NetApp, Pure Storage, HPE. + +## Enterprise Storage Arrays + +### All-Flash Arrays (AFA) + +| Vendor | Product | Notes | +|--------|---------|-------| +| **Pure Storage** | FlashArray | Evergreen model | +| **NetApp** | AFF | ONTAP software | +| **Dell EMC** | PowerStore | Unified block/file | +| **HPE** | Alletra** | dHCI, cloud-native | + +### Hybrid Arrays + +Mix of SSD and HDD with automated tiering. + +| Vendor | Product | +|--------|---------| +| **Dell EMC** | PowerVault, Unity | +| **NetApp** | FAS | +| **HPE** | Primera | + +### Key Features + +| Feature | Description | +|---------|-------------| +| **Deduplication** | Eliminate redundant data | +| **Compression** | Reduce storage footprint | +| **Thin Provisioning** | Allocate on demand | +| **Snapshots** | Point-in-time copies | +| **Replication** | Sync/async remote copies | +| **Tiering** | Move data between SSD/HDD | +| **Encryption** | At-rest and in-flight | + +## Object Storage + +### Concepts + +| Term | Description | +|------|-------------| +| **Object** | Data + metadata + unique ID | +| **Bucket** | Container for objects | +| **Key** | Object identifier (path-like) | +| **Metadata** | Custom attributes | + +### S3 API + +**De facto standard for object storage.** + +```python +import boto3 + +s3 = boto3.client('s3') +s3.upload_file('data.csv', 'my-bucket', 'path/to/data.csv') +s3.download_file('my-bucket', 'path/to/data.csv', 'local.csv') +``` + +### Implementations + +| Product | Type | Notes | +|---------|------|-------| +| **AWS S3** | Cloud | Original, most popular | +| **Azure Blob** | Cloud | Microsoft | +| **GCS** | Cloud | Google | +| **MinIO** | Self-hosted | S3-compatible | +| **Ceph RGW** | Self-hosted | S3/Swift compatible | +| **Wasabi** | Cloud | S3-compatible, cheaper | + +### Use Cases + +- **Data lakes** — Unstructured data at scale +- **Backup/archive** — Long-term retention +- **Static assets** — Web content, media +- **ML datasets** — Training data storage + +## Distributed Storage + +### Software-Defined Storage + +| System | Type | Use Case | +|--------|------|----------| +| **Ceph** | Block/Object/File | OpenStack, Kubernetes | +| **GlusterFS** | File | Scale-out NAS | +| **MinIO** | Object | S3-compatible | +| **HDFS** | File | Hadoop/Big Data | +| **Longhorn** | Block | Kubernetes | + +### Ceph Architecture + +``` +┌──────────────────────────────────────────┐ +│ Client Access │ +│ RBD (Block) | RGW (S3) | CephFS │ +├──────────────────────────────────────────┤ +│ RADOS │ +│ (Reliable Autonomic Distributed Object │ +│ Storage) │ +├──────────────────────────────────────────┤ +│ ┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐ │ +│ │ OSD │ │ OSD │ │ OSD │ │ OSD │ │ +│ └─────┘ └─────┘ └─────┘ └─────┘ │ +│ (Object Storage Daemons) │ +└──────────────────────────────────────────┘ +``` + +### Kubernetes Storage + +| Solution | Type | +|----------|------| +| **Rook-Ceph** | Block, Object, File | +| **Longhorn** | Block (simple) | +| **OpenEBS** | Block | +| **Portworx** | Enterprise | +| **NetApp Trident** | NetApp integration | + +**CSI (Container Storage Interface):** + +- Standard for K8s storage plugins +- Dynamic provisioning +- Snapshot support + +## Comparison + +### Block vs File vs Object + +| Aspect | Block | File | Object | +|--------|-------|------|--------| +| **Access** | Raw blocks | Hierarchy (paths) | Key-value | +| **Protocol** | FC, iSCSI, NVMe-oF | NFS, SMB | S3, Swift | +| **Use Case** | Databases, VMs | File sharing | Web, backup, data lakes | +| **Scalability** | Limited | Moderate | Massive | +| **Metadata** | Minimal | File attributes | Rich, custom | + +### Performance Tiers + +| Tier | Media | IOPS | Latency | Use Case | +|------|-------|------|---------|----------| +| **Ultra** | NVMe/Optane | 1M+ | <100 μs | Real-time | +| **Performance** | NVMe | 100K+ | <500 μs | Databases | +| **Capacity** | TLC SSD | 10K+ | ~1 ms | General | +| **Archive** | HDD/QLC | 100s | ~10 ms | Cold data | + +### Cost Comparison + +| Storage Type | $/TB/Month (approx) | +|--------------|---------------------| +| **NVMe SSD (enterprise)** | $50-100 | +| **SATA SSD (enterprise)** | $20-50 | +| **HDD (enterprise)** | $5-15 | +| **S3 Standard** | $23 | +| **S3 Glacier** | $4 | +| **Tape (LTO-9)** | $1-3 | + +## Data Protection + +### RAID Levels + +| Level | Description | Overhead | Use Case | +|-------|-------------|----------|----------| +| **RAID 0** | Stripe, no redundancy | 0% | Performance only | +| **RAID 1** | Mirror | 50% | Simple redundancy | +| **RAID 5** | Stripe + parity | 1 disk | Read-heavy | +| **RAID 6** | Stripe + 2 parity | 2 disks | Write-heavy | +| **RAID 10** | Mirror + stripe | 50% | Performance + safety | + +### Erasure Coding + +**Alternative to RAID for distributed storage:** + +- Lower overhead than mirroring +- Configurable redundancy (e.g., 8+3) +- Used in object storage, Ceph + +### Backup Strategies + +| Type | Frequency | Retention | +|------|-----------|-----------| +| **Full** | Weekly/Monthly | Long-term | +| **Incremental** | Daily | Medium-term | +| **Differential** | Daily | Medium-term | +| **Continuous (CDP)** | Real-time | Short-term | + +### 3-2-1 Rule + +- **3** copies of data +- **2** different storage types +- **1** offsite/cloud copy + +## When to Choose + +| Scenario | Recommendation | +|----------|----------------| +| **Database (OLTP)** | NVMe SSD, SAN, or local | +| **File Sharing** | NAS (Synology, NetApp) | +| **Web Assets** | Object storage (S3) | +| **Big Data/Analytics** | HDFS, object storage | +| **Kubernetes** | Rook-Ceph, Longhorn | +| **Backup** | Object storage, tape | +| **Archive** | S3 Glacier, tape | +| **Mainframe** | DASD, tape (IBM) | + +## Related + +- [[Database Engines]] — Databases on storage +- [[Mainframes]] — Enterprise storage (DASD) +- [[Kubernetes]] — Container storage +- [[File Storage]] — Cloud file storage options diff --git a/Hardware/Virtual Networking.md b/Hardware/Virtual Networking.md new file mode 100644 index 0000000..78f9622 --- /dev/null +++ b/Hardware/Virtual Networking.md @@ -0,0 +1,424 @@ +--- +title: Virtual Networking +aliases: + - Software-Defined Networking + - SDN + - Network Virtualization +tags: + - hardware + - networking + - virtualization +type: reference +status: complete +created: "2025-12-18" +--- + +# Virtual Networking + +Network infrastructure implemented in software, abstracting physical network hardware. + +## Overview + +| Aspect | Details | +|--------|---------| +| **Purpose** | Flexible, programmable networking | +| **Abstraction** | Logical networks over physical | +| **Control** | Centralized, software-defined | +| **Use Cases** | Cloud, data center, multi-tenant | + +## Virtual Network Components + +### Virtual Switch (vSwitch) + +``` +┌───────────────────────────────────────────────────────────┐ +│ Physical Host │ +│ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ +│ │ VM 1 │ │ VM 2 │ │ VM 3 │ │ +│ │ vNIC │ │ vNIC │ │ vNIC │ │ +│ └────┬────┘ └────┬────┘ └────┬────┘ │ +│ │ │ │ │ +│ ┌────┴────────────┴────────────┴────┐ │ +│ │ Virtual Switch │ │ +│ │ (OVS, VMware vSwitch, Hyper-V) │ │ +│ └──────────────────┬─────────────────┘ │ +│ │ │ +│ ┌──────────────────┴─────────────────┐ │ +│ │ Physical NIC (pNIC) │ │ +│ └──────────────────┬─────────────────┘ │ +└─────────────────────┼─────────────────────────────────────┘ + │ + Physical Network +``` + +### Virtual NIC (vNIC) + +| Type | Description | +|------|-------------| +| **Emulated** | Software emulates real NIC (e1000) | +| **Paravirtual** | Guest-aware drivers (virtio, vmxnet3) | +| **SR-IOV** | Direct hardware access (VF) | +| **DPDK** | User-space packet processing | + +## Open vSwitch (OVS) + +### Architecture + +``` +┌────────────────────────────────────────────────────────┐ +│ User Space │ +│ ┌──────────────────────────────────────────────────┐ │ +│ │ ovs-vswitchd │ │ +│ │ ┌──────────┐ ┌──────────┐ ┌──────────────┐ │ │ +│ │ │ OpenFlow │ │ OVSDB │ │ Port/VLAN │ │ │ +│ │ │ Protocol │ │ Protocol │ │ Management │ │ │ +│ │ └──────────┘ └──────────┘ └──────────────┘ │ │ +│ └──────────────────────────────────────────────────┘ │ +│ │ │ +│ ┌──────────────────────────────────────────────────┐ │ +│ │ ovsdb-server (config DB) │ │ +│ └──────────────────────────────────────────────────┘ │ +└────────────────────────────┬───────────────────────────┘ + │ +┌────────────────────────────┴───────────────────────────┐ +│ Kernel Space │ +│ ┌──────────────────────────────────────────────────┐ │ +│ │ OVS Kernel Module (datapath) │ │ +│ │ Fast path packet forwarding │ │ +│ └──────────────────────────────────────────────────┘ │ +└─────────────────────────────────────────────────────────┘ +``` + +### OVS Commands + +```bash +# Create bridge +ovs-vsctl add-br br0 + +# Add port +ovs-vsctl add-port br0 eth0 +ovs-vsctl add-port br0 veth0 + +# Set VLAN +ovs-vsctl set port veth0 tag=100 + +# Create trunk port +ovs-vsctl set port eth0 trunks=100,200,300 + +# Show configuration +ovs-vsctl show + +# Flow management +ovs-ofctl dump-flows br0 +ovs-ofctl add-flow br0 "in_port=1,actions=output:2" + +# Port statistics +ovs-ofctl dump-ports br0 +``` + +## Network Modes + +### Bridge Mode + +``` +┌─────────────────────────────────────────┐ +│ Physical Host │ +│ ┌────────┐ ┌────────┐ │ +│ │ VM 1 │ │ VM 2 │ │ +│ │10.0.0.2│ │10.0.0.3│ │ +│ └───┬────┘ └───┬────┘ │ +│ │ │ │ +│ ┌───┴───────────┴───┐ │ +│ │ Bridge (br0) │ │ +│ └─────────┬─────────┘ │ +│ │ │ +│ ┌─────────┴─────────┐ │ +│ │ Physical NIC │ │ +│ │ 10.0.0.1 │ │ +│ └─────────┬─────────┘ │ +└────────────┼────────────────────────────┘ + │ + Physical Network (10.0.0.0/24) + +• VMs appear as separate hosts on physical network +• VMs get IPs from same range as host +``` + +### NAT Mode + +``` +┌─────────────────────────────────────────┐ +│ Physical Host │ +│ ┌────────┐ ┌────────┐ │ +│ │ VM 1 │ │ VM 2 │ │ +│ │192.168.│ │192.168.│ │ +│ │122.10 │ │122.11 │ │ +│ └───┬────┘ └───┬────┘ │ +│ │ │ │ +│ ┌───┴───────────┴───┐ │ +│ │ Virtual Network │ │ +│ │ (192.168.122.0) │ │ +│ └─────────┬─────────┘ │ +│ │ NAT │ +│ ┌─────────┴─────────┐ │ +│ │ Physical NIC │ │ +│ │ 10.0.0.1 │ │ +│ └─────────┬─────────┘ │ +└────────────┼────────────────────────────┘ + │ + Physical Network + +• VMs share host's IP for outbound +• Port forwarding for inbound +• VMs isolated from physical network +``` + +### Host-Only Mode + +``` +┌─────────────────────────────────────────┐ +│ Physical Host │ +│ ┌────────┐ ┌────────┐ ┌─────────┐ │ +│ │ VM 1 │ │ VM 2 │ │ Host │ │ +│ │192.168.│ │192.168.│ │ 192.168.│ │ +│ │56.10 │ │56.11 │ │ 56.1 │ │ +│ └───┬────┘ └───┬────┘ └────┬────┘ │ +│ │ │ │ │ +│ ┌───┴───────────┴────────────┴───┐ │ +│ │ Host-Only Network │ │ +│ │ (192.168.56.0/24) │ │ +│ └────────────────────────────────┘ │ +│ │ +│ ┌─────────────────────────────────┐ │ +│ │ Physical NIC (no connection) │ │ +│ └─────────────────────────────────┘ │ +└─────────────────────────────────────────┘ + +• VMs can talk to each other and host +• No external network access +• Good for isolated testing +``` + +## Overlay Networks + +### VXLAN + +``` +┌─────────────────────────────────────────────────────────────┐ +│ VXLAN Overlay │ +│ ┌─────────────────────────────────────────────────────┐ │ +│ │ Virtual Network (VNI 1000) │ │ +│ │ ┌──────┐ ┌──────┐ │ │ +│ │ │ VM A │ │ VM B │ │ │ +│ │ │10.0.1│ │10.0.1│ │ │ +│ │ │ .10 │ │ .11 │ │ │ +│ │ └──┬───┘ └──┬───┘ │ │ +│ └──────┼──────────────────────┼───────────────────────┘ │ +│ │ │ │ +│ ┌──────┴────┐ ┌──────┴────┐ │ +│ │ VTEP │ │ VTEP │ │ +│ │ 192.168. │ │ 192.168. │ │ +│ │ 1.10 │ │ 1.20 │ │ +│ └──────┬────┘ └──────┬────┘ │ +│ │ │ │ +└───────────┼──────────────────────┼───────────────────────────┘ + │ │ + ┌──────┴──────────────────────┴──────┐ + │ Physical Network │ + │ (Underlay 192.168.1.0/24) │ + └─────────────────────────────────────┘ + +VXLAN encapsulates L2 frames in UDP (port 4789) +VNI (VXLAN Network Identifier) = 24 bits = 16M networks +``` + +### VXLAN Packet Structure + +``` +┌─────────────────────────────────────────────────────┐ +│ Outer Ethernet │ Outer IP │ UDP │ VXLAN │ Inner │ +│ Header │ Header │ Hdr │ Header│ Ethernet │ +│ │ │ │ │ Frame │ +└─────────────────────────────────────────────────────┘ + │ │ + │ └─ VNI (24 bits) + └─ VTEP IP addresses +``` + +### GRE (Generic Routing Encapsulation) + +```bash +# Linux GRE tunnel +ip tunnel add gre1 mode gre remote 10.0.0.2 local 10.0.0.1 +ip link set gre1 up +ip addr add 192.168.100.1/24 dev gre1 +``` + +## Software-Defined Networking (SDN) + +### SDN Architecture + +``` +┌─────────────────────────────────────────────────────────────┐ +│ Application Layer │ +│ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ +│ │ Network │ │ Security│ │ Monitor │ │ +│ │ Apps │ │ Apps │ │ Apps │ │ +│ └────┬────┘ └────┬────┘ └────┬────┘ │ +└─────────┼────────────┼────────────┼─────────────────────────┘ + │ │ │ + └────────────┼────────────┘ + │ Northbound API (REST) +┌──────────────────────┼──────────────────────────────────────┐ +│ │ Control Layer │ +│ ┌─────────────────┴─────────────────┐ │ +│ │ SDN Controller │ │ +│ │ (OpenDaylight, ONOS, Floodlight)│ │ +│ └─────────────────┬─────────────────┘ │ +│ │ Southbound API (OpenFlow) │ +└──────────────────────┼──────────────────────────────────────┘ + │ +┌──────────────────────┼──────────────────────────────────────┐ +│ │ Data Layer │ +│ ┌─────────┐ ┌────┴────┐ ┌─────────┐ │ +│ │ Switch │ │ Switch │ │ Switch │ │ +│ │ A │ │ B │ │ C │ │ +│ └─────────┘ └─────────┘ └─────────┘ │ +└─────────────────────────────────────────────────────────────┘ +``` + +### OpenFlow + +``` +# OpenFlow rules (conceptual) +Match: src_ip=10.0.0.0/24, dst_port=80 +Action: forward to port 2 + +Match: src_ip=192.168.1.0/24 +Action: drop + +Match: any +Action: send to controller +``` + +## Cloud Virtual Networking + +### AWS VPC + +``` +┌─────────────────────────────────────────────────────────────┐ +│ VPC (10.0.0.0/16) │ +│ ┌─────────────────────────┐ ┌────────────────────────┐ │ +│ │ Public Subnet │ │ Private Subnet │ │ +│ │ (10.0.1.0/24) │ │ (10.0.2.0/24) │ │ +│ │ ┌──────┐ ┌──────┐ │ │ ┌──────┐ ┌──────┐ │ │ +│ │ │ EC2 │ │ EC2 │ │ │ │ EC2 │ │ RDS │ │ │ +│ │ └──────┘ └──────┘ │ │ └──────┘ └──────┘ │ │ +│ │ │ │ │ │ │ │ +│ │ ┌─────┴─────┐ │ │ ┌─────┴─────┐ │ │ +│ │ │ Internet │ │ │ │ NAT │ │ │ +│ │ │ Gateway │ │ │ │ Gateway │ │ │ +│ │ └─────┬─────┘ │ │ └─────┬─────┘ │ │ +│ └────────┼────────────────┘ └────────┼──────────────┘ │ +│ │ │ │ +│ ┌────────┴────────────────────────────┴────────────┐ │ +│ │ Route Tables │ │ +│ └────────────────────────┬─────────────────────────┘ │ +└───────────────────────────┼─────────────────────────────────┘ + │ + Internet +``` + +### Key Cloud Networking Concepts + +| Concept | AWS | Azure | GCP | +|---------|-----|-------|-----| +| **Virtual Network** | VPC | VNet | VPC | +| **Subnets** | Subnet | Subnet | Subnet | +| **Firewall** | Security Groups, NACLs | NSG | Firewall Rules | +| **Load Balancer** | ALB/NLB/ELB | Load Balancer | Cloud Load Balancing | +| **NAT** | NAT Gateway | NAT Gateway | Cloud NAT | +| **VPN** | VPN Gateway | VPN Gateway | Cloud VPN | +| **Peering** | VPC Peering | VNet Peering | VPC Peering | + +## Network Function Virtualization (NFV) + +``` +Traditional: NFV: +┌─────────┐ ┌─────────────────────┐ +│Firewall │ │ NFV Platform │ +│Appliance│ │ ┌───────────────┐ │ +└─────────┘ │ │ Virtual FW │ │ +┌─────────┐ │ ├───────────────┤ │ +│ Load │ ────────▶ │ │ Virtual LB │ │ +│Balancer │ │ ├───────────────┤ │ +└─────────┘ │ │ Virtual Router│ │ +┌─────────┐ │ └───────────────┘ │ +│ Router │ │ (Standard servers) │ +│Appliance│ └─────────────────────┘ +└─────────┘ +``` + +| VNF Type | Function | +|----------|----------| +| vFirewall | Security filtering | +| vRouter | Routing | +| vLB | Load balancing | +| vWAN Optimizer | WAN acceleration | +| vIDS/IPS | Intrusion detection | + +## Container Networking + +### CNI (Container Network Interface) + +| CNI Plugin | Description | +|------------|-------------| +| **Calico** | L3 networking, network policies | +| **Flannel** | Simple overlay (VXLAN) | +| **Weave** | Mesh networking | +| **Cilium** | eBPF-based, security | +| **AWS VPC CNI** | Native AWS VPC | + +### Kubernetes Networking + +``` +┌───────────────────────────────────────────────────────────┐ +│ Kubernetes Cluster │ +│ ┌─────────────────────────────────────────────────────┐ │ +│ │ Pod Network │ │ +│ │ ┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐ │ │ +│ │ │ Pod A │ │ Pod B │ │ Pod C │ │ Pod D │ │ │ +│ │ │10.244. │ │10.244. │ │10.244. │ │10.244. │ │ │ +│ │ │ 1.10 │ │ 1.11 │ │ 2.10 │ │ 2.11 │ │ │ +│ │ └────┬───┘ └────┬───┘ └────┬───┘ └────┬───┘ │ │ +│ │ │ │ │ │ │ │ +│ │ ┌────┴───────────┴───┐ ┌────┴───────────┴───┐ │ │ +│ │ │ Node 1 │ │ Node 2 │ │ │ +│ │ │ (10.244.1.0/24) │ │ (10.244.2.0/24) │ │ │ +│ │ └────────────────────┘ └────────────────────┘ │ │ +│ └─────────────────────────────────────────────────────┘ │ +│ │ │ +│ ┌─────────────────────────┴───────────────────────────┐ │ +│ │ Service Network (ClusterIP) │ │ +│ │ 10.96.0.0/12 │ │ +│ └─────────────────────────────────────────────────────┘ │ +└───────────────────────────────────────────────────────────┘ +``` + +## Performance Optimization + +| Technique | Description | +|-----------|-------------| +| **SR-IOV** | Direct NIC access bypassing hypervisor | +| **DPDK** | User-space packet processing | +| **virtio** | Paravirtualized drivers | +| **Jumbo Frames** | Larger MTU (9000) for efficiency | +| **RSS** | Receive Side Scaling | +| **TSO/LRO** | TCP Segmentation Offload | + +## Related + +- [[Hypervisors]] — VM management +- [[Routers and Switches]] — Physical networking +- [[Network Topologies]] — Network design +- [[Hardware MOC]] — Hardware overview diff --git a/Languages MOC.md b/Languages MOC.md index 237ccc1..73b52cc 100644 --- a/Languages MOC.md +++ b/Languages MOC.md @@ -46,6 +46,17 @@ Programming languages and their ecosystems. | [[Languages/Swift|Swift]] | Multi-paradigm | Apple platforms | | [[Languages/Dart|Dart]] | OOP | Flutter | +### Legacy & Mainframe + +| Language | Paradigm | Use Cases | +|----------|----------|-----------| +| [[Languages/COBOL|COBOL]] | Procedural | Banking, insurance, government | +| [[Languages/Fortran|Fortran]] | Array-oriented | Scientific computing, HPC | +| [[Languages/JCL|JCL]] | Job control | Mainframe batch processing | +| [[Languages/BASIC|BASIC]] | Procedural | Education, VBA, legacy apps | +| [[Languages/Pascal|Pascal]] | Structured | Education, Delphi desktop | +| [[Languages/Ada|Ada]] | Contract-based | Aerospace, defense, safety-critical | + --- ## By Domain @@ -57,6 +68,8 @@ Programming languages and their ecosystems. | [[Domains/Desktop Development|Desktop]] | C#, C++, Swift, Rust, TypeScript | | [[Domains/Game Development|Game]] | C#, C++, Rust | | [[Domains/Embedded & Systems|Systems]] | C++, Rust, C | +| Legacy & Mainframe | COBOL, Fortran, JCL, Ada | +| Scientific & HPC | Fortran, Python, C++ | --- @@ -90,6 +103,6 @@ SORT file.name ASC ```dataview LIST -FROM #comparison AND (#csharp OR #go OR #python OR #typescript OR #rust OR #java OR #kotlin OR #swift OR #cpp OR #dart OR #php OR #ruby) +FROM #comparison AND (#csharp OR #go OR #python OR #typescript OR #rust OR #java OR #kotlin OR #swift OR #cpp OR #dart OR #php OR #ruby OR #legacy OR #mainframe OR #scientific) SORT file.name ASC ``` diff --git a/Languages/AS400.md b/Languages/AS400.md new file mode 100644 index 0000000..5004a04 --- /dev/null +++ b/Languages/AS400.md @@ -0,0 +1,378 @@ +--- +title: AS/400 +aliases: + - IBM i + - iSeries + - System i + - IBM Midrange +tags: + - language + - legacy + - mainframe + - enterprise + - ibm +type: reference +status: complete +created: "2025-12-16" +--- + +# AS/400 (IBM i) + +IBM's integrated midrange system combining hardware, OS, database, and security — known for extreme reliability and backwards compatibility. + +## Overview + +| Aspect | Details | +|--------|---------| +| **Original Name** | AS/400 (Application System/400) | +| **Current Name** | IBM i (on Power Systems) | +| **First Released** | 1988 | +| **Architecture** | Single-Level Storage, object-based | +| **Database** | DB2 for i (integrated) | +| **Primary Languages** | RPG, COBOL, CL, SQL | +| **Market** | SMB to enterprise, manufacturing, distribution | + +## Name Evolution + +| Era | Name | Hardware | +|-----|------|----------| +| 1988-2000 | AS/400 | CISC (IMPI), then PowerPC | +| 2000-2006 | iSeries | POWER4/5 | +| 2006-2008 | System i | POWER5/6 | +| 2008-present | IBM i | Power Systems (shared with AIX) | + +## Architecture + +### Single-Level Storage + +**Unique memory model** — All storage appears as one address space. + +``` +┌─────────────────────────────────────────┐ +│ Single-Level Storage │ +│ ┌─────────────────────────────────┐ │ +│ │ Virtual Address Space │ │ +│ │ (All objects addressable) │ │ +│ └─────────────────────────────────┘ │ +│ ↕ │ +│ ┌──────────┐ ┌──────────────────┐ │ +│ │ RAM │ │ Disk Storage │ │ +│ └──────────┘ └──────────────────┘ │ +│ (System manages movement) │ +└─────────────────────────────────────────┘ +``` + +**Benefits:** + +- No file system in traditional sense +- Objects persist automatically +- No manual memory management +- Simplified programming model + +### Object-Based Architecture + +**Everything is an object** with type and attributes. + +| Object Type | Code | Description | +|-------------|------|-------------| +| **\*PGM** | Program | Compiled executable | +| **\*FILE** | File | Physical/Logical file (database) | +| **\*DTAARA** | Data Area | Shared data storage | +| **\*USRSPC** | User Space | Raw memory area | +| **\*MSGQ** | Message Queue | Inter-process communication | +| **\*JOBD** | Job Description | Job attributes | +| **\*LIB** | Library | Object container | + +### Machine Interface (MI) + +**Hardware abstraction layer** — Programs compile to MI, not native code. + +``` +┌─────────────────┐ +│ Application │ +│ (RPG, COBOL) │ +├─────────────────┤ +│ MI Instructions│ ← Technology Independent +├─────────────────┤ +│ SLIC (OS) │ ← System Licensed Internal Code +├─────────────────┤ +│ Power Hardware │ +└─────────────────┘ +``` + +**Result:** Programs from 1988 run on 2024 hardware unchanged. + +## Integrated Database (DB2 for i) + +### Physical Files + +**Database tables** stored as system objects. + +```sql +CREATE TABLE CUSTMAST ( + CUSTNO DECIMAL(8,0) NOT NULL, + CUSTNAME CHAR(50), + BALANCE DECIMAL(11,2), + PRIMARY KEY (CUSTNO) +) +``` + +### Logical Files (Views/Indexes) + +```sql +CREATE INDEX CUSTIDX ON CUSTMAST (CUSTNAME) + +CREATE VIEW ACTIVECUST AS + SELECT * FROM CUSTMAST WHERE BALANCE > 0 +``` + +### DDS (Data Description Specifications) + +**Traditional file definition** (pre-SQL): + +```dds + A R CUSTREC + A CUSTNO 8S 0 TEXT('Customer Number') + A CUSTNAME 50A TEXT('Customer Name') + A BALANCE 11S 2 TEXT('Balance') + A K CUSTNO +``` + +### SQL vs Native I/O + +| Approach | Use Case | +|----------|----------| +| **SQL** | Ad-hoc queries, reports, modern apps | +| **Native I/O** | High-volume transaction processing | + +Native I/O can be 10x faster for simple key lookups. + +## Programming Languages + +### RPG (Report Program Generator) + +**Primary AS/400 language** — See [[RPG]] for details. + +```rpg +**free +dcl-f CUSTMAST disk usage(*update); + +dcl-ds custRec likerec(CUSTREC); + +chain (custNo) CUSTMAST custRec; +if %found(CUSTMAST); + custRec.BALANCE += payment; + update CUSTREC custRec; +endif; +``` + +### CL (Control Language) + +**System command scripting:** + +```cl +PGM + DCLF FILE(CUSTMAST) + + CHGJOB CCSID(37) + CRTDUPOBJ OBJ(CUSTMAST) FROMLIB(PROD) TOLIB(TEST) + CALL PGM(NIGHTBATCH) + + MONMSG MSGID(CPF0000) EXEC(GOTO CMDLBL(ERROR)) + + SNDMSG MSG('Batch complete') TOUSR(OPERATOR) + RETURN + +ERROR: + SNDMSG MSG('Batch failed') TOUSR(OPERATOR) +ENDPGM +``` + +### COBOL on IBM i + +```cobol + IDENTIFICATION DIVISION. + PROGRAM-ID. CUSTUPD. + + ENVIRONMENT DIVISION. + INPUT-OUTPUT SECTION. + FILE-CONTROL. + SELECT CUSTMAST ASSIGN TO DATABASE-CUSTMAST + ORGANIZATION IS INDEXED + ACCESS IS RANDOM + RECORD KEY IS CUSTNO. +``` + +### SQL Embedded in RPG + +```rpg +**free +exec sql + SELECT CUSTNAME, BALANCE INTO :custName, :balance + FROM CUSTMAST + WHERE CUSTNO = :custNo; + +if sqlcode = 0; + // Process customer +endif; +``` + +## Job System + +### Job Types + +| Type | Description | +|------|-------------| +| **Interactive** | Terminal sessions (5250) | +| **Batch** | Submitted jobs (SBMJOB) | +| **Autostart** | Start with subsystem | +| **Prestart** | Pool of waiting jobs | +| **Server** | System servers (HTTP, SQL) | + +### Subsystems + +**Job execution environments:** + +| Subsystem | Purpose | +|-----------|---------| +| **QINTER** | Interactive jobs | +| **QBATCH** | Batch processing | +| **QSYSWRK** | System work | +| **QSERVER** | File serving | +| **QUSRWRK** | User server jobs | + +### Job Queues + +```cl +SBMJOB CMD(CALL NIGHTBATCH) JOB(NIGHTLY) JOBQ(QBATCH) +``` + +## Security + +### Object-Level Security + +**Granular permissions on every object:** + +| Authority | Description | +|-----------|-------------| +| **\*ALL** | Full control | +| **\*CHANGE** | Read, add, update, delete | +| **\*USE** | Read and execute | +| **\*EXCLUDE** | No access | + +```cl +GRTOBJAUT OBJ(CUSTMAST) OBJTYPE(*FILE) USER(WEBUSER) AUT(*USE) +``` + +### User Profiles + +```cl +CRTUSRPRF USRPRF(NEWUSER) PASSWORD(temp123) + USRCLS(*USER) SPCAUT(*NONE) +``` + +### Security Levels + +| Level | Description | +|-------|-------------| +| **10** | No security (not recommended) | +| **20** | Password security | +| **30** | Resource security | +| **40** | Operating system integrity | +| **50** | Enhanced integrity (C2) | + +## Modern IBM i + +### ILE (Integrated Language Environment) + +**Modern program model** — Procedures, modules, service programs. + +``` +┌─────────────────────────────────────┐ +│ Service Program │ +│ ┌─────────┐ ┌─────────┐ │ +│ │ Module │ │ Module │ │ +│ │ (RPG) │ │ (C) │ │ +│ └─────────┘ └─────────┘ │ +└─────────────────────────────────────┘ +``` + +### PASE (Portable Application Solutions Environment) + +**AIX runtime on IBM i** — Run Unix/Linux binaries. + +- Python, Node.js, PHP +- Open-source packages +- SSH, Git, compilers + +### Web Integration + +| Technology | Purpose | +|------------|---------| +| **IWS (Integrated Web Services)** | REST/SOAP from RPG | +| **Apache** | Web server (PASE) | +| **Node.js** | Modern web apps | +| **PHP** | Web scripting | + +### Database Modernization + +```sql +-- JSON support +SELECT JSON_OBJECT('name': CUSTNAME, 'balance': BALANCE) +FROM CUSTMAST; + +-- Temporal tables +CREATE TABLE CUSTMAST ( + ... + SYS_START TIMESTAMP(12) GENERATED ALWAYS AS ROW BEGIN, + SYS_END TIMESTAMP(12) GENERATED ALWAYS AS ROW END, + PERIOD SYSTEM_TIME (SYS_START, SYS_END) +) WITH SYSTEM VERSIONING; +``` + +## Comparison with Mainframes + +| Aspect | IBM i (AS/400) | z/OS (Mainframe) | +|--------|----------------|------------------| +| **Target Market** | SMB to large enterprise | Large enterprise | +| **Database** | Integrated DB2 | Separate DB2 | +| **Complexity** | Simpler, integrated | More complex | +| **Primary Language** | RPG | COBOL | +| **Transaction Volume** | Millions/day | Billions/day | +| **Cost** | Lower | Higher | +| **Learning Curve** | Easier | Steeper | + +## When to Use IBM i + +### Strengths + +| Strength | Rationale | +|----------|-----------| +| **Reliability** | 99.99%+ uptime common | +| **Integration** | OS, DB, security unified | +| **Backwards Compatibility** | 35+ years of code runs | +| **Security** | Object-level, no root exploits | +| **TCO** | Lower admin overhead | + +### Considerations + +| Consideration | Impact | +|---------------|--------| +| **Skills Gap** | Aging workforce | +| **Perception** | "Legacy" stigma | +| **Modern Tooling** | Catching up | +| **Cloud Options** | Limited (IBM Power VS) | + +### Best For + +- **ERP Systems** — Manufacturing, distribution +- **Financial Applications** — Banks, credit unions +- **Healthcare** — Hospital systems +- **Retail** — POS, inventory + +## Related + +- [[RPG]] — Primary IBM i programming language +- [[COBOL]] — Also runs on IBM i +- [[Mainframes]] — IBM's larger systems +- [[Languages MOC]] — All language references diff --git a/Languages/Ada.md b/Languages/Ada.md new file mode 100644 index 0000000..4498994 --- /dev/null +++ b/Languages/Ada.md @@ -0,0 +1,382 @@ +--- +title: Ada +aliases: + - Ada Programming Language + - Ada 2012 + - Ada 2022 + - SPARK Ada +tags: + - language + - legacy + - safety-critical + - defense + - aerospace +type: reference +status: complete +created: "2025-12-16" +--- + +# Ada + +Strongly-typed, safety-focused language designed for mission-critical and real-time systems where failure is not an option. + +## Overview + +| Aspect | Details | +|--------|---------| +| **Paradigm** | Procedural, OOP, concurrent, contract-based | +| **Typing** | Static, strong, nominal | +| **Memory Model** | Manual + controlled types + optional GC | +| **First Appeared** | 1980 (DoD mandate, Jean Ichbiah) | +| **Current Standard** | Ada 2022 | +| **Primary Use Cases** | Aerospace, defense, rail, medical, avionics | +| **Named After** | Ada Lovelace, first programmer | +| **Key Feature** | Built-in concurrency and contracts | + +## Core Concepts + +### Program Structure + +```ada +with Ada.Text_IO; use Ada.Text_IO; + +procedure Hello is + Name : String(1..50); + Last : Natural; +begin + Put("Enter your name: "); + Get_Line(Name, Last); + Put_Line("Hello, " & Name(1..Last) & "!"); +end Hello; +``` + +### Strong Typing + +**Distinct Types** — Same underlying representation, different types: + +```ada +type Meters is new Float; +type Feet is new Float; + +Distance_M : Meters := 100.0; +Distance_F : Feet := 328.0; + +-- Distance_M := Distance_F; -- Compile error! Type mismatch +Distance_M := Meters(Distance_F * 0.3048); -- Explicit conversion +``` + +### Subtypes and Ranges + +```ada +type Day_Number is range 1 .. 31; +type Month_Number is range 1 .. 12; +type Year_Number is range 1900 .. 2100; + +subtype Working_Day is Day_Number range 1 .. 5; +subtype Percentage is Float range 0.0 .. 100.0; + +-- Runtime constraint check +Today : Day_Number := 32; -- Raises Constraint_Error +``` + +### Records + +```ada +type Date is record + Day : Day_Number; + Month : Month_Number; + Year : Year_Number; +end record; + +type Person is record + Name : String(1..50); + Birth : Date; + Is_Active : Boolean := True; +end record; + +Employee : Person := ( + Name => "Alice Smith" & (12..50 => ' '), + Birth => (Day => 15, Month => 6, Year => 1990), + Is_Active => True +); +``` + +### Arrays + +```ada +type Vector is array (1 .. 3) of Float; +type Matrix is array (1 .. 3, 1 .. 3) of Float; + +-- Unconstrained array type +type Int_Array is array (Positive range <>) of Integer; + +Numbers : Int_Array(1..100); +Identity : constant Matrix := ( + (1.0, 0.0, 0.0), + (0.0, 1.0, 0.0), + (0.0, 0.0, 1.0) +); +``` + +## Packages + +### Package Specification (Interface) + +```ada +-- math_utils.ads +package Math_Utils is + type Complex is private; + + function Create(Re, Im : Float) return Complex; + function "+"(Left, Right : Complex) return Complex; + function Magnitude(C : Complex) return Float; + +private + type Complex is record + Re, Im : Float; + end record; +end Math_Utils; +``` + +### Package Body (Implementation) + +```ada +-- math_utils.adb +with Ada.Numerics.Elementary_Functions; +use Ada.Numerics.Elementary_Functions; + +package body Math_Utils is + function Create(Re, Im : Float) return Complex is + begin + return (Re => Re, Im => Im); + end Create; + + function "+"(Left, Right : Complex) return Complex is + begin + return (Re => Left.Re + Right.Re, + Im => Left.Im + Right.Im); + end "+"; + + function Magnitude(C : Complex) return Float is + begin + return Sqrt(C.Re**2 + C.Im**2); + end Magnitude; +end Math_Utils; +``` + +## Concurrency (Tasks) + +### Built-In Concurrency + +```ada +task type Worker is + entry Start(Job_ID : Integer); + entry Stop; +end Worker; + +task body Worker is + ID : Integer; +begin + accept Start(Job_ID : Integer) do + ID := Job_ID; + end Start; + + loop + select + accept Stop; + exit; + or + delay 1.0; + Put_Line("Worker" & ID'Image & " working..."); + end select; + end loop; +end Worker; +``` + +### Protected Objects + +**Thread-Safe Data:** + +```ada +protected type Shared_Counter is + procedure Increment; + procedure Decrement; + function Value return Integer; +private + Count : Integer := 0; +end Shared_Counter; + +protected body Shared_Counter is + procedure Increment is + begin + Count := Count + 1; + end Increment; + + procedure Decrement is + begin + Count := Count - 1; + end Decrement; + + function Value return Integer is + begin + return Count; + end Value; +end Shared_Counter; +``` + +## Design by Contract + +### Pre/Post Conditions (Ada 2012+) + +```ada +function Divide(A, B : Float) return Float + with Pre => B /= 0.0, + Post => Divide'Result * B = A; + +procedure Push(S : in out Stack; Item : Element) + with Pre => not Is_Full(S), + Post => not Is_Empty(S) and Top(S) = Item; +``` + +### Type Invariants + +```ada +type Account is private + with Type_Invariant => Balance(Account) >= 0.0; +``` + +### Subtype Predicates + +```ada +subtype Even is Integer + with Dynamic_Predicate => Even mod 2 = 0; + +subtype Valid_Email is String + with Dynamic_Predicate => + (for some I in Valid_Email'Range => Valid_Email(I) = '@'); +``` + +## SPARK Ada + +### Formal Verification Subset + +```ada +package Stack + with SPARK_Mode +is + Max_Size : constant := 100; + type Stack is private; + + function Is_Empty(S : Stack) return Boolean; + function Is_Full(S : Stack) return Boolean; + + procedure Push(S : in out Stack; Item : Integer) + with Pre => not Is_Full(S), + Post => not Is_Empty(S); + + procedure Pop(S : in out Stack; Item : out Integer) + with Pre => not Is_Empty(S), + Post => not Is_Full(S); + +private + type Stack is record + Data : array (1 .. Max_Size) of Integer; + Top : Natural := 0; + end record; +end Stack; +``` + +**SPARK Guarantees:** + +- No runtime errors (proven statically) +- No undefined behavior +- No data races +- Functional correctness proofs + +## Ada Evolution + +| Version | Year | Key Features | +|---------|------|--------------| +| **Ada 83** | 1983 | Original DoD standard | +| **Ada 95** | 1995 | OOP, protected types, child packages | +| **Ada 2005** | 2005 | Interfaces, containers, scheduling | +| **Ada 2012** | 2012 | Contracts, expressions, iterators | +| **Ada 2022** | 2022 | Parallel blocks, delta aggregates, 'Image | + +## Real-World Usage + +| Domain | Examples | +|--------|----------| +| **Aviation** | Boeing 777 fly-by-wire, Airbus | +| **Space** | Ariane rockets, ISS, Mars missions | +| **Rail** | Train control systems (TGV, metros) | +| **Defense** | Military systems worldwide | +| **Medical** | Life-critical devices | +| **Finance** | Trading systems | + +## Tooling + +| Tool | Purpose | +|------|---------| +| **GNAT** | Free Ada compiler (GCC-based) | +| **GNAT Studio** | IDE for Ada/SPARK | +| **GNATprove** | SPARK formal verification | +| **AdaCore** | Commercial support | +| **Alire** | Package manager | + +## Comparison with Alternatives + +| Aspect | Ada | Rust | C++ | Java | +|--------|-----|------|-----|------| +| **Memory Safety** | ✅ Strong typing | ✅ Borrow checker | ❌ Manual | ✅ GC | +| **Concurrency** | ✅ Native tasks | ✅ Ownership | ⚠️ Libraries | ✅ Threads | +| **Contracts** | ✅ Native | ⚠️ Limited | ⚠️ Assertions | ⚠️ Assertions | +| **Formal Verification** | ✅ SPARK | ⚠️ Tools exist | ⚠️ Limited | ⚠️ Limited | +| **Real-Time** | ✅ Ravenscar profile | ⚠️ Possible | ⚠️ Possible | ❌ GC pauses | +| **Learning Curve** | ⚠️ Steep | ⚠️ Steep | ❌ Very steep | ✅ Moderate | +| **Community Size** | ⚠️ Small | ✅ Growing | ✅ Large | ✅ Large | + +## When to Use Ada + +### Strengths + +| Strength | Rationale | +|----------|-----------| +| **Safety** | Catches errors at compile time | +| **Reliability** | Proven in mission-critical systems | +| **Readability** | Verbose but clear | +| **Concurrency** | First-class language support | +| **Long-Term Maintenance** | 40+ year codebases | +| **Certification** | DO-178C, IEC 61508 compliance | + +### Considerations + +| Consideration | Impact | +|---------------|--------| +| **Verbosity** | More code than modern languages | +| **Small Community** | Fewer libraries and resources | +| **Learning Curve** | Complex type system | +| **Niche Market** | Limited outside safety-critical | + +### Best For + +- **Avionics** — Flight control systems +- **Space Systems** — Satellites, rockets +- **Rail** — Signaling, train control +- **Medical Devices** — Life-critical equipment +- **Defense** — Military applications +- **High-Integrity Systems** — Where failure costs lives + +**Avoid For:** + +- Web applications +- Mobile apps +- Rapid prototyping +- Startups (unless safety-critical) + +## Related + +- [[Rust]] — Modern systems language with safety focus +- [[Pascal]] — Syntactic ancestor +- [[C++]] — Alternative for systems programming +- [[Languages MOC]] — Overview of all language references diff --git a/Languages/BASIC.md b/Languages/BASIC.md new file mode 100644 index 0000000..518987b --- /dev/null +++ b/Languages/BASIC.md @@ -0,0 +1,290 @@ +--- +title: BASIC +aliases: + - Beginner's All-purpose Symbolic Instruction Code + - Visual Basic + - QuickBASIC +tags: + - language + - legacy + - educational + - scripting +type: reference +status: complete +created: "2025-12-16" +--- + +# BASIC + +Beginner-friendly language that introduced programming to millions, evolving from mainframe timesharing to the foundation of Visual Basic. + +## Overview + +| Aspect | Details | +|--------|---------| +| **Paradigm** | Procedural, event-driven (VB) | +| **Typing** | Dynamic (classic), Static (VB.NET) | +| **Memory Model** | Managed | +| **First Appeared** | 1964 (Kemeny & Kurtz, Dartmouth) | +| **Primary Use Cases** | Education, rapid prototyping, Windows apps | +| **Notable Variants** | GW-BASIC, QuickBASIC, Visual Basic, VBA | +| **Modern Successor** | VB.NET (still supported) | + +## Historical Significance + +| Era | Variant | Impact | +|-----|---------|--------| +| **1964-1980** | Dartmouth BASIC | Timesharing, education revolution | +| **1975-1985** | Microcomputer BASIC | Home computers (Apple II, C64, TRS-80) | +| **1985-1991** | QuickBASIC/Turbo BASIC | Structured programming, compilation | +| **1991-2002** | Visual Basic 1-6 | Windows GUI development revolution | +| **2002-present** | VB.NET | Modern .NET language | + +## Classic BASIC (Line Numbers) + +### Original Syntax + +```basic +10 REM This is a comment +20 PRINT "What is your name?" +30 INPUT NAME$ +40 PRINT "Hello, "; NAME$ +50 LET X = 0 +60 FOR I = 1 TO 10 +70 X = X + I +80 NEXT I +90 PRINT "Sum 1-10 ="; X +100 END +``` + +### Key Characteristics + +- **Line Numbers** — Required for every statement +- **GOTO** — Primary flow control +- **Single-Letter Variables** — Limited namespace +- **String Variables** — Suffix `$` (e.g., `NAME$`) +- **Arrays** — DIM statement + +### Common Statements + +| Statement | Purpose | +|-----------|---------| +| **PRINT** | Output to screen | +| **INPUT** | Read user input | +| **LET** | Variable assignment | +| **IF...THEN** | Conditional | +| **FOR...NEXT** | Loop | +| **GOTO** | Unconditional jump | +| **GOSUB...RETURN** | Subroutine call | +| **DIM** | Array declaration | +| **REM** | Comment | +| **DATA/READ** | Inline data | + +## Structured BASIC (QuickBASIC) + +### Modern Syntax + +```basic +' QuickBASIC / QBasic example +DECLARE SUB ProcessData (filename AS STRING) +DIM numbers(100) AS INTEGER +DIM total AS LONG + +INPUT "Enter filename: ", filename$ +CALL ProcessData(filename$) + +FOR i = 1 TO 100 + total = total + numbers(i) +NEXT i + +PRINT "Total:"; total + +SUB ProcessData (filename AS STRING) + OPEN filename FOR INPUT AS #1 + ' ... process file + CLOSE #1 +END SUB +``` + +### Improvements Over Classic + +- No line numbers required +- Named subroutines (SUB/FUNCTION) +- Structured loops (DO...LOOP, SELECT CASE) +- User-defined types (TYPE...END TYPE) +- Multiple modules +- Compiled executables + +## Visual Basic (VB6) + +### Event-Driven Programming + +```vb +' Form with button +Private Sub btnCalculate_Click() + Dim result As Double + result = Val(txtInput.Text) * 2 + lblOutput.Caption = "Result: " & result +End Sub + +Private Sub Form_Load() + txtInput.Text = "0" +End Sub +``` + +### Key Features + +| Feature | Description | +|---------|-------------| +| **RAD** | Rapid Application Development | +| **GUI Designer** | Drag-and-drop interface builder | +| **COM Integration** | ActiveX controls and automation | +| **Database** | ADO/DAO data access | +| **Events** | Click, Load, Change, etc. | + +### VB6 Impact + +- Democratized Windows development +- Millions of business applications +- VBA embedded in Office suite +- Estimated 3+ million developers at peak + +## VB.NET (Modern) + +### Current Syntax + +```vb +Imports System + +Module Program + Sub Main(args As String()) + Dim numbers = New List(Of Integer) From {1, 2, 3, 4, 5} + + Dim sum = numbers.Where(Function(n) n Mod 2 = 0).Sum() + Console.WriteLine($"Sum of evens: {sum}") + + Dim person = New Person With { + .Name = "Alice", + .Age = 30 + } + Console.WriteLine(person) + End Sub +End Module + +Public Class Person + Public Property Name As String + Public Property Age As Integer + + Public Overrides Function ToString() As String + Return $"{Name} ({Age})" + End Function +End Class +``` + +### .NET Features + +- Full OOP (classes, inheritance, interfaces) +- LINQ support +- Async/Await +- Generics +- .NET ecosystem access +- Cross-platform (.NET Core/5+) + +## VBA (Visual Basic for Applications) + +### Office Automation + +```vb +Sub FormatReport() + ' Excel VBA example + Dim ws As Worksheet + Set ws = ActiveSheet + + With ws.Range("A1:D1") + .Font.Bold = True + .Interior.Color = RGB(200, 200, 255) + End With + + Dim lastRow As Long + lastRow = ws.Cells(Rows.Count, 1).End(xlUp).Row + + ws.Range("D2:D" & lastRow).Formula = "=B2*C2" +End Sub +``` + +### VBA Hosts + +- Microsoft Excel +- Microsoft Word +- Microsoft Access +- Microsoft Outlook +- AutoCAD +- Many CAD/engineering tools + +## BASIC Family Tree + +``` +Dartmouth BASIC (1964) + ├── Microsoft BASIC (1975) + │ ├── GW-BASIC / BASICA + │ ├── QuickBASIC (1985) + │ │ └── QBasic (1991) + │ └── Visual Basic (1991) + │ ├── VBA (1993) + │ └── VB.NET (2002) + ├── BBC BASIC + ├── Commodore BASIC + ├── Applesoft BASIC + └── FreeBASIC (open source) +``` + +## Comparison with Alternatives + +| Aspect | BASIC/VB | Python | C# | +|--------|----------|--------|-----| +| **Learning Curve** | ✅ Very easy | ✅ Easy | ⚠️ Moderate | +| **Syntax Verbosity** | ⚠️ Verbose | ✅ Concise | ⚠️ Moderate | +| **Modern Relevance** | ⚠️ Declining | ✅ Growing | ✅ Strong | +| **RAD/GUI** | ✅ Historical strength | ⚠️ Frameworks needed | ✅ WinForms/WPF | +| **Office Integration** | ✅ VBA native | ⚠️ Libraries | ⚠️ COM interop | + +## When to Use + +### Strengths + +| Strength | Rationale | +|----------|-----------| +| **Learning** | Designed for beginners | +| **Office Automation** | VBA is the native scripting language | +| **Legacy Maintenance** | Millions of VB6/VBA apps in production | +| **RAD** | Quick prototyping | + +### Considerations + +| Consideration | Impact | +|---------------|--------| +| **Declining Community** | Fewer resources, packages | +| **VB6 End of Life** | No longer supported | +| **Performance** | Interpreted variants slower | +| **Modern Paradigms** | Limited functional programming | + +### Best For + +- **Office Automation** — VBA for Excel/Word macros +- **Legacy Systems** — Maintaining VB6 applications +- **Education** — Teaching programming concepts +- **Quick Prototypes** — Rapid Windows tools + +**Avoid For:** + +- New enterprise applications (use C#) +- Web development +- Mobile applications +- Performance-critical systems + +## Related + +- [[C Sharp|C#]] — Modern .NET alternative +- [[Python]] — Current beginner-friendly language +- [[COBOL]] — Another enterprise legacy language +- [[Languages MOC]] — Overview of all language references diff --git a/Languages/COBOL.md b/Languages/COBOL.md new file mode 100644 index 0000000..684c74c --- /dev/null +++ b/Languages/COBOL.md @@ -0,0 +1,256 @@ +--- +title: COBOL +aliases: + - Common Business-Oriented Language +tags: + - language + - legacy + - mainframe + - enterprise + - business +type: reference +status: complete +created: "2025-12-16" +--- + +# COBOL + +Verbose, English-like mainframe language powering critical financial and government systems for 60+ years. + +## Overview + +| Aspect | Details | +|--------|---------| +| **Paradigm** | Procedural, imperative | +| **Typing** | Static, strong | +| **Memory Model** | Managed by runtime | +| **First Appeared** | 1959 (Grace Hopper's team) | +| **Current Standard** | COBOL 2023 (ISO/IEC 1989:2023) | +| **Primary Use Cases** | Banking, insurance, government, batch processing | +| **Platforms** | IBM z/OS, AS/400, Unix, Linux, Windows | +| **Estimated Lines in Production** | 200+ billion | + +## Core Concepts + +### Program Structure + +**Divisions** — COBOL programs are organized into four mandatory divisions: + +| Division | Purpose | +|----------|---------| +| **IDENTIFICATION** | Program metadata (name, author, date) | +| **ENVIRONMENT** | Hardware/software configuration, file mappings | +| **DATA** | Variable and file definitions | +| **PROCEDURE** | Executable code (paragraphs and sections) | + +### Data Division + +**Hierarchical Data Definition** — Level numbers define structure. + +```cobol +01 CUSTOMER-RECORD. + 05 CUSTOMER-ID PIC 9(8). + 05 CUSTOMER-NAME. + 10 FIRST-NAME PIC X(20). + 10 LAST-NAME PIC X(30). + 05 ACCOUNT-BALANCE PIC S9(7)V99. +``` + +**PICTURE Clause (PIC)** — Defines data format: + +- `9` — Numeric digit +- `X` — Alphanumeric character +- `V` — Implied decimal point +- `S` — Sign indicator +- `(n)` — Repetition count + +### File Handling + +**Record-Oriented I/O** — Native support for sequential, indexed, and relative files. + +```cobol +SELECT CUSTOMER-FILE ASSIGN TO "CUSTMAST" + ORGANIZATION IS INDEXED + ACCESS MODE IS RANDOM + RECORD KEY IS CUSTOMER-ID. +``` + +**File Operations:** + +- `OPEN INPUT/OUTPUT/I-O/EXTEND` +- `READ ... INTO ... AT END` +- `WRITE ... FROM ...` +- `REWRITE` — Update existing record +- `DELETE` — Remove record +- `CLOSE` + +### Procedure Division + +**English-Like Syntax** — Designed for business readability. + +```cobol +PROCEDURE DIVISION. + PERFORM INITIALIZE-PROGRAM + PERFORM PROCESS-RECORDS UNTIL END-OF-FILE + PERFORM FINALIZE-PROGRAM + STOP RUN. + +PROCESS-RECORDS. + READ CUSTOMER-FILE INTO CUSTOMER-RECORD + AT END SET END-OF-FILE TO TRUE + END-READ + IF NOT END-OF-FILE + PERFORM CALCULATE-INTEREST + PERFORM UPDATE-BALANCE + END-IF. +``` + +### COPYBOOK + +**Reusable Data Definitions** — Shared record layouts across programs. + +```cobol +COPY CUSTREC. +``` + +Promotes consistency in multi-program systems. + +## COBOL Evolution + +| Version | Year | Key Features | +|---------|------|--------------| +| **COBOL-60** | 1960 | Original specification | +| **COBOL-68** | 1968 | First ANSI standard | +| **COBOL-74** | 1974 | Structured programming | +| **COBOL-85** | 1985 | END-IF, inline PERFORM, nested programs | +| **COBOL 2002** | 2002 | OOP, Unicode, pointers | +| **COBOL 2014** | 2014 | XML/JSON support, dynamic tables | +| **COBOL 2023** | 2023 | Modern features, deprecations | + +## Mainframe Environment + +### JCL Integration + +COBOL programs run via [[JCL]] job streams: + +```jcl +//STEP1 EXEC PGM=CUSTUPDT +//CUSTFILE DD DSN=PROD.CUSTOMER.MASTER,DISP=SHR +//SYSOUT DD SYSOUT=* +``` + +### Common Tools + +| Tool | Purpose | +|------|---------| +| **CICS** | Online transaction processing | +| **IMS** | Hierarchical database system | +| **DB2** | Relational database | +| **VSAM** | File access method | +| **Batch** | Scheduled job processing | + +### CICS (Online) + +**Transaction Processing** — Real-time, terminal-driven programs. + +- `EXEC CICS RECEIVE` — Get terminal input +- `EXEC CICS SEND` — Display output +- `EXEC CICS READ` — Database access +- `EXEC CICS RETURN` — End transaction + +## Modern COBOL + +### Object-Oriented COBOL + +COBOL 2002+ supports classes and methods: + +```cobol +CLASS-ID. Customer INHERITS Base. + DATA DIVISION. + WORKING-STORAGE SECTION. + 01 customer-id PIC 9(8). + + METHOD-ID. GetBalance. + ... + END METHOD GetBalance. +END CLASS Customer. +``` + +### Modern Tooling + +| Tool | Description | +|------|-------------| +| **GnuCOBOL** | Free, open-source compiler | +| **Micro Focus** | Enterprise IDE, debugging | +| **IBM Developer for z/OS** | Eclipse-based mainframe IDE | +| **Raincode** | .NET COBOL compiler | + +### Cloud and Modernization + +- **AWS Mainframe Modernization** — Run COBOL on cloud +- **Micro Focus Enterprise Server** — Containerized mainframe +- **Refactoring to Java/.NET** — Automated conversion tools +- **API Wrappers** — Expose COBOL as REST services + +## Comparison with Alternatives + +| Aspect | COBOL | Java | Python | +|--------|-------|------|--------| +| **Decimal Precision** | ✅ Native fixed-point | ⚠️ BigDecimal needed | ⚠️ Decimal module | +| **Verbosity** | ❌ Extremely verbose | ⚠️ Moderate | ✅ Concise | +| **Batch Processing** | ✅ Native file handling | ⚠️ Framework needed | ⚠️ Library needed | +| **Learning Curve** | ⚠️ Unusual syntax | ✅ Widely taught | ✅ Beginner-friendly | +| **Job Market** | ⚠️ Niche but lucrative | ✅ Abundant | ✅ Abundant | +| **Modern Tooling** | ⚠️ Limited | ✅ Excellent | ✅ Excellent | + +## When to Use COBOL + +### Strengths + +| Strength | Rationale | +|----------|-----------| +| **Decimal Arithmetic** | Native fixed-point math, no floating-point errors | +| **Batch Processing** | Built for high-volume data processing | +| **Stability** | Programs run unchanged for decades | +| **Mainframe Integration** | Native to z/OS ecosystem | +| **Transaction Volume** | Handles billions of transactions daily | + +### Considerations + +| Consideration | Impact | +|---------------|--------| +| **Verbose Syntax** | More lines of code for simple operations | +| **Limited Talent Pool** | Aging workforce, fewer new learners | +| **Tooling Gap** | Modern development practices harder to adopt | +| **Testing Complexity** | Mainframe testing requires specialized setup | + +### Best For + +- **Banking Systems** — Core banking, transaction processing +- **Insurance** — Policy administration, claims processing +- **Government** — Tax systems, social security, benefits +- **Batch Processing** — High-volume overnight processing +- **Legacy Maintenance** — Existing COBOL systems + +**Avoid For:** + +- New greenfield projects (unless mainframe-specific) +- Web applications +- Mobile development +- Real-time analytics + +## Industry Impact + +| Metric | Value | +|--------|-------| +| **Daily Transactions** | 95% of ATM transactions | +| **Banking** | 80%+ of global banking systems | +| **Fortune 500** | 70%+ use COBOL systems | +| **Lines in Production** | More than any other language | + +## Related + +- [[JCL]] — Job Control Language for running COBOL programs +- [[Fortran]] — Contemporary scientific computing language +- [[Java]] — Common modernization target +- [[Languages MOC]] — Overview of all language references diff --git a/Languages/Fortran.md b/Languages/Fortran.md new file mode 100644 index 0000000..81b6633 --- /dev/null +++ b/Languages/Fortran.md @@ -0,0 +1,316 @@ +--- +title: Fortran +aliases: + - FORTRAN + - Formula Translation +tags: + - language + - legacy + - scientific + - numerical + - hpc +type: reference +status: complete +created: "2025-12-16" +--- + +# Fortran + +The oldest high-level programming language, still dominant in scientific computing and high-performance numerical simulation. + +## Overview + +| Aspect | Details | +|--------|---------| +| **Paradigm** | Procedural, array-oriented, OOP (2003+) | +| **Typing** | Static, strong | +| **Memory Model** | Automatic arrays, allocatable, pointers | +| **First Appeared** | 1957 (John Backus, IBM) | +| **Current Standard** | Fortran 2023 | +| **Primary Use Cases** | Scientific simulation, HPC, numerical computing | +| **Compilers** | gfortran, Intel Fortran, NVIDIA HPC, NAG | +| **Key Domains** | Physics, chemistry, weather, climate, CFD | + +## Core Concepts + +### Array-First Design + +**Native Multidimensional Arrays** — First-class support for numerical computing. + +```fortran +real :: matrix(100, 100) +real :: vector(1000) + +! Whole-array operations +matrix = 0.0 +vector = vector * 2.0 + +! Array slicing +matrix(1:10, :) = 1.0 +``` + +**Column-Major Order** — Arrays stored column-first (opposite of C). + +### Array Operations + +**Intrinsic Functions** — Operate on entire arrays: + +```fortran +sum(array) ! Sum all elements +maxval(array) ! Maximum value +matmul(A, B) ! Matrix multiplication +transpose(matrix) ! Matrix transpose +dot_product(a, b) ! Vector dot product +``` + +**WHERE Statement** — Conditional array operations: + +```fortran +where (temperature > 100.0) + state = "gas" +elsewhere + state = "liquid" +end where +``` + +### Program Structure + +```fortran +program heat_simulation + implicit none + real :: temperature(100, 100) + integer :: i, j + + call initialize(temperature) + call simulate(temperature, 1000) + call output_results(temperature) + +end program heat_simulation + +subroutine initialize(grid) + real, intent(out) :: grid(:,:) + grid = 20.0 ! Room temperature +end subroutine initialize +``` + +### Modules + +**Encapsulation and Namespacing** — Modern code organization. + +```fortran +module physics_constants + implicit none + real, parameter :: pi = 3.14159265359 + real, parameter :: c = 299792458.0 ! Speed of light + real, parameter :: G = 6.67430e-11 ! Gravitational constant +end module physics_constants + +program simulation + use physics_constants + ! pi, c, G now available +end program +``` + +### Derived Types + +**User-Defined Types** — Struct-like data structures. + +```fortran +type :: particle + real :: position(3) + real :: velocity(3) + real :: mass +end type particle + +type(particle) :: p +p%mass = 1.0 +p%position = [0.0, 0.0, 0.0] +``` + +## Fortran Evolution + +| Version | Year | Key Features | +|---------|------|--------------| +| **FORTRAN** | 1957 | First high-level language, arithmetic expressions | +| **FORTRAN IV** | 1962 | Logical IF, subprograms | +| **FORTRAN 77** | 1977 | Structured IF-THEN-ELSE, CHARACTER type | +| **Fortran 90** | 1990 | Free-form, modules, array syntax, allocatable | +| **Fortran 95** | 1995 | FORALL, pure/elemental procedures | +| **Fortran 2003** | 2003 | OOP, C interoperability, IEEE floating-point | +| **Fortran 2008** | 2008 | Coarrays, submodules, DO CONCURRENT | +| **Fortran 2018** | 2018 | Enhanced coarrays, C interop improvements | +| **Fortran 2023** | 2023 | Generics (templates), enumeration types | + +**Naming Convention:** + +- FORTRAN (all caps) — Versions before 90 +- Fortran (capitalized) — Modern versions (90+) + +## Parallel Computing + +### Coarrays + +**Built-In Parallelism** — Native distributed memory model. + +```fortran +real :: local_sum[*] ! Coarray - one per image +real :: global_sum + +local_sum = sum(my_data) +sync all +if (this_image() == 1) then + global_sum = sum(local_sum[1:num_images()]) +end if +``` + +### DO CONCURRENT + +**Parallel Loops** — Compiler-optimized parallel iteration. + +```fortran +do concurrent (i = 1:n, j = 1:m) + result(i,j) = compute(data(i,j)) +end do +``` + +### OpenMP Integration + +**Directive-Based Parallelism:** + +```fortran +!$omp parallel do +do i = 1, n + a(i) = b(i) + c(i) +end do +!$omp end parallel do +``` + +### MPI + +**Message Passing** — Industry standard for distributed computing. + +```fortran +use mpi +call MPI_Init(ierr) +call MPI_Comm_rank(MPI_COMM_WORLD, rank, ierr) +call MPI_Send(data, count, MPI_REAL, dest, tag, MPI_COMM_WORLD, ierr) +call MPI_Finalize(ierr) +``` + +## Scientific Libraries + +| Library | Domain | +|---------|--------| +| **BLAS** | Basic Linear Algebra Subprograms | +| **LAPACK** | Linear algebra (eigenvalues, SVD, systems) | +| **ScaLAPACK** | Distributed LAPACK | +| **FFTW** | Fast Fourier transforms | +| **PETSc** | Parallel solvers | +| **NetCDF** | Scientific data format I/O | +| **HDF5** | Hierarchical data format | + +**Performance Note:** Fortran's array model allows compilers to generate highly optimized SIMD code. + +## Compilers + +| Compiler | Vendor | Notes | +|----------|--------|-------| +| **gfortran** | GNU | Free, widely available | +| **ifort/ifx** | Intel | Highly optimized for Intel CPUs | +| **nvfortran** | NVIDIA | GPU offloading support | +| **NAG Fortran** | NAG | Excellent standards compliance | +| **Flang** | LLVM | Modern LLVM-based compiler | + +**Recommended Flags (gfortran):** + +- `-O2` / `-O3` — Optimization +- `-Wall -Wextra` — Warnings +- `-fcheck=all` — Runtime checks (debug) +- `-fopenmp` — OpenMP support +- `-std=f2018` — Standard compliance + +## Comparison with Alternatives + +| Aspect | Fortran | C/C++ | Python (NumPy) | Julia | +|--------|---------|-------|----------------|-------| +| **Array Syntax** | ✅ Native, elegant | ❌ Manual loops | ✅ NumPy arrays | ✅ Native arrays | +| **Performance** | ✅ Excellent | ✅ Excellent | ⚠️ C extensions | ✅ JIT compiled | +| **Parallel** | ✅ Coarrays, OpenMP | ⚠️ Libraries | ⚠️ Limited | ✅ Native tasks | +| **Legacy Code** | ✅ Decades of libraries | ⚠️ Some | ❌ None | ❌ None | +| **Learning Curve** | ⚠️ Unusual syntax | ⚠️ Complex | ✅ Easy | ✅ Easy | +| **Modern Features** | ⚠️ Catching up | ✅ Rapid evolution | ✅ Extensive | ✅ Modern design | + +### Key Distinctions + +**Fortran vs C/C++:** + +- Fortran arrays are bounds-checked (optionally), C is not +- Fortran uses column-major, C uses row-major +- Fortran has no pointers in the C sense (safer) +- Fortran's complex number type is native + +**Fortran vs Python/NumPy:** + +- Fortran compiles to native code (much faster) +- NumPy backends often call Fortran (BLAS/LAPACK) +- Python better for prototyping, Fortran for production HPC + +**Fortran vs Julia:** + +- Julia is newer with modern syntax +- Fortran has 60+ years of optimized libraries +- Julia's JIT adds startup overhead + +## When to Use Fortran + +### Strengths + +| Strength | Rationale | +|----------|-----------| +| **Numerical Performance** | Compilers generate highly optimized numerical code | +| **Array Operations** | First-class multidimensional arrays | +| **Scientific Libraries** | BLAS, LAPACK, decades of validated code | +| **HPC Ecosystem** | Native parallelism, MPI/OpenMP integration | +| **Backwards Compatibility** | Old code still compiles and runs | + +### Considerations + +| Consideration | Impact | +|---------------|--------| +| **Syntax** | Different from mainstream languages | +| **String Handling** | Historically awkward | +| **General Programming** | Not suited for web, apps, GUIs | +| **Declining Popularity** | Smaller community than Python/C++ | + +### Best For + +- **Climate Modeling** — Weather prediction, earth sciences +- **Computational Physics** — Particle simulations, quantum mechanics +- **Computational Chemistry** — Molecular dynamics, DFT +- **Aerospace/CFD** — Fluid dynamics, structural analysis +- **Financial Modeling** — Risk analysis, derivatives pricing +- **HPC Centers** — Supercomputer workloads + +**Avoid For:** + +- Web development +- Mobile applications +- System programming +- General application development + +## Industry Usage + +| Domain | Examples | +|--------|----------| +| **Weather** | ECMWF, NOAA models | +| **Physics** | CERN simulations | +| **Aerospace** | NASA, Boeing CFD | +| **Energy** | Nuclear reactor simulations | +| **Finance** | Quantitative modeling | + +## Related + +- [[COBOL]] — Contemporary business-oriented language +- [[C]] — Systems programming alternative +- [[Python]] — Modern scientific computing (with NumPy) +- [[Languages MOC]] — Overview of all language references diff --git a/Languages/JCL.md b/Languages/JCL.md new file mode 100644 index 0000000..ce02f4a --- /dev/null +++ b/Languages/JCL.md @@ -0,0 +1,342 @@ +--- +title: JCL +aliases: + - Job Control Language + - z/OS JCL + - MVS JCL +tags: + - language + - legacy + - mainframe + - scripting + - batch +type: reference +status: complete +created: "2025-12-16" +--- + +# JCL + +Job Control Language for IBM mainframes — defines jobs, allocates resources, and controls program execution in batch processing environments. + +## Overview + +| Aspect | Details | +|--------|---------| +| **Type** | Job control / scripting language | +| **Platform** | IBM z/OS, MVS, OS/390 | +| **First Appeared** | 1964 (OS/360) | +| **Primary Use** | Batch job submission, resource allocation | +| **Syntax Style** | Fixed-format, positional/keyword parameters | +| **File Extension** | None (members in PDS) | +| **Modern Equivalent** | Shell scripts, workflow managers | + +## Core Concepts + +### Statement Types + +JCL consists of three primary statement types: + +| Statement | Prefix | Purpose | +|-----------|--------|---------| +| **JOB** | `//name JOB` | Defines job, sets accounting | +| **EXEC** | `//name EXEC` | Executes program or procedure | +| **DD** | `//name DD` | Data Definition — allocates datasets | + +**Comment:** `//*` starts a comment line. + +### Basic Job Structure + +```jcl +//MYJOB JOB (ACCT),'PROGRAMMER NAME',CLASS=A,MSGCLASS=X +//* +//* This job compiles and runs a COBOL program +//* +//STEP1 EXEC PGM=IGYCRCTL +//SYSPRINT DD SYSOUT=* +//SYSIN DD DSN=USER.SOURCE(PROGRAM1),DISP=SHR +//SYSLIN DD DSN=&&LOADSET,DISP=(MOD,PASS), +// UNIT=SYSDA,SPACE=(TRK,(3,3)) +//* +//STEP2 EXEC PGM=IEWL +//SYSLIN DD DSN=&&LOADSET,DISP=(OLD,DELETE) +//SYSLMOD DD DSN=USER.LOAD(PROGRAM1),DISP=SHR +//SYSPRINT DD SYSOUT=* +//* +//STEP3 EXEC PGM=PROGRAM1 +//INFILE DD DSN=PROD.INPUT.DATA,DISP=SHR +//OUTFILE DD DSN=PROD.OUTPUT.DATA,DISP=(NEW,CATLG,DELETE), +// UNIT=SYSDA,SPACE=(CYL,(10,5)), +// DCB=(RECFM=FB,LRECL=80,BLKSIZE=8000) +//SYSOUT DD SYSOUT=* +``` + +### JOB Statement + +**Identifies and Configures the Job:** + +```jcl +//PAYROLL JOB (ACCT123,DEPT456),'JOHN SMITH', +// CLASS=A, +// MSGCLASS=X, +// MSGLEVEL=(1,1), +// NOTIFY=&SYSUID, +// TIME=(5,30), +// REGION=0M +``` + +| Parameter | Purpose | +|-----------|---------| +| **CLASS** | Job class (priority/resource pool) | +| **MSGCLASS** | Output class for job messages | +| **MSGLEVEL** | Verbosity of JCL listing | +| **NOTIFY** | User to notify on completion | +| **TIME** | Maximum CPU time (minutes,seconds) | +| **REGION** | Memory allocation | + +### EXEC Statement + +**Executes Programs or Procedures:** + +```jcl +//STEP1 EXEC PGM=IEFBR14 +//STEP2 EXEC PGM=SORT,PARM='SIZE=MAX' +//STEP3 EXEC PROC=COBCL +``` + +| Parameter | Purpose | +|-----------|---------| +| **PGM** | Program name to execute | +| **PROC** | Procedure name to invoke | +| **PARM** | Parameters passed to program | +| **COND** | Conditional execution | +| **TIME** | Step time limit | +| **REGION** | Step memory limit | + +### DD Statement + +**Data Definition — The Heart of JCL:** + +```jcl +//INPUT DD DSN=PROD.CUSTOMER.MASTER,DISP=SHR +//OUTPUT DD DSN=PROD.CUSTOMER.BACKUP, +// DISP=(NEW,CATLG,DELETE), +// UNIT=SYSDA, +// SPACE=(CYL,(100,50),RLSE), +// DCB=(RECFM=FB,LRECL=200,BLKSIZE=20000) +//SYSOUT DD SYSOUT=* +//SYSIN DD * + Control card data goes here +/* +``` + +### DISP Parameter + +**Dataset Disposition** — Controls access and handling: + +```jcl +DISP=(status,normal-end,abnormal-end) +``` + +| Status | Meaning | +|--------|---------| +| **NEW** | Create new dataset | +| **OLD** | Exclusive access to existing | +| **SHR** | Shared read access | +| **MOD** | Append to existing | + +| Disposition | Meaning | +|-------------|---------| +| **DELETE** | Remove after use | +| **KEEP** | Retain without cataloging | +| **CATLG** | Catalog the dataset | +| **PASS** | Pass to next step | + +### SPACE Parameter + +**Disk Space Allocation:** + +```jcl +SPACE=(unit,(primary,secondary,directory),RLSE) +``` + +| Unit | Size | +|------|------| +| **TRK** | Tracks | +| **CYL** | Cylinders | +| **bytes** | Exact bytes | + +### DCB Parameter + +**Data Control Block** — Record format: + +```jcl +DCB=(RECFM=FB,LRECL=80,BLKSIZE=8000) +``` + +| Attribute | Values | +|-----------|--------| +| **RECFM** | F (fixed), V (variable), FB (fixed blocked), VB | +| **LRECL** | Logical record length | +| **BLKSIZE** | Physical block size | + +## Conditional Execution + +### COND Parameter + +**Skip Step Based on Return Codes:** + +```jcl +//STEP2 EXEC PGM=PROG2,COND=(4,LT) +``` + +Meaning: Skip if ANY prior return code is **less than** 4. + +| Operator | Meaning | +|----------|---------| +| **GT** | Greater than | +| **GE** | Greater than or equal | +| **EQ** | Equal | +| **NE** | Not equal | +| **LT** | Less than | +| **LE** | Less than or equal | + +### IF/THEN/ELSE (Modern) + +```jcl +// IF (STEP1.RC = 0) THEN +//STEP2 EXEC PGM=PROG2 +// ELSE +//STEP2A EXEC PGM=ERRORHANDLER +// ENDIF +``` + +## Procedures (PROCs) + +**Reusable JCL Templates:** + +### Cataloged Procedure + +```jcl +//COBCL PROC +//COMPILE EXEC PGM=IGYCRCTL +//SYSPRINT DD SYSOUT=* +//SYSIN DD DSN=&SOURCE,DISP=SHR +//SYSLIN DD DSN=&&OBJ,DISP=(,PASS) +//LINK EXEC PGM=IEWL +//SYSLIN DD DSN=&&OBJ,DISP=(OLD,DELETE) +//SYSLMOD DD DSN=&LOAD,DISP=SHR +// PEND +``` + +### Invoking with Overrides + +```jcl +//STEP1 EXEC PROC=COBCL,SOURCE='MY.SOURCE(PROG1)', +// LOAD='MY.LOADLIB(PROG1)' +//COMPILE.SYSPRINT DD SYSOUT=A +``` + +## Common Programs + +| Program | Purpose | +|---------|---------| +| **IEFBR14** | Do nothing (allocate/delete datasets) | +| **IEBGENER** | Copy sequential datasets | +| **IEBCOPY** | Copy/compress PDS members | +| **SORT** | Sort/merge data (DFSORT, Syncsort) | +| **IDCAMS** | VSAM utility (define, delete, repro) | +| **IKJEFT01** | TSO batch | + +### IEFBR14 Example (Create Dataset) + +```jcl +//CREATE EXEC PGM=IEFBR14 +//NEWFILE DD DSN=USER.NEW.DATASET, +// DISP=(NEW,CATLG,DELETE), +// UNIT=SYSDA, +// SPACE=(TRK,(10,5)), +// DCB=(RECFM=FB,LRECL=80,BLKSIZE=8000) +``` + +### SORT Example + +```jcl +//SORT EXEC PGM=SORT +//SORTIN DD DSN=INPUT.DATA,DISP=SHR +//SORTOUT DD DSN=OUTPUT.DATA,DISP=(NEW,CATLG) +//SYSIN DD * + SORT FIELDS=(1,10,CH,A,15,5,ZD,D) + INCLUDE COND=(20,2,CH,EQ,C'US') +/* +//SYSOUT DD SYSOUT=* +``` + +## Symbolic Parameters + +**Variable Substitution:** + +```jcl +//MYJOB JOB ... +// SET DATE=20251216 +// SET ENV=PROD +//STEP1 EXEC PGM=REPORT +//INPUT DD DSN=&ENV..DATA.&DATE,DISP=SHR +``` + +**Note:** Double periods (`..`) separate symbolic from literal text. + +## Common Pitfalls + +| Issue | Cause | Solution | +|-------|-------|----------| +| **JCL Error** | Syntax error, invalid parameter | Check column positions, spelling | +| **S806** | Program not found | Verify STEPLIB/JOBLIB | +| **S0C7** | Data exception | Invalid numeric data | +| **S0C4** | Protection exception | Memory/addressing error | +| **S913** | Security violation | Check RACF permissions | +| **S322** | Time exceeded | Increase TIME parameter | +| **SB37** | Out of space | Increase SPACE, compress | + +## Comparison with Modern Equivalents + +| Aspect | JCL | Bash/Shell | Kubernetes Jobs | +|--------|-----|------------|-----------------| +| **Resource Allocation** | DD statements | Volumes/mounts | Resource specs | +| **Conditional Execution** | COND/IF | `if`/`&&`/`||` | Init containers | +| **Parameterization** | Symbolic & | Environment vars | ConfigMaps | +| **Job Dependencies** | Return codes | Exit codes | Job dependencies | +| **Scheduling** | JES2/JES3 | Cron | CronJobs | + +## When to Use JCL + +### Strengths + +| Strength | Rationale | +|----------|-----------| +| **Resource Control** | Fine-grained dataset allocation | +| **Job Management** | Native batch scheduling | +| **COBOL Integration** | Standard for mainframe programs | +| **Audit Trail** | Comprehensive job logging | + +### Considerations + +| Consideration | Impact | +|---------------|--------| +| **Cryptic Syntax** | Steep learning curve | +| **Fixed Format** | Column-sensitive (mostly) | +| **Mainframe-Only** | No portability | +| **Verbose** | Simple tasks require many lines | + +### Best For + +- Running [[COBOL]] batch programs +- Mainframe data processing +- Enterprise batch scheduling +- Integration with CICS/DB2 + +## Related + +- [[COBOL]] — Primary language executed via JCL +- [[Fortran]] — Also runs on mainframes +- [[Languages MOC]] — Overview of all language references diff --git a/Languages/PL-I.md b/Languages/PL-I.md new file mode 100644 index 0000000..7aa39f7 --- /dev/null +++ b/Languages/PL-I.md @@ -0,0 +1,500 @@ +--- +title: PL/I +aliases: + - PL/1 + - Programming Language One +tags: + - language + - legacy + - mainframe + - ibm + - enterprise +type: reference +status: complete +created: "2025-12-16" +--- + +# PL/I + +IBM's "do everything" language from the 1960s, combining features from FORTRAN, COBOL, and ALGOL for scientific and business programming. + +## Overview + +| Aspect | Details | +|--------|---------| +| **Full Name** | Programming Language One | +| **First Released** | 1964 (IBM) | +| **Paradigm** | Procedural, structured | +| **Typing** | Static, with extensive conversions | +| **Platforms** | z/OS, z/VM, OS/2, Windows, Unix | +| **Primary Use** | Mainframe systems, scientific, business | +| **Design Goal** | One language for all purposes | + +## History and Context + +### Why PL/I Was Created + +**1960s Problem:** + +- FORTRAN — Scientific, poor I/O +- COBOL — Business, poor math +- ALGOL — Academic, limited adoption + +**IBM's Solution:** Create one language with everything. + +### Influence on Later Languages + +| Feature | PL/I (1964) | Later Adoption | +|---------|-------------|----------------| +| Exception handling | `ON` conditions | Java, C++, Python | +| Pointers | `POINTER` | C (1972) | +| Structures | `STRUCTURE` | C structs | +| Dynamic allocation | `ALLOCATE` | malloc/new | +| String operations | Built-in | Most languages | +| Preprocessor | `%` statements | C preprocessor | + +## Basic Syntax + +### Program Structure + +```pli +EXAMPLE: PROCEDURE OPTIONS(MAIN); + /* Variable declarations */ + DECLARE CUSTOMER_NAME CHARACTER(50); + DECLARE BALANCE DECIMAL FIXED(11,2); + DECLARE COUNT BINARY FIXED(31); + + /* Executable statements */ + CUSTOMER_NAME = 'JOHN SMITH'; + BALANCE = 1234.56; + COUNT = 0; + + PUT SKIP LIST('Customer:', CUSTOMER_NAME); + PUT SKIP LIST('Balance:', BALANCE); + +END EXAMPLE; +``` + +### Data Types + +#### Arithmetic Types + +```pli +/* Fixed-point decimal (like COBOL) */ +DECLARE PRICE DECIMAL FIXED(7,2); /* 99999.99 */ +DECLARE QUANTITY DECIMAL FIXED(5); /* 99999 */ + +/* Binary (integers) */ +DECLARE INDEX BINARY FIXED(15); /* 16-bit int */ +DECLARE COUNTER BINARY FIXED(31); /* 32-bit int */ + +/* Floating-point */ +DECLARE RATE DECIMAL FLOAT(16); /* Decimal float */ +DECLARE RESULT BINARY FLOAT(53); /* Binary float (double) */ + +/* Complex numbers */ +DECLARE Z COMPLEX BINARY FLOAT; +Z = 3 + 4I; +``` + +#### String Types + +```pli +/* Fixed-length strings */ +DECLARE NAME CHARACTER(30); +DECLARE CODE CHARACTER(5); + +/* Varying-length strings */ +DECLARE MESSAGE CHARACTER(100) VARYING; + +/* Bit strings */ +DECLARE FLAGS BIT(8); +DECLARE MASK BIT(32); +``` + +#### Structured Types + +```pli +DECLARE 1 CUSTOMER, + 2 CUST_ID DECIMAL FIXED(8), + 2 NAME, + 3 FIRST CHARACTER(20), + 3 LAST CHARACTER(30), + 2 ADDRESS, + 3 STREET CHARACTER(40), + 3 CITY CHARACTER(25), + 3 STATE CHARACTER(2), + 3 ZIP CHARACTER(10), + 2 BALANCE DECIMAL FIXED(11,2); + +/* Access */ +CUSTOMER.NAME.FIRST = 'JOHN'; +CUSTOMER.BALANCE = 1000.00; +``` + +### Arrays + +```pli +/* Single dimension */ +DECLARE SCORES(100) DECIMAL FIXED(5,2); + +/* Multi-dimensional */ +DECLARE MATRIX(10,10) BINARY FLOAT; + +/* With bounds */ +DECLARE TABLE(-5:5, 1:12) DECIMAL FIXED(7,2); + +/* Array operations */ +DECLARE A(100) FIXED, B(100) FIXED, C(100) FIXED; +C = A + B; /* Element-wise addition */ +C = A * 2; /* Scalar multiplication */ +``` + +## Control Structures + +### Conditionals + +```pli +/* IF statement */ +IF BALANCE > 0 THEN + PUT LIST('Credit'); +ELSE IF BALANCE < 0 THEN + PUT LIST('Debit'); +ELSE + PUT LIST('Zero'); + +/* SELECT (like CASE) */ +SELECT (GRADE); + WHEN ('A') PUT LIST('Excellent'); + WHEN ('B') PUT LIST('Good'); + WHEN ('C') PUT LIST('Average'); + OTHERWISE PUT LIST('Below Average'); +END; +``` + +### Loops + +```pli +/* DO loop (like FOR) */ +DO I = 1 TO 100; + TOTAL = TOTAL + ARRAY(I); +END; + +/* DO WHILE */ +DO WHILE (NOT END_OF_FILE); + READ FILE(INPUT) INTO(RECORD); + PROCESS(RECORD); +END; + +/* DO UNTIL */ +DO UNTIL (RESPONSE = 'Q'); + GET LIST(RESPONSE); +END; + +/* Iterative with increment */ +DO I = 1 TO 100 BY 2; /* Odd numbers */ + PUT LIST(I); +END; + +/* Multiple iteration */ +DO I = 1, 3, 5, 7 TO 20 BY 2; + PUT LIST(I); +END; +``` + +## Procedures and Functions + +### Internal Procedures + +```pli +MAIN: PROCEDURE OPTIONS(MAIN); + DECLARE RESULT DECIMAL FIXED(11,2); + + RESULT = CALCULATE_TAX(1000.00, 0.08); + PUT LIST('Tax:', RESULT); + + CALCULATE_TAX: PROCEDURE(AMOUNT, RATE) + RETURNS(DECIMAL FIXED(11,2)); + DECLARE AMOUNT DECIMAL FIXED(11,2); + DECLARE RATE DECIMAL FIXED(5,4); + + RETURN(AMOUNT * RATE); + END CALCULATE_TAX; + +END MAIN; +``` + +### Parameter Passing + +```pli +/* By reference (default) */ +PROCEDURE UPDATE_BALANCE(BALANCE); + DECLARE BALANCE DECIMAL FIXED(11,2); + BALANCE = BALANCE + 100; /* Modifies caller's variable */ +END; + +/* By value (copy) */ +PROCEDURE SAFE_CALC(VALUE) BYVALUE; + DECLARE VALUE DECIMAL FIXED(11,2); + VALUE = VALUE * 2; /* Only local copy modified */ +END; +``` + +### Recursive Procedures + +```pli +FACTORIAL: PROCEDURE(N) RETURNS(FIXED BINARY(31)) RECURSIVE; + DECLARE N FIXED BINARY(31); + + IF N <= 1 THEN + RETURN(1); + ELSE + RETURN(N * FACTORIAL(N - 1)); +END FACTORIAL; +``` + +## Exception Handling (ON Conditions) + +### Built-in Conditions + +```pli +/* Handle zero divide */ +ON ZERODIVIDE BEGIN; + PUT LIST('Division by zero!'); + RESULT = 0; +END; + +/* Handle overflow */ +ON OVERFLOW BEGIN; + PUT LIST('Numeric overflow'); + /* SIGNAL or GOTO */ +END; + +/* Handle end of file */ +ON ENDFILE(INFILE) EOF_FLAG = '1'B; + +/* Handle conversion error */ +ON CONVERSION BEGIN; + PUT LIST('Invalid data conversion'); + /* Take corrective action */ +END; + +/* Multiple conditions */ +ON ERROR BEGIN; + PUT LIST('An error occurred'); + PUT LIST('In:', ONLOC()); + PUT LIST('Condition:', ONCODE()); +END; +``` + +### Condition Codes + +| Condition | Trigger | +|-----------|---------| +| **ZERODIVIDE** | Division by zero | +| **OVERFLOW** | Numeric overflow | +| **UNDERFLOW** | Numeric underflow | +| **CONVERSION** | Invalid type conversion | +| **STRINGRANGE** | String bounds exceeded | +| **SUBSCRIPTRANGE** | Array bounds exceeded | +| **ENDFILE** | End of file reached | +| **KEY** | Key not found | +| **ERROR** | Catch-all | + +### SIGNAL and REVERT + +```pli +/* Manually raise condition */ +SIGNAL ENDFILE(MYFILE); + +/* Disable handler */ +REVERT ZERODIVIDE; + +/* Re-enable system default */ +ON ZERODIVIDE SYSTEM; +``` + +## File I/O + +### Stream I/O + +```pli +/* Print output */ +PUT LIST('Hello, World!'); +PUT SKIP LIST('Value:', X, 'Total:', Y); +PUT EDIT(NAME, BALANCE) (A(30), F(11,2)); + +/* Read input */ +GET LIST(A, B, C); +GET EDIT(NAME, AMOUNT) (A(30), F(10,2)); + +/* File I/O */ +DECLARE REPORT FILE PRINT; +OPEN FILE(REPORT) OUTPUT; +PUT FILE(REPORT) LIST('Report Title'); +CLOSE FILE(REPORT); +``` + +### Record I/O + +```pli +/* Declare file */ +DECLARE CUSTFILE FILE RECORD SEQUENTIAL; +DECLARE 1 CUSTREC, + 2 CUSTNO FIXED DECIMAL(8), + 2 NAME CHARACTER(50), + 2 BALANCE FIXED DECIMAL(11,2); + +/* Sequential read */ +OPEN FILE(CUSTFILE) INPUT; +ON ENDFILE(CUSTFILE) MORE_RECORDS = '0'B; + +READ FILE(CUSTFILE) INTO(CUSTREC); +DO WHILE(MORE_RECORDS); + /* Process record */ + READ FILE(CUSTFILE) INTO(CUSTREC); +END; +CLOSE FILE(CUSTFILE); + +/* Keyed access */ +DECLARE INDEXFILE FILE RECORD KEYED; +READ FILE(INDEXFILE) INTO(RECORD) KEY(CUSTNO); +REWRITE FILE(INDEXFILE) FROM(RECORD) KEY(CUSTNO); +``` + +## Pointers and Storage + +### Pointer Operations + +```pli +DECLARE PTR POINTER; +DECLARE X FIXED DECIMAL(11,2) BASED(PTR); + +/* Allocate storage */ +ALLOCATE X SET(PTR); +X = 1234.56; + +/* Free storage */ +FREE X; + +/* Null pointer */ +IF PTR = NULL() THEN + PUT LIST('Pointer is null'); +``` + +### Linked Structures + +```pli +DECLARE 1 NODE BASED(P), + 2 VALUE FIXED DECIMAL(11,2), + 2 NEXT POINTER; + +DECLARE (HEAD, P, NEWNODE) POINTER; + +/* Create node */ +ALLOCATE NODE SET(NEWNODE); +NEWNODE->VALUE = 100; +NEWNODE->NEXT = HEAD; +HEAD = NEWNODE; + +/* Traverse list */ +P = HEAD; +DO WHILE(P ^= NULL()); + PUT LIST(P->VALUE); + P = P->NEXT; +END; +``` + +## Preprocessor + +```pli +%DECLARE DEBUG CHARACTER; +%DEBUG = 'YES'; + +%IF DEBUG = 'YES' %THEN + %ACTIVATE TRACE; +%ELSE; + %DEACTIVATE TRACE; +%END; + +%INCLUDE COPYBOOK; + +%MACRO SWAP(A, B); + TEMP = A; + A = B; + B = TEMP; +%END; +``` + +## Modern PL/I + +### Enterprise PL/I for z/OS + +**Current IBM Compiler Features:** + +- Unicode support +- XML processing +- SQL/DB2 integration +- 64-bit support +- Improved optimization + +### Integration with Modern Systems + +```pli +/* DB2 SQL */ +EXEC SQL + SELECT NAME, BALANCE INTO :NAME, :BALANCE + FROM CUSTOMERS + WHERE CUSTNO = :CUSTNO; + +/* CICS */ +EXEC CICS READ FILE('CUSTFILE') + INTO(CUSTREC) + RIDFLD(CUSTNO); + +/* XML */ +DECLARE DOC XMLCHAR(10000); +``` + +## Comparison with Other Languages + +| Aspect | PL/I | COBOL | C | +|--------|------|-------|---| +| **Decimal Arithmetic** | ✅ Native | ✅ Native | ❌ Libraries | +| **Pointers** | ✅ Full | ⚠️ Limited | ✅ Full | +| **Structures** | ✅ Yes | ✅ Yes | ✅ Yes | +| **Exception Handling** | ✅ ON conditions | ❌ None | ❌ None (C) | +| **String Handling** | ✅ Built-in | ✅ Built-in | ⚠️ Library | +| **Arrays** | ✅ Flexible bounds | ⚠️ Fixed | ⚠️ Zero-based | + +## When to Use PL/I + +### Strengths + +| Strength | Rationale | +|----------|-----------| +| **Versatility** | Scientific and business in one language | +| **Exception Handling** | Pioneering error handling | +| **Data Types** | Rich numeric and string types | +| **Mainframe Integration** | First-class z/OS support | + +### Considerations + +| Consideration | Impact | +|---------------|--------| +| **Complexity** | Many features, large language | +| **Niche Usage** | Mainly mainframes | +| **Skills** | Very limited talent pool | + +### Best For + +- Legacy mainframe systems +- Mixed scientific/business applications +- Systems requiring structured exception handling + +## Related + +- [[COBOL]] — Business-focused mainframe language +- [[Fortran]] — Scientific computing predecessor +- [[Mainframes]] — Primary platform +- [[Languages MOC]] — All language references diff --git a/Languages/Pascal.md b/Languages/Pascal.md new file mode 100644 index 0000000..d7db8bc --- /dev/null +++ b/Languages/Pascal.md @@ -0,0 +1,346 @@ +--- +title: Pascal +aliases: + - Object Pascal + - Delphi + - Free Pascal +tags: + - language + - legacy + - educational + - structured +type: reference +status: complete +created: "2025-12-16" +--- + +# Pascal + +Influential teaching language emphasizing structured programming, spawning Delphi and influencing generations of language design. + +## Overview + +| Aspect | Details | +|--------|---------| +| **Paradigm** | Procedural, structured (OOP in Object Pascal) | +| **Typing** | Static, strong | +| **Memory Model** | Manual (stack + heap) | +| **First Appeared** | 1970 (Niklaus Wirth) | +| **Primary Use Cases** | Education, desktop applications (Delphi) | +| **Notable Implementations** | Turbo Pascal, Delphi, Free Pascal, Lazarus | +| **Design Goal** | Teaching good programming practices | + +## Core Concepts + +### Program Structure + +```pascal +program HelloWorld; +uses + SysUtils; +var + name: string; +begin + Write('Enter your name: '); + ReadLn(name); + WriteLn('Hello, ', name, '!'); + WriteLn('Today is ', DateToStr(Date)); +end. +``` + +### Structure Overview + +| Section | Purpose | +|---------|---------| +| **program** | Program name declaration | +| **uses** | Unit imports | +| **const** | Constants | +| **type** | Type definitions | +| **var** | Variable declarations | +| **begin...end** | Main code block | + +### Data Types + +**Built-in Types:** + +| Type | Description | +|------|-------------| +| **Integer** | Whole numbers | +| **Real** | Floating-point | +| **Char** | Single character | +| **String** | Character sequence | +| **Boolean** | True/False | +| **Array** | Fixed-size collection | +| **Record** | Structured data | +| **Set** | Collection of values | +| **Pointer** | Memory address | + +### Type Definitions + +```pascal +type + TMonth = 1..12; { Subrange } + TDays = (Mon, Tue, Wed, Thu, Fri); { Enumeration } + TName = string[50]; { Bounded string } + TMatrix = array[1..10, 1..10] of Real; + + TPerson = record + Name: string; + Age: Integer; + Active: Boolean; + end; +``` + +### Procedures and Functions + +```pascal +{ Procedure - no return value } +procedure Greet(name: string); +begin + WriteLn('Hello, ', name); +end; + +{ Function - returns a value } +function Factorial(n: Integer): Integer; +begin + if n <= 1 then + Result := 1 + else + Result := n * Factorial(n - 1); +end; + +{ Var parameter - pass by reference } +procedure Swap(var a, b: Integer); +var + temp: Integer; +begin + temp := a; + a := b; + b := temp; +end; +``` + +### Control Structures + +```pascal +{ If statement } +if x > 0 then + WriteLn('Positive') +else if x < 0 then + WriteLn('Negative') +else + WriteLn('Zero'); + +{ Case statement } +case grade of + 'A': WriteLn('Excellent'); + 'B': WriteLn('Good'); + 'C': WriteLn('Average'); +else + WriteLn('Below average'); +end; + +{ For loop } +for i := 1 to 10 do + WriteLn(i); + +{ While loop } +while not EOF(inputFile) do +begin + ReadLn(inputFile, line); + ProcessLine(line); +end; + +{ Repeat-until } +repeat + ReadLn(value); +until value > 0; +``` + +## Pascal Evolution + +| Version | Year | Key Features | +|---------|------|--------------| +| **Original Pascal** | 1970 | Structured programming, strong typing | +| **UCSD Pascal** | 1978 | Units, p-code portability | +| **Turbo Pascal** | 1983 | Fast compilation, integrated IDE | +| **Object Pascal** | 1986 | Object-oriented extensions | +| **Delphi** | 1995 | RAD, VCL, Windows development | +| **Free Pascal** | 1997 | Open source, cross-platform | +| **Delphi 2009** | 2008 | Unicode, generics | +| **Delphi 11+** | 2021+ | Cross-platform (FMX), modern features | + +## Object Pascal (Delphi) + +### Class Definition + +```pascal +type + TAnimal = class + private + FName: string; + protected + procedure SetName(const Value: string); + public + constructor Create(const AName: string); + destructor Destroy; override; + property Name: string read FName write SetName; + procedure Speak; virtual; abstract; + end; + + TDog = class(TAnimal) + public + procedure Speak; override; + end; + +implementation + +constructor TAnimal.Create(const AName: string); +begin + inherited Create; + FName := AName; +end; + +procedure TDog.Speak; +begin + WriteLn(FName, ' says: Woof!'); +end; +``` + +### Interfaces + +```pascal +type + ILogger = interface + ['{GUID-HERE}'] + procedure Log(const Message: string); + end; + + TFileLogger = class(TInterfacedObject, ILogger) + public + procedure Log(const Message: string); + end; +``` + +### Generics (Delphi 2009+) + +```pascal +type + TStack = class + private + FItems: array of T; + public + procedure Push(const Item: T); + function Pop: T; + end; + +var + IntStack: TStack; + StrStack: TStack; +``` + +## Delphi VCL/FMX + +### Visual Component Library + +```pascal +procedure TForm1.Button1Click(Sender: TObject); +begin + ShowMessage('Hello, ' + Edit1.Text); + ListBox1.Items.Add(Edit1.Text); +end; + +procedure TForm1.FormCreate(Sender: TObject); +begin + Caption := 'My Application'; + Edit1.Text := ''; +end; +``` + +### Key Components + +| Component | Purpose | +|-----------|---------| +| **TForm** | Window/dialog | +| **TButton** | Clickable button | +| **TEdit** | Text input field | +| **TLabel** | Static text | +| **TListBox** | Scrollable list | +| **TDataSet** | Database access | +| **TTimer** | Scheduled events | + +## Implementations + +| Implementation | License | Platforms | Notes | +|----------------|---------|-----------|-------| +| **Delphi** | Commercial | Windows, macOS, iOS, Android, Linux | RAD Studio, VCL/FMX | +| **Free Pascal** | GPL | 20+ platforms | Compiler-only | +| **Lazarus** | GPL | Cross-platform | Free Pascal + IDE | +| **PascalABC.NET** | Free | Windows (.NET) | Educational | +| **Turbo Pascal** | Historical | DOS | Borland (discontinued) | + +## Comparison with Alternatives + +| Aspect | Pascal/Delphi | C# | Java | +|--------|---------------|-----|------| +| **Compilation** | Native | IL/JIT | Bytecode/JIT | +| **Memory** | Manual + ARC | GC | GC | +| **Performance** | ✅ Excellent native | ✅ Good | ✅ Good | +| **RAD/GUI** | ✅ VCL/FMX | ✅ WinForms/WPF | ⚠️ Swing/JavaFX | +| **Community Size** | ⚠️ Small | ✅ Large | ✅ Large | +| **Learning Resources** | ⚠️ Limited | ✅ Abundant | ✅ Abundant | +| **Job Market** | ⚠️ Niche | ✅ Strong | ✅ Strong | + +## Design Influence + +Pascal influenced many languages: + +| Language | Influence | +|----------|-----------| +| **Modula-2** | Direct successor by Wirth | +| **Ada** | Syntax and structure | +| **Oberon** | Simplified Pascal by Wirth | +| **Java** | Some syntax elements | +| **TypeScript** | `type` keyword, enum syntax | + +## When to Use + +### Strengths + +| Strength | Rationale | +|----------|-----------| +| **Readability** | English-like, structured code | +| **Strong Typing** | Catches errors at compile time | +| **Fast Compilation** | Quick edit-compile-run cycle | +| **Native Performance** | No VM overhead | +| **RAD (Delphi)** | Rapid Windows development | + +### Considerations + +| Consideration | Impact | +|---------------|--------| +| **Small Community** | Fewer libraries, resources | +| **Niche Market** | Limited job opportunities | +| **Commercial Tools** | Delphi is expensive | +| **Platform Focus** | Historically Windows-centric | + +### Best For + +- **Legacy Delphi Apps** — Maintaining existing systems +- **Desktop Applications** — Windows business software +- **Teaching** — Structured programming concepts +- **Embedded (Free Pascal)** — Resource-constrained systems + +**Avoid For:** + +- Web development (prefer TypeScript, C#) +- Mobile-first development (prefer Kotlin, Swift) +- New enterprise projects (prefer C#, Java) +- Machine learning (prefer Python) + +## Related + +- [[C Sharp|C#]] — Modern alternative with similar RAD capabilities +- [[BASIC]] — Contemporary educational language +- [[Ada]] — Pascal-influenced safety-critical language +- [[Languages MOC]] — Overview of all language references diff --git a/Languages/RPG.md b/Languages/RPG.md new file mode 100644 index 0000000..035b21a --- /dev/null +++ b/Languages/RPG.md @@ -0,0 +1,437 @@ +--- +title: RPG +aliases: + - RPG IV + - RPGLE + - ILE RPG + - Report Program Generator +tags: + - language + - legacy + - mainframe + - ibm + - enterprise +type: reference +status: complete +created: "2025-12-16" +--- + +# RPG + +IBM's business programming language for AS/400 (IBM i), evolved from fixed-format report generation to modern free-format programming. + +## Overview + +| Aspect | Details | +|--------|---------| +| **Full Name** | Report Program Generator | +| **Platform** | IBM i (AS/400, iSeries) | +| **First Released** | 1959 (RPG I), 1994 (RPG IV) | +| **Current Version** | RPG IV (ILE RPG, free-format) | +| **Paradigm** | Procedural, now supports procedures | +| **Primary Use** | Business applications, ERP, database processing | + +## Evolution + +| Version | Year | Key Features | +|---------|------|--------------| +| **RPG I** | 1959 | Report generation | +| **RPG II** | 1969 | S/36, S/38 | +| **RPG III** | 1979 | Structured operations | +| **RPG/400** | 1988 | AS/400 version | +| **RPG IV** | 1994 | ILE, procedures, long names | +| **Free-format** | 2001 | C-style free-format (partial) | +| **Fully free** | 2013 | **FREE - no column restrictions | + +## Fixed vs Free Format + +### Fixed Format (Legacy) + +**Columns matter** — Each column has specific meaning. + +```rpg + C CUSTNO CHAIN CUSTMAST + C IF %FOUND(CUSTMAST) + C EVAL BALANCE = BALANCE + PAYMENT + C UPDATE CUSTREC + C ENDIF +``` + +| Columns | Purpose | +|---------|---------| +| 1-5 | Sequence number | +| 6 | Form type (H, F, D, I, C, O, P) | +| 7-8 | Control level | +| 9-11 | Indicators | +| 12+ | Operation-specific | + +### Free Format (Modern) + +**No column restrictions** — Modern syntax. + +```rpg +**free +ctl-opt dftactgrp(*no) actgrp(*caller); + +dcl-f CUSTMAST disk usage(*update) keyed; + +dcl-s custNo packed(8); +dcl-s payment packed(11:2); + +chain (custNo) CUSTMAST; +if %found(CUSTMAST); + BALANCE += payment; + update CUSTREC; +endif; + +*inlr = *on; +``` + +## Program Structure + +### Control Options + +```rpg +**free +ctl-opt dftactgrp(*no) // Not default activation group + actgrp(*caller) // Use caller's activation group + option(*srcstmt) // Source statement numbers + debug(*yes); // Debug mode +``` + +### File Declarations + +```rpg +// Physical file (table) - update access +dcl-f CUSTMAST disk usage(*update) keyed; + +// Display file (screen) +dcl-f CUSTDSP workstn; + +// Printer file +dcl-f CUSTRPT printer oflind(*in90); + +// Program-described file +dcl-f FLATFILE disk usage(*input); +``` + +### Data Definitions + +```rpg +// Standalone variables +dcl-s custNo packed(8); +dcl-s custName char(50); +dcl-s balance packed(11:2); +dcl-s isActive ind; + +// Constants +dcl-c MAX_RECORDS 1000; +dcl-c COMPANY_NAME 'ACME Corp'; + +// Data structures +dcl-ds custRec qualified; + custNo packed(8); + custName char(50); + balance packed(11:2); +end-ds; + +// Based on file record format +dcl-ds custFile likerec(CUSTREC); + +// Array +dcl-s codes char(3) dim(100); +dcl-ds custArray likeds(custRec) dim(500); +``` + +### Procedures + +```rpg +// Main procedure +dcl-proc main; + dcl-pi *n; + inputCustNo packed(8) const; + end-pi; + + dcl-s result packed(11:2); + + result = getBalance(inputCustNo); + + return; +end-proc; + +// Subprocedure with return value +dcl-proc getBalance; + dcl-pi *n packed(11:2); + custNo packed(8) const; + end-pi; + + chain (custNo) CUSTMAST; + if %found(CUSTMAST); + return BALANCE; + endif; + + return 0; +end-proc; + +// Exported procedure (for service programs) +dcl-proc calculateTax export; + dcl-pi *n packed(11:2); + amount packed(11:2) const; + rate packed(5:4) const; + end-pi; + + return amount * rate; +end-proc; +``` + +## Database Operations + +### Native I/O + +```rpg +// Read by key +chain (custNo) CUSTMAST; +if %found(CUSTMAST); + // Process record +endif; + +// Sequential read +read CUSTMAST; +dow not %eof(CUSTMAST); + // Process record + read CUSTMAST; +enddo; + +// Partial key read +setll (stateCode) CUSTBYSTATE; +reade (stateCode) CUSTBYSTATE; +dow not %eof(CUSTBYSTATE); + // Process records for state + reade (stateCode) CUSTBYSTATE; +enddo; + +// Update +chain (custNo) CUSTMAST; +if %found(CUSTMAST); + BALANCE += payment; + update CUSTREC; +endif; + +// Write new record +custRec.CUSTNO = newCustNo; +custRec.CUSTNAME = newName; +custRec.BALANCE = 0; +write CUSTREC custRec; + +// Delete +chain (custNo) CUSTMAST; +if %found(CUSTMAST); + delete CUSTREC; +endif; +``` + +### Embedded SQL + +```rpg +// Single row fetch +exec sql + SELECT CUSTNAME, BALANCE + INTO :custName, :balance + FROM CUSTMAST + WHERE CUSTNO = :custNo; + +if sqlcode = 0; + // Found +elseif sqlcode = 100; + // Not found +else; + // Error +endif; + +// Cursor for multiple rows +exec sql DECLARE custCursor CURSOR FOR + SELECT CUSTNO, CUSTNAME, BALANCE + FROM CUSTMAST + WHERE STATE = :stateCode + ORDER BY CUSTNAME; + +exec sql OPEN custCursor; + +exec sql FETCH custCursor INTO :custNo, :custName, :balance; +dow sqlcode = 0; + // Process row + exec sql FETCH custCursor INTO :custNo, :custName, :balance; +enddo; + +exec sql CLOSE custCursor; + +// Insert +exec sql + INSERT INTO CUSTMAST (CUSTNO, CUSTNAME, BALANCE) + VALUES (:custNo, :custName, :balance); + +// Update +exec sql + UPDATE CUSTMAST + SET BALANCE = BALANCE + :payment + WHERE CUSTNO = :custNo; +``` + +## Built-In Functions + +### String Functions + +| Function | Description | +|----------|-------------| +| `%trim(string)` | Remove leading/trailing blanks | +| `%subst(string:start:len)` | Substring | +| `%scan(needle:haystack)` | Find position | +| `%replace(new:old:start:len)` | Replace substring | +| `%xlate(from:to:string)` | Translate characters | +| `%len(string)` | Length | +| `%upper(string)` | Uppercase | +| `%lower(string)` | Lowercase | + +### Numeric Functions + +| Function | Description | +|----------|-------------| +| `%dec(value:digits:decimals)` | Convert to decimal | +| `%int(value)` | Convert to integer | +| `%abs(number)` | Absolute value | +| `%rem(dividend:divisor)` | Remainder | +| `%div(dividend:divisor)` | Integer division | +| `%editc(number:editcode)` | Format number | + +### Date Functions + +| Function | Description | +|----------|-------------| +| `%date(value)` | Convert to date | +| `%time(value)` | Convert to time | +| `%timestamp(value)` | Convert to timestamp | +| `%diff(date1:date2:unit)` | Date difference | +| `%days(number)` | Duration in days | + +### File Functions + +| Function | Description | +|----------|-------------| +| `%found(file)` | Record found | +| `%eof(file)` | End of file | +| `%equal(file)` | Exact key match | +| `%open(file)` | File is open | + +## ILE Concepts + +### Modules and Programs + +``` +Source (.rpgle) + ↓ CRTRPGMOD +Module (.module) + ↓ CRTPGM +Program (.pgm) +``` + +### Service Programs + +**Reusable procedure libraries:** + +``` +Module A + Module B + ↓ CRTSRVPGM +Service Program (.srvpgm) + ↓ Bind to +Multiple Programs +``` + +### Binding + +```cl +CRTPGM PGM(MYPGM) MODULE(MYMOD) + BNDSRVPGM(MYUTIL MATHLIB) +``` + +## Error Handling + +### Monitor Blocks + +```rpg +monitor; + result = numerator / denominator; +on-error *all; + result = 0; + errorOccurred = *on; +endmon; + +// Specific errors +monitor; + chain (custNo) CUSTMAST; +on-error 1218; // Record locked + // Handle lock +on-error 1211; // I/O error + // Handle I/O error +endmon; +``` + +### Program Status Data Structure + +```rpg +dcl-ds pgmStatus psds; + pgmName *proc; + pgmStatus *status; + prevStatus char(5) pos(16); + lineNum char(8) pos(21); + routine *routine; + parms *parms; + exceptionId char(7) pos(40); + user char(10) pos(254); + jobNumber char(6) pos(264); +end-ds; +``` + +## Comparison with Modern Languages + +| Aspect | RPG | Java | C# | +|--------|-----|------|-----| +| **Typing** | Static | Static | Static | +| **Database** | Integrated | JDBC | ADO.NET | +| **Performance** | ✅ Excellent | ✅ Good | ✅ Good | +| **Learning Curve** | ⚠️ Unusual syntax | ✅ Familiar | ✅ Familiar | +| **Modern Tooling** | ⚠️ Limited | ✅ Excellent | ✅ Excellent | +| **Community** | ⚠️ Small | ✅ Large | ✅ Large | + +## When to Use RPG + +### Strengths + +| Strength | Rationale | +|----------|-----------| +| **Database Performance** | Native I/O extremely fast | +| **IBM i Integration** | First-class platform language | +| **Business Logic** | Designed for business apps | +| **Stability** | Decades of production code | + +### Considerations + +| Consideration | Impact | +|---------------|--------| +| **Platform Lock-in** | IBM i only | +| **Skills Availability** | Shrinking talent pool | +| **Modern Practices** | TDD, CI/CD harder | +| **Perception** | "Legacy" stigma | + +### Best For + +- IBM i application development +- High-performance database operations +- Maintaining existing systems +- ERP and business applications + +## Related + +- [[AS400]] — The platform RPG runs on +- [[COBOL]] — Similar legacy business language +- [[JCL]] — Mainframe job control +- [[Languages MOC]] — All language references diff --git a/Security/JWT.md b/Security/JWT.md new file mode 100644 index 0000000..10f47ae --- /dev/null +++ b/Security/JWT.md @@ -0,0 +1,357 @@ +--- +title: JWT +aliases: + - JSON Web Token + - JSON Web Tokens +tags: + - security + - authentication + - api + - tokens +type: reference +status: complete +created: "2025-12-16" +--- + +# JWT + +JSON Web Tokens — compact, URL-safe tokens for securely transmitting claims between parties. + +## Overview + +| Aspect | Details | +|--------|---------| +| **Standard** | RFC 7519 | +| **Format** | Base64URL encoded JSON | +| **Structure** | Header.Payload.Signature | +| **Use Cases** | Authentication, authorization, information exchange | + +## Structure + +``` +eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9. +eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ. +SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c + +│ Header │ +│ │ +│ Payload │ +│ │ +│ Signature │ +``` + +### Header + +```json +{ + "alg": "HS256", + "typ": "JWT" +} +``` + +| Field | Description | +|-------|-------------| +| `alg` | Signing algorithm (HS256, RS256, ES256) | +| `typ` | Token type (always "JWT") | +| `kid` | Key ID (for key rotation) | + +### Payload (Claims) + +```json +{ + "sub": "user123", + "name": "Alice Smith", + "email": "alice@example.com", + "iat": 1704063600, + "exp": 1704067200, + "iss": "https://auth.example.com", + "aud": "https://api.example.com", + "roles": ["admin", "user"] +} +``` + +#### Registered Claims + +| Claim | Name | Description | +|-------|------|-------------| +| `iss` | Issuer | Who created the token | +| `sub` | Subject | Who the token is about | +| `aud` | Audience | Intended recipient | +| `exp` | Expiration | When token expires (Unix timestamp) | +| `nbf` | Not Before | Token not valid before this time | +| `iat` | Issued At | When token was created | +| `jti` | JWT ID | Unique token identifier | + +### Signature + +``` +HMACSHA256( + base64UrlEncode(header) + "." + base64UrlEncode(payload), + secret +) +``` + +## Signing Algorithms + +### Symmetric (HMAC) + +**Same key for signing and verification.** + +| Algorithm | Description | +|-----------|-------------| +| `HS256` | HMAC with SHA-256 | +| `HS384` | HMAC with SHA-384 | +| `HS512` | HMAC with SHA-512 | + +```javascript +// Node.js +const jwt = require('jsonwebtoken'); + +const token = jwt.sign( + { sub: 'user123', name: 'Alice' }, + 'your-256-bit-secret', + { algorithm: 'HS256', expiresIn: '1h' } +); +``` + +### Asymmetric (RSA, ECDSA) + +**Private key signs, public key verifies.** + +| Algorithm | Description | +|-----------|-------------| +| `RS256` | RSA + SHA-256 | +| `RS384` | RSA + SHA-384 | +| `RS512` | RSA + SHA-512 | +| `ES256` | ECDSA + P-256 | +| `ES384` | ECDSA + P-384 | +| `ES512` | ECDSA + P-521 | + +```javascript +const fs = require('fs'); +const jwt = require('jsonwebtoken'); + +const privateKey = fs.readFileSync('private.pem'); +const publicKey = fs.readFileSync('public.pem'); + +// Sign with private key +const token = jwt.sign( + { sub: 'user123' }, + privateKey, + { algorithm: 'RS256', expiresIn: '1h' } +); + +// Verify with public key +const decoded = jwt.verify(token, publicKey, { algorithms: ['RS256'] }); +``` + +### Algorithm Comparison + +| Type | Algorithms | Key Distribution | Use Case | +|------|------------|------------------|----------| +| **Symmetric** | HS256/384/512 | Shared secret | Single service | +| **RSA** | RS256/384/512 | Public/private | Distributed systems | +| **ECDSA** | ES256/384/512 | Public/private | Smaller tokens | + +## Validation + +### Validation Steps + +```javascript +function validateJWT(token, publicKey) { + // 1. Split token + const [headerB64, payloadB64, signature] = token.split('.'); + + // 2. Decode header and payload + const header = JSON.parse(base64UrlDecode(headerB64)); + const payload = JSON.parse(base64UrlDecode(payloadB64)); + + // 3. Check algorithm (prevent alg:none attack) + if (!['RS256', 'ES256'].includes(header.alg)) { + throw new Error('Invalid algorithm'); + } + + // 4. Verify signature + const valid = verifySignature(headerB64, payloadB64, signature, publicKey); + if (!valid) throw new Error('Invalid signature'); + + // 5. Check expiration + if (payload.exp && Date.now() >= payload.exp * 1000) { + throw new Error('Token expired'); + } + + // 6. Check not before + if (payload.nbf && Date.now() < payload.nbf * 1000) { + throw new Error('Token not yet valid'); + } + + // 7. Check issuer + if (payload.iss !== 'https://auth.example.com') { + throw new Error('Invalid issuer'); + } + + // 8. Check audience + if (payload.aud !== 'https://api.example.com') { + throw new Error('Invalid audience'); + } + + return payload; +} +``` + +### Library Validation + +```javascript +// Node.js with jsonwebtoken +const jwt = require('jsonwebtoken'); + +try { + const decoded = jwt.verify(token, publicKey, { + algorithms: ['RS256'], + issuer: 'https://auth.example.com', + audience: 'https://api.example.com', + clockTolerance: 30 // 30 seconds leeway + }); + console.log(decoded); +} catch (err) { + console.error('Invalid token:', err.message); +} +``` + +## Security Best Practices + +### Do + +| Practice | Rationale | +|----------|-----------| +| **Verify signature** | Prevent tampering | +| **Check expiration** | Limit token lifetime | +| **Validate issuer/audience** | Prevent token misuse | +| **Use asymmetric for distributed** | No shared secrets | +| **Keep tokens short-lived** | 5-15 minutes typical | +| **Use HTTPS** | Prevent interception | + +### Don't + +| Anti-Pattern | Risk | +|--------------|------| +| **Store in localStorage** | XSS can steal tokens | +| **Include sensitive data** | Payload is readable | +| **Trust without verifying** | Accept forged tokens | +| **Allow `alg: none`** | Signature bypass | +| **Long expiration** | Extended exposure window | +| **Expose in URLs** | Logged, bookmarked, cached | + +## Common Attacks + +### Algorithm Confusion + +```json +// Attacker changes header +{ "alg": "none", "typ": "JWT" } +// or +{ "alg": "HS256", "typ": "JWT" } // When server expects RS256 +``` + +**Prevention:** Always specify allowed algorithms. + +```javascript +jwt.verify(token, key, { algorithms: ['RS256'] }); // Only RS256 +``` + +### Token Sidejacking + +**Attack:** XSS steals token from localStorage. + +**Prevention:** Use HttpOnly cookies or secure token binding. + +### Brute Force (Weak Secrets) + +**Attack:** Guess HMAC secret for HS256. + +**Prevention:** Use strong secrets (256+ bits) or asymmetric algorithms. + +## Storage Options + +| Storage | XSS Safe | CSRF Safe | Best For | +|---------|----------|-----------|----------| +| **HttpOnly Cookie** | ✅ | ❌ (needs protection) | Web apps | +| **localStorage** | ❌ | ✅ | Not recommended | +| **sessionStorage** | ❌ | ✅ | Short sessions | +| **Memory** | ✅ | ✅ | SPAs (refresh from cookie) | + +### Recommended Pattern (Web) + +```javascript +// Server sets token in HttpOnly cookie +res.cookie('token', jwt, { + httpOnly: true, + secure: true, + sameSite: 'strict', + maxAge: 15 * 60 * 1000 // 15 minutes +}); + +// Client includes automatically (no JS access) +// Add CSRF token for state-changing requests +``` + +## JWK (JSON Web Key) + +**Public key format for distribution.** + +```json +{ + "keys": [ + { + "kty": "RSA", + "kid": "key-id-1", + "use": "sig", + "alg": "RS256", + "n": "0vx7agoebGc...", + "e": "AQAB" + } + ] +} +``` + +**JWKS Endpoint:** `/.well-known/jwks.json` + +## Token Lifecycle + +``` +┌─────────┐ ┌─────────────┐ ┌──────────┐ +│ Login │───▶│ Access Token│───▶│ API │ +└─────────┘ │ (15 min) │ │ Requests │ + └──────┬──────┘ └──────────┘ + │ + │ Expires + ▼ + ┌─────────────┐ + │Refresh Token│ + │ (7 days) │ + └──────┬──────┘ + │ + ▼ + ┌─────────────┐ + │New Access │ + │Token │ + └─────────────┘ +``` + +## Libraries + +| Language | Library | +|----------|---------| +| **Node.js** | jsonwebtoken, jose | +| **Python** | PyJWT, python-jose | +| **Go** | golang-jwt | +| **Java** | jjwt, auth0-java-jwt | +| **C#** | System.IdentityModel.Tokens.Jwt | +| **Ruby** | ruby-jwt | + +## Related + +- [[OAuth and OIDC]] — Authentication protocols +- [[Identity and Access Management]] — IAM overview +- [[Cryptography]] — Signing algorithms +- [[Security MOC]] — Security topics diff --git a/Security/OAuth and OIDC.md b/Security/OAuth and OIDC.md new file mode 100644 index 0000000..04c38f5 --- /dev/null +++ b/Security/OAuth and OIDC.md @@ -0,0 +1,329 @@ +--- +title: OAuth and OIDC +aliases: + - OAuth 2.0 + - OpenID Connect + - OAuth +tags: + - security + - authentication + - authorization + - api +type: reference +status: complete +created: "2025-12-16" +--- + +# OAuth and OIDC + +Industry-standard protocols for authorization (OAuth 2.0) and authentication (OpenID Connect). + +## Overview + +| Protocol | Purpose | Result | +|----------|---------|--------| +| **OAuth 2.0** | Authorization | Access Token | +| **OpenID Connect** | Authentication | ID Token + Access Token | + +``` +OAuth 2.0: "Can this app access my photos?" +OIDC: "Who is this user?" + OAuth 2.0 +``` + +## Key Concepts + +| Term | Definition | +|------|------------| +| **Resource Owner** | User who owns the data | +| **Client** | Application requesting access | +| **Authorization Server** | Issues tokens (Google, Auth0) | +| **Resource Server** | API hosting protected resources | +| **Access Token** | Credential to access resources | +| **Refresh Token** | Credential to get new access tokens | +| **ID Token** | JWT containing user identity (OIDC) | +| **Scope** | Permission being requested | + +## OAuth 2.0 Flows + +### Authorization Code Flow (Recommended) + +**Best for:** Server-side web apps + +``` +┌──────────┐ ┌───────────────┐ +│ User │ │ Authorization │ +│ (Browser)│ │ Server │ +└────┬─────┘ └───────┬───────┘ + │ │ + │ 1. Click "Login with Google" │ + │────────────────────────────────────────────▶ + │ │ + │ 2. Redirect to Auth Server │ + │◀──────────────────────────────────────────── + │ │ + │ 3. User logs in & consents │ + │────────────────────────────────────────────▶ + │ │ + │ 4. Redirect back with code │ + │◀──────────────────────────────────────────── + │ │ +┌────┴─────┐ │ +│ Client │ 5. Exchange code for tokens │ +│ (Server) │─────────────────────────────────────▶│ +│ │ │ +│ │ 6. Access Token + Refresh Token │ +│ │◀─────────────────────────────────────│ +└──────────┘ │ +``` + +**Request:** + +``` +GET /authorize? + response_type=code& + client_id=CLIENT_ID& + redirect_uri=https://app.com/callback& + scope=openid profile email& + state=RANDOM_STATE +``` + +**Token Exchange:** + +```bash +POST /token +Content-Type: application/x-www-form-urlencoded + +grant_type=authorization_code& +code=AUTH_CODE& +redirect_uri=https://app.com/callback& +client_id=CLIENT_ID& +client_secret=CLIENT_SECRET +``` + +### Authorization Code with PKCE + +**Best for:** Mobile apps, SPAs (public clients) + +``` +1. Generate code_verifier (random string) +2. Create code_challenge = BASE64URL(SHA256(code_verifier)) +3. Include code_challenge in authorize request +4. Include code_verifier in token request +``` + +```javascript +// Generate PKCE values +const codeVerifier = generateRandomString(64); +const codeChallenge = base64UrlEncode(sha256(codeVerifier)); + +// Authorization request +const authUrl = `https://auth.example.com/authorize? + response_type=code& + client_id=${clientId}& + redirect_uri=${redirectUri}& + scope=openid profile& + code_challenge=${codeChallenge}& + code_challenge_method=S256& + state=${state}`; + +// Token request (no client_secret needed) +const tokenResponse = await fetch('https://auth.example.com/token', { + method: 'POST', + body: new URLSearchParams({ + grant_type: 'authorization_code', + code: authorizationCode, + redirect_uri: redirectUri, + client_id: clientId, + code_verifier: codeVerifier + }) +}); +``` + +### Client Credentials Flow + +**Best for:** Machine-to-machine (no user) + +```bash +POST /token +Content-Type: application/x-www-form-urlencoded + +grant_type=client_credentials& +client_id=CLIENT_ID& +client_secret=CLIENT_SECRET& +scope=api:read api:write +``` + +### Implicit Flow (Deprecated) + +**Don't use for new applications.** Replaced by Authorization Code + PKCE. + +## OpenID Connect + +**OIDC = OAuth 2.0 + Identity Layer** + +### ID Token (JWT) + +```json +{ + "iss": "https://auth.example.com", + "sub": "user123", + "aud": "client_id", + "exp": 1704067200, + "iat": 1704063600, + "nonce": "random_nonce", + "name": "Alice Smith", + "email": "alice@example.com", + "email_verified": true, + "picture": "https://example.com/alice.jpg" +} +``` + +### Standard Scopes + +| Scope | Claims Returned | +|-------|-----------------| +| `openid` | Required for OIDC (sub) | +| `profile` | name, family_name, picture, etc. | +| `email` | email, email_verified | +| `address` | address | +| `phone` | phone_number, phone_number_verified | + +### UserInfo Endpoint + +```bash +GET /userinfo +Authorization: Bearer ACCESS_TOKEN + +Response: +{ + "sub": "user123", + "name": "Alice Smith", + "email": "alice@example.com" +} +``` + +## Token Types + +### Access Token + +``` +Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9... +``` + +| Property | Typical Value | +|----------|---------------| +| **Lifetime** | 5-60 minutes | +| **Format** | JWT or opaque | +| **Use** | API authorization | + +### Refresh Token + +| Property | Typical Value | +|----------|---------------| +| **Lifetime** | Days to months | +| **Format** | Opaque | +| **Use** | Get new access tokens | + +```bash +POST /token +Content-Type: application/x-www-form-urlencoded + +grant_type=refresh_token& +refresh_token=REFRESH_TOKEN& +client_id=CLIENT_ID& +client_secret=CLIENT_SECRET +``` + +### ID Token + +| Property | Value | +|----------|-------| +| **Format** | Always JWT | +| **Use** | User authentication | +| **Validate** | Signature, exp, aud, iss | + +## Token Validation + +### JWT Validation Steps + +1. **Parse JWT** — Split header.payload.signature +2. **Verify signature** — Using issuer's public key +3. **Check `exp`** — Token not expired +4. **Check `iat`** — Issued at reasonable time +5. **Check `aud`** — Matches your client_id +6. **Check `iss`** — Matches expected issuer +7. **Check `nonce`** — Matches sent nonce (OIDC) + +```javascript +const jwt = require('jsonwebtoken'); +const jwksClient = require('jwks-rsa'); + +const client = jwksClient({ + jwksUri: 'https://auth.example.com/.well-known/jwks.json' +}); + +function getKey(header, callback) { + client.getSigningKey(header.kid, (err, key) => { + callback(null, key.getPublicKey()); + }); +} + +jwt.verify(token, getKey, { + audience: 'CLIENT_ID', + issuer: 'https://auth.example.com', + algorithms: ['RS256'] +}, (err, decoded) => { + if (err) throw err; + console.log(decoded); +}); +``` + +## Security Best Practices + +### Do + +| Practice | Rationale | +|----------|-----------| +| **Use PKCE** | Prevents code interception | +| **Validate state** | Prevents CSRF | +| **Use short-lived tokens** | Limit exposure window | +| **Store tokens securely** | HttpOnly cookies or secure storage | +| **Use HTTPS** | Prevent token interception | +| **Validate all claims** | Prevent token misuse | + +### Don't + +| Anti-Pattern | Risk | +|--------------|------| +| **Store tokens in localStorage** | XSS can steal them | +| **Include secrets in SPAs** | Visible to users | +| **Use Implicit flow** | Token in URL, no refresh | +| **Skip state validation** | CSRF vulnerability | +| **Long-lived access tokens** | Extended exposure if stolen | + +## Common Providers + +| Provider | OIDC Discovery URL | +|----------|-------------------| +| **Google** | `https://accounts.google.com/.well-known/openid-configuration` | +| **Microsoft** | `https://login.microsoftonline.com/{tenant}/v2.0/.well-known/openid-configuration` | +| **Auth0** | `https://{domain}/.well-known/openid-configuration` | +| **Okta** | `https://{domain}/.well-known/openid-configuration` | +| **Keycloak** | `https://{host}/realms/{realm}/.well-known/openid-configuration` | + +## Flow Decision Guide + +| Scenario | Recommended Flow | +|----------|------------------| +| **Server-side web app** | Authorization Code | +| **SPA (Single Page App)** | Authorization Code + PKCE | +| **Mobile app** | Authorization Code + PKCE | +| **Machine-to-machine** | Client Credentials | +| **First-party highly trusted** | Resource Owner Password (rare) | + +## Related + +- [[JWT]] — Token format details +- [[Identity and Access Management]] — IAM overview +- [[API Design Patterns]] — API security +- [[Security MOC]] — Security topics diff --git a/Tools MOC.md b/Tools MOC.md index ed37dda..33f507a 100644 --- a/Tools MOC.md +++ b/Tools MOC.md @@ -43,6 +43,22 @@ Development tools, libraries, and infrastructure across languages. - [[Package Managers]] — apt, dnf, pacman, brew, winget - [[Process Managers]] — systemd, launchd, supervisord, PM2 +### Hardware + +- [[Hardware/Mainframes|Mainframes]] — IBM z/Series enterprise systems +- [[Hardware/Server Hardware|Server Hardware]] — Rack servers, blade systems +- [[Hardware/CPU Architectures|CPU Architectures]] — x86, ARM, RISC-V +- [[Hardware/GPUs|GPUs]] — Graphics and compute accelerators +- [[Hardware/Storage Systems|Storage Systems]] — SAN, NAS, object storage +- [[Hardware/Routers and Switches|Routers and Switches]] — Network infrastructure +- [[Hardware/Firewalls|Firewalls]] — Network security devices +- [[Hardware/Network Topologies|Network Topologies]] — Network design patterns + +### Virtualization + +- [[Hardware/Hypervisors|Hypervisors]] — VMware, Hyper-V, KVM, Xen +- [[Hardware/Virtual Networking|Virtual Networking]] — SDN, VXLAN, overlay networks + ### Runtimes - [[JavaScript Runtimes]] — Node.js vs Deno vs Bun @@ -55,6 +71,11 @@ Development tools, libraries, and infrastructure across languages. - [[Container Tools]] — Portainer, Traefik, Watchtower, registries - [[Kubernetes]] — K8s, K3s, kind, minikube, Helm +### Local Development + +- [[Tools/Local Kubernetes|Local Kubernetes]] — KIND, K3s, k3d, minikube +- [[Tools/Local Observability|Local Observability]] — Grafana, Prometheus, Loki locally + ### Infrastructure & Security - [[API Gateways]] — Traffic routing, rate limiting, auth @@ -68,6 +89,16 @@ Development tools, libraries, and infrastructure across languages. - [[ORMs & Database Access]] — Data access patterns - [[Serialization]] — JSON, YAML, binary formats +### Database Deep-Dives + +- [[Tools/PostgreSQL|PostgreSQL]] — Advanced SQL features, extensions +- [[Tools/MySQL|MySQL]] — InnoDB, replication, performance +- [[Tools/SQLite|SQLite]] — Embedded database, zero-config +- [[Tools/MongoDB|MongoDB]] — Document database, aggregation +- [[Tools/Redis|Redis]] — In-memory data structures, caching +- [[Tools/Elasticsearch|Elasticsearch]] — Full-text search, analytics +- [[Tools/Database Indexing|Database Indexing]] — B-tree, hash, GIN indexes + ### Observability - [[Observability Stack]] — Metrics, logs, traces comparison @@ -77,6 +108,14 @@ Development tools, libraries, and infrastructure across languages. - [[Prometheus]] — Metrics monitoring - [[Grafana]] — Visualization and dashboards +### DevOps & SRE + +- [[Tools/SLOs and SLIs|SLOs and SLIs]] — Service level objectives, error budgets +- [[Tools/Incident Management|Incident Management]] — On-call, postmortems, blameless culture +- [[Tools/Chaos Engineering|Chaos Engineering]] — Failure injection, resilience testing +- [[Tools/Runbooks|Runbooks]] — Operational procedures, automation +- [[Tools/CICS|CICS]] — Mainframe transaction processing + ### UI - [[Terminal UI & Language Features]] — CLI and TUI libraries diff --git a/Tools/CICS.md b/Tools/CICS.md new file mode 100644 index 0000000..576171b --- /dev/null +++ b/Tools/CICS.md @@ -0,0 +1,482 @@ +--- +title: CICS +aliases: + - Customer Information Control System + - CICS TS + - CICS Transaction Server +tags: + - tools + - mainframe + - legacy + - enterprise + - transactions +type: reference +status: complete +created: "2025-12-16" +--- + +# CICS + +IBM's mainframe transaction processing system, handling billions of transactions daily across banking, retail, and enterprise systems. + +## Overview + +| Aspect | Details | +|--------|---------| +| **Full Name** | Customer Information Control System | +| **Current Version** | CICS Transaction Server (TS) 6.x | +| **First Released** | 1969 | +| **Platform** | z/OS | +| **Primary Use** | Online Transaction Processing (OLTP) | +| **Transaction Volume** | 1+ million transactions/second capable | +| **Market Reach** | Used by 90%+ of Fortune 500 | + +## Core Concepts + +### What CICS Does + +``` +┌─────────────────────────────────────────────┐ +│ Terminals │ +│ ┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐ │ +│ │3270 │ │3270 │ │ Web │ │ MQ │ │ API │ │ +│ └──┬──┘ └──┬──┘ └──┬──┘ └──┬──┘ └──┬──┘ │ +└─────┼───────┼───────┼───────┼───────┼───────┘ + │ │ │ │ │ + └───────┴───────┼───────┴───────┘ + ▼ +┌─────────────────────────────────────────────┐ +│ CICS │ +│ ┌─────────────────────────────────────┐ │ +│ │ Transaction Manager │ │ +│ │ • Task Management │ │ +│ │ • Program Control │ │ +│ │ • Terminal Control │ │ +│ │ • File Control │ │ +│ │ • Temporary Storage │ │ +│ └─────────────────────────────────────┘ │ +│ │ │ +│ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ +│ │ VSAM │ │ DB2 │ │ IMS │ │ +│ └─────────┘ └─────────┘ └─────────┘ │ +└─────────────────────────────────────────────┘ +``` + +### Key Services + +| Service | Purpose | +|---------|---------| +| **Task Management** | Concurrent transaction execution | +| **Program Control** | LINK, XCTL, RETURN | +| **Terminal Control** | Screen I/O (SEND/RECEIVE MAP) | +| **File Control** | VSAM file access | +| **Temporary Storage** | Scratch pad data | +| **Transient Data** | Queued I/O | +| **Interval Control** | Time-based operations | + +## Transaction Model + +### ACID Properties + +| Property | CICS Implementation | +|----------|---------------------| +| **Atomicity** | Syncpoint/Rollback | +| **Consistency** | Program logic | +| **Isolation** | Record-level locking | +| **Durability** | Logging, recovery | + +### Transaction Lifecycle + +``` +Terminal Input (TRANID) + ↓ +Program Load + ↓ +Execute (EXEC CICS commands) + ↓ +SYNCPOINT (commit) or ROLLBACK + ↓ +Response to Terminal +``` + +## Programming Model + +### COBOL with CICS + +```cobol + IDENTIFICATION DIVISION. + PROGRAM-ID. CUSTINQ. + + DATA DIVISION. + WORKING-STORAGE SECTION. + 01 WS-CUSTNO PIC 9(8). + 01 WS-CUSTNAME PIC X(50). + 01 WS-BALANCE PIC S9(9)V99. + 01 WS-RESP PIC S9(8) COMP. + + COPY DFHAID. + COPY CUSTMAP. + + PROCEDURE DIVISION. + EXEC CICS RECEIVE MAP('CUSTMAP') + MAPSET('CUSTSET') + INTO(CUSTMAPI) + RESP(WS-RESP) + END-EXEC + + IF WS-RESP = DFHRESP(NORMAL) + MOVE CUSTNOI TO WS-CUSTNO + PERFORM READ-CUSTOMER + PERFORM SEND-RESPONSE + END-IF + + EXEC CICS RETURN END-EXEC. + + READ-CUSTOMER. + EXEC CICS READ FILE('CUSTMAST') + INTO(CUSTOMER-RECORD) + RIDFLD(WS-CUSTNO) + RESP(WS-RESP) + END-EXEC. + + SEND-RESPONSE. + MOVE WS-CUSTNAME TO CUSTNAMEO + MOVE WS-BALANCE TO BALANCEO + EXEC CICS SEND MAP('CUSTMAP') + MAPSET('CUSTSET') + FROM(CUSTMAPO) + ERASE + END-EXEC. +``` + +### Common EXEC CICS Commands + +#### Terminal I/O + +```cobol +* Receive screen input +EXEC CICS RECEIVE MAP('mapname') + MAPSET('mapset') + INTO(data-area) +END-EXEC + +* Send screen output +EXEC CICS SEND MAP('mapname') + MAPSET('mapset') + FROM(data-area) + ERASE +END-EXEC + +* Send text +EXEC CICS SEND TEXT FROM(message) + LENGTH(msg-len) + ERASE +END-EXEC +``` + +#### File Operations + +```cobol +* Read by key +EXEC CICS READ FILE('filename') + INTO(record-area) + RIDFLD(key-field) + RESP(response) +END-EXEC + +* Read for update +EXEC CICS READ FILE('filename') + INTO(record-area) + RIDFLD(key-field) + UPDATE +END-EXEC + +* Rewrite (after READ UPDATE) +EXEC CICS REWRITE FILE('filename') + FROM(record-area) +END-EXEC + +* Write new record +EXEC CICS WRITE FILE('filename') + FROM(record-area) + RIDFLD(key-field) +END-EXEC + +* Delete +EXEC CICS DELETE FILE('filename') + RIDFLD(key-field) +END-EXEC + +* Browse +EXEC CICS STARTBR FILE('filename') + RIDFLD(key-field) + GTEQ +END-EXEC + +EXEC CICS READNEXT FILE('filename') + INTO(record-area) + RIDFLD(key-field) +END-EXEC + +EXEC CICS ENDBR FILE('filename') END-EXEC +``` + +#### Program Control + +```cobol +* Call another program, return here +EXEC CICS LINK PROGRAM('progname') + COMMAREA(data-area) + LENGTH(data-len) +END-EXEC + +* Transfer control, don't return +EXEC CICS XCTL PROGRAM('progname') + COMMAREA(data-area) +END-EXEC + +* Return to CICS +EXEC CICS RETURN END-EXEC + +* Return with next transaction +EXEC CICS RETURN TRANSID('MENU') + COMMAREA(data-area) +END-EXEC +``` + +#### Temporary Storage + +```cobol +* Write to TS queue +EXEC CICS WRITEQ TS QUEUE('queuename') + FROM(data-area) + LENGTH(data-len) + ITEM(item-number) +END-EXEC + +* Read from TS queue +EXEC CICS READQ TS QUEUE('queuename') + INTO(data-area) + LENGTH(data-len) + ITEM(item-number) +END-EXEC + +* Delete queue +EXEC CICS DELETEQ TS QUEUE('queuename') +END-EXEC +``` + +#### Error Handling + +```cobol +* Using RESP +EXEC CICS READ FILE('CUSTFILE') + INTO(CUST-REC) + RIDFLD(CUST-KEY) + RESP(WS-RESP) + RESP2(WS-RESP2) +END-EXEC + +EVALUATE WS-RESP + WHEN DFHRESP(NORMAL) + CONTINUE + WHEN DFHRESP(NOTFND) + PERFORM RECORD-NOT-FOUND + WHEN DFHRESP(LENGERR) + PERFORM LENGTH-ERROR + WHEN OTHER + PERFORM GENERAL-ERROR +END-EVALUATE + +* Using HANDLE CONDITION (legacy) +EXEC CICS HANDLE CONDITION + NOTFND(NOT-FOUND-PARA) + ERROR(ERROR-PARA) +END-EXEC +``` + +## BMS (Basic Mapping Support) + +### Map Definition + +``` +CUSTSET DFHMSD TYPE=&SYSPARM,LANG=COBOL,MODE=INOUT, * + CTRL=FREEKB,STORAGE=AUTO,TIOAPFX=YES +* +CUSTMAP DFHMDI SIZE=(24,80),LINE=1,COLUMN=1 +* + DFHMDF POS=(1,30),LENGTH=20, * + ATTRB=(ASKIP,BRT), * + INITIAL='CUSTOMER INQUIRY' +* + DFHMDF POS=(5,1),LENGTH=11, * + ATTRB=ASKIP, * + INITIAL='CUST NUMBER' +* +CUSTNO DFHMDF POS=(5,13),LENGTH=8, * + ATTRB=(UNPROT,NUM,IC) +* + DFHMDF POS=(7,1),LENGTH=11, * + ATTRB=ASKIP, * + INITIAL='CUST NAME' +* +CUSTNAME DFHMDF POS=(7,13),LENGTH=50, * + ATTRB=(ASKIP,BRT) +* + DFHMSD TYPE=FINAL + END +``` + +### Attribute Bytes + +| Attribute | Meaning | +|-----------|---------| +| **ASKIP** | Auto-skip (protected) | +| **UNPROT** | Unprotected (input) | +| **NUM** | Numeric only | +| **BRT** | Bright (highlighted) | +| **DRK** | Dark (hidden) | +| **IC** | Initial cursor position | + +## Modern CICS + +### Web Services + +```cobol +* RESTful service +EXEC CICS WEB RECEIVE + INTO(WS-REQUEST) + LENGTH(WS-REQ-LEN) +END-EXEC + +* Process JSON +EXEC CICS JSON PARSE CHANNEL('JSONCHNL') + CONTAINER('JSONDATA') + INTO(WS-DATA-AREA) +END-EXEC + +* Send response +EXEC CICS WEB SEND + FROM(WS-RESPONSE) + LENGTH(WS-RESP-LEN) + MEDIATYPE('application/json') + STATUSCODE(200) +END-EXEC +``` + +### Liberty Integration + +CICS can host Java/Liberty for modern apps: + +- JAX-RS REST APIs +- MicroProfile +- Spring Boot + +### Containers (Channels and Containers) + +```cobol +* Put data in container +EXEC CICS PUT CONTAINER('CUSTDATA') + CHANNEL('CUSTCHNL') + FROM(CUSTOMER-RECORD) +END-EXEC + +* Get data from container +EXEC CICS GET CONTAINER('CUSTDATA') + CHANNEL('CUSTCHNL') + INTO(CUSTOMER-RECORD) +END-EXEC + +* Link with channel +EXEC CICS LINK PROGRAM('CUSTPROC') + CHANNEL('CUSTCHNL') +END-EXEC +``` + +## Administration + +### Key Resources + +| Resource | Definition | Purpose | +|----------|------------|---------| +| **PROGRAM** | PPT entry | Program attributes | +| **TRANSACTION** | PCT entry | Transaction → Program mapping | +| **FILE** | FCT entry | File access definitions | +| **TDQUEUE** | DCT entry | Transient data queues | +| **TSMODEL** | TST entry | Temp storage models | + +### CEDA Commands + +``` +CEDA DEFINE PROGRAM(CUSTINQ) GROUP(CUSTGRP) +CEDA DEFINE TRANSACTION(CUST) PROGRAM(CUSTINQ) GROUP(CUSTGRP) +CEDA INSTALL GROUP(CUSTGRP) +``` + +### CEMT Commands + +``` +CEMT INQUIRE PROGRAM(CUSTINQ) +CEMT SET PROGRAM(CUSTINQ) NEWCOPY +CEMT INQUIRE TRANSACTION(CUST) +CEMT SET FILE(CUSTMAST) OPEN +``` + +## High Availability + +### Sysplex Support + +- **Workload balancing** across regions +- **Data sharing** with DB2 +- **Recovery** and restart + +### CICSPlex SM + +Central management of multiple CICS regions: + +- Workload management +- Business application management +- Real-time analysis + +## Comparison with Alternatives + +| Aspect | CICS | IMS TM | Tuxedo | +|--------|------|--------|--------| +| **Platform** | z/OS | z/OS | Unix/Linux | +| **Scale** | Massive | Massive | Large | +| **Language** | COBOL, Java, + | COBOL | C, COBOL | +| **Integration** | DB2, VSAM, MQ | IMS DB | Oracle | + +## When to Use CICS + +### Strengths + +| Strength | Rationale | +|----------|-----------| +| **Scale** | Billions of transactions daily | +| **Reliability** | Decades of proven uptime | +| **ACID** | Full transaction support | +| **Integration** | Deep z/OS ecosystem | + +### Considerations + +| Consideration | Impact | +|---------------|--------| +| **Cost** | z/OS licensing | +| **Skills** | Specialized knowledge | +| **Complexity** | Many moving parts | + +### Best For + +- High-volume OLTP +- Banking and finance +- Insurance claims +- Airline reservations +- Retail POS backend + +## Related + +- [[COBOL]] — Primary CICS language +- [[JCL]] — Batch job control +- [[Mainframes]] — Platform overview +- [[Tools MOC]] — All tools diff --git a/Tools/Chaos Engineering.md b/Tools/Chaos Engineering.md new file mode 100644 index 0000000..26c9754 --- /dev/null +++ b/Tools/Chaos Engineering.md @@ -0,0 +1,409 @@ +--- +title: Chaos Engineering +aliases: + - Chaos Monkey + - Failure Injection + - Resilience Testing + - Game Days +tags: + - tools + - devops + - sre + - testing + - reliability +type: reference +status: complete +created: "2025-12-16" +--- + +# Chaos Engineering + +The discipline of experimenting on distributed systems to build confidence in their ability to withstand turbulent conditions in production. + +## Overview + +| Aspect | Description | +|--------|-------------| +| **Definition** | Proactive failure injection to find weaknesses | +| **Origin** | Netflix (2011) - Chaos Monkey | +| **Goal** | Build resilient systems, not fragile ones | +| **Principle** | Better to fail controlled than unexpectedly | + +## Core Principles + +### The Chaos Engineering Manifesto + +1. **Build a hypothesis around steady state** +2. **Vary real-world events** +3. **Run experiments in production** +4. **Automate to run continuously** +5. **Minimize blast radius** + +### Steady State Hypothesis + +``` +"Under normal conditions, our payment service maintains +99.9% success rate and p99 latency under 200ms. + +Hypothesis: If we terminate 1 database replica, the +service will maintain these SLOs." +``` + +## Experiment Design + +### Experiment Structure + +``` +┌─────────────────────────────────────────────────────┐ +│ Chaos Experiment │ +├─────────────────────────────────────────────────────┤ +│ │ +│ 1. STEADY STATE │ +│ Define normal behavior metrics │ +│ Example: Error rate < 1%, Latency p99 < 200ms │ +│ │ +│ 2. HYPOTHESIS │ +│ What we expect to happen │ +│ Example: "System will auto-failover" │ +│ │ +│ 3. INTRODUCE CHAOS │ +│ Inject the failure │ +│ Example: Terminate database primary │ +│ │ +│ 4. OBSERVE │ +│ Monitor system behavior │ +│ Example: Watch metrics, logs, alerts │ +│ │ +│ 5. ANALYZE │ +│ Did hypothesis hold? │ +│ Example: Failover took 45s, SLO breached │ +│ │ +│ 6. IMPROVE │ +│ Fix weaknesses found │ +│ Example: Tune failover timeout │ +│ │ +└─────────────────────────────────────────────────────┘ +``` + +### Experiment Template + +```yaml +name: Database Failover Test +description: Verify system handles database primary failure +owner: platform-team +environment: staging # Start here before production + +steady_state: + metrics: + - name: success_rate + query: sum(rate(requests_success[5m])) / sum(rate(requests_total[5m])) + threshold: "> 0.999" + - name: latency_p99 + query: histogram_quantile(0.99, rate(request_duration_bucket[5m])) + threshold: "< 0.2" # 200ms + +hypothesis: > + When the database primary is terminated, the system will + failover to replica within 30 seconds and maintain SLOs. + +method: + - action: terminate_instance + target: database-primary + +rollback: + - action: restore_instance + target: database-primary + +abort_conditions: + - metric: error_rate + threshold: "> 0.1" # Abort if >10% errors + duration: 60s +``` + +## Types of Chaos Experiments + +### Infrastructure Failures + +| Experiment | Description | +|------------|-------------| +| **Instance termination** | Kill VMs/containers randomly | +| **Network partition** | Isolate services from each other | +| **Latency injection** | Add artificial delay | +| **Disk failure** | Fill disk, corrupt data | +| **CPU stress** | Exhaust compute resources | +| **Memory pressure** | Consume available memory | +| **DNS failure** | Break name resolution | + +### Application Failures + +| Experiment | Description | +|------------|-------------| +| **Exception injection** | Force error paths | +| **Dependency failure** | Mock failing upstream | +| **Resource exhaustion** | Connection pool, thread pool | +| **State corruption** | Invalid cache data | + +### Network Chaos + +| Experiment | Description | +|------------|-------------| +| **Packet loss** | Drop % of packets | +| **Latency** | Add delay to requests | +| **Bandwidth** | Limit throughput | +| **DNS manipulation** | Wrong/slow DNS | +| **TLS failures** | Certificate issues | + +## Tools + +### Chaos Monkey (Netflix) + +**The original chaos tool.** + +```yaml +# Simian Army configuration +simianarmy.chaos.enabled = true +simianarmy.chaos.probability = 1.0 +simianarmy.chaos.leashed = false +simianarmy.chaos.ASG.enabled = true +``` + +### Gremlin + +**Commercial chaos platform.** + +| Feature | Description | +|---------|-------------| +| **Attack Library** | Pre-built failure scenarios | +| **Scenarios** | Multi-step experiments | +| **Targets** | Hosts, containers, K8s | +| **Safety** | Automatic rollback | + +### Chaos Mesh (Kubernetes) + +**Cloud-native chaos for K8s.** + +```yaml +apiVersion: chaos-mesh.org/v1alpha1 +kind: PodChaos +metadata: + name: pod-failure-example +spec: + action: pod-failure + mode: one + selector: + namespaces: + - production + labelSelectors: + app: payment-service + duration: "60s" +``` + +### Litmus (Kubernetes) + +**CNCF chaos engineering project.** + +```yaml +apiVersion: litmuschaos.io/v1alpha1 +kind: ChaosEngine +metadata: + name: nginx-chaos +spec: + appinfo: + appns: default + applabel: "app=nginx" + chaosServiceAccount: litmus-admin + experiments: + - name: pod-delete + spec: + components: + env: + - name: TOTAL_CHAOS_DURATION + value: "30" + - name: CHAOS_INTERVAL + value: "10" +``` + +### AWS Fault Injection Simulator + +**AWS-native chaos service.** + +```json +{ + "description": "Terminate random EC2 instances", + "targets": { + "instances": { + "resourceType": "aws:ec2:instance", + "selectionMode": "COUNT(1)", + "resourceTags": { + "Environment": "production" + } + } + }, + "actions": { + "terminateInstances": { + "actionId": "aws:ec2:terminate-instances", + "targets": { + "Instances": "instances" + } + } + } +} +``` + +### tc (Traffic Control) - Linux + +**Network chaos with Linux tools.** + +```bash +# Add 100ms latency to eth0 +tc qdisc add dev eth0 root netem delay 100ms + +# Add 10% packet loss +tc qdisc add dev eth0 root netem loss 10% + +# Limit bandwidth to 1mbit +tc qdisc add dev eth0 root tbf rate 1mbit burst 32kbit latency 400ms + +# Remove chaos +tc qdisc del dev eth0 root +``` + +### Toxiproxy (Shopify) + +**Proxy for simulating network conditions.** + +```bash +# Create proxy +toxiproxy-cli create redis -l localhost:26379 -u localhost:6379 + +# Add latency +toxiproxy-cli toxic add redis -t latency -a latency=1000 + +# Add timeout (connection drop) +toxiproxy-cli toxic add redis -t timeout -a timeout=5000 +``` + +## Tool Comparison + +| Tool | Platform | OSS | Key Feature | +|------|----------|-----|-------------| +| **Chaos Monkey** | AWS | ✅ | Original, simple | +| **Gremlin** | Multi | ❌ | Enterprise features | +| **Chaos Mesh** | K8s | ✅ | K8s native | +| **Litmus** | K8s | ✅ | CNCF project | +| **AWS FIS** | AWS | ❌ | AWS native | +| **Toxiproxy** | Any | ✅ | Network simulation | +| **Pumba** | Docker | ✅ | Container chaos | + +## Game Days + +### What is a Game Day? + +**Scheduled chaos exercise** with the team. + +``` +┌─────────────────────────────────────────────────────┐ +│ Game Day Structure │ +├─────────────────────────────────────────────────────┤ +│ │ +│ BEFORE (1 week prior) │ +│ • Define scenarios │ +│ • Notify stakeholders │ +│ • Prepare rollback procedures │ +│ • Brief participants │ +│ │ +│ DURING (2-4 hours) │ +│ • Run experiments │ +│ • Observe team response │ +│ • Document findings │ +│ • Practice incident response │ +│ │ +│ AFTER (1 week following) │ +│ • Review results │ +│ • Create action items │ +│ • Update runbooks │ +│ • Share learnings │ +│ │ +└─────────────────────────────────────────────────────┘ +``` + +### Game Day Scenarios + +| Scenario | Tests | +|----------|-------| +| **Zone failure** | Multi-AZ resilience | +| **Database failover** | HA configuration | +| **Dependency outage** | Circuit breakers | +| **Traffic spike** | Auto-scaling | +| **On-call handoff** | Process and documentation | +| **Secret rotation** | Credential management | + +## Best Practices + +### Start Small + +``` +Progression: +1. Development environment +2. Staging environment +3. Production (canary) +4. Production (wider) +``` + +### Safety Mechanisms + +| Mechanism | Purpose | +|-----------|---------| +| **Abort conditions** | Stop if impact exceeds threshold | +| **Blast radius limits** | Contain experiment scope | +| **Time limits** | Maximum experiment duration | +| **Rollback automation** | Quick recovery | +| **Runbook ready** | Manual recovery documented | + +### What Not to Do + +| Anti-Pattern | Risk | +|--------------|------| +| **No hypothesis** | Can't evaluate results | +| **No monitoring** | Can't observe impact | +| **Skip staging** | Unknown production risk | +| **No communication** | Surprise outages | +| **No rollback plan** | Extended incidents | + +## Maturity Model + +| Level | Description | +|-------|-------------| +| **1. Ad-hoc** | Manual, occasional experiments | +| **2. Defined** | Documented experiments, game days | +| **3. Automated** | Continuous chaos in staging | +| **4. Advanced** | Continuous chaos in production | +| **5. Optimized** | AI-driven chaos, self-healing | + +## Metrics + +### Experiment Metrics + +| Metric | Measures | +|--------|----------| +| **Experiments run** | Volume of testing | +| **Weaknesses found** | Effectiveness | +| **MTTR during chaos** | Recovery capability | +| **False positives** | Alert quality | + +### Program Metrics + +| Metric | Measures | +|--------|----------| +| **Coverage** | % of services tested | +| **Frequency** | Experiments per month | +| **Fix rate** | Issues resolved after finding | +| **Production incidents** | Should decrease over time | + +## Related + +- [[Incident Management]] — When experiments reveal issues +- [[SLOs and SLIs]] — Defining steady state +- [[Runbooks]] — Recovery procedures +- [[Testing Strategies]] — Testing approaches +- [[Tools MOC]] — All tools diff --git a/Tools/Database Indexing.md b/Tools/Database Indexing.md new file mode 100644 index 0000000..e97e68d --- /dev/null +++ b/Tools/Database Indexing.md @@ -0,0 +1,378 @@ +--- +title: Database Indexing +aliases: + - Indexing Strategies + - Index Types + - Query Optimization +tags: + - tools + - database + - performance + - optimization +type: reference +status: complete +created: "2025-12-16" +--- + +# Database Indexing + +Strategies for creating and using indexes to optimize database query performance. + +## Overview + +| Aspect | Description | +|--------|-------------| +| **What** | Data structures for fast lookups | +| **Trade-off** | Faster reads vs slower writes + storage | +| **Goal** | Minimize disk I/O and row scans | + +## How Indexes Work + +### Without Index (Full Table Scan) + +``` +Query: WHERE email = 'alice@example.com' + +Table: users (1M rows) +┌────┬────────────────────┬─────────┐ +│ id │ email │ name │ ← Must scan ALL rows +├────┼────────────────────┼─────────┤ +│ 1 │ bob@example.com │ Bob │ Check +│ 2 │ alice@example.com │ Alice │ ✓ Found! +│ 3 │ carol@example.com │ Carol │ Check (continues...) +│... │ ... │ ... │ +│ 1M │ zack@example.com │ Zack │ +└────┴────────────────────┴─────────┘ +``` + +### With B-tree Index + +``` +Query: WHERE email = 'alice@example.com' + +B-tree Index on email: + ┌─────────────┐ + │ m... │ + └──────┬──────┘ + ┌────────────┼────────────┐ + ▼ ▼ ▼ + ┌─────────┐ ┌─────────┐ ┌─────────┐ + │ a-f │ │ g-l │ │ m-z │ + └────┬────┘ └─────────┘ └─────────┘ + │ + ┌────▼────┐ + │ alice@ │ → Row pointer → Directly to row + │ bob@ │ + └─────────┘ + +O(log n) lookups instead of O(n) +``` + +## Index Types + +### B-tree Index + +**Default in most databases. Good for:** + +- Equality: `WHERE x = 5` +- Range: `WHERE x > 5` +- Sorting: `ORDER BY x` +- Prefix: `WHERE x LIKE 'abc%'` + +```sql +-- PostgreSQL, MySQL, SQL Server +CREATE INDEX idx_email ON users(email); + +-- Works with: +SELECT * FROM users WHERE email = 'test@example.com'; +SELECT * FROM users WHERE email > 'a' AND email < 'b'; +SELECT * FROM users ORDER BY email; +SELECT * FROM users WHERE email LIKE 'test%'; +``` + +### Hash Index + +**Fast equality lookups only.** + +```sql +-- PostgreSQL +CREATE INDEX idx_email_hash ON users USING HASH(email); + +-- Works with: +SELECT * FROM users WHERE email = 'test@example.com'; + +-- Does NOT work with: +SELECT * FROM users WHERE email > 'a'; -- No range +SELECT * FROM users ORDER BY email; -- No sorting +``` + +### GIN (Generalized Inverted Index) + +**For values containing multiple elements.** + +```sql +-- PostgreSQL: Arrays, JSONB, full-text +CREATE INDEX idx_tags ON posts USING GIN(tags); +CREATE INDEX idx_data ON events USING GIN(data jsonb_path_ops); +CREATE INDEX idx_search ON articles USING GIN(to_tsvector('english', body)); + +-- Works with: +SELECT * FROM posts WHERE tags @> ARRAY['redis']; +SELECT * FROM events WHERE data @> '{"type": "click"}'; +SELECT * FROM articles WHERE to_tsvector('english', body) @@ to_tsquery('database'); +``` + +### GiST (Generalized Search Tree) + +**For geometric and range data.** + +```sql +-- PostgreSQL: PostGIS, ranges, full-text +CREATE INDEX idx_location ON places USING GIST(location); +CREATE INDEX idx_schedule ON events USING GIST(time_range); + +-- Works with: +SELECT * FROM places WHERE location <@ box '((0,0),(10,10))'; +SELECT * FROM events WHERE time_range && '[2024-01-01, 2024-01-31]'; +``` + +### BRIN (Block Range Index) + +**For large, naturally ordered data.** + +```sql +-- PostgreSQL: Time-series, logs +CREATE INDEX idx_created ON logs USING BRIN(created_at); + +-- Very small index size +-- Best when data is inserted in order +``` + +### Full-Text Index + +```sql +-- PostgreSQL +CREATE INDEX idx_search ON articles USING GIN(to_tsvector('english', title || ' ' || body)); + +-- MySQL +CREATE FULLTEXT INDEX idx_search ON articles(title, body); + +-- Query +SELECT * FROM articles WHERE MATCH(title, body) AGAINST('database performance'); +``` + +## Composite Indexes + +### Index Column Order Matters + +```sql +CREATE INDEX idx_composite ON orders(customer_id, status, created_at); +``` + +**Leftmost Prefix Rule:** + +| Query | Uses Index? | +|-------|-------------| +| `WHERE customer_id = 1` | ✅ Yes | +| `WHERE customer_id = 1 AND status = 'pending'` | ✅ Yes | +| `WHERE customer_id = 1 AND status = 'pending' AND created_at > '2024-01-01'` | ✅ Yes | +| `WHERE status = 'pending'` | ❌ No (missing leftmost) | +| `WHERE customer_id = 1 AND created_at > '2024-01-01'` | ⚠️ Partial (uses customer_id only) | + +### Choosing Column Order + +```sql +-- Put equality conditions first, range conditions last +-- High cardinality columns first (usually) + +-- Good for: WHERE customer_id = ? AND status = ? ORDER BY created_at +CREATE INDEX idx_orders ON orders(customer_id, status, created_at); + +-- Good for: WHERE customer_id = ? ORDER BY created_at DESC +CREATE INDEX idx_orders_recent ON orders(customer_id, created_at DESC); +``` + +## Covering Indexes + +**Index contains all columns needed—no table lookup required.** + +```sql +-- PostgreSQL (INCLUDE) +CREATE INDEX idx_orders_covering ON orders(customer_id) + INCLUDE (total, status); + +-- MySQL (columns in index) +CREATE INDEX idx_orders_covering ON orders(customer_id, total, status); + +-- Query satisfied entirely from index: +SELECT total, status FROM orders WHERE customer_id = 123; +``` + +## Partial Indexes + +**Index only a subset of rows.** + +```sql +-- PostgreSQL +CREATE INDEX idx_active_users ON users(email) + WHERE active = true; + +-- Only 10% of users are active = much smaller index +-- Query must match the WHERE clause to use it: +SELECT * FROM users WHERE email = 'test@example.com' AND active = true; +``` + +## Expression Indexes + +**Index on a computed value.** + +```sql +-- PostgreSQL +CREATE INDEX idx_lower_email ON users(LOWER(email)); + +-- MySQL +CREATE INDEX idx_lower_email ON users((LOWER(email))); + +-- Query must use same expression: +SELECT * FROM users WHERE LOWER(email) = 'test@example.com'; +``` + +## Index Anti-Patterns + +### Functions on Indexed Columns + +```sql +-- ❌ Bad: Index not used +SELECT * FROM orders WHERE YEAR(created_at) = 2024; + +-- ✅ Good: Index can be used +SELECT * FROM orders +WHERE created_at >= '2024-01-01' AND created_at < '2025-01-01'; +``` + +### Leading Wildcards + +```sql +-- ❌ Bad: Full scan +SELECT * FROM users WHERE email LIKE '%@gmail.com'; + +-- ✅ Good: Uses index +SELECT * FROM users WHERE email LIKE 'john%'; + +-- Alternative: Full-text search or reverse index +``` + +### OR Conditions + +```sql +-- ❌ Potentially slow: May not use index +SELECT * FROM orders WHERE customer_id = 1 OR status = 'pending'; + +-- ✅ Better: UNION (if both columns indexed) +SELECT * FROM orders WHERE customer_id = 1 +UNION +SELECT * FROM orders WHERE status = 'pending'; +``` + +### Over-Indexing + +```sql +-- ❌ Too many indexes: +CREATE INDEX idx1 ON users(email); +CREATE INDEX idx2 ON users(email, name); +CREATE INDEX idx3 ON users(email, name, created_at); +CREATE INDEX idx4 ON users(name); +CREATE INDEX idx5 ON users(name, email); + +-- Problems: +-- - Slow INSERT/UPDATE/DELETE +-- - Increased storage +-- - Optimizer confusion +``` + +## Analyzing Index Usage + +### PostgreSQL + +```sql +-- Explain query +EXPLAIN (ANALYZE, BUFFERS) SELECT * FROM users WHERE email = 'test@example.com'; + +-- Index usage stats +SELECT relname, indexrelname, idx_scan, idx_tup_read, idx_tup_fetch +FROM pg_stat_user_indexes +ORDER BY idx_scan DESC; + +-- Unused indexes +SELECT * FROM pg_stat_user_indexes WHERE idx_scan = 0; + +-- Table statistics +ANALYZE users; +SELECT * FROM pg_stats WHERE tablename = 'users'; +``` + +### MySQL + +```sql +-- Explain query +EXPLAIN SELECT * FROM users WHERE email = 'test@example.com'; + +-- Index usage +SELECT * FROM sys.schema_index_statistics +WHERE table_schema = 'mydb'; + +-- Unused indexes +SELECT * FROM sys.schema_unused_indexes; + +-- Update statistics +ANALYZE TABLE users; +``` + +## Index Maintenance + +### When to Rebuild + +| Scenario | Action | +|----------|--------| +| **Large deletes** | Rebuild to reclaim space | +| **Statistics stale** | ANALYZE table | +| **Index bloat (PostgreSQL)** | REINDEX or pg_repack | +| **Fragmentation (SQL Server)** | REORGANIZE or REBUILD | + +### Commands + +```sql +-- PostgreSQL +REINDEX INDEX idx_email; +REINDEX TABLE users; +VACUUM ANALYZE users; + +-- MySQL +OPTIMIZE TABLE users; +ALTER TABLE users ENGINE=InnoDB; -- Rebuild + +-- SQL Server +ALTER INDEX idx_email ON users REORGANIZE; +ALTER INDEX idx_email ON users REBUILD; +``` + +## Decision Guide + +| Scenario | Index Type | +|----------|------------| +| **Equality lookups** | B-tree or Hash | +| **Range queries** | B-tree | +| **Full-text search** | GIN/Full-text | +| **JSON/Array contains** | GIN | +| **Geospatial** | GiST | +| **Time-series (ordered)** | BRIN | +| **Frequently filtered subset** | Partial index | +| **Case-insensitive search** | Expression index | +| **Query needs specific columns** | Covering index | + +## Related + +- [[PostgreSQL]] — PostgreSQL specifics +- [[MySQL]] — MySQL specifics +- [[Database Engines]] — Database comparison +- [[Tools MOC]] — All tools diff --git a/Tools/Elasticsearch.md b/Tools/Elasticsearch.md new file mode 100644 index 0000000..33d7418 --- /dev/null +++ b/Tools/Elasticsearch.md @@ -0,0 +1,508 @@ +--- +title: Elasticsearch +aliases: + - ES + - Elastic + - ELK Stack +tags: + - tool + - database + - search + - analytics +type: reference +status: complete +created: "2025-12-18" +--- + +# Elasticsearch + +Distributed search and analytics engine built on Apache Lucene. + +## Overview + +| Aspect | Details | +|--------|---------| +| **Type** | Search engine / Document store | +| **Based On** | Apache Lucene | +| **Protocol** | REST API (JSON) | +| **License** | SSPL / Elastic License | +| **Written In** | Java | +| **Use Cases** | Full-text search, log analytics, APM | + +## Core Concepts + +| Concept | Description | +|---------|-------------| +| **Index** | Collection of documents (like a database) | +| **Document** | JSON record (like a row) | +| **Field** | Key-value pair in document | +| **Mapping** | Schema definition for fields | +| **Shard** | Horizontal partition of index | +| **Replica** | Copy of shard for redundancy | +| **Node** | Single Elasticsearch instance | +| **Cluster** | Group of nodes | + +## Architecture + +``` +┌─────────────────────────────────────────────────────┐ +│ Cluster │ +│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ +│ │ Node 1 │ │ Node 2 │ │ Node 3 │ │ +│ │ (Master) │ │ (Data) │ │ (Data) │ │ +│ │ ┌─────────┐ │ │ ┌─────────┐ │ │ ┌─────────┐ │ │ +│ │ │Shard P0 │ │ │ │Shard P1 │ │ │ │Shard P2 │ │ │ +│ │ │Shard R1 │ │ │ │Shard R2 │ │ │ │Shard R0 │ │ │ +│ │ └─────────┘ │ │ └─────────┘ │ │ └─────────┘ │ │ +│ └─────────────┘ └─────────────┘ └─────────────┘ │ +└─────────────────────────────────────────────────────┘ + +P = Primary shard, R = Replica shard +``` + +## Basic Operations + +### Index Management + +```bash +# Create index +PUT /products +{ + "settings": { + "number_of_shards": 3, + "number_of_replicas": 1 + } +} + +# Check index exists +HEAD /products + +# Get index info +GET /products + +# Delete index +DELETE /products + +# List all indices +GET /_cat/indices?v +``` + +### Document CRUD + +```bash +# Create document (auto ID) +POST /products/_doc +{ + "name": "Laptop", + "price": 999.99, + "category": "electronics" +} + +# Create document (specific ID) +PUT /products/_doc/1 +{ + "name": "Laptop", + "price": 999.99, + "category": "electronics" +} + +# Get document +GET /products/_doc/1 + +# Update document (partial) +POST /products/_update/1 +{ + "doc": { + "price": 899.99 + } +} + +# Delete document +DELETE /products/_doc/1 + +# Bulk operations +POST /_bulk +{"index": {"_index": "products", "_id": "1"}} +{"name": "Laptop", "price": 999} +{"index": {"_index": "products", "_id": "2"}} +{"name": "Mouse", "price": 29} +{"delete": {"_index": "products", "_id": "3"}} +``` + +## Mappings + +### Define Schema + +```bash +PUT /products +{ + "mappings": { + "properties": { + "name": { "type": "text" }, + "sku": { "type": "keyword" }, + "price": { "type": "float" }, + "quantity": { "type": "integer" }, + "created": { "type": "date" }, + "in_stock": { "type": "boolean" }, + "tags": { "type": "keyword" }, + "description": { + "type": "text", + "analyzer": "english" + }, + "location": { "type": "geo_point" } + } + } +} +``` + +### Field Types + +| Type | Use Case | +|------|----------| +| `text` | Full-text search (analyzed) | +| `keyword` | Exact match, aggregations | +| `long/integer/short/byte` | Numeric | +| `float/double` | Floating point | +| `boolean` | True/false | +| `date` | Datetime | +| `geo_point` | Lat/lon coordinates | +| `nested` | Arrays of objects | +| `object` | JSON objects | + +## Search Queries + +### Match Queries (Full-Text) + +```bash +# Simple match +GET /products/_search +{ + "query": { + "match": { + "description": "wireless bluetooth" + } + } +} + +# Match phrase +GET /products/_search +{ + "query": { + "match_phrase": { + "description": "noise cancelling" + } + } +} + +# Multi-match +GET /products/_search +{ + "query": { + "multi_match": { + "query": "laptop", + "fields": ["name^3", "description"] + } + } +} +``` + +### Term Queries (Exact Match) + +```bash +# Term (exact match on keyword) +GET /products/_search +{ + "query": { + "term": { + "category": "electronics" + } + } +} + +# Terms (multiple values) +GET /products/_search +{ + "query": { + "terms": { + "tags": ["sale", "new"] + } + } +} + +# Range +GET /products/_search +{ + "query": { + "range": { + "price": { + "gte": 100, + "lte": 500 + } + } + } +} + +# Exists +GET /products/_search +{ + "query": { + "exists": { + "field": "discount" + } + } +} +``` + +### Compound Queries + +```bash +# Bool query +GET /products/_search +{ + "query": { + "bool": { + "must": [ + { "match": { "description": "laptop" } } + ], + "filter": [ + { "term": { "in_stock": true } }, + { "range": { "price": { "lte": 1000 } } } + ], + "should": [ + { "term": { "brand": "apple" } } + ], + "must_not": [ + { "term": { "category": "refurbished" } } + ], + "minimum_should_match": 1 + } + } +} +``` + +| Clause | Behavior | Affects Score | +|--------|----------|---------------| +| `must` | Required | ✅ Yes | +| `filter` | Required | ❌ No (cached) | +| `should` | Optional boost | ✅ Yes | +| `must_not` | Excluded | ❌ No | + +## Aggregations + +### Metric Aggregations + +```bash +GET /products/_search +{ + "size": 0, + "aggs": { + "avg_price": { "avg": { "field": "price" } }, + "max_price": { "max": { "field": "price" } }, + "min_price": { "min": { "field": "price" } }, + "total_value": { "sum": { "field": "price" } }, + "price_stats": { "stats": { "field": "price" } } + } +} +``` + +### Bucket Aggregations + +```bash +GET /products/_search +{ + "size": 0, + "aggs": { + "by_category": { + "terms": { + "field": "category", + "size": 10 + }, + "aggs": { + "avg_price": { "avg": { "field": "price" } } + } + }, + "price_ranges": { + "range": { + "field": "price", + "ranges": [ + { "to": 100 }, + { "from": 100, "to": 500 }, + { "from": 500 } + ] + } + }, + "by_date": { + "date_histogram": { + "field": "created", + "calendar_interval": "month" + } + } + } +} +``` + +## Analyzers + +### Built-in Analyzers + +| Analyzer | Description | +|----------|-------------| +| `standard` | Default, Unicode text | +| `simple` | Lowercase, split on non-letters | +| `whitespace` | Split on whitespace | +| `english` | English stemming, stopwords | +| `keyword` | No analysis (exact) | + +### Test Analyzer + +```bash +GET /_analyze +{ + "analyzer": "standard", + "text": "The Quick Brown Fox" +} +# Result: ["the", "quick", "brown", "fox"] + +GET /_analyze +{ + "analyzer": "english", + "text": "running runners ran" +} +# Result: ["run", "runner", "ran"] +``` + +### Custom Analyzer + +```bash +PUT /my_index +{ + "settings": { + "analysis": { + "analyzer": { + "my_analyzer": { + "type": "custom", + "tokenizer": "standard", + "filter": ["lowercase", "asciifolding", "my_stemmer"] + } + }, + "filter": { + "my_stemmer": { + "type": "stemmer", + "language": "english" + } + } + } + } +} +``` + +## Index Templates + +```bash +PUT /_index_template/logs_template +{ + "index_patterns": ["logs-*"], + "template": { + "settings": { + "number_of_shards": 1 + }, + "mappings": { + "properties": { + "@timestamp": { "type": "date" }, + "message": { "type": "text" }, + "level": { "type": "keyword" }, + "service": { "type": "keyword" } + } + } + } +} +``` + +## ELK Stack + +``` +┌───────────┐ ┌───────────────┐ ┌─────────────┐ +│ Beats │───▶│ Logstash │───▶│Elasticsearch│ +│(Filebeat) │ │ (Processing) │ │ (Storage) │ +└───────────┘ └───────────────┘ └──────┬──────┘ + │ + ┌──────▼──────┐ + │ Kibana │ + │(Visualize) │ + └─────────────┘ +``` + +| Component | Purpose | +|-----------|---------| +| **Elasticsearch** | Store and search data | +| **Logstash** | Ingest and transform data | +| **Kibana** | Visualize and explore data | +| **Beats** | Lightweight data shippers | + +## Client Libraries + +### Node.js + +```javascript +const { Client } = require('@elastic/elasticsearch'); + +const client = new Client({ node: 'http://localhost:9200' }); + +// Index document +await client.index({ + index: 'products', + id: '1', + document: { name: 'Laptop', price: 999 } +}); + +// Search +const result = await client.search({ + index: 'products', + query: { match: { name: 'laptop' } } +}); + +console.log(result.hits.hits); +``` + +### Python + +```python +from elasticsearch import Elasticsearch + +es = Elasticsearch(['http://localhost:9200']) + +# Index document +es.index(index='products', id=1, document={'name': 'Laptop', 'price': 999}) + +# Search +result = es.search(index='products', query={'match': {'name': 'laptop'}}) +print(result['hits']['hits']) +``` + +## Performance Tips + +| Tip | Rationale | +|-----|-----------| +| **Use filter context** | Cached, faster than query | +| **Avoid wildcard prefix** | `*foo` scans entire index | +| **Use keyword for exact** | Don't analyze when unnecessary | +| **Tune shard size** | 10-50GB per shard ideal | +| **Use bulk API** | Much faster than individual requests | +| **Set refresh_interval** | Increase for write-heavy loads | + +## When to Use + +| Good For | Not Ideal For | +|----------|---------------| +| ✅ Full-text search | ❌ Primary data store | +| ✅ Log analytics | ❌ Transactions | +| ✅ Real-time dashboards | ❌ Frequent updates | +| ✅ Autocomplete | ❌ Small datasets | +| ✅ Geospatial queries | ❌ Simple key-value | + +## Related + +- [[PostgreSQL]] — SQL full-text search alternative +- [[Redis]] — Caching layer +- [[Prometheus]] — Metrics (not logs) +- [[Tools MOC]] — All tools diff --git a/Tools/Incident Management.md b/Tools/Incident Management.md new file mode 100644 index 0000000..941637d --- /dev/null +++ b/Tools/Incident Management.md @@ -0,0 +1,393 @@ +--- +title: Incident Management +aliases: + - Incident Response + - On-Call + - Postmortems + - Blameless Culture +tags: + - tools + - devops + - sre + - operations + - reliability +type: reference +status: complete +created: "2025-12-16" +--- + +# Incident Management + +Structured approach to detecting, responding to, resolving, and learning from production incidents. + +## Overview + +| Phase | Focus | +|-------|-------| +| **Detection** | Monitoring, alerting, customer reports | +| **Response** | Triage, communication, assembly | +| **Mitigation** | Stop the bleeding | +| **Resolution** | Full fix and recovery | +| **Learning** | Postmortem, action items | + +## Incident Lifecycle + +``` +Detection → Triage → Response → Mitigation → Resolution → Postmortem + │ │ │ │ │ │ + ▼ ▼ ▼ ▼ ▼ ▼ + Alert Severity Assemble Stop the Full fix Learn & + fires assign team bleeding deployed improve +``` + +## Severity Levels + +| Level | Name | Description | Response | +|-------|------|-------------|----------| +| **SEV1** | Critical | Complete outage, data loss | Immediate, all hands | +| **SEV2** | Major | Significant degradation | Immediate, on-call team | +| **SEV3** | Minor | Limited impact | Business hours | +| **SEV4** | Low | Minimal impact | Ticket queue | + +### Severity Matrix + +| Impact | Widespread | Limited | Minimal | +|--------|------------|---------|---------| +| **Complete loss** | SEV1 | SEV2 | SEV3 | +| **Degraded** | SEV2 | SEV3 | SEV4 | +| **Inconvenience** | SEV3 | SEV4 | SEV4 | + +## Incident Roles + +### Core Roles + +| Role | Responsibilities | +|------|-----------------| +| **Incident Commander (IC)** | Overall coordination, decisions | +| **Communications Lead** | Status updates, stakeholder comms | +| **Operations Lead** | Technical investigation, mitigation | +| **Scribe** | Document timeline, actions, decisions | + +### Incident Commander Duties + +``` +┌─────────────────────────────────────────────────────┐ +│ Incident Commander │ +├─────────────────────────────────────────────────────┤ +│ │ +│ • Declare incident severity │ +│ • Assign roles │ +│ • Coordinate responders │ +│ • Make decisions (or delegate) │ +│ • Keep focus on mitigation │ +│ • Call for escalation when needed │ +│ • Declare incident resolved │ +│ │ +│ ❌ NOT responsible for: │ +│ • Doing the technical work │ +│ • Knowing all the answers │ +│ │ +└─────────────────────────────────────────────────────┘ +``` + +## Incident Response Process + +### 1. Detection & Declaration + +```markdown +## Detection Checklist +- [ ] Alert fired (or customer report received) +- [ ] Verify alert is valid (not false positive) +- [ ] Assess initial scope and impact +- [ ] Declare incident and severity +- [ ] Create incident channel/bridge +- [ ] Page appropriate responders +``` + +### 2. Assembly & Triage + +```markdown +## Initial Response (First 5 Minutes) +- [ ] IC identified and announced +- [ ] Roles assigned (Comms, Ops, Scribe) +- [ ] Initial status posted +- [ ] Known information shared +- [ ] Initial theories identified +``` + +### 3. Investigation & Mitigation + +```markdown +## Investigation +- [ ] Check recent deployments +- [ ] Review recent config changes +- [ ] Examine metrics and logs +- [ ] Check dependencies +- [ ] Identify blast radius + +## Mitigation Options +- [ ] Rollback deployment +- [ ] Toggle feature flag +- [ ] Scale up resources +- [ ] Failover to backup +- [ ] Enable degraded mode +``` + +### 4. Communication + +**Internal Status Updates:** + +```markdown +## Status Update - 14:35 UTC + +**Severity:** SEV1 +**Status:** Investigating +**Duration:** 15 minutes + +**Impact:** +- Payment processing failing for ~30% of users +- Error rate: 28% (normally <1%) + +**Current Actions:** +- Investigating correlation with deploy at 14:15 +- Preparing rollback as mitigation option + +**Next Update:** 14:45 UTC or on significant change +``` + +**External Communication:** + +```markdown +## Statuspage Update + +**Investigating Payment Issues** + +We are currently investigating issues with payment +processing. Some users may experience failures when +completing purchases. + +Our team is actively working on resolution. We will +provide updates as we have more information. + +Posted: 14:35 UTC +``` + +### 5. Resolution + +```markdown +## Resolution Checklist +- [ ] Service restored to normal operation +- [ ] Metrics confirm recovery +- [ ] Customer-facing comms updated +- [ ] Incident timeline documented +- [ ] IC declares incident resolved +- [ ] Schedule postmortem +``` + +## On-Call Best Practices + +### On-Call Rotation + +| Practice | Rationale | +|----------|-----------| +| **1-week rotations** | Long enough to learn, short enough to not burn out | +| **Primary + Secondary** | Backup for coverage gaps | +| **Follow-the-sun** | 24/7 without night shifts | +| **Minimum team size: 6** | Sustainable rotation | + +### On-Call Responsibilities + +```markdown +## On-Call Engineer Duties + +**During Shift:** +- Respond to pages within SLA (typically 5-15 min) +- Acknowledge or escalate all alerts +- Update incident channels +- Perform handoff at rotation end + +**Page Response:** +1. Acknowledge the page +2. Assess severity +3. Begin investigation or escalate +4. Communicate status +``` + +### Reducing On-Call Burden + +| Action | Impact | +|--------|--------| +| **Fix noisy alerts** | Fewer false positives | +| **Automate remediation** | Reduce human intervention | +| **Improve runbooks** | Faster resolution | +| **Invest in reliability** | Fewer incidents overall | + +## Postmortems + +### Blameless Culture + +```markdown +## Blameless Postmortem Principles + +1. **Assume good intent** - People tried to do the right thing +2. **Focus on systems** - What allowed this to happen? +3. **Learn, don't blame** - Goal is improvement, not punishment +4. **Share widely** - Transparency builds trust +5. **Follow through** - Action items must be completed +``` + +### Postmortem Template + +```markdown +# Incident Postmortem: [Title] + +**Date:** 2024-01-15 +**Duration:** 45 minutes +**Severity:** SEV2 +**Author:** @jane +**Reviewers:** @ops-team + +## Summary +Brief description of what happened and impact. + +## Impact +- 30% of payment requests failed +- ~500 affected transactions +- Estimated revenue impact: $X + +## Timeline (UTC) +| Time | Event | +|------|-------| +| 14:15 | Deploy of payment-service v2.3.4 | +| 14:20 | First alert: Error rate > 10% | +| 14:22 | On-call acknowledges, begins investigation | +| 14:30 | Correlation with deploy identified | +| 14:35 | Rollback initiated | +| 14:40 | Rollback complete, errors decreasing | +| 14:50 | Error rate normal, incident resolved | + +## Root Cause +The deploy included a database query that performed a +full table scan instead of using an index, causing +connection pool exhaustion under load. + +## Contributing Factors +- Query was tested but not at production scale +- Load testing does not cover payment flow +- No query performance monitoring in CI + +## What Went Well +- Fast detection (5 minutes) +- Quick correlation with deploy +- Rollback was smooth + +## What Could Be Improved +- Detection could have been faster with better metrics +- Load testing should cover critical paths +- Query review process needed + +## Action Items +| Action | Owner | Due | Status | +|--------|-------|-----|--------| +| Add query performance tests | @jane | 2024-01-22 | TODO | +| Implement load testing for payments | @bob | 2024-01-29 | TODO | +| Add connection pool metrics | @alice | 2024-01-20 | TODO | +| Review deploy process | @ops | 2024-01-25 | TODO | + +## Lessons Learned +- Database queries need production-scale testing +- Connection pool exhaustion is a common failure mode +- Quick rollback capability is essential +``` + +### Postmortem Meeting + +| Phase | Duration | Focus | +|-------|----------|-------| +| **Review timeline** | 10 min | What happened when | +| **Root cause analysis** | 15 min | Why did this happen | +| **Contributing factors** | 10 min | What enabled this | +| **Action items** | 15 min | What will we do | +| **Wrap up** | 5 min | Next steps, owners | + +## Tools + +### Incident Management Platforms + +| Tool | Features | +|------|----------| +| **PagerDuty** | Alerting, on-call, incident response | +| **Opsgenie** | Alerting, on-call management | +| **Incident.io** | Incident response, postmortems | +| **Rootly** | Incident management, Slack-native | +| **FireHydrant** | Incident response automation | +| **Jira Service Management** | ITSM, incident tracking | + +### Status Pages + +| Tool | Type | +|------|------| +| **Statuspage (Atlassian)** | Hosted | +| **Cachet** | Self-hosted | +| **Instatus** | Hosted | +| **Status.io** | Hosted | + +### Communication + +| Tool | Use | +|------|-----| +| **Slack/Teams** | Incident channels | +| **Zoom/Meet** | War room bridges | +| **Statuspage** | External communication | + +## Metrics + +### Incident Metrics + +| Metric | Definition | +|--------|------------| +| **MTTD** | Mean Time to Detect | +| **MTTA** | Mean Time to Acknowledge | +| **MTTR** | Mean Time to Resolve | +| **MTBF** | Mean Time Between Failures | +| **Incident Count** | By severity, by service | + +### Tracking Example + +``` +┌─────────────────────────────────────────────────────┐ +│ Incident Metrics - Q4 2024 │ +├─────────────────────────────────────────────────────┤ +│ │ +│ Total Incidents: 23 │ +│ SEV1: 2 SEV2: 8 SEV3: 13 │ +│ │ +│ MTTD: 4.2 min (target: <5 min) ✅ │ +│ MTTA: 6.8 min (target: <15 min) ✅ │ +│ MTTR: 38 min (target: <60 min) ✅ │ +│ │ +│ Top Causes: │ +│ 1. Deploy issues (35%) │ +│ 2. Dependency failures (26%) │ +│ 3. Resource exhaustion (22%) │ +│ │ +└─────────────────────────────────────────────────────┘ +``` + +## Anti-Patterns + +| Anti-Pattern | Problem | Solution | +|--------------|---------|----------| +| **Hero culture** | Burnout, single point of failure | Spread knowledge, rotate | +| **Alert fatigue** | Ignored alerts | Fix noisy alerts | +| **Blame** | Hidden problems, fear | Blameless culture | +| **No postmortems** | Same incidents repeat | Always learn | +| **Postmortem theater** | Action items ignored | Track completion | + +## Related + +- [[SLOs and SLIs]] — Reliability targets +- [[Runbooks]] — Operational procedures +- [[Chaos Engineering]] — Proactive resilience +- [[Observability Stack]] — Detection capabilities +- [[Tools MOC]] — All tools diff --git a/Tools/Local Kubernetes.md b/Tools/Local Kubernetes.md new file mode 100644 index 0000000..eeb99a3 --- /dev/null +++ b/Tools/Local Kubernetes.md @@ -0,0 +1,423 @@ +--- +title: Local Kubernetes +aliases: + - Local K8s + - KIND + - K3s + - minikube +tags: + - tool + - kubernetes + - containers + - local-dev +type: reference +status: complete +created: "2025-12-18" +--- + +# Local Kubernetes + +Running Kubernetes clusters on your development machine. + +## Overview + +| Tool | Best For | Resource Usage | Multi-Node | +|------|----------|----------------|------------| +| **KIND** | CI/CD, testing | Medium | Yes | +| **K3s** | Edge, lightweight | Low | Yes | +| **minikube** | Learning, simple | Medium-High | Limited | +| **Docker Desktop** | Convenience | High | No | +| **Rancher Desktop** | Docker alternative | Medium | No | + +## KIND (Kubernetes IN Docker) + +**Runs K8s nodes as Docker containers.** + +### Installation + +```bash +# macOS +brew install kind + +# Linux +curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.20.0/kind-linux-amd64 +chmod +x ./kind +sudo mv ./kind /usr/local/bin/kind + +# Windows +choco install kind +``` + +### Basic Usage + +```bash +# Create cluster +kind create cluster + +# Create named cluster +kind create cluster --name my-cluster + +# List clusters +kind get clusters + +# Delete cluster +kind delete cluster --name my-cluster + +# Get kubeconfig +kind get kubeconfig --name my-cluster +``` + +### Multi-Node Cluster + +```yaml +# kind-config.yaml +kind: Cluster +apiVersion: kind.x-k8s.io/v1alpha4 +nodes: + - role: control-plane + - role: worker + - role: worker +``` + +```bash +kind create cluster --config kind-config.yaml +``` + +### With Ingress + +```yaml +# kind-ingress.yaml +kind: Cluster +apiVersion: kind.x-k8s.io/v1alpha4 +nodes: + - role: control-plane + kubeadmConfigPatches: + - | + kind: InitConfiguration + nodeRegistration: + kubeletExtraArgs: + node-labels: "ingress-ready=true" + extraPortMappings: + - containerPort: 80 + hostPort: 80 + protocol: TCP + - containerPort: 443 + hostPort: 443 + protocol: TCP +``` + +```bash +kind create cluster --config kind-ingress.yaml + +# Install nginx ingress +kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/kind/deploy.yaml +``` + +### Load Local Images + +```bash +# Build image +docker build -t myapp:latest . + +# Load into KIND (no registry needed!) +kind load docker-image myapp:latest --name my-cluster + +# Use in deployment +# image: myapp:latest +# imagePullPolicy: Never +``` + +## K3s + +**Lightweight Kubernetes for edge/IoT, also great for local dev.** + +### Installation + +```bash +# Linux (single node) +curl -sfL https://get.k3s.io | sh - + +# Check status +sudo systemctl status k3s + +# Get kubeconfig +sudo cat /etc/rancher/k3s/k3s.yaml + +# Copy to local kubeconfig +mkdir -p ~/.kube +sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config +sudo chown $USER ~/.kube/config +``` + +### K3d (K3s in Docker) + +```bash +# Install k3d (runs K3s in Docker) +curl -s https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh | bash +# or +brew install k3d + +# Create cluster +k3d cluster create my-cluster + +# With port mapping +k3d cluster create my-cluster -p "8080:80@loadbalancer" + +# Multi-node +k3d cluster create my-cluster --agents 2 + +# List clusters +k3d cluster list + +# Delete +k3d cluster delete my-cluster +``` + +### K3d with Registry + +```bash +# Create cluster with local registry +k3d cluster create my-cluster --registry-create my-registry:5000 + +# Tag and push image +docker tag myapp:latest localhost:5000/myapp:latest +docker push localhost:5000/myapp:latest + +# Use in deployment +# image: my-registry:5000/myapp:latest +``` + +### K3d Config File + +```yaml +# k3d-config.yaml +apiVersion: k3d.io/v1alpha5 +kind: Simple +metadata: + name: my-cluster +servers: 1 +agents: 2 +ports: + - port: 8080:80 + nodeFilters: + - loadbalancer +registries: + create: + name: my-registry + host: "0.0.0.0" + hostPort: "5000" +options: + k3s: + extraArgs: + - arg: --disable=traefik + nodeFilters: + - server:* +``` + +```bash +k3d cluster create --config k3d-config.yaml +``` + +## minikube + +**Original local K8s, good for learning.** + +### Installation + +```bash +# macOS +brew install minikube + +# Linux +curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 +sudo install minikube-linux-amd64 /usr/local/bin/minikube + +# Windows +choco install minikube +``` + +### Basic Usage + +```bash +# Start (uses Docker by default) +minikube start + +# With specific driver +minikube start --driver=docker +minikube start --driver=hyperkit # macOS +minikube start --driver=hyperv # Windows + +# With resources +minikube start --cpus=4 --memory=8192 + +# Status +minikube status + +# Stop +minikube stop + +# Delete +minikube delete +``` + +### Useful Features + +```bash +# Dashboard +minikube dashboard + +# Access service via tunnel +minikube service my-service + +# SSH into node +minikube ssh + +# Mount local directory +minikube mount /local/path:/container/path + +# Enable addons +minikube addons list +minikube addons enable ingress +minikube addons enable metrics-server +``` + +### Local Images + +```bash +# Point Docker CLI to minikube's Docker +eval $(minikube docker-env) + +# Now builds go directly to minikube +docker build -t myapp:latest . + +# Use in deployment (no push needed) +# image: myapp:latest +# imagePullPolicy: Never +``` + +## Comparison + +### When to Use What + +| Scenario | Best Choice | +|----------|-------------| +| **CI/CD pipelines** | KIND | +| **Quick local testing** | KIND or k3d | +| **Learning Kubernetes** | minikube | +| **Multi-node testing** | KIND or k3d | +| **Resource-constrained** | k3d (K3s) | +| **Edge/IoT simulation** | K3s | +| **Just need kubectl** | Docker Desktop | + +### Resource Comparison + +| Tool | RAM (1 node) | Startup Time | +|------|--------------|--------------| +| **KIND** | ~1-2 GB | ~30-60s | +| **k3d** | ~500MB-1GB | ~20-30s | +| **minikube** | ~2-4 GB | ~60-120s | +| **Docker Desktop** | ~2-4 GB | Already running | + +## Common Patterns + +### Development Workflow + +```bash +# 1. Create cluster +kind create cluster --name dev + +# 2. Set context +kubectl config use-context kind-dev + +# 3. Deploy app +kubectl apply -f k8s/ + +# 4. Port forward for testing +kubectl port-forward svc/my-service 8080:80 + +# 5. Make changes, rebuild +docker build -t myapp:latest . +kind load docker-image myapp:latest --name dev +kubectl rollout restart deployment/my-app + +# 6. Cleanup when done +kind delete cluster --name dev +``` + +### Tilt (Hot Reload) + +```python +# Tiltfile +docker_build('myapp', '.') +k8s_yaml('k8s/deployment.yaml') +k8s_resource('my-app', port_forwards=8080) +``` + +```bash +# Install Tilt +curl -fsSL https://raw.githubusercontent.com/tilt-dev/tilt/master/scripts/install.sh | bash + +# Run (watches files, auto-rebuilds) +tilt up +``` + +### Skaffold (CI/CD focused) + +```yaml +# skaffold.yaml +apiVersion: skaffold/v4beta6 +kind: Config +build: + artifacts: + - image: myapp +deploy: + kubectl: + manifests: + - k8s/*.yaml +``` + +```bash +# Dev mode (watch + rebuild) +skaffold dev + +# One-time deploy +skaffold run +``` + +## Local Registry + +### Docker Registry + +```bash +# Run local registry +docker run -d -p 5000:5000 --name registry registry:2 + +# Connect KIND to registry +# (add to kind config) +containerdConfigPatches: + - |- + [plugins."io.containerd.grpc.v1.cri".registry.mirrors."localhost:5000"] + endpoint = ["http://registry:5000"] + +# Push images +docker tag myapp localhost:5000/myapp +docker push localhost:5000/myapp +``` + +## Tips + +| Tip | Details | +|-----|---------| +| **Use contexts** | `kubectl config get-contexts` to avoid accidents | +| **Resource limits** | Set in your manifests for realistic testing | +| **Clean up** | Delete clusters when not in use | +| **Persistence** | Use PVCs to test storage | +| **Network policies** | KIND supports Calico for realistic networking | +| **Multiple clusters** | Easy to spin up for testing upgrades | + +## Related + +- [[Kubernetes]] — Kubernetes overview +- [[Container Runtimes]] — Docker, Podman +- [[Local Observability]] — Grafana, Prometheus locally +- [[Tools MOC]] — All tools diff --git a/Tools/Local Observability.md b/Tools/Local Observability.md new file mode 100644 index 0000000..0bcd513 --- /dev/null +++ b/Tools/Local Observability.md @@ -0,0 +1,482 @@ +--- +title: Local Observability +aliases: + - Local Monitoring + - Local Grafana + - Local Prometheus +tags: + - tool + - observability + - local-dev + - monitoring +type: reference +status: complete +created: "2025-12-18" +--- + +# Local Observability + +Running monitoring and observability tools on your development machine. + +## Overview + +Why run observability locally? + +- Test dashboards before production +- Develop metrics/traces in your app +- Learn the tools without cloud costs +- Debug performance issues locally + +## Quick Start Options + +| Method | Best For | Complexity | +|--------|----------|------------| +| **Docker Compose** | Simple, standalone | Low | +| **Kubernetes (KIND/k3d)** | Production-like | Medium | +| **Binary/brew** | Single tool testing | Low | +| **All-in-one (Grafana LGTM)** | Full stack quickly | Low | + +## Docker Compose Stack + +### Full Observability Stack + +```yaml +# docker-compose.yaml +version: "3.8" + +services: + # Metrics + prometheus: + image: prom/prometheus:latest + ports: + - "9090:9090" + volumes: + - ./prometheus.yml:/etc/prometheus/prometheus.yml + - prometheus-data:/prometheus + command: + - '--config.file=/etc/prometheus/prometheus.yml' + - '--storage.tsdb.path=/prometheus' + + # Visualization + grafana: + image: grafana/grafana:latest + ports: + - "3000:3000" + environment: + - GF_SECURITY_ADMIN_PASSWORD=admin + - GF_USERS_ALLOW_SIGN_UP=false + volumes: + - grafana-data:/var/lib/grafana + - ./grafana/provisioning:/etc/grafana/provisioning + depends_on: + - prometheus + - loki + + # Logs + loki: + image: grafana/loki:latest + ports: + - "3100:3100" + command: -config.file=/etc/loki/local-config.yaml + + # Log shipper + promtail: + image: grafana/promtail:latest + volumes: + - ./promtail.yml:/etc/promtail/config.yml + - /var/log:/var/log:ro + command: -config.file=/etc/promtail/config.yml + + # Traces + tempo: + image: grafana/tempo:latest + ports: + - "3200:3200" # tempo + - "4317:4317" # otlp grpc + - "4318:4318" # otlp http + command: -config.file=/etc/tempo/tempo.yml + volumes: + - ./tempo.yml:/etc/tempo/tempo.yml + +volumes: + prometheus-data: + grafana-data: +``` + +### Prometheus Config + +```yaml +# prometheus.yml +global: + scrape_interval: 15s + +scrape_configs: + # Prometheus itself + - job_name: 'prometheus' + static_configs: + - targets: ['localhost:9090'] + + # Your application + - job_name: 'myapp' + static_configs: + - targets: ['host.docker.internal:8080'] + metrics_path: /metrics + + # Node exporter (if running) + - job_name: 'node' + static_configs: + - targets: ['node-exporter:9100'] +``` + +### Start Stack + +```bash +docker-compose up -d + +# Access: +# Grafana: http://localhost:3000 (admin/admin) +# Prometheus: http://localhost:9090 +# Loki: http://localhost:3100 +``` + +## Prometheus Only + +### Docker + +```bash +# Simple prometheus +docker run -d \ + --name prometheus \ + -p 9090:9090 \ + -v $(pwd)/prometheus.yml:/etc/prometheus/prometheus.yml \ + prom/prometheus +``` + +### Binary + +```bash +# Download +wget https://github.com/prometheus/prometheus/releases/download/v2.47.0/prometheus-2.47.0.linux-amd64.tar.gz +tar xvf prometheus-*.tar.gz +cd prometheus-* + +# Run +./prometheus --config.file=prometheus.yml + +# macOS +brew install prometheus +prometheus --config.file=/usr/local/etc/prometheus.yml +``` + +### Test Your Metrics + +```bash +# Check Prometheus is scraping +curl http://localhost:9090/api/v1/targets + +# Query metrics +curl 'http://localhost:9090/api/v1/query?query=up' + +# Your app metrics +curl http://localhost:8080/metrics +``` + +## Grafana Only + +### Docker + +```bash +docker run -d \ + --name grafana \ + -p 3000:3000 \ + grafana/grafana + +# With persistent storage +docker run -d \ + --name grafana \ + -p 3000:3000 \ + -v grafana-storage:/var/lib/grafana \ + grafana/grafana +``` + +### Binary + +```bash +# macOS +brew install grafana +brew services start grafana + +# Linux +wget https://dl.grafana.com/oss/release/grafana-10.0.0.linux-amd64.tar.gz +tar -zxvf grafana-*.tar.gz +cd grafana-* +./bin/grafana-server +``` + +### Provisioning Dashboards + +```yaml +# grafana/provisioning/datasources/datasources.yml +apiVersion: 1 +datasources: + - name: Prometheus + type: prometheus + url: http://prometheus:9090 + isDefault: true + + - name: Loki + type: loki + url: http://loki:3100 + + - name: Tempo + type: tempo + url: http://tempo:3200 +``` + +```yaml +# grafana/provisioning/dashboards/dashboards.yml +apiVersion: 1 +providers: + - name: 'default' + folder: '' + type: file + options: + path: /etc/grafana/provisioning/dashboards +``` + +## On Kubernetes (KIND/K3s) + +### Using Helm + +```bash +# Add repos +helm repo add prometheus-community https://prometheus-community.github.io/helm-charts +helm repo add grafana https://grafana.github.io/helm-charts +helm repo update + +# Install prometheus-stack (includes Grafana) +helm install monitoring prometheus-community/kube-prometheus-stack \ + --namespace monitoring \ + --create-namespace + +# Port forward +kubectl port-forward -n monitoring svc/monitoring-grafana 3000:80 +kubectl port-forward -n monitoring svc/monitoring-kube-prometheus-prometheus 9090:9090 + +# Get Grafana password +kubectl get secret -n monitoring monitoring-grafana -o jsonpath="{.data.admin-password}" | base64 -d +``` + +### Minimal Setup + +```yaml +# prometheus-k8s.yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: prometheus-config +data: + prometheus.yml: | + global: + scrape_interval: 15s + scrape_configs: + - job_name: 'kubernetes-pods' + kubernetes_sd_configs: + - role: pod + relabel_configs: + - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape] + action: keep + regex: true +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: prometheus +spec: + replicas: 1 + selector: + matchLabels: + app: prometheus + template: + metadata: + labels: + app: prometheus + spec: + containers: + - name: prometheus + image: prom/prometheus:latest + ports: + - containerPort: 9090 + volumeMounts: + - name: config + mountPath: /etc/prometheus + volumes: + - name: config + configMap: + name: prometheus-config +--- +apiVersion: v1 +kind: Service +metadata: + name: prometheus +spec: + selector: + app: prometheus + ports: + - port: 9090 +``` + +## Grafana LGTM Stack + +**All-in-one: Loki, Grafana, Tempo, Mimir** + +```bash +# Single container with everything +docker run -d \ + --name lgtm \ + -p 3000:3000 \ + -p 4317:4317 \ + -p 4318:4318 \ + grafana/otel-lgtm + +# Grafana at http://localhost:3000 +# OTLP gRPC at localhost:4317 +# OTLP HTTP at localhost:4318 +``` + +## Application Instrumentation + +### Send Metrics + +```python +# Python with prometheus_client +from prometheus_client import Counter, Histogram, start_http_server + +# Define metrics +REQUEST_COUNT = Counter('http_requests_total', 'Total requests', ['method', 'endpoint']) +REQUEST_LATENCY = Histogram('http_request_duration_seconds', 'Request latency') + +# Use in code +REQUEST_COUNT.labels(method='GET', endpoint='/api').inc() +with REQUEST_LATENCY.time(): + # do work + pass + +# Start metrics server +start_http_server(8080) # /metrics endpoint +``` + +```go +// Go with prometheus/client_golang +import ( + "github.com/prometheus/client_golang/prometheus" + "github.com/prometheus/client_golang/prometheus/promhttp" +) + +var requestCount = prometheus.NewCounterVec( + prometheus.CounterOpts{ + Name: "http_requests_total", + }, + []string{"method", "endpoint"}, +) + +func init() { + prometheus.MustRegister(requestCount) +} + +// Expose metrics +http.Handle("/metrics", promhttp.Handler()) +``` + +### Send Traces (OpenTelemetry) + +```python +# Python with OpenTelemetry +from opentelemetry import trace +from opentelemetry.sdk.trace import TracerProvider +from opentelemetry.sdk.trace.export import BatchSpanProcessor +from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter + +# Setup +provider = TracerProvider() +processor = BatchSpanProcessor(OTLPSpanExporter(endpoint="localhost:4317")) +provider.add_span_processor(processor) +trace.set_tracer_provider(provider) + +# Use +tracer = trace.get_tracer(__name__) +with tracer.start_as_current_span("my-operation"): + # do work + pass +``` + +### Send Logs (to Loki) + +```python +# Python with logging to Loki +import logging +import logging_loki + +handler = logging_loki.LokiHandler( + url="http://localhost:3100/loki/api/v1/push", + tags={"application": "myapp"}, + version="1", +) +logger = logging.getLogger("myapp") +logger.addHandler(handler) + +logger.info("Application started", extra={"user": "john"}) +``` + +## Useful Dashboards + +### Import Pre-built + +| Dashboard | ID | For | +|-----------|-----|-----| +| Node Exporter Full | 1860 | System metrics | +| Go Metrics | 6671 | Go applications | +| JVM Micrometer | 4701 | Java/Spring | +| NGINX | 9614 | NGINX | +| PostgreSQL | 9628 | PostgreSQL | +| Kubernetes Cluster | 7249 | K8s overview | + +```bash +# Import via API +curl -X POST http://admin:admin@localhost:3000/api/dashboards/import \ + -H "Content-Type: application/json" \ + -d '{"dashboard":{"id":null},"overwrite":true,"inputs":[],"folderId":0,"gnetId":1860}' +``` + +## Cleanup + +```bash +# Docker Compose +docker-compose down -v # -v removes volumes + +# Kubernetes +helm uninstall monitoring -n monitoring +kubectl delete namespace monitoring + +# Docker +docker rm -f prometheus grafana loki +docker volume rm grafana-storage +``` + +## Tips + +| Tip | Details | +|-----|---------| +| **Persist data** | Use volumes to keep metrics across restarts | +| **Network access** | Use `host.docker.internal` to reach host apps | +| **Resource limits** | Set memory limits for Prometheus | +| **Retention** | Configure `--storage.tsdb.retention.time=15d` | +| **Provisioning** | Pre-configure dashboards via files | + +## Related + +- [[Prometheus]] — Prometheus in depth +- [[Grafana]] — Grafana in depth +- [[OpenTelemetry]] — Instrumentation +- [[Local Kubernetes]] — K8s locally +- [[Tools MOC]] — All tools diff --git a/Tools/MongoDB.md b/Tools/MongoDB.md new file mode 100644 index 0000000..ef4515c --- /dev/null +++ b/Tools/MongoDB.md @@ -0,0 +1,501 @@ +--- +title: MongoDB +aliases: + - Mongo + - MongoDB Database +tags: + - tool + - database + - nosql + - document +type: reference +status: complete +created: "2025-12-18" +--- + +# MongoDB + +Document-oriented NoSQL database for flexible, scalable data storage. + +## Overview + +| Aspect | Details | +|--------|---------| +| **Type** | Document database (NoSQL) | +| **Data Model** | JSON-like documents (BSON) | +| **Query Language** | MongoDB Query Language (MQL) | +| **License** | SSPL (Server Side Public License) | +| **Written In** | C++ | +| **Use Cases** | Content management, real-time analytics, IoT | + +## Core Concepts + +| Concept | SQL Equivalent | Description | +|---------|---------------|-------------| +| **Database** | Database | Container for collections | +| **Collection** | Table | Group of documents | +| **Document** | Row | Individual record (BSON) | +| **Field** | Column | Key-value pair in document | +| **_id** | Primary Key | Unique document identifier | +| **Index** | Index | Performance optimization | + +## Document Structure + +```javascript +// Document example +{ + _id: ObjectId("507f1f77bcf86cd799439011"), + name: "Alice Smith", + email: "alice@example.com", + age: 28, + address: { + street: "123 Main St", + city: "Boston", + state: "MA" + }, + tags: ["developer", "mongodb"], + orders: [ + { product: "Laptop", price: 999.99 }, + { product: "Mouse", price: 29.99 } + ], + createdAt: ISODate("2024-01-15T10:30:00Z") +} +``` + +### BSON Data Types + +| Type | Description | +|------|-------------| +| `String` | UTF-8 string | +| `Int32/Int64` | Integer values | +| `Double` | Floating point | +| `Boolean` | true/false | +| `ObjectId` | 12-byte unique identifier | +| `Date` | UTC datetime | +| `Array` | Ordered list | +| `Object` | Embedded document | +| `Binary` | Binary data | +| `Null` | Null value | + +## CRUD Operations + +### Create + +```javascript +// Insert one +db.users.insertOne({ + name: "Alice", + email: "alice@example.com" +}); + +// Insert many +db.users.insertMany([ + { name: "Bob", email: "bob@example.com" }, + { name: "Carol", email: "carol@example.com" } +]); +``` + +### Read + +```javascript +// Find all +db.users.find(); + +// Find with filter +db.users.find({ name: "Alice" }); + +// Find one +db.users.findOne({ email: "alice@example.com" }); + +// Projection (select fields) +db.users.find({}, { name: 1, email: 1, _id: 0 }); + +// Sorting and limiting +db.users.find().sort({ name: 1 }).limit(10).skip(20); +``` + +### Update + +```javascript +// Update one +db.users.updateOne( + { _id: ObjectId("...") }, + { $set: { name: "Alice Smith" } } +); + +// Update many +db.users.updateMany( + { status: "inactive" }, + { $set: { status: "archived" } } +); + +// Replace document +db.users.replaceOne( + { _id: ObjectId("...") }, + { name: "New Doc", email: "new@example.com" } +); + +// Upsert (insert if not exists) +db.users.updateOne( + { email: "new@example.com" }, + { $set: { name: "New User" } }, + { upsert: true } +); +``` + +### Delete + +```javascript +// Delete one +db.users.deleteOne({ _id: ObjectId("...") }); + +// Delete many +db.users.deleteMany({ status: "archived" }); + +// Delete all +db.users.deleteMany({}); +``` + +## Query Operators + +### Comparison + +```javascript +db.products.find({ + price: { $gt: 100 }, // Greater than + quantity: { $gte: 10 }, // Greater than or equal + rating: { $lt: 5 }, // Less than + stock: { $lte: 0 }, // Less than or equal + status: { $eq: "active" }, // Equal + category: { $ne: "archived" }, // Not equal + tags: { $in: ["sale", "new"] },// In array + type: { $nin: ["draft"] } // Not in array +}); +``` + +### Logical + +```javascript +// AND (implicit) +db.products.find({ price: { $gt: 100 }, status: "active" }); + +// AND (explicit) +db.products.find({ + $and: [ + { price: { $gt: 100 } }, + { price: { $lt: 500 } } + ] +}); + +// OR +db.products.find({ + $or: [ + { status: "sale" }, + { price: { $lt: 50 } } + ] +}); + +// NOT +db.products.find({ + price: { $not: { $gt: 100 } } +}); + +// NOR +db.products.find({ + $nor: [{ status: "draft" }, { status: "archived" }] +}); +``` + +### Element & Array + +```javascript +// Field exists +db.users.find({ phone: { $exists: true } }); + +// Type check +db.users.find({ age: { $type: "int" } }); + +// Array contains +db.users.find({ tags: "mongodb" }); + +// Array contains all +db.users.find({ tags: { $all: ["mongodb", "nodejs"] } }); + +// Array size +db.users.find({ tags: { $size: 3 } }); + +// Array element match +db.orders.find({ + items: { $elemMatch: { product: "Laptop", quantity: { $gt: 1 } } } +}); +``` + +## Update Operators + +```javascript +db.users.updateOne( + { _id: ObjectId("...") }, + { + $set: { name: "New Name" }, // Set field + $unset: { temp: "" }, // Remove field + $inc: { loginCount: 1 }, // Increment + $mul: { price: 1.1 }, // Multiply + $min: { lowScore: 50 }, // Set if less + $max: { highScore: 100 }, // Set if greater + $push: { tags: "new" }, // Add to array + $pull: { tags: "old" }, // Remove from array + $addToSet: { tags: "unique" }, // Add if not exists + $pop: { queue: -1 }, // Remove first (-1) or last (1) + $rename: { oldField: "newField" }, // Rename field + $currentDate: { updatedAt: true } // Set to current date + } +); +``` + +## Aggregation Pipeline + +```javascript +db.orders.aggregate([ + // Stage 1: Filter + { $match: { status: "completed" } }, + + // Stage 2: Join + { $lookup: { + from: "products", + localField: "productId", + foreignField: "_id", + as: "product" + }}, + + // Stage 3: Unwind array + { $unwind: "$product" }, + + // Stage 4: Group + { $group: { + _id: "$product.category", + totalSales: { $sum: "$amount" }, + avgPrice: { $avg: "$product.price" }, + count: { $sum: 1 } + }}, + + // Stage 5: Sort + { $sort: { totalSales: -1 } }, + + // Stage 6: Limit + { $limit: 10 }, + + // Stage 7: Project + { $project: { + category: "$_id", + totalSales: 1, + avgPrice: { $round: ["$avgPrice", 2] }, + _id: 0 + }} +]); +``` + +### Common Aggregation Stages + +| Stage | Purpose | +|-------|---------| +| `$match` | Filter documents | +| `$group` | Group by field, aggregate | +| `$sort` | Sort results | +| `$project` | Reshape documents | +| `$lookup` | Join collections | +| `$unwind` | Flatten arrays | +| `$limit` / `$skip` | Pagination | +| `$count` | Count documents | +| `$facet` | Multiple pipelines | +| `$bucket` | Categorize into ranges | + +## Indexes + +```javascript +// Single field index +db.users.createIndex({ email: 1 }); // 1 = ascending, -1 = descending + +// Compound index +db.users.createIndex({ lastName: 1, firstName: 1 }); + +// Unique index +db.users.createIndex({ email: 1 }, { unique: true }); + +// Text index (full-text search) +db.articles.createIndex({ title: "text", content: "text" }); +db.articles.find({ $text: { $search: "mongodb tutorial" } }); + +// TTL index (auto-expire documents) +db.sessions.createIndex({ createdAt: 1 }, { expireAfterSeconds: 3600 }); + +// Partial index +db.users.createIndex( + { email: 1 }, + { partialFilterExpression: { status: "active" } } +); + +// List indexes +db.users.getIndexes(); + +// Drop index +db.users.dropIndex("email_1"); +``` + +## Schema Validation + +```javascript +db.createCollection("users", { + validator: { + $jsonSchema: { + bsonType: "object", + required: ["name", "email"], + properties: { + name: { + bsonType: "string", + description: "must be a string and is required" + }, + email: { + bsonType: "string", + pattern: "^.+@.+$", + description: "must be a valid email" + }, + age: { + bsonType: "int", + minimum: 0, + maximum: 150 + }, + status: { + enum: ["active", "inactive", "pending"] + } + } + } + }, + validationLevel: "strict", // or "moderate" + validationAction: "error" // or "warn" +}); +``` + +## Replication + +### Replica Set + +``` +┌─────────────┐ +│ Primary │ ←── All writes +└──────┬──────┘ + │ Replication + ┌───┴───┐ + ▼ ▼ +┌─────┐ ┌─────┐ +│Sec 1│ │Sec 2│ ←── Read replicas (optional) +└─────┘ └─────┘ +``` + +| Role | Description | +|------|-------------| +| **Primary** | Receives all writes | +| **Secondary** | Replicates from primary | +| **Arbiter** | Votes in elections (no data) | + +```javascript +// Read from secondary (eventually consistent) +db.users.find().readPref("secondary"); + +// Write concern +db.users.insertOne( + { name: "Alice" }, + { writeConcern: { w: "majority", j: true } } +); +``` + +## Sharding + +``` + ┌─────────────┐ + │ mongos │ ←── Query Router + └──────┬──────┘ + │ + ┌──────────────────┼──────────────────┐ + ▼ ▼ ▼ + ┌─────────┐ ┌─────────┐ ┌─────────┐ + │ Shard 1 │ │ Shard 2 │ │ Shard 3 │ + │ (RS) │ │ (RS) │ │ (RS) │ + └─────────┘ └─────────┘ └─────────┘ +``` + +```javascript +// Enable sharding on database +sh.enableSharding("mydb"); + +// Shard collection +sh.shardCollection("mydb.users", { region: 1 }); // Range +sh.shardCollection("mydb.logs", { _id: "hashed" }); // Hashed +``` + +## Drivers + +### Node.js + +```javascript +const { MongoClient } = require('mongodb'); + +const client = new MongoClient('mongodb://localhost:27017'); +await client.connect(); + +const db = client.db('mydb'); +const users = db.collection('users'); + +// CRUD +const result = await users.insertOne({ name: 'Alice' }); +const user = await users.findOne({ _id: result.insertedId }); +await users.updateOne({ _id: user._id }, { $set: { verified: true } }); + +await client.close(); +``` + +### Python (PyMongo) + +```python +from pymongo import MongoClient + +client = MongoClient('mongodb://localhost:27017') +db = client.mydb +users = db.users + +# CRUD +result = users.insert_one({'name': 'Alice'}) +user = users.find_one({'_id': result.inserted_id}) +users.update_one({'_id': user['_id']}, {'$set': {'verified': True}}) + +client.close() +``` + +## MongoDB vs SQL + +| SQL | MongoDB | +|-----|---------| +| `SELECT * FROM users` | `db.users.find()` | +| `SELECT name FROM users` | `db.users.find({}, {name: 1})` | +| `WHERE age > 25` | `{age: {$gt: 25}}` | +| `ORDER BY name` | `.sort({name: 1})` | +| `LIMIT 10` | `.limit(10)` | +| `JOIN` | `$lookup` (aggregation) | +| `GROUP BY` | `$group` (aggregation) | +| `COUNT(*)` | `countDocuments()` | + +## When to Use + +| Good For | Not Ideal For | +|----------|---------------| +| ✅ Flexible schemas | ❌ Complex transactions | +| ✅ Rapid development | ❌ Heavy JOINs | +| ✅ Hierarchical data | ❌ Strict ACID requirements | +| ✅ Horizontal scaling | ❌ Small, simple datasets | +| ✅ Real-time analytics | ❌ Existing SQL expertise | + +## Related + +- [[Redis]] — In-memory data store +- [[PostgreSQL]] — SQL with JSON support +- [[Database Engines]] — Database comparison +- [[Tools MOC]] — All tools diff --git a/Tools/MySQL.md b/Tools/MySQL.md new file mode 100644 index 0000000..cfe6d01 --- /dev/null +++ b/Tools/MySQL.md @@ -0,0 +1,323 @@ +--- +title: MySQL +aliases: + - MariaDB +tags: + - tools + - database + - sql + - relational +type: reference +status: complete +created: "2025-12-16" +--- + +# MySQL + +World's most popular open-source relational database, known for simplicity, speed, and web application dominance. + +## Overview + +| Aspect | Details | +|--------|---------| +| **Type** | Relational (RDBMS) | +| **License** | GPL (Community) / Commercial | +| **Owner** | Oracle Corporation | +| **Fork** | MariaDB (community-driven) | +| **Default Port** | 3306 | +| **Strengths** | Simplicity, read performance, web apps | + +## Storage Engines + +| Engine | Use Case | +|--------|----------| +| **InnoDB** | Default, ACID, transactions | +| **MyISAM** | Read-heavy, full-text (legacy) | +| **Memory** | Temporary tables, caching | +| **Archive** | Compressed historical data | + +### InnoDB Features + +| Feature | Description | +|---------|-------------| +| **ACID** | Full transaction support | +| **Row-level locking** | High concurrency | +| **Foreign keys** | Referential integrity | +| **MVCC** | Non-blocking reads | +| **Crash recovery** | Automatic recovery | + +## Data Types + +```sql +-- Numeric +TINYINT, SMALLINT, MEDIUMINT, INT, BIGINT +DECIMAL(10,2) -- Exact +FLOAT, DOUBLE -- Approximate + +-- String +CHAR(50) -- Fixed length +VARCHAR(255) -- Variable length +TEXT, MEDIUMTEXT, LONGTEXT + +-- Date/Time +DATE, TIME, DATETIME, TIMESTAMP +YEAR + +-- Binary +BLOB, MEDIUMBLOB, LONGBLOB +BINARY, VARBINARY + +-- JSON (MySQL 5.7+) +JSON +``` + +## Indexing + +### Index Types + +| Type | Description | +|------|-------------| +| **B-tree** | Default, range queries | +| **Hash** | Memory engine only | +| **Full-text** | Text search | +| **Spatial** | Geographic data | + +### Index Examples + +```sql +-- Primary key +CREATE TABLE users ( + id INT AUTO_INCREMENT PRIMARY KEY, + email VARCHAR(255) UNIQUE, + name VARCHAR(100) +); + +-- Composite index +CREATE INDEX idx_name_email ON users(name, email); + +-- Prefix index (for long strings) +CREATE INDEX idx_email_prefix ON users(email(20)); + +-- Full-text index +CREATE FULLTEXT INDEX idx_content ON articles(title, body); + +-- Using EXPLAIN +EXPLAIN SELECT * FROM users WHERE email = 'test@example.com'; +``` + +### Index Best Practices + +| Practice | Rationale | +|----------|-----------| +| **Index WHERE columns** | Speed up filtering | +| **Leftmost prefix rule** | Composite index usage | +| **Avoid over-indexing** | Slow writes | +| **Use covering indexes** | Avoid table lookups | + +## Query Optimization + +### EXPLAIN Output + +```sql +EXPLAIN FORMAT=JSON SELECT * FROM orders +WHERE customer_id = 123 AND status = 'pending'; + +-- Key fields: +-- type: ALL (bad) → index → range → ref → eq_ref → const (best) +-- key: Which index used +-- rows: Estimated rows scanned +-- Extra: Using where, Using index, Using filesort +``` + +### Common Optimizations + +```sql +-- Use indexes for ORDER BY +SELECT * FROM orders ORDER BY created_at DESC LIMIT 10; +-- Needs: INDEX(created_at) + +-- Avoid SELECT * +SELECT id, name FROM users WHERE id = 123; + +-- Use LIMIT with OFFSET carefully +-- Bad for large offsets: +SELECT * FROM logs ORDER BY id LIMIT 10 OFFSET 1000000; +-- Better (keyset pagination): +SELECT * FROM logs WHERE id > 1000000 ORDER BY id LIMIT 10; + +-- Avoid functions on indexed columns +-- Bad: +SELECT * FROM users WHERE YEAR(created_at) = 2024; +-- Good: +SELECT * FROM users WHERE created_at >= '2024-01-01' + AND created_at < '2025-01-01'; +``` + +## Replication + +### Master-Slave (Source-Replica) + +``` +Source (Write) ──binlog──► Replica (Read) + │ + ├──► Replica 2 + └──► Replica 3 +``` + +### Replication Types + +| Type | Description | +|------|-------------| +| **Async** | Default, some lag possible | +| **Semi-sync** | At least one replica confirms | +| **Group Replication** | Multi-primary, Paxos-based | + +### Setup Example + +```sql +-- On source +CREATE USER 'repl'@'%' IDENTIFIED BY 'password'; +GRANT REPLICATION SLAVE ON *.* TO 'repl'@'%'; +SHOW MASTER STATUS; + +-- On replica +CHANGE MASTER TO + MASTER_HOST='source.example.com', + MASTER_USER='repl', + MASTER_PASSWORD='password', + MASTER_LOG_FILE='mysql-bin.000001', + MASTER_LOG_POS=123; +START SLAVE; +``` + +## JSON Support + +```sql +-- Create table with JSON +CREATE TABLE events ( + id INT AUTO_INCREMENT PRIMARY KEY, + data JSON +); + +-- Insert JSON +INSERT INTO events (data) VALUES ('{"type": "click", "page": "/home"}'); + +-- Query JSON +SELECT data->>'$.type' AS event_type FROM events; +SELECT * FROM events WHERE data->>'$.type' = 'click'; + +-- JSON functions +SELECT JSON_EXTRACT(data, '$.type') FROM events; +SELECT JSON_KEYS(data) FROM events; +SELECT JSON_ARRAY_LENGTH(data->'$.items') FROM events; + +-- Generated columns for indexing +ALTER TABLE events ADD COLUMN event_type VARCHAR(50) + GENERATED ALWAYS AS (data->>'$.type') STORED; +CREATE INDEX idx_event_type ON events(event_type); +``` + +## Performance Tuning + +### Key Configuration + +```ini +# my.cnf + +[mysqld] +# InnoDB Buffer Pool (70-80% of RAM for dedicated server) +innodb_buffer_pool_size = 12G +innodb_buffer_pool_instances = 8 + +# Logging +innodb_log_file_size = 2G +innodb_flush_log_at_trx_commit = 1 # ACID (2 for performance) + +# Connections +max_connections = 500 +thread_cache_size = 50 + +# Query cache (disabled in 8.0+) +# query_cache_type = 0 + +# Temp tables +tmp_table_size = 256M +max_heap_table_size = 256M +``` + +### Monitoring Queries + +```sql +-- Slow query log +SET GLOBAL slow_query_log = 'ON'; +SET GLOBAL long_query_time = 1; + +-- Process list +SHOW PROCESSLIST; +SHOW FULL PROCESSLIST; + +-- InnoDB status +SHOW ENGINE INNODB STATUS; + +-- Table statistics +SHOW TABLE STATUS LIKE 'orders'; + +-- Index usage +SELECT * FROM sys.schema_unused_indexes; +SELECT * FROM sys.schema_redundant_indexes; +``` + +## MySQL vs MariaDB + +| Aspect | MySQL | MariaDB | +|--------|-------|---------| +| **Owner** | Oracle | Community | +| **Compatibility** | N/A | Drop-in (mostly) | +| **Features** | Conservative | More experimental | +| **Storage Engines** | InnoDB focus | More options | +| **JSON** | Native | Native | + +## Comparison with PostgreSQL + +| Feature | MySQL | PostgreSQL | +|---------|-------|------------| +| **Simplicity** | ✅ Easier | ⚠️ More complex | +| **Read Performance** | ✅ Excellent | ✅ Excellent | +| **Write Performance** | ✅ Good | ✅ Good | +| **SQL Standards** | ⚠️ Deviations | ✅ Excellent | +| **JSON** | ✅ Good | ✅ Better (JSONB) | +| **Replication** | ✅ Many options | ✅ Logical + Physical | +| **Extensions** | ⚠️ Limited | ✅ Rich | +| **Window Functions** | ✅ 8.0+ | ✅ Long-standing | + +## When to Use + +### Strengths + +| Strength | Rationale | +|----------|-----------| +| **Web applications** | LAMP stack, WordPress, etc. | +| **Read-heavy workloads** | Efficient reads | +| **Simplicity** | Easy to set up and manage | +| **Ecosystem** | Massive tooling support | + +### Considerations + +| Consideration | Impact | +|---------------|--------| +| **Complex queries** | PostgreSQL may be better | +| **Oracle ownership** | Some prefer MariaDB | + +### Best For + +- Web applications (WordPress, Drupal) +- Read-heavy workloads +- Simple CRUD applications +- Teams familiar with MySQL + +## Related + +- [[PostgreSQL]] — Alternative RDBMS +- [[Database Engines]] — Database comparison +- [[Database Indexing]] — Indexing strategies +- [[Tools MOC]] — All tools diff --git a/Tools/PostgreSQL.md b/Tools/PostgreSQL.md new file mode 100644 index 0000000..6ed4f01 --- /dev/null +++ b/Tools/PostgreSQL.md @@ -0,0 +1,350 @@ +--- +title: PostgreSQL +aliases: + - Postgres + - PG +tags: + - tools + - database + - sql + - relational +type: reference +status: complete +created: "2025-12-16" +--- + +# PostgreSQL + +Advanced open-source relational database known for standards compliance, extensibility, and robust feature set. + +## Overview + +| Aspect | Details | +|--------|---------| +| **Type** | Relational (ORDBMS) | +| **License** | PostgreSQL License (permissive) | +| **Current Version** | 16.x (2024) | +| **Default Port** | 5432 | +| **Written In** | C | +| **Strengths** | Standards, extensibility, data integrity | + +## Key Features + +| Feature | Description | +|---------|-------------| +| **ACID Compliant** | Full transaction support | +| **MVCC** | Multi-Version Concurrency Control | +| **Extensible** | Custom types, operators, functions | +| **JSON/JSONB** | Native JSON with indexing | +| **Full-Text Search** | Built-in FTS capabilities | +| **PostGIS** | Geospatial extension | +| **Partitioning** | Table partitioning | +| **Logical Replication** | Fine-grained replication | + +## Data Types + +### Common Types + +| Type | Description | +|------|-------------| +| `INTEGER` / `BIGINT` | Whole numbers | +| `NUMERIC(p,s)` | Exact decimal | +| `TEXT` / `VARCHAR(n)` | Strings | +| `BOOLEAN` | True/false | +| `TIMESTAMP[TZ]` | Date and time | +| `UUID` | Universally unique ID | +| `JSONB` | Binary JSON | +| `ARRAY` | Array of any type | + +### PostgreSQL-Specific Types + +```sql +-- Arrays +CREATE TABLE tags ( + id SERIAL PRIMARY KEY, + name TEXT, + keywords TEXT[] +); +INSERT INTO tags VALUES (1, 'tech', ARRAY['postgres', 'database']); +SELECT * FROM tags WHERE 'postgres' = ANY(keywords); + +-- JSONB +CREATE TABLE events ( + id SERIAL PRIMARY KEY, + data JSONB +); +INSERT INTO events (data) VALUES ('{"type": "click", "page": "/home"}'); +SELECT data->>'type' FROM events; +SELECT * FROM events WHERE data @> '{"type": "click"}'; + +-- Range types +CREATE TABLE reservations ( + id SERIAL PRIMARY KEY, + room_id INT, + during TSRANGE +); +INSERT INTO reservations VALUES (1, 101, '[2024-01-01 14:00, 2024-01-01 16:00)'); +SELECT * FROM reservations WHERE during && '[2024-01-01 15:00, 2024-01-01 17:00)'; + +-- UUID +CREATE TABLE users ( + id UUID DEFAULT gen_random_uuid() PRIMARY KEY, + email TEXT UNIQUE +); +``` + +## Indexing + +### Index Types + +| Type | Use Case | +|------|----------| +| **B-tree** | Default, equality and range | +| **Hash** | Equality only | +| **GiST** | Geometric, full-text search | +| **GIN** | Arrays, JSONB, full-text | +| **BRIN** | Large sorted tables | + +### Index Examples + +```sql +-- B-tree (default) +CREATE INDEX idx_users_email ON users(email); + +-- Partial index +CREATE INDEX idx_active_users ON users(email) + WHERE active = true; + +-- Expression index +CREATE INDEX idx_users_lower_email ON users(LOWER(email)); + +-- GIN for JSONB +CREATE INDEX idx_events_data ON events USING GIN(data); + +-- GIN for array +CREATE INDEX idx_tags_keywords ON tags USING GIN(keywords); + +-- BRIN for time-series +CREATE INDEX idx_logs_created ON logs USING BRIN(created_at); + +-- Covering index (INCLUDE) +CREATE INDEX idx_orders_customer ON orders(customer_id) + INCLUDE (total, status); +``` + +## Performance + +### EXPLAIN ANALYZE + +```sql +EXPLAIN (ANALYZE, BUFFERS, FORMAT TEXT) +SELECT * FROM orders WHERE customer_id = 123; + +-- Output: +-- Index Scan using idx_orders_customer on orders +-- Index Cond: (customer_id = 123) +-- Buffers: shared hit=3 +-- Planning Time: 0.1 ms +-- Execution Time: 0.05 ms +``` + +### Query Optimization Tips + +| Technique | When to Use | +|-----------|-------------| +| **Use indexes** | Filter/sort columns | +| **Partial indexes** | Frequently filtered subset | +| **Covering indexes** | Avoid table lookups | +| **LIMIT** | Pagination | +| **Materialized views** | Complex aggregations | +| **Connection pooling** | High connection count | + +### Configuration Tuning + +```ini +# postgresql.conf + +# Memory +shared_buffers = 4GB # 25% of RAM +effective_cache_size = 12GB # 75% of RAM +work_mem = 256MB # Per operation +maintenance_work_mem = 1GB # For VACUUM, CREATE INDEX + +# WAL +wal_buffers = 64MB +checkpoint_completion_target = 0.9 + +# Planner +random_page_cost = 1.1 # For SSD +effective_io_concurrency = 200 # For SSD + +# Connections +max_connections = 200 +``` + +## Advanced Features + +### CTEs and Window Functions + +```sql +-- CTE (Common Table Expression) +WITH monthly_sales AS ( + SELECT DATE_TRUNC('month', created_at) AS month, + SUM(total) AS revenue + FROM orders + GROUP BY 1 +) +SELECT month, revenue, + LAG(revenue) OVER (ORDER BY month) AS prev_month +FROM monthly_sales; + +-- Window functions +SELECT customer_id, total, + ROW_NUMBER() OVER (PARTITION BY customer_id ORDER BY total DESC) AS rank, + SUM(total) OVER (PARTITION BY customer_id) AS customer_total, + AVG(total) OVER () AS overall_avg +FROM orders; +``` + +### Full-Text Search + +```sql +-- Create search column +ALTER TABLE articles ADD COLUMN search_vector tsvector; + +UPDATE articles SET search_vector = + to_tsvector('english', title || ' ' || body); + +CREATE INDEX idx_articles_search ON articles USING GIN(search_vector); + +-- Search +SELECT title, ts_rank(search_vector, query) AS rank +FROM articles, to_tsquery('english', 'postgres & performance') query +WHERE search_vector @@ query +ORDER BY rank DESC; +``` + +### Partitioning + +```sql +-- Range partitioning +CREATE TABLE logs ( + id BIGSERIAL, + created_at TIMESTAMP NOT NULL, + message TEXT +) PARTITION BY RANGE (created_at); + +CREATE TABLE logs_2024_01 PARTITION OF logs + FOR VALUES FROM ('2024-01-01') TO ('2024-02-01'); + +CREATE TABLE logs_2024_02 PARTITION OF logs + FOR VALUES FROM ('2024-02-01') TO ('2024-03-01'); + +-- Automatic partition creation (pg_partman extension) +``` + +### LISTEN/NOTIFY + +```sql +-- Publisher +NOTIFY order_created, '{"order_id": 123}'; + +-- Subscriber (application code) +LISTEN order_created; +-- Receives: {"order_id": 123} +``` + +## Replication + +### Streaming Replication + +``` +Primary ──WAL──► Standby (sync/async) + │ + ├──► Standby 2 + └──► Standby 3 +``` + +### Logical Replication + +```sql +-- Publisher +CREATE PUBLICATION my_pub FOR TABLE orders, customers; + +-- Subscriber +CREATE SUBSCRIPTION my_sub + CONNECTION 'host=primary dbname=mydb' + PUBLICATION my_pub; +``` + +## Extensions + +| Extension | Purpose | +|-----------|---------| +| **PostGIS** | Geospatial data | +| **pg_trgm** | Fuzzy string matching | +| **pgcrypto** | Cryptographic functions | +| **pg_stat_statements** | Query statistics | +| **timescaledb** | Time-series | +| **pgvector** | Vector similarity search | +| **pg_partman** | Partition management | + +```sql +-- Enable extension +CREATE EXTENSION pg_stat_statements; +CREATE EXTENSION pgvector; + +-- Vector search example +CREATE TABLE items ( + id SERIAL PRIMARY KEY, + embedding vector(1536) +); + +SELECT * FROM items +ORDER BY embedding <-> '[0.1, 0.2, ...]'::vector +LIMIT 10; +``` + +## Comparison + +| Feature | PostgreSQL | MySQL | +|---------|------------|-------| +| **Standards** | ✅ Excellent | ⚠️ Good | +| **JSON** | ✅ JSONB with indexes | ✅ JSON | +| **Full-text** | ✅ Built-in | ✅ Built-in | +| **Replication** | ✅ Logical + Physical | ✅ Multiple options | +| **Extensions** | ✅ Rich ecosystem | ⚠️ Limited | +| **Performance** | ✅ Complex queries | ✅ Simple queries | + +## When to Use + +### Strengths + +| Strength | Rationale | +|----------|-----------| +| **Complex queries** | Advanced SQL, CTEs, window functions | +| **Data integrity** | Strong constraints, transactions | +| **Extensibility** | Custom types, PostGIS, vectors | +| **Standards** | Most SQL-compliant | + +### Considerations + +| Consideration | Impact | +|---------------|--------| +| **Connection overhead** | Need connection pooling | +| **Replication complexity** | More setup than MySQL | + +### Best For + +- Complex analytical queries +- Geospatial applications (PostGIS) +- JSON document storage with SQL +- Applications requiring strong data integrity +- Vector/AI applications (pgvector) + +## Related + +- [[MySQL]] — Alternative RDBMS +- [[Database Engines]] — Database comparison +- [[Database Indexing]] — Indexing strategies +- [[Tools MOC]] — All tools diff --git a/Tools/Redis.md b/Tools/Redis.md new file mode 100644 index 0000000..4c9307d --- /dev/null +++ b/Tools/Redis.md @@ -0,0 +1,337 @@ +--- +title: Redis +aliases: + - Redis Cache + - Redis Database +tags: + - tools + - database + - cache + - nosql + - key-value +type: reference +status: complete +created: "2025-12-16" +--- + +# Redis + +In-memory data structure store used as cache, database, message broker, and streaming engine. + +## Overview + +| Aspect | Details | +|--------|---------| +| **Type** | Key-value / Data structure store | +| **Storage** | In-memory (with persistence options) | +| **License** | BSD (Redis 7.2+: RSALv2/SSPLv1) | +| **Default Port** | 6379 | +| **Protocol** | RESP (Redis Serialization Protocol) | +| **Strengths** | Speed, data structures, simplicity | + +## Data Structures + +### Strings + +```redis +SET user:1:name "Alice" +GET user:1:name # "Alice" + +INCR page:views # Atomic increment +INCRBY page:views 10 # Increment by 10 + +SETEX session:abc 3600 "data" # Expires in 1 hour +SETNX lock:resource "holder" # Set if not exists + +MSET key1 "v1" key2 "v2" # Multiple set +MGET key1 key2 # Multiple get +``` + +### Lists + +```redis +LPUSH queue:jobs "job1" # Push left +RPUSH queue:jobs "job2" # Push right +LPOP queue:jobs # Pop left +RPOP queue:jobs # Pop right +BRPOP queue:jobs 30 # Blocking pop (30s timeout) + +LRANGE queue:jobs 0 -1 # Get all +LLEN queue:jobs # Length +LINDEX queue:jobs 0 # Get by index +``` + +### Sets + +```redis +SADD tags:post:1 "redis" "database" "cache" +SMEMBERS tags:post:1 # Get all members +SISMEMBER tags:post:1 "redis" # Check membership + +SINTER tags:post:1 tags:post:2 # Intersection +SUNION tags:post:1 tags:post:2 # Union +SDIFF tags:post:1 tags:post:2 # Difference + +SCARD tags:post:1 # Cardinality (count) +SRANDMEMBER tags:post:1 # Random member +``` + +### Sorted Sets + +```redis +ZADD leaderboard 100 "alice" 85 "bob" 92 "charlie" +ZRANGE leaderboard 0 -1 WITHSCORES # Ascending +ZREVRANGE leaderboard 0 2 WITHSCORES # Top 3 + +ZSCORE leaderboard "alice" # Get score +ZINCRBY leaderboard 5 "bob" # Increment score +ZRANK leaderboard "alice" # Rank (0-indexed) + +ZRANGEBYSCORE leaderboard 80 100 # By score range +ZCOUNT leaderboard 80 100 # Count in range +``` + +### Hashes + +```redis +HSET user:1 name "Alice" email "alice@example.com" age 30 +HGET user:1 name # "Alice" +HGETALL user:1 # All fields and values +HMGET user:1 name email # Multiple fields + +HINCRBY user:1 age 1 # Increment field +HDEL user:1 email # Delete field +HEXISTS user:1 name # Check field exists +HKEYS user:1 # All field names +``` + +### Streams + +```redis +XADD events * type "click" page "/home" +XADD events * type "view" page "/products" + +XREAD COUNT 10 STREAMS events 0 # Read from beginning +XREAD BLOCK 5000 STREAMS events $ # Block for new + +# Consumer groups +XGROUP CREATE events mygroup $ MKSTREAM +XREADGROUP GROUP mygroup consumer1 COUNT 1 STREAMS events > +XACK events mygroup 1234567890-0 +``` + +## Pub/Sub + +```redis +# Subscriber +SUBSCRIBE news:tech news:sports +PSUBSCRIBE news:* # Pattern subscribe + +# Publisher +PUBLISH news:tech "Redis 8.0 released!" +``` + +## Caching Patterns + +### Cache-Aside (Lazy Loading) + +```python +def get_user(user_id): + # Try cache first + cached = redis.get(f"user:{user_id}") + if cached: + return json.loads(cached) + + # Cache miss - load from DB + user = db.query("SELECT * FROM users WHERE id = ?", user_id) + redis.setex(f"user:{user_id}", 3600, json.dumps(user)) + return user +``` + +### Write-Through + +```python +def update_user(user_id, data): + # Update DB + db.execute("UPDATE users SET ... WHERE id = ?", user_id) + + # Update cache + redis.setex(f"user:{user_id}", 3600, json.dumps(data)) +``` + +### Write-Behind (Write-Back) + +```python +def update_user(user_id, data): + # Update cache immediately + redis.setex(f"user:{user_id}", 3600, json.dumps(data)) + + # Queue DB write for later + redis.rpush("db:write:queue", json.dumps({ + "table": "users", + "id": user_id, + "data": data + })) +``` + +## Distributed Locking + +### Basic Lock + +```redis +SET lock:resource "owner" NX EX 30 +# NX = only if not exists +# EX = expire in 30 seconds + +# Release (only if owner) +# Use Lua script for atomicity +``` + +### Redlock Algorithm + +```python +# Acquire lock on N/2+1 instances +# Use if: Distributed system, need strong guarantees +# Libraries: redlock-py, redisson +``` + +## Persistence + +### RDB (Snapshots) + +```redis +# redis.conf +save 900 1 # Save if 1 key changed in 900s +save 300 10 # Save if 10 keys changed in 300s +save 60 10000 # Save if 10000 keys changed in 60s + +dbfilename dump.rdb +``` + +### AOF (Append-Only File) + +```redis +# redis.conf +appendonly yes +appendfsync everysec # Options: always, everysec, no +``` + +### Comparison + +| Aspect | RDB | AOF | +|--------|-----|-----| +| **Recovery Speed** | ✅ Fast | ⚠️ Slower | +| **Data Safety** | ⚠️ May lose recent | ✅ More durable | +| **File Size** | ✅ Compact | ⚠️ Larger | +| **Recommended** | Backups | Durability | + +## High Availability + +### Replication + +```redis +# On replica +REPLICAOF master.example.com 6379 +``` + +### Redis Sentinel + +``` +┌─────────┐ ┌─────────┐ ┌─────────┐ +│Sentinel │ │Sentinel │ │Sentinel │ +└────┬────┘ └────┬────┘ └────┬────┘ + │ │ │ + └───────────────┼───────────────┘ + │ + ┌──────┴──────┐ + │ Master │ + └──────┬──────┘ + ┌───────────┼───────────┐ + ▼ ▼ ▼ + ┌────────┐ ┌────────┐ ┌────────┐ + │Replica │ │Replica │ │Replica │ + └────────┘ └────────┘ └────────┘ +``` + +### Redis Cluster + +``` +┌─────────────────────────────────────────┐ +│ Redis Cluster │ +│ │ +│ Slots 0-5460 5461-10922 10923-16383│ +│ ┌────────┐ ┌────────┐ ┌────────┐│ +│ │Master 1│ │Master 2│ │Master 3││ +│ └────┬───┘ └────┬───┘ └────┬───┘│ +│ │ │ │ │ +│ ┌────▼───┐ ┌────▼───┐ ┌────▼───┐│ +│ │Replica │ │Replica │ │Replica ││ +│ └────────┘ └────────┘ └────────┘│ +└─────────────────────────────────────────┘ +``` + +## Lua Scripting + +```redis +-- Atomic increment if below limit +local current = redis.call('GET', KEYS[1]) +if current and tonumber(current) >= tonumber(ARGV[1]) then + return 0 +else + return redis.call('INCR', KEYS[1]) +end +``` + +```python +# Python client +script = """ +local current = redis.call('GET', KEYS[1]) +... +""" +rate_limit = redis.register_script(script) +result = rate_limit(keys=['rate:user:1'], args=[100]) +``` + +## Use Cases + +| Use Case | Data Structure | +|----------|----------------| +| **Session storage** | Strings + TTL | +| **Caching** | Strings, Hashes | +| **Rate limiting** | Strings + INCR | +| **Leaderboards** | Sorted Sets | +| **Queues** | Lists (BRPOP) | +| **Pub/Sub** | Pub/Sub | +| **Real-time analytics** | HyperLogLog, Streams | +| **Geospatial** | Geo commands | + +## Performance Tips + +| Tip | Rationale | +|-----|-----------| +| **Use pipelining** | Reduce round trips | +| **Avoid large keys** | Memory + network | +| **Use SCAN over KEYS** | Non-blocking iteration | +| **Set TTLs** | Prevent memory bloat | +| **Use connection pooling** | Reduce connection overhead | + +## Monitoring + +```redis +INFO # Server info +INFO memory # Memory stats +INFO replication # Replication info + +MONITOR # Real-time commands (debug only) +SLOWLOG GET 10 # Slow queries + +CLIENT LIST # Connected clients +DBSIZE # Key count +``` + +## Related + +- [[Database Engines]] — Database comparison +- [[Caching Strategies]] — Caching patterns +- [[Message Queues]] — Queue comparison +- [[Tools MOC]] — All tools diff --git a/Tools/Runbooks.md b/Tools/Runbooks.md new file mode 100644 index 0000000..98fac2c --- /dev/null +++ b/Tools/Runbooks.md @@ -0,0 +1,522 @@ +--- +title: Runbooks +aliases: + - Operations Runbooks + - Playbooks + - SOPs + - Standard Operating Procedures +tags: + - tools + - devops + - sre + - operations + - documentation +type: reference +status: complete +created: "2025-12-16" +--- + +# Runbooks + +Documented procedures for operating systems, responding to incidents, and performing routine tasks. + +## Overview + +| Type | Purpose | When Used | +|------|---------|-----------| +| **Alert Runbooks** | Respond to specific alerts | Incident response | +| **Operations Runbooks** | Routine maintenance | Scheduled tasks | +| **Troubleshooting Guides** | Diagnose issues | Investigation | +| **Deployment Runbooks** | Release procedures | Deployments | + +## Runbook Structure + +### Standard Template + +```markdown +# [Runbook Title] + +**Service:** payment-service +**Owner:** @payments-team +**Last Updated:** 2024-01-15 +**Review Frequency:** Quarterly + +## Overview +Brief description of what this runbook covers. + +## Prerequisites +- Access requirements +- Tools needed +- Knowledge required + +## Procedure +Step-by-step instructions. + +## Verification +How to confirm success. + +## Rollback +How to undo if needed. + +## Escalation +When and who to escalate to. + +## Related +Links to related runbooks and documentation. +``` + +## Alert Runbook Example + +```markdown +# High Error Rate - Payment Service + +**Alert:** payment-service-high-error-rate +**Severity:** SEV2 +**Service:** payment-service +**Owner:** @payments-team + +## Alert Details + +**Query:** +```promql +sum(rate(http_requests_total{service="payment",status=~"5.."}[5m])) +/ +sum(rate(http_requests_total{service="payment"}[5m])) > 0.01 +``` + +**Threshold:** Error rate > 1% +**Duration:** 5 minutes + +## Impact + +Payment processing may be failing for users. Revenue impact +possible if not resolved quickly. + +## Quick Diagnosis + +### 1. Check Error Distribution + +```bash +# View error breakdown by endpoint +kubectl logs -l app=payment-service --tail=100 | \ + grep "ERROR" | \ + jq '.endpoint' | sort | uniq -c | sort -rn +``` + +### 2. Check Dependencies + +- [ ] Database: [DB Dashboard](https://grafana/db) +- [ ] Payment Gateway: [Gateway Status](https://status.stripe.com) +- [ ] Redis: [Cache Dashboard](https://grafana/redis) + +### 3. Check Recent Changes + +```bash +# Recent deployments +kubectl rollout history deployment/payment-service + +# Recent config changes +git log --oneline -10 -- config/payment/ +``` + +## Common Causes & Fixes + +### Database Connection Issues + +**Symptoms:** Timeout errors, connection refused +**Check:** + +```bash +kubectl exec -it payment-service-xxx -- \ + pg_isready -h $DB_HOST +``` + +**Fix:** Scale up connection pool or restart pods + +### Payment Gateway Timeout + +**Symptoms:** Gateway timeout errors +**Check:** [Stripe Status](https://status.stripe.com) +**Fix:** Enable circuit breaker fallback + +### Memory Pressure + +**Symptoms:** OOMKilled pods, slow responses +**Check:** + +```bash +kubectl top pods -l app=payment-service +``` + +**Fix:** Scale horizontally or increase memory limits + +## Mitigation Steps + +### Option 1: Rollback (if recent deploy) + +```bash +kubectl rollout undo deployment/payment-service +kubectl rollout status deployment/payment-service +``` + +### Option 2: Scale Up + +```bash +kubectl scale deployment/payment-service --replicas=10 +``` + +### Option 3: Enable Degraded Mode + +```bash +# Toggle feature flag +curl -X POST https://feature-flags/api/flags/payment-degraded-mode \ + -d '{"enabled": true}' +``` + +## Verification + +- [ ] Error rate returned to < 1% +- [ ] Latency p99 < 200ms +- [ ] No new errors in logs +- [ ] Payments processing successfully + +## Escalation Paths + +| Condition | Escalate To | +|-----------|-------------| +| Cannot diagnose in 15 min | Senior on-call | +| Database issues confirmed | DBA on-call | +| Payment gateway issue | Stripe support | +| Revenue impact > $10k | Engineering management | + +## Related Runbooks + +- [[Database Failover Runbook]] +- [[Payment Gateway Fallback]] +- [[Scaling Procedures]] + +``` + +## Operations Runbook Example + +```markdown +# Database Backup Verification + +**Frequency:** Weekly +**Owner:** @platform-team +**Duration:** 30 minutes + +## Overview +Verify database backups are completing successfully and +can be restored. + +## Prerequisites +- [ ] Access to backup storage (S3) +- [ ] Access to staging environment +- [ ] Database admin credentials + +## Procedure + +### 1. Check Backup Completion +```bash +# List recent backups +aws s3 ls s3://backups/postgres/ --recursive | \ + sort | tail -7 +``` + +Expected: Daily backups for last 7 days + +### 2. Verify Backup Integrity + +```bash +# Download latest backup +aws s3 cp s3://backups/postgres/latest.sql.gz ./ + +# Check file integrity +gunzip -t latest.sql.gz + +# Check file size (should be > 1GB for production) +ls -lh latest.sql.gz +``` + +### 3. Test Restore (Staging) + +```bash +# Restore to staging +gunzip -c latest.sql.gz | \ + psql -h staging-db.internal -U admin -d restore_test + +# Verify row counts +psql -h staging-db.internal -d restore_test -c \ + "SELECT COUNT(*) FROM users;" +``` + +### 4. Document Results + +Record in [Backup Verification Log](https://wiki/backup-log): + +- Date verified +- Backup size +- Restore time +- Any issues found + +## Success Criteria + +- [ ] Backup files exist for all scheduled days +- [ ] Files are not corrupted (gunzip passes) +- [ ] Restore completes without errors +- [ ] Data spot-check passes + +## Failure Actions + +| Issue | Action | +|-------|--------| +| Missing backup | Check backup job logs, alert DBA | +| Corrupted file | Restore from previous day, investigate | +| Restore fails | Page DBA on-call | + +## Escalation Procedures + +- Missing >1 day of backups: Page DBA +- Restore failure: SEV3 incident + +``` + +## Troubleshooting Guide Example + +```markdown +# Troubleshooting: Slow API Response Times + +**Service:** api-gateway +**Owner:** @platform-team + +## Symptoms +- API latency > 500ms (normally < 100ms) +- User complaints about slow page loads +- Timeout errors increasing + +## Diagnosis Flowchart + +``` + +Start + │ + ▼ +Is it all endpoints? ──No──► Check specific endpoint + │ (see "Single Endpoint" below) + Yes + │ + ▼ +Check dependencies ──Slow──► Diagnose dependency + │ (Database? Cache? External?) + Healthy + │ + ▼ +Check resource usage ──High──► Scale or optimize + │ (CPU? Memory? Connections?) + Normal + │ + ▼ +Check recent changes ──Found──► Consider rollback + │ + None + │ + ▼ +Deep dive with profiling + +``` + +## Diagnostic Commands + +### Overall Health +```bash +# Check pod health +kubectl get pods -l app=api-gateway + +# Check resource usage +kubectl top pods -l app=api-gateway + +# Check recent events +kubectl get events --sort-by='.lastTimestamp' | \ + grep api-gateway +``` + +### Network Diagnostics + +```bash +# Check DNS resolution time +time nslookup database.internal + +# Check connection to dependencies +kubectl exec api-gateway-xxx -- \ + curl -w "@curl-format.txt" -o /dev/null -s \ + http://user-service/health +``` + +### Database Diagnostics + +```bash +# Check active connections +psql -c "SELECT count(*) FROM pg_stat_activity;" + +# Check slow queries +psql -c "SELECT query, calls, mean_time + FROM pg_stat_statements + ORDER BY mean_time DESC LIMIT 10;" +``` + +### Application Diagnostics + +```bash +# Get thread dump (Java) +kubectl exec api-gateway-xxx -- \ + jstack $(pgrep java) + +# Check GC logs +kubectl logs api-gateway-xxx | grep "GC" + +# Enable debug logging temporarily +kubectl set env deployment/api-gateway LOG_LEVEL=DEBUG +``` + +## Common Causes + +### 1. Database Slow Queries + +**Indicators:** High DB latency in traces +**Fix:** + +- Add missing indexes +- Optimize queries +- Scale read replicas + +### 2. Connection Pool Exhaustion + +**Indicators:** Connection timeout errors +**Fix:** + +- Increase pool size +- Check for connection leaks +- Add connection timeout + +### 3. Memory Pressure / GC + +**Indicators:** High GC pause times, OOM events +**Fix:** + +- Increase heap size +- Tune GC parameters +- Fix memory leaks + +### 4. Noisy Neighbor + +**Indicators:** CPU throttling, variable latency +**Fix:** + +- Request dedicated nodes +- Implement resource limits +- Scale horizontally + +## Resolution Verification + +- [ ] Latency returned to baseline +- [ ] Error rate normal +- [ ] No ongoing resource pressure +- [ ] Root cause identified and tracked + +``` + +## Best Practices + +### Writing Good Runbooks + +| Practice | Rationale | +|----------|-----------| +| **Use simple language** | Stressful situations impair reading | +| **Be specific** | Exact commands, not descriptions | +| **Include copy-paste** | Reduce typing errors | +| **Add verification steps** | Confirm each action worked | +| **Show expected output** | So operators know what's normal | +| **Keep current** | Outdated runbooks cause harm | + +### Runbook Maintenance + +```markdown +## Runbook Review Checklist (Quarterly) + +- [ ] Run through procedure in staging +- [ ] Verify all commands still work +- [ ] Update screenshots/output examples +- [ ] Check all links are valid +- [ ] Review escalation contacts +- [ ] Update based on recent incidents +- [ ] Get peer review +``` + +### Runbook Automation + +| Level | Description | +|-------|-------------| +| **Manual** | Human follows document | +| **Assisted** | Scripts for complex steps | +| **Semi-automated** | Human triggers, automation runs | +| **Fully automated** | No human intervention | + +```bash +# Example: Assisted runbook script +#!/bin/bash +# runbook-high-error-rate.sh + +echo "=== Payment Service High Error Rate Runbook ===" + +echo -e "\n[1/4] Checking error distribution..." +kubectl logs -l app=payment-service --tail=100 2>/dev/null | \ + grep "ERROR" | head -20 + +echo -e "\n[2/4] Checking recent deployments..." +kubectl rollout history deployment/payment-service | tail -5 + +echo -e "\n[3/4] Checking resource usage..." +kubectl top pods -l app=payment-service + +echo -e "\n[4/4] Checking dependencies..." +for dep in database redis payment-gateway; do + echo -n "$dep: " + kubectl exec deploy/payment-service -- \ + curl -s -o /dev/null -w "%{http_code}" http://$dep/health + echo +done + +echo -e "\n=== Suggested Actions ===" +echo "1. If recent deploy: kubectl rollout undo deployment/payment-service" +echo "2. If high resource: kubectl scale deployment/payment-service --replicas=10" +echo "3. If dependency down: Check dependency runbook" +``` + +## Tools + +| Tool | Purpose | +|------|---------| +| **Confluence/Notion** | Documentation hosting | +| **Rundeck** | Runbook automation | +| **StackStorm** | Event-driven automation | +| **PagerDuty** | Runbook links in alerts | +| **Jupyter Notebooks** | Interactive runbooks | + +## Anti-Patterns + +| Anti-Pattern | Problem | +|--------------|---------| +| **Runbook rot** | Outdated procedures cause mistakes | +| **Too complex** | Won't be followed under stress | +| **Not tested** | May not work when needed | +| **Only one person knows** | Knowledge silos | +| **No verification steps** | Can't confirm success | + +## Related + +- [[Incident Management]] — Using runbooks in incidents +- [[SLOs and SLIs]] — What runbooks protect +- [[Chaos Engineering]] — Testing runbooks +- [[Technical Writing]] — Writing better docs +- [[Tools MOC]] — All tools diff --git a/Tools/SLOs and SLIs.md b/Tools/SLOs and SLIs.md new file mode 100644 index 0000000..6b685a6 --- /dev/null +++ b/Tools/SLOs and SLIs.md @@ -0,0 +1,353 @@ +--- +title: SLOs and SLIs +aliases: + - Service Level Objectives + - Service Level Indicators + - SLAs + - Error Budgets +tags: + - tools + - devops + - sre + - observability + - reliability +type: reference +status: complete +created: "2025-12-16" +--- + +# SLOs and SLIs + +Quantitative approach to defining and measuring service reliability, forming the foundation of Site Reliability Engineering (SRE). + +## Overview + +| Term | Full Name | Definition | +|------|-----------|------------| +| **SLI** | Service Level Indicator | Metric measuring service behavior | +| **SLO** | Service Level Objective | Target value for an SLI | +| **SLA** | Service Level Agreement | Contract with consequences | +| **Error Budget** | 100% - SLO | Allowable unreliability | + +## The Hierarchy + +``` +┌─────────────────────────────────────────────┐ +│ SLA (Contract) │ +│ "99.9% uptime or service credits" │ +│ │ +│ ┌───────────────────────────────────────┐ │ +│ │ SLO (Target) │ │ +│ │ "99.95% of requests < 200ms" │ │ +│ │ │ │ +│ │ ┌─────────────────────────────────┐ │ │ +│ │ │ SLI (Measurement) │ │ │ +│ │ │ "Request latency at p99" │ │ │ +│ │ └─────────────────────────────────┘ │ │ +│ └───────────────────────────────────────┘ │ +└─────────────────────────────────────────────┘ +``` + +**Key Insight:** SLOs should be stricter than SLAs to provide a safety buffer. + +## Service Level Indicators (SLIs) + +### Common SLI Categories + +| Category | Measures | Examples | +|----------|----------|----------| +| **Availability** | Is it up? | Success rate, uptime | +| **Latency** | How fast? | Response time (p50, p99) | +| **Throughput** | How much? | Requests per second | +| **Error Rate** | How often failing? | 5xx errors / total requests | +| **Correctness** | Is it right? | Data integrity, accuracy | +| **Freshness** | How recent? | Data staleness | + +### Defining Good SLIs + +**Good SLI Properties:** + +| Property | Description | +|----------|-------------| +| **Measurable** | Can be quantified objectively | +| **User-centric** | Reflects user experience | +| **Actionable** | Team can influence it | +| **Understandable** | Clear to all stakeholders | + +### SLI Specifications + +``` +SLI: Request Latency +──────────────────────────────────────── +Measurement: Time from request received to response sent +Source: Application metrics (Prometheus) +Aggregation: 99th percentile over 5-minute windows +Good event: Response time < 200ms +Valid event: All HTTP requests (excluding health checks) +``` + +### Calculating SLIs + +**Availability SLI:** + +``` +Availability = (Successful Requests / Total Requests) × 100 + +Example: (9,950 / 10,000) × 100 = 99.5% +``` + +**Latency SLI:** + +``` +Latency SLI = (Requests < Threshold / Total Requests) × 100 + +Example: (9,800 requests < 200ms / 10,000 total) × 100 = 98% +``` + +## Service Level Objectives (SLOs) + +### Setting SLOs + +| Consideration | Guidance | +|---------------|----------| +| **User expectations** | What do users actually need? | +| **Historical data** | What are you achieving now? | +| **Business requirements** | What can you afford to provide? | +| **Technical constraints** | What's realistically achievable? | + +### SLO Examples + +| Service | SLI | SLO | +|---------|-----|-----| +| **API Gateway** | Availability | 99.95% over 30 days | +| **API Gateway** | Latency (p99) | < 100ms for 99% of requests | +| **Database** | Availability | 99.99% over 30 days | +| **Search** | Latency (p50) | < 50ms for 95% of requests | +| **Batch Job** | Success rate | 99.9% of jobs complete | +| **Data Pipeline** | Freshness | < 5 minutes old 99.9% of time | + +### SLO Document Template + +```yaml +service: payment-api +owner: payments-team +slos: + - name: Availability + description: Payment API responds successfully + sli: + type: availability + good_event: HTTP status 2xx or 4xx + valid_event: All requests (excluding /health) + objective: 99.95% + window: 30 days + + - name: Latency + description: Payment API responds quickly + sli: + type: latency + threshold: 200ms + percentile: 99 + objective: 99% + window: 30 days +``` + +## Error Budgets + +### The Concept + +``` +Error Budget = 100% - SLO + +If SLO = 99.9% +Then Error Budget = 0.1% +``` + +### Error Budget Calculation + +| SLO | Error Budget | Monthly Downtime Allowed | +|-----|--------------|--------------------------| +| 99% | 1% | 7 hours 18 minutes | +| 99.9% | 0.1% | 43 minutes 50 seconds | +| 99.95% | 0.05% | 21 minutes 55 seconds | +| 99.99% | 0.01% | 4 minutes 23 seconds | +| 99.999% | 0.001% | 26 seconds | + +### Using Error Budgets + +``` +┌─────────────────────────────────────────────────────┐ +│ Error Budget Status │ +├─────────────────────────────────────────────────────┤ +│ │ +│ Budget: 0.1% (43 min/month) │ +│ Used: 0.03% (13 min) │ +│ Remaining: 0.07% (30 min) │ +│ │ +│ ██████████░░░░░░░░░░░░░░░░░░░░ 30% consumed │ +│ │ +│ Status: ✅ HEALTHY - Ship features! │ +│ │ +└─────────────────────────────────────────────────────┘ +``` + +### Error Budget Policies + +| Budget Status | Action | +|---------------|--------| +| **> 50% remaining** | Ship features, experiment | +| **25-50% remaining** | Normal development, monitor | +| **< 25% remaining** | Slow down, focus on reliability | +| **Exhausted** | Freeze features, fix reliability | + +### Error Budget Policy Example + +```markdown +## Error Budget Policy + +### When budget is healthy (>50% remaining): +- Normal feature development proceeds +- Risky changes allowed with appropriate review + +### When budget is concerning (25-50% remaining): +- Feature development continues with caution +- Extra scrutiny on changes that might affect reliability + +### When budget is critical (<25% remaining): +- Feature freeze except for reliability improvements +- Post-incident reviews required for all incidents + +### When budget is exhausted (0% remaining): +- Hard feature freeze +- All engineering effort on reliability +- Executive escalation required for exceptions +``` + +## SLAs vs SLOs + +### Key Differences + +| Aspect | SLO | SLA | +|--------|-----|-----| +| **Audience** | Internal teams | External customers | +| **Consequence** | Policy-based | Contractual (financial) | +| **Target** | What we aim for | What we guarantee | +| **Flexibility** | Can be adjusted | Legally binding | + +### Relationship + +``` +SLA: 99.9% availability (contractual) + ↑ + │ Buffer (0.05%) + │ +SLO: 99.95% availability (internal target) + ↑ + │ Buffer (0.04%) + │ +Actual: 99.99% availability (current performance) +``` + +## Implementation + +### Prometheus SLO Recording Rules + +```yaml +groups: + - name: slo_availability + rules: + # Total requests + - record: sli:http_requests:rate5m + expr: sum(rate(http_requests_total[5m])) + + # Successful requests + - record: sli:http_requests_success:rate5m + expr: sum(rate(http_requests_total{status=~"2.."}[5m])) + + # Availability SLI + - record: sli:availability:ratio + expr: sli:http_requests_success:rate5m / sli:http_requests:rate5m + + # Error budget consumption (30 day window) + - record: slo:error_budget:remaining + expr: | + 1 - ( + (1 - avg_over_time(sli:availability:ratio[30d])) + / + (1 - 0.999) # SLO target + ) +``` + +### Alerting on Error Budget + +```yaml +groups: + - name: slo_alerts + rules: + # Alert when burning budget too fast (will exhaust in 2 hours) + - alert: HighErrorBudgetBurn + expr: | + slo:error_budget:burn_rate:1h > 14.4 + for: 5m + labels: + severity: critical + annotations: + summary: "High error budget burn rate" + description: "At current rate, error budget will be exhausted in 2 hours" + + # Alert when budget is low + - alert: ErrorBudgetLow + expr: slo:error_budget:remaining < 0.25 + for: 5m + labels: + severity: warning + annotations: + summary: "Error budget below 25%" +``` + +### Multi-Window, Multi-Burn-Rate Alerts + +| Window | Burn Rate | Exhaustion Time | Alert Type | +|--------|-----------|-----------------|------------| +| 5m | 14.4× | 2 hours | Page | +| 30m | 6× | 5 hours | Page | +| 6h | 1× | 30 days | Ticket | + +## Tools for SLOs + +| Tool | Type | Features | +|------|------|----------| +| **Prometheus** | Metrics | Recording rules, alerts | +| **Grafana** | Visualization | SLO dashboards | +| **Nobl9** | SLO Platform | Error budgets, burn rates | +| **Datadog** | Observability | SLO tracking | +| **Google SLO Generator** | OSS | Automated SLO from metrics | +| **Sloth** | OSS | Prometheus SLO generator | + +## Best Practices + +### Do + +| Practice | Rationale | +|----------|-----------| +| **Start simple** | 1-2 SLOs per service initially | +| **Focus on user experience** | External behavior, not internals | +| **Review regularly** | Adjust as service evolves | +| **Make SLOs visible** | Dashboards, status pages | +| **Act on error budgets** | Actually slow down when exhausted | + +### Don't + +| Anti-Pattern | Problem | +|--------------|---------| +| **100% SLO** | Impossible, blocks all change | +| **Too many SLOs** | Dilutes focus | +| **Internal metrics as SLIs** | CPU usage ≠ user experience | +| **Ignoring error budgets** | Defeats the purpose | +| **SLO without consequences** | Just becomes a vanity metric | + +## Related + +- [[Incident Management]] — When SLOs are breached +- [[Observability Stack]] — Measuring SLIs +- [[Prometheus]] — SLO implementation +- [[Tools MOC]] — All tools diff --git a/Tools/SQLite.md b/Tools/SQLite.md new file mode 100644 index 0000000..70409a6 --- /dev/null +++ b/Tools/SQLite.md @@ -0,0 +1,400 @@ +--- +title: SQLite +aliases: + - SQLite3 + - SQLite Database +tags: + - tool + - database + - sql + - embedded +type: reference +status: complete +created: "2025-12-18" +--- + +# SQLite + +Self-contained, serverless, zero-configuration SQL database engine. + +## Overview + +| Aspect | Details | +|--------|---------| +| **Type** | Embedded relational database | +| **Storage** | Single file | +| **License** | Public domain | +| **Written In** | C | +| **Size** | ~700 KB library | +| **Use Cases** | Mobile apps, browsers, IoT, testing | + +## Key Characteristics + +| Feature | Description | +|---------|-------------| +| **Serverless** | No separate server process | +| **Zero-config** | No setup or administration | +| **Self-contained** | Single library, no dependencies | +| **Transactional** | ACID compliant | +| **Cross-platform** | Runs everywhere | + +## Architecture + +``` +┌─────────────────────────────────────┐ +│ Application │ +├─────────────────────────────────────┤ +│ SQLite Library │ +│ ┌────────────────────────────────┐ │ +│ │ SQL Parser → Bytecode Compiler │ │ +│ │ ↓ │ │ +│ │ Virtual Machine (VDBE) │ │ +│ │ ↓ │ │ +│ │ B-Tree Module │ │ +│ │ ↓ │ │ +│ │ Pager Module │ │ +│ │ ↓ │ │ +│ │ OS Interface (VFS) │ │ +│ └────────────────────────────────┘ │ +├─────────────────────────────────────┤ +│ Database File (.db) │ +└─────────────────────────────────────┘ +``` + +## Data Types + +### Type Affinity + +| Affinity | Description | +|----------|-------------| +| `TEXT` | Character strings | +| `NUMERIC` | Integer or real based on value | +| `INTEGER` | Signed integer | +| `REAL` | Floating point | +| `BLOB` | Binary data as-is | + +```sql +-- Dynamic typing +CREATE TABLE flexible ( + id INTEGER PRIMARY KEY, + data ANY -- Can store any type +); + +INSERT INTO flexible VALUES (1, 'text'); +INSERT INTO flexible VALUES (2, 42); +INSERT INTO flexible VALUES (3, 3.14); +``` + +## Common Operations + +### Creating Database + +```bash +# Create or open database +sqlite3 mydb.db + +# In-memory database +sqlite3 :memory: +``` + +### Basic SQL + +```sql +-- Create table +CREATE TABLE users ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + name TEXT NOT NULL, + email TEXT UNIQUE, + created_at TEXT DEFAULT CURRENT_TIMESTAMP +); + +-- Insert +INSERT INTO users (name, email) VALUES ('Alice', 'alice@example.com'); + +-- Query +SELECT * FROM users WHERE name LIKE 'A%'; + +-- Update +UPDATE users SET email = 'new@example.com' WHERE id = 1; + +-- Delete +DELETE FROM users WHERE id = 1; +``` + +### Indexes + +```sql +-- Create index +CREATE INDEX idx_users_email ON users(email); + +-- Unique index +CREATE UNIQUE INDEX idx_users_name ON users(name); + +-- Composite index +CREATE INDEX idx_users_name_email ON users(name, email); + +-- Partial index +CREATE INDEX idx_active_users ON users(name) WHERE active = 1; + +-- Check query plan +EXPLAIN QUERY PLAN SELECT * FROM users WHERE email = 'test@example.com'; +``` + +## Advanced Features + +### JSON Support + +```sql +-- Store JSON +CREATE TABLE events ( + id INTEGER PRIMARY KEY, + data JSON +); + +INSERT INTO events VALUES (1, '{"type": "click", "x": 100, "y": 200}'); + +-- Query JSON +SELECT json_extract(data, '$.type') FROM events; +SELECT data->>'$.type' FROM events; -- SQLite 3.38+ + +-- JSON functions +SELECT json_array(1, 2, 3); -- [1,2,3] +SELECT json_object('a', 1, 'b', 2); -- {"a":1,"b":2} +``` + +### Full-Text Search (FTS5) + +```sql +-- Create FTS table +CREATE VIRTUAL TABLE articles_fts USING fts5( + title, + content, + tokenize='porter' +); + +-- Populate +INSERT INTO articles_fts VALUES ('Hello World', 'This is the content...'); + +-- Search +SELECT * FROM articles_fts WHERE articles_fts MATCH 'hello'; +SELECT * FROM articles_fts WHERE articles_fts MATCH 'title:hello'; + +-- Ranking +SELECT *, rank FROM articles_fts WHERE articles_fts MATCH 'search term' +ORDER BY rank; +``` + +### Window Functions + +```sql +SELECT + name, + department, + salary, + ROW_NUMBER() OVER (PARTITION BY department ORDER BY salary DESC) as rank, + SUM(salary) OVER (PARTITION BY department) as dept_total, + AVG(salary) OVER () as company_avg +FROM employees; +``` + +### Common Table Expressions + +```sql +-- Recursive CTE for hierarchical data +WITH RECURSIVE subordinates AS ( + SELECT id, name, manager_id, 0 as level + FROM employees + WHERE id = 1 -- Start with CEO + + UNION ALL + + SELECT e.id, e.name, e.manager_id, s.level + 1 + FROM employees e + JOIN subordinates s ON e.manager_id = s.id +) +SELECT * FROM subordinates; +``` + +## Transactions & Locking + +### Transaction Modes + +```sql +-- Deferred (default) - lock acquired on first read/write +BEGIN DEFERRED TRANSACTION; + +-- Immediate - write lock acquired immediately +BEGIN IMMEDIATE TRANSACTION; + +-- Exclusive - exclusive lock immediately +BEGIN EXCLUSIVE TRANSACTION; + +COMMIT; +-- or +ROLLBACK; +``` + +### Write-Ahead Logging (WAL) + +```sql +-- Enable WAL mode (recommended for concurrency) +PRAGMA journal_mode = WAL; + +-- Check mode +PRAGMA journal_mode; + +-- Checkpoint +PRAGMA wal_checkpoint(TRUNCATE); +``` + +| Mode | Readers During Write | Writers Concurrent | Crash Recovery | +|------|---------------------|-------------------|----------------| +| **DELETE** | Blocked | No | Rollback journal | +| **WAL** | Allowed | No (but non-blocking) | WAL file | + +## Performance Tuning + +### Pragmas + +```sql +-- Memory settings +PRAGMA cache_size = -64000; -- 64MB cache +PRAGMA temp_store = MEMORY; -- Temp tables in RAM + +-- Synchronization (trade durability for speed) +PRAGMA synchronous = NORMAL; -- or OFF for max speed + +-- Enable foreign keys +PRAGMA foreign_keys = ON; + +-- Analyze for query planner +ANALYZE; + +-- Optimize database +VACUUM; +``` + +### Bulk Operations + +```sql +-- Wrap in transaction (much faster) +BEGIN TRANSACTION; +INSERT INTO users VALUES (...); +INSERT INTO users VALUES (...); +-- ... thousands more +COMMIT; + +-- Use prepared statements (in code) +-- Disable sync for bulk loads +PRAGMA synchronous = OFF; +-- Re-enable after +PRAGMA synchronous = NORMAL; +``` + +## Language Bindings + +### Python + +```python +import sqlite3 + +# Connect +conn = sqlite3.connect('mydb.db') +conn.row_factory = sqlite3.Row # Dict-like access + +# Execute +cursor = conn.cursor() +cursor.execute('SELECT * FROM users WHERE id = ?', (1,)) +user = cursor.fetchone() + +# Context manager +with conn: + conn.execute('INSERT INTO users (name) VALUES (?)', ('Bob',)) + # Auto-commits on success, rollback on exception + +conn.close() +``` + +### Node.js (better-sqlite3) + +```javascript +const Database = require('better-sqlite3'); +const db = new Database('mydb.db'); + +// Synchronous API (better-sqlite3) +const user = db.prepare('SELECT * FROM users WHERE id = ?').get(1); +const users = db.prepare('SELECT * FROM users').all(); + +// Transaction +const insertMany = db.transaction((users) => { + const stmt = db.prepare('INSERT INTO users (name) VALUES (?)'); + for (const user of users) stmt.run(user.name); +}); +insertMany([{ name: 'Alice' }, { name: 'Bob' }]); + +db.close(); +``` + +### Go + +```go +import ( + "database/sql" + _ "github.com/mattn/go-sqlite3" +) + +db, err := sql.Open("sqlite3", "./mydb.db") +defer db.Close() + +// Query +rows, _ := db.Query("SELECT id, name FROM users") +defer rows.Close() + +for rows.Next() { + var id int + var name string + rows.Scan(&id, &name) +} + +// Exec +db.Exec("INSERT INTO users (name) VALUES (?)", "Alice") +``` + +## Limitations + +| Limitation | Details | +|------------|---------| +| **Concurrent writes** | Single writer at a time | +| **Network access** | Local file only (no client-server) | +| **Database size** | Theoretical 281 TB, practical ~1 TB | +| **Row size** | Default 1 GB max | +| **Columns** | 2000 max per table | + +## Use Cases + +| Use Case | Why SQLite | +|----------|------------| +| **Mobile apps** | iOS/Android default, offline-first | +| **Desktop apps** | Browsers, editors, games | +| **Embedded systems** | Small footprint, zero-config | +| **Testing** | Fast, in-memory, disposable | +| **Data analysis** | Import CSV, query with SQL | +| **Application cache** | Structured data caching | +| **File format** | Application file format | + +## SQLite vs Server Databases + +| Aspect | SQLite | PostgreSQL/MySQL | +|--------|--------|------------------| +| **Setup** | None | Server installation | +| **Concurrency** | Limited | High | +| **Network** | Local only | Client-server | +| **Scalability** | Single machine | Distributed possible | +| **Best for** | Embedded, edge | Multi-user, web | + +## Related + +- [[PostgreSQL]] — Enterprise SQL database +- [[MySQL]] — Popular web database +- [[Database Engines]] — Database comparison +- [[Tools MOC]] — All tools