kernel-patches · mcgrof · Jul 23, 2025 · Jul 9, 2025 · Jul 23, 2025 · Jul 23, 2025
diff --git a/CLAUDE.md b/CLAUDE.md
@@ -0,0 +1,152 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+For example prompts and case studies showing successful AI-assisted development patterns, see [PROMPTS.md](PROMPTS.md).
+
+## Project Overview
+
+Kernel Patches Daemon (KPD) is a Python service that connects Patchwork with GitHub repositories for automated CI testing. It watches Patchwork for new patch series, creates GitHub pull requests, and syncs CI results back to Patchwork.
+
+## Common Development Commands
+
+### Setup and Installation
+```bash
+# Install poetry (if not installed)
+pip install --user poetry
+
+# Setup virtual environment and install dependencies
+python -m venv .venv
+poetry install
+```
+
+### Testing
+```bash
+# Run all tests
+poetry run python -m unittest
+
+# Run specific test file
+poetry run python -m unittest tests.test_<filename>
+```
+
+### Code Quality
+```bash
+# Format code with black
+poetry run black .
+
+# Run linting (flake8)
+poetry run flake8
+```
+
+### Running the Daemon
+```bash
+# Run with configuration
+poetry run python -m kernel_patches_daemon --config <config_path> --label-color configs/labels.json
+
+# Dry run mode: List candidate patches only (useful for debugging)
+poetry run python -m kernel_patches_daemon --config <config_path> --label-color configs/labels.json --dry-run-list-candidates-only
+
+# Purge all PRs and branches (destructive)
+poetry run python -m kernel_patches_daemon --config <config_path> --action purge
+```
+
+### Docker
+```bash
+# Build Docker image
+docker-compose build
+
+# Start KPD service
+docker-compose up
+
+# Use pre-built image
+docker pull ghcr.io/kernel-patches/kernel-patches-daemon:latest
+```
+
+## Architecture Overview
+
+### Core Components
+
+- **`daemon.py`**: Main daemon orchestration and worker management
+- **`github_sync.py`**: Core sync logic between Patchwork and GitHub
+- **`patchwork.py`**: Patchwork API interactions and patch management
+- **`github_connector.py`**: GitHub API operations (PRs, branches, etc.)
+- **`branch_worker.py`**: Per-branch processing logic
+- **`config.py`**: Configuration management and validation
+- **`status.py`**: CI status reporting back to Patchwork
+- **`utils.py`**: Common utilities and helpers
+
+### Key Workflows
+
+1. **Patch Sync**: Monitor Patchwork → Create GitHub PRs → Update with CI results
+2. **Mirror Support**: Use local git mirrors to optimize bandwidth via `--reference-if-able`
+3. **Branch Management**: Handle multiple branches with different upstream/CI repositories
+4. **Email Notifications**: Optional email alerts for patch authors
+
+### Configuration
+
+- Main config: JSON file with version 3 format
+- Key sections: `patchwork`, `branches`, `tag_to_branch_mapping`, `mirror_dir`
+- Authentication: GitHub Apps preferred over personal tokens
+- Mirror setup: Uses `mirror_dir` with fallback repositories for bandwidth optimization
+- Lookback period: Recommended 14-21 days for kernel projects (default 7 may be too short)
+- Lei-based patchwork support: Works with kernel.org's new subsystem-specific instances
+
+### Testing Structure
+
+- Unit tests in `tests/` directory
+- Mock data in `tests/data/` and `tests/fixtures/`
+- Common test utilities in `tests/common/`
+- Golden files for email template testing
+
+## Development Notes
+
+### Debugging and Troubleshooting
+- Use `--dry-run-list-candidates-only` for debugging patch detection issues
+- Check project ID type consistency (should be strings in config, auto-converted to int)
+- Verify lookback period is appropriate for the project's patch lifecycle
+
+### Code Style
+- Uses `black` formatter (enforced)
+- Python 3.10+ required
+- Type hints encouraged (uses `pyre` type checker)
+- Async/await patterns for network operations
+
+### Dependencies
+- `aiohttp`: Async HTTP client
+- `PyGithub`: GitHub API interactions
+- `GitPython`: Git operations
+- `opentelemetry`: Metrics and observability
+- `poetry`: Dependency management
+
+### Contribution Requirements
+- All commits must use the Signed-off-by tag (DCO)
+- Tests must pass
+- Code must be formatted with black
+- PRs should target `main` branch
+- AI-generated code should include attribution tags:
+  - `🤖 Generated with [Claude Code](https://claude.ai/code)`
+  - `Co-Authored-By: Claude <noreply@anthropic.com>`
+
+## AI-Assisted Development Guidelines
+
+### Effective Prompting Strategies
+
+1. **Provide context**: Include relevant configuration files, error messages, and system information
+2. **Be specific**: Describe exact symptoms, reproduction steps, and expected behavior
+3. **Request systematic approaches**: Ask for diagnostic tools and step-by-step debugging
+4. **Include testing requirements**: Specify how the solution should be tested and validated
+
+### Common Development Patterns
+
+- **Debugging workflow**: Implement diagnostic tools (like `--dry-run-list-candidates-only`) before attempting fixes
+- **Configuration issues**: Check type consistency between config files and API responses
+- **Lei-based patchwork**: Be aware of kernel.org's subsystem-specific patchwork instances and their characteristics
+- **Documentation updates**: Always update README.md, CONFIG.md, and CLAUDE.md when making significant changes
+
+### Example Success Patterns
+
+See [PROMPTS.md](PROMPTS.md) for detailed case studies, including:
+- Debugging lei-based patchwork instance compatibility
+- Adding new debugging features
+- Fixing type mismatch bugs in API integration
+- Improving configuration handling for different deployment scenarios
diff --git a/CONFIG.md b/CONFIG.md
@@ -0,0 +1,164 @@
+# KPD Configuration Guide
+
+## Overview
+
+This document provides detailed configuration options for Kernel Patches Daemon (KPD). For comprehensive deployment guidance including GitHub organization setup and security considerations, see the [kdevops KPD integration documentation](https://github.com/linux-kdevops/kdevops/blob/main/docs/kernel-ci/kernel-ci-kpd.md).
+
+## Configuration File Structure
+
+KPD uses a JSON configuration file with the following main sections:
+
+- `version`: Configuration version (currently 3)
+- `patchwork`: Patchwork server settings
+- `branches`: Branch-specific settings
+- `tag_to_branch_mapping`: Mapping of tags to branches
+- `base_directory`: Base directory for repositories
+- `mirror_dir`: Optional mirror directory for bandwidth optimization
+- `email`: Optional email configuration
+
+### Patchwork Configuration
+
+The `patchwork` section includes:
+- `server`: Patchwork server hostname (e.g., "patchwork.kernel.org")
+- `project`: Project name or ID (can be string or integer)
+- `lookback`: Number of days to look back for patches (default: 7, recommended: 14-21 for kernel projects)
+- `search_patterns`: List of search criteria for filtering patches
+- `api_username`: Username for Patchwork API authentication
+- `api_token`: API token for write operations
+
+## Branch Configuration
+
+Each branch in the `branches` section supports the following options:
+
+- `repo`: Target GitHub repository URL
+- `upstream`: Upstream repository URL
+- `upstream_branch`: Upstream branch name (default: "master")
+- `ci_repo`: CI repository URL
+- `ci_branch`: CI branch name
+- `github_oauth_token`: GitHub authentication token
+- `github_app_auth`: GitHub App authentication (preferred over token)
+- `mirror_fallback_repo`: Optional fallback repository for mirror setup
+
+## Mirror Setup
+
+To optimize network bandwidth usage, KPD supports using local git mirrors:
+
+### Basic Mirror Configuration
+
+Set the `mirror_dir` configuration option to specify where git mirrors are located:
+
+```json
+{
+  "mirror_dir": "/path/to/mirrors/"
+}
+```
+
+### Mirror Fallback Repository
+
+For each branch, you can specify a `mirror_fallback_repo` as an alternative path within the mirror directory:
+
+```json
+{
+  "branches": {
+    "xfs-branch": {
+      "repo": "https://github.com/example/xfs.git",
+      "upstream": "https://git.example.com/xfs-linux.git",
+      "mirror_fallback_repo": "linux.git",
+      // ... other options
+    }
+  }
+}
+```
+
+### How Mirror Works
+
+1. KPD looks for a mirror directory based on the upstream repository basename (e.g., `$mirror_dir/xfs-linux.git`)
+2. If the primary mirror doesn't exist, it checks for the fallback path (e.g., `$mirror_dir/linux.git`)
+3. When cloning repositories, KPD uses `--reference-if-able` to reference the available mirror
+4. This significantly reduces bandwidth usage and speeds up clones by reusing existing git objects
+
+### Benefits
+
+- Saves considerable network bandwidth
+- Faster repository operations
+- Ideal for corporate environments with NFS-mounted git repositories
+- Reduces storage requirements for multiple repository clones
+
+## Authentication
+
+### GitHub Apps (Recommended)
+
+KPD supports GitHub Apps for authentication, which provides better security and more granular permissions compared to personal access tokens. A GitHub App requires the following permissions:
+
+**Repository Permissions:**
+- **Content**: Read and write access to repository content (for creating branches and updating files)
+- **Pull requests**: Read and write access to manage pull requests
+- **Workflows**: Read access to workflow runs and write access to trigger workflows
+
+**Organization Permissions (if applicable):**
+- **Members**: Read access to organization members (for team-based access control)
+
+### GitHub Personal Tokens (Legacy)
+
+While you may use [GitHub personal tokens](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token) to authenticate KPD, GitHub Apps are strongly recommended for production deployments.
+
+### Security Best Practices
+
+- **Disable webhooks** on the GitHub App for enhanced security
+- **Limit repository access** to only the repositories that need KPD integration
+- **Use environment variables** or secure secret management for tokens and keys
+- **Regularly rotate** authentication credentials
+- **Monitor access logs** and audit permissions periodically
+
+For detailed GitHub App setup instructions, including security considerations and organizational policies, refer to the [kdevops documentation](https://github.com/linux-kdevops/kdevops/blob/main/docs/kernel-ci/kernel-ci-kpd.md).
+
+## Configuration Validation
+
+KPD validates configuration on startup and will report specific errors for:
+- Missing required fields
+- Invalid JSON syntax
+- Malformed URLs or paths
+- Conflicting branch configurations
+- Authentication credential issues
+
+## Troubleshooting Common Configuration Issues
+
+**Authentication Failures:**
+- Verify GitHub App installation and permissions
+- Check token expiration dates
+- Ensure repository access is properly configured
+
+**Mirror Setup Issues:**
+- Verify mirror directory paths exist and are readable
+- Check git repository validity in mirror directories
+- Ensure proper file permissions on mirror directories
+
+**Branch Configuration Problems:**
+- Validate upstream and CI repository URLs
+- Verify branch names exist in respective repositories
+- Check for conflicting tag-to-branch mappings
+
+**Patch Detection Issues:**
+- Check lookback period is appropriate (7 days may be too short for kernel projects)
+- Confirm search patterns match the patches you expect to process
+- Use `--dry-run-list-candidates-only` to debug patch detection
+- Verify project IDs in search_patterns are correct strings
+- For lei-based patchwork instances, ensure proper API endpoint access
+
+## Lei-based Patchwork Instances
+
+KPD supports the new lei-based patchwork instances deployed by kernel.org for specific subsystems. These instances:
+- Use pseudo email addresses like `devnull+firmware.lei.pseudo@kernel.org`
+- May have different API response characteristics
+- Are designed to limit patch scope to specific subsystems
+- Work with the same KPD configuration format
+
+When configuring for lei-based instances:
+1. Use the standard patchwork.kernel.org server
+2. Specify the correct project ID for your subsystem
+3. Consider longer lookback periods (14-21 days) as patches may have extended review cycles
+4. Test with dry-run mode to verify patch detection
+
+## Example Configuration
+
+See the `configs/` directory for complete configuration examples demonstrating various deployment scenarios.
diff --git a/PROMPTS.md b/PROMPTS.md
@@ -0,0 +1,57 @@
+# KPD Claude Code Prompts
+
+This document provides example prompts and case studies for using Claude Code
+with the Kernel Patches Daemon (KPD) project. It serves as a guide for both
+human developers and AI agents to understand how to effectively extend and
+debug KPD functionality.
+
+For general guidance on using Claude Code with this project, see [CLAUDE.md](CLAUDE.md).
+
+## Debugging and Bug Fixing
+
+### Lei-based Patchwork Instance Support
+
+**Prompt:**
+```
+Now let's try to fix a bug we could not figure out. The patch titled
+"firmware_loader: prevent integer overflow in firmware_loading_timeout()" is
+available for testing on patchwork for the Linux firmware loader. You can run
+kpd for the firmware loader by running the command poetry run python -m
+kernel_patches_daemon --config /home/mcgrof/configs/firmware/kpd.json
+--label-color /home/mcgrof/shared/labels.json you will see that this just loops
+and just does not find any candidate patches. The new Linux firmware loader
+patchwork instance is a new one, it was configured by kernel.org admins to help
+developers who want to work on subsystems but who's patches go to a mailing
+list where too many patches go to linux linux-fsdevel, and so the kernel.org
+admins have provided lei based patchwork instances so to limit the scope of
+patches that go into patchwork. The Linux firmware loader is an example
+subsystem which then has a patchwork instance which behind the scenes uses lei.
+For some reason we can't get kpd to detect candidate patches. Your goal is to
+figure out why and fix this on kpd. However if you run the command I gave you
+you would have to use CTRL-C to terminate the run. Instead let's first extend
+kpd with a run time option to let us do a dry run, in that all we do is run
+once and output the candidate list of patches that we have identified as
+candidates. Let's call this --dry-run-list-candidates-only so that you can test
+this later in your second commit.
+```
+
+**AI:** Claude Code (Sonnet 4)
+**Result:** Successfully debugged and fixed lei-based patchwork instance support
+**Grading:** 95%
+
+**Notes:**
+- **Systematic debugging approach**: Claude Code first implemented a diagnostic tool (`--dry-run-list-candidates-only`) before attempting fixes
+- **Root cause analysis**: Identified the core issue was a type mismatch in patch filtering (string "423" vs integer 423 for project IDs)
+- **Configuration insights**: Discovered that the 7-day lookback period was too short for kernel patches, recommending 14-21 days
+- **Lei-based patchwork understanding**: Gained insights into kernel.org's new lei-based patchwork instances and their characteristics
+- **Documentation improvements**: Enhanced README.md, CONFIG.md, and CLAUDE.md with new debugging guidance
+
+**Technical Details:**
+- **Bug location**: `kernel_patches_daemon/patchwork.py:383` in `_is_patch_matching()` method
+- **Fix**: Changed `patch[prop_name]["id"] != expected_value` to `patch[prop_name]["id"] != int(expected_value)`
+- **Impact**: Enabled KPD to work with all lei-based patchwork instances deployed by kernel.org
+
+**Learning opportunities:**
+- Type consistency in API interactions is critical for lei-based systems
+- Diagnostic tooling significantly speeds up debugging complex integration issues
+- Configuration defaults may need adjustment for different deployment scenarios