Add comprehensive SDK engine documentation and migration guide

[Copilot]

Creates complete documentation for the Copilot SDK engine mode, enabling users to build advanced agentic workflows with multi-turn conversations, custom inline tools, and multi-agent orchestration.

Documentation Structure

Reference Documentation (1 file, 13 KB)

• reference/engines-sdk.md - SDK engine specification with architecture, configuration options, performance characteristics, and security considerations

Guides (5 files, 101 KB)

• guides/sdk-sessions.md - Session management patterns, state persistence, restoration strategies
• guides/sdk-custom-tools.md - Inline tool development with JSON Schema parameters, implementation patterns, security
• guides/sdk-events.md - Real-time event handling, custom handlers, budget monitoring
• guides/sdk-multi-agent.md - Multi-agent coordination strategies (sequential, parallel, pipeline, hierarchical)
• guides/migrate-to-sdk.md - Step-by-step migration from CLI mode with decision framework, mapping tables, rollback strategy

Migration Examples (5 files, 28 KB)

• examples/sdk-migrations/ - Four complete migration examples ranging from simple (5 min) to complex (1-2 days)
• examples/sdk-migrations.md - Index with complexity assessment and best practices

Example: Basic SDK Configuration

CLI Mode (Before):

engine: copilot
tools:
  github:
    allowed: [issue_read, add_issue_comment]

SDK Mode (After):

engine:
  id: copilot
  mode: sdk
  session:
    persistent: true    # Context retention across turns
    max-turns: 10
  budget:
    max-tokens: 50000   # Cost controls
  events:
    streaming: true     # Real-time monitoring
    handlers:
      - on: token_usage
        action: log
tools:
  github:
    allowed: [issue_read, add_issue_comment]

Example: Custom Inline Tool

engine:
  mode: sdk
  tools:
    inline:
      - name: validate_code
        description: "Run custom validation checks"
        parameters:
          type: object
          required: [file_path]
          properties:
            file_path:
              type: string
        implementation: |
          const fs = require('fs');
          const content = fs.readFileSync(file_path, 'utf8');
          return {
            valid: !content.includes('console.log'),
            issues: []
          };

Example: Multi-Agent Orchestration

engine:
  mode: sdk
  agents:
    - name: security_expert
      role: "Security vulnerability analysis"
      parallel: true
      budget:
        max-tokens: 50000
    
    - name: performance_expert
      role: "Performance optimization analysis"
      parallel: true
      budget:
        max-tokens: 50000
    
    - name: coordinator
      role: "Aggregate findings from specialists"
      dependencies: [security_expert, performance_expert]
      session:
        restore: merge  # Combines context from all agents
  
  coordination:
    strategy: hybrid

Updates to Existing Documentation

• reference/engines.md - Added SDK mode section with feature comparison and links to comprehensive SDK documentation

Coverage

• 142 KB total documentation
• 100+ YAML examples demonstrating SDK features
• 10+ migration patterns from CLI to SDK
• 4 coordination strategies for multi-agent workflows
• 10+ event types with handler patterns
• 7-step migration process with compatibility checker

All documentation follows Diátaxis framework with cross-referenced links.

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Create documentation and migration guide for SDK workflows</issue_title>
<issue_description>## Description

Create comprehensive documentation for the Copilot SDK engine, including migration guide for converting existing CLI workflows to SDK mode.

Part of Epic

#10154 - Copilot SDK Integration for Advanced Agentic Workflows

Documentation Scope

1. SDK Engine Overview

New documentation page: docs/src/content/docs/reference/engines-sdk.md

Content:

• What is the SDK engine and how it differs from CLI
• When to use SDK vs CLI
• Architecture diagram
• Feature comparison table
• Performance characteristics
• Limitations and known issues

2. Session Management Guide

New documentation page: docs/src/content/docs/guides/sdk-sessions.md

Content:

• Multi-turn conversation patterns
• State persistence options
• Session restoration
• State size management
• Best practices for long-running workflows

3. Custom Tools Guide

New documentation page: docs/src/content/docs/guides/sdk-custom-tools.md

Content:

• Inline tool definition syntax
• Parameter schema specification
• Security considerations
• Tool implementation patterns
• Common tool examples
• Testing custom tools

4. Event Handling Guide

New documentation page: docs/src/content/docs/guides/sdk-events.md

Content:

• Event types reference
• Handler registration
• Real-time monitoring patterns
• Budget control with events
• Custom handler implementation

5. Multi-Agent Guide

New documentation page: docs/src/content/docs/guides/sdk-multi-agent.md

Content:

• Multi-agent architecture patterns
• Coordination strategies
• Shared context management
• Agent specialization examples
• Resource management

6. Migration Guide

New documentation page: docs/src/content/docs/guides/migrate-to-sdk.md

Content:

• Decision framework: Should I migrate?
• Step-by-step migration process
• CLI to SDK mapping table
• Example migrations
• Common pitfalls and solutions
• Rollback strategy

Migration Guide Content

Decision Framework

## Should I migrate to SDK mode?

**Migrate if you need:**
- ✅ Multi-turn conversations with context retention
- ✅ Custom inline tools with workflow-specific logic
- ✅ Real-time event handling and streaming
- ✅ Multi-agent orchestration
- ✅ Programmatic control flow (retry, branching)
- ✅ Cost-aware execution with budget controls

**Stay with CLI if:**
- ✅ Simple single-pass workflows
- ✅ No need for conversation context
- ✅ Standard MCP tools are sufficient
- ✅ Prefer battle-tested stability

Migration Steps

## Step-by-Step Migration

### 1. Evaluate Compatibility

Run compatibility checker:
```bash
gh aw check-sdk-compatibility workflow.md

2. Update Frontmatter

Before (CLI):

engine: copilot
tools:
  github:
    allowed: [issue_read]

After (SDK):

engine:
  id: copilot
  mode: sdk
  session:
    persistent: true
    max-turns: 5
tools:
  github:
    allowed: [issue_read]

3. Convert Custom Logic to Inline Tools

If you have bash scripts doing custom logic, convert to inline tools.

4. Test in Parallel

Run both CLI and SDK versions side-by-side to compare results.

5. Monitor and Tune

Use event handlers to monitor performance and adjust configuration.

## Example Migrations

### Example 1: Simple Workflow

**Before (CLI):**
```yaml
---
engine: copilot
tools:
  github:
    allowed: [issue_read, add_issue_comment]
---
# Triage Issue

Read issue and add triage label.

After (SDK):

---
engine:
  id: copilot
  mode: sdk
tools:
  github:
    allowed: [issue_read, add_issue_comment]
---
# Triage Issue

Read issue and add triage label.

Notes: Minimal changes needed for simple workflows.

Example 2: Multi-Stage Review

Before (CLI - not possible):

# Cannot implement multi-stage review with context retention

After (SDK):

---
engine:
  id: copilot
  mode: sdk
  session:
    persistent: true
    max-turns: 10
tools:
  github:
    allowed: [issue_read, add_issue_comment]
---
# Progressive Code Review

Stage 1: Initial review of PR
Stage 2: Follow-up after developer changes
Stage 3: Final approval check

Example 3: Custom Validation

Before (CLI with bash):

---
engine: copilot
tools:
  bash: ["validate.sh"]
---
# Review with Custom Validation

Review code and run ./validate.sh for custom checks.

After (SDK with inline tool):

---
engine:
  id: copilot
  mode: sdk
  tools:
    inline:
      - name: validate_code
        description: "Run custom validation"
        implementation: |
          const result = await exec('./validate.sh');
          return { valid: result.exitCode === 0 };
tools:
  github:
    allowed: [issue_read]
---
# Review with Custom Validation

Review code and use validate_code tool for custo...

</details>



<!-- START COPILOT CODING AGENT SUFFIX -->

- Fixes githubnext/gh-aw#10159

<!-- START COPILOT CODING AGENT TIPS -->
---

✨ Let Copilot coding agent [set things up for you](https://github.com/githubnext/gh-aw/issues/new?title=✨+Set+up+Copilot+instructions&body=Configure%20instructions%20for%20this%20repository%20as%20documented%20in%20%5BBest%20practices%20for%20Copilot%20coding%20agent%20in%20your%20repository%5D%28https://gh.io/copilot-coding-agent-tips%29%2E%0A%0A%3COnboard%20this%20repo%3E&assignees=copilot) — coding agent works faster and does higher quality work when set up for your repo.