📚 Documentation Noob Test Report - January 26, 2026

[github-actions[bot]]

I tested the GitHub Agentic Workflows documentation from a complete beginner's perspective to identify pain points, confusing areas, and opportunities for improvement.

Summary

Date of test: January 26, 2026
Testing method: Direct documentation analysis
Pages analyzed:

• Home page (index.mdx)
• Quick Start Guide (setup/quick-start.md)
• Creating Workflows (setup/agentic-authoring.mdx)
• CLI Commands (setup/cli.md)
• Glossary (reference/glossary.md)

Overall impression: The documentation is comprehensive and well-structured, but contains several barriers that could frustrate complete beginners trying to get started for the first time.

---

🔴 Critical Issues (Block getting started)

1. Terminology Overload Without Early Glossary Link

Location: Quick Start Guide, first steps
Issue: The Quick Start immediately uses jargon like "frontmatter", "MCP", "compilation", and "lock files" before explaining them.

Problem: While a glossary link appears in a tip box, beginners scanning the steps might miss it. The first paragraph throws 4+ technical terms at new users.

Recommendation:

• Move the glossary tip to the VERY TOP of the page, before step 1
• Make it more visually prominent (different color/style)
• Consider inline tooltips or hover definitions for first use of technical terms

2. "Happy Path" Assumption May Not Be Happy

Location: Quick Start Guide - "Adding a Daily Status Workflow to Your Repo"
Issue: The guide says "This is a happy path guide" but prerequisites might fail.

Problem for beginners:

"If you stumble on one of these steps, go to the Prerequisites section"
```

But which step requires which prerequisite? A beginner won't know if they need:
- GitHub CLI already installed?
- Admin access to a repo?
- An active Copilot subscription?
- Write permissions?

**Recommendation:**
Add a **"Before You Begin" checklist** at the very top:
```
✅ GitHub CLI installed (check with `gh --version`)
✅ Admin or write access to a repository
✅ Active GitHub Copilot subscription
✅ Ability to create Personal Access Tokens
```
</details>

<details>
<summary><strong>3. PAT Creation Process Is Confusing</strong> ⚠️ HIGHEST PRIORITY</summary>

**Location:** Quick Start Guide - Step 4 "Create a Personal Access Token"  
**Issue:** The PAT creation instructions contain gotchas that will frustrate beginners.

**Specific problems:**
1. "Resource owner: Your personal account (required for Copilot Requests permission)" - Why? What if I'm in an org?
2. "Repository access: 'Public repositories' (required for Copilot Requests permission to appear)" - This is VERY confusing. Why does the repo access type affect which permissions appear?
3. The troubleshooting tip comes AFTER the steps, so users won't see it until they're already stuck.

**Missing information:**
- What happens if I select "Private repositories only"? (Copilot Requests won't appear)
- What if I work in an organization? (Different PAT requirements)
- Can I use a classic PAT? (No, but this isn't stated clearly)

**Recommendation:**
- Add a visual flowchart or decision tree for PAT creation
- Show what the GitHub UI should look like at each step (screenshots would help immensely)
- Move troubleshooting information INLINE with the steps, not at the end
- Add a clear warning: "⚠️ If you don't see 'Copilot Requests', ensure you selected 'Public repositories' or 'All repositories'"

**This is the #1 place where beginners will get stuck and give up.**
</details>

<details>
<summary><strong>4. Secret Setup Lacks Verification</strong></summary>

**Location:** Quick Start Guide - Step 4 "Add the token to your repository"  
**Issue:** After adding the secret, there's no way to verify it was set up correctly before running a workflow.

**Problem:** A beginner adds the secret `COPILOT_GITHUB_TOKEN` but doesn't know if:
- They copied it correctly
- It has the right permissions
- It will work with workflows

**Recommendation:**
- Add a verification step using `gh aw secrets bootstrap --engine copilot`
- Explain what a successful verification looks like
- Provide clear error messages and fixes if verification fails
</details>

---

## 🟡 Confusing Areas (Slow down learning)

<details>
<summary><strong>1. Unclear Relationship Between .md and .lock.yml Files</strong></summary>

**Location:** Quick Start Guide - Step 3  
**Issue:** The guide mentions ".lock.yml (the generated GitHub Actions workflow file)" in parentheses but doesn't explain WHY there are two files.

**What beginners are thinking:**
- "Do I edit the .md or .lock.yml file?"
- "What if they get out of sync?"
- "Can I delete one of them?"
- "Why do I need to commit both?"

**Recommendation:**
Add a clear explanation BEFORE step 3:
```
📝 Understanding Workflow Files

Agentic workflows use TWO files:
- **`.md` file** - Your source file written in natural language (you edit this)
- **`.lock.yml` file** - Generated GitHub Actions YAML (never edit directly)

Think of it like TypeScript → JavaScript: you write .md, the compiler creates .lock.yml
```
</details>

<details>
<summary><strong>2. "Compilation" Term Is Confusing</strong></summary>

**Location:** Multiple places throughout docs  
**Issue:** The term "compilation" is used without explaining what's being compiled or why.

**Problem:** In programming, compilation means translating source code to machine code. Here it means:
- Markdown → YAML
- Natural language → GitHub Actions
- Validating security settings
- Resolving imports

But beginners think "Wait, this is markdown, why does it need to compile?"

**Recommendation:**
First use of "compile" should have an inline explanation:
```
Run `gh aw compile` to transform your markdown workflow into a GitHub Actions YAML file 
(this process validates your workflow and applies security hardening)
```
</details>

<details>
<summary><strong>3. MCP Terminology Is Unexplained Early</strong></summary>

**Location:** Quick Start Guide - Step 2  
**Issue:** "Initialize your repository to configure custom agents and MCP server" - what's an MCP server?

**Problem:** The glossary defines MCP, but beginners won't know to look there. The term just appears without context.

**Recommendation:**
Add inline explanation on first use:
```
MCP (Model Context Protocol) - the system that connects AI agents to tools like GitHub's API
```
</details>

<details>
<summary><strong>4. "The Agentics" Collection Is Mentioned Without Context</strong></summary>

**Location:** Quick Start Guide - Step 3  
**Issue:** "Add a sample from the agentics collection" - what is "the agentics"?

**Problem:** Beginners don't know:
- What "the agentics" collection is
- Where it's hosted (github.com/githubnext/agentics)
- What other workflows are available
- Whether it's official or community-maintained

**Recommendation:**
Expand first mention:
```
Add a sample from The Agentics collection (a curated library of ready-to-use workflows 
maintained by GitHub Next at github.com/githubnext/agentics)

5. Agent File Terminology Causes Confusion

Location: Creating Workflows - "What is the agentic-workflows agent?"
Issue: Multiple uses of "agent" in different contexts:

• "Agentic workflows"
• "AI agents"
• "Agent files"
• "Custom agents"
• "The agentic-workflows agent"
• "Coding agent"

Problem: Beginners will be confused about what each "agent" means and how they differ.

Recommendation:
Create a clear table distinguishing agent types:

Term	Meaning	Example
Agentic workflow	A workflow with AI capabilities	daily-team-status.md
Agent file	Custom AI instructions (.agent.md)	agentic-workflows.agent.md
Coding agent	The AI that executes tasks	GitHub Copilot CLI
AI agent	Generic term for AI system	-

---

🟢 What Worked Well

1. Clear Step-by-Step Structure

The Quick Start Guide uses numbered steps with clear command examples. This is excellent for beginners who need hand-holding.

2. Code Examples Are Copy-Pasteable

All bash commands are in code blocks that are easy to copy. The wrap class helps with long commands.

3. Visual Hierarchy with Callouts

The use of tips, warnings, and callouts helps important information stand out.

4. Glossary Is Comprehensive

The glossary defines terms clearly with examples. Linking technical terms to the glossary is excellent practice.

5. CLI Command Reference Is Well-Organized

The "Most Common Commands" table at the top of the CLI docs is perfect for beginners. Shows what 95% of users need immediately.

---

📊 Recommendations (Prioritized)

Quick Wins (Can be done immediately)

1. Add "Prerequisites Checklist" to top of Quick Start (5 mins)

   • Shows exactly what you need before starting
   • Prevents confusion and failed attempts

2. Move Glossary tip to TOP of Quick Start (2 mins)

   • Make it the first thing beginners see
   • Larger, more prominent styling

3. Add inline definitions for jargon (30 mins)

   • First use of technical terms gets tooltip/explanation
   • Reduces need to constantly reference glossary

4. Add "Before/After" clarification for .md/.lock.yml files (10 mins)

   • Clear explanation of which file to edit
   • Diagram showing relationship

Medium-Term Improvements (Requires more work)

5. Add screenshots for PAT creation (1-2 hours)

   • Show actual GitHub UI at each step
   • Highlight what to click and what options to select

6. Create visual flowchart for PAT setup (1 hour)

   • Decision tree for different user situations
   • Covers personal accounts, orgs, private/public repos

7. Add verification steps after setup (2 hours)

   • Commands to check if secrets work
   • What success/failure looks like

8. Create "Compilation Explained" diagram (1-2 hours)

   • Visual showing markdown → YAML transformation
   • What happens during compilation
   • Why both files are needed

Long-Term Documentation Improvements

9. Interactive Tutorial/Wizard (Days/weeks)

   • Step-by-step guided setup in browser
   • Validates each step before moving to next
   • Catches common mistakes automatically

10. Video Walkthrough (Days)

    • 5-10 minute video showing complete setup
    • Covers common pitfalls
    • Shows what success looks like

11. Common Error Messages Guide (Days)

    • Every error message gets a troubleshooting page
    • Explains what went wrong and how to fix
    • Links from error messages to solutions

12. Terminology Consistency Review (Days)

    • Audit all uses of "agent" and clarify
    • Create style guide for documentation
    • Ensure consistent terminology across all pages

---

🎯 Most Important Issue to Fix IMMEDIATELY

PAT Creation Process (Critical Issue #3)

This is the #1 place where beginners will get stuck and give up. The "Copilot Requests" permission not appearing is a non-obvious UI quirk that will frustrate users.

Immediate action:

1. Add clear warning BEFORE step 3: "⚠️ Important: You MUST select 'Public repositories' or 'All repositories' for the Copilot Requests permission to appear. This is a GitHub UI requirement."
2. Add screenshot showing the correct selection
3. Add troubleshooting section INLINE (not at the end)

This single change could reduce support requests by 50%+.

---

Testing Methodology Note

Due to network isolation in the testing environment (Playwright browser couldn't access localhost), this analysis was conducted by examining documentation source files directly and applying beginner-perspective heuristics. While visual testing with screenshots would have been ideal, the source code analysis revealed significant content and structure issues that impact the beginner experience.

---

Labels: documentation, user-experience, automated-testing

AI generated by Documentation Noob Tester

• expires on Feb 2, 2026, 7:47 AM UTC

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-02-02T07:47:50.002Z.