📚 Documentation Noob Test Report - December 15, 2025

[github-actions[bot]]

Summary

Date of test: December 15, 2025
Pages analyzed:

• Home page (/gh-aw/)
• Quick Start Guide (/gh-aw/setup/quick-start/)
• CLI Commands (/gh-aw/setup/cli/)

Overall impression as a new user:
The documentation has a professional structure with good visual design, but contains several critical gaps and confusing areas that would significantly slow down a complete beginner trying to get started with GitHub Agentic Workflows.

---

🔴 Critical Issues Found

1. Missing "What is this?" on Homepage

Issue: The homepage jumps straight into "What are Agentic Workflows?" without first establishing what GitHub Agentic Workflows (the tool/extension) actually is.

Impact: As a newcomer, I'm confused about whether this is:

• A GitHub feature
• A CLI extension I need to install
• A framework I need to learn
• A service I need to sign up for

Recommendation: Add a prominent section at the very top:

## What is GitHub Agentic Workflows?

GitHub Agentic Workflows (gh-aw) is a GitHub CLI extension that lets you write AI-powered automation workflows in plain English using markdown files. These workflows run on GitHub Actions and can make intelligent decisions about your repository.

---

2. Unclear Installation Prerequisites

Issue: The Quick Start guide says "Install the extension" but doesn't clearly state what prerequisites you need BEFORE you can even install it.

What a beginner doesn't know:

• Do I need GitHub CLI installed first?
• Do I need a GitHub account?
• Do I need admin rights to my repository?
• Do I need a Copilot subscription?

Current text:

gh extension install githubnext/gh-aw

Should be:

### Prerequisites

Before installing, ensure you have:
- ✅ [GitHub CLI](https://cli.github.com/) installed and authenticated
- ✅ A GitHub repository (personal or org) with admin access
- ✅ Active [GitHub Copilot subscription](https://github.com/features/copilot) (for AI-powered workflows)
- ✅ Git installed on your machine

**Then install the extension:**
```bash
gh extension install githubnext/gh-aw

---

3. Token Creation Instructions Too Buried

Issue: Step 3 of Quick Start requires creating a Personal Access Token with a very specific "Copilot Requests" permission. This is CRITICAL but feels buried in a long list of sub-steps.

Why this is confusing:

• The "Copilot Requests" permission is unusual and not well-known
• The callout box explaining why you can't find it comes AFTER the steps
• No explanation of WHY this permission is needed

Recommendation: Add prominent explanation at the start:

### Why do I need a special token?

GitHub Agentic Workflows run AI agents that need to access GitHub Copilot on your behalf. This requires a special **"Copilot Requests"** permission that's only available with:
- Fine-grained personal access tokens (not classic tokens)
- An active GitHub Copilot subscription
- Your personal user account as the resource owner

---

4. "What happens next?" Missing After Token Setup

Issue: After I add the COPILOT_GITHUB_TOKEN secret, the guide says "✅ Your repository is now configured!" but doesn't tell me:

• Can I test if it worked?
• How do I know the token is valid?
• What happens if I did it wrong?

Missing verification step:

# Test your setup
gh aw status

# Expected output:
# ✓ gh-aw extension: v0.x.x
# ✓ GitHub CLI: authenticated
# ✓ Repository: githubnext/gh-aw
# ✓ Copilot token: configured

---

🟡 Confusing Areas

1. "Agentic" Jargon Without Definition

Issue: The term "agentic" appears everywhere but is never explicitly defined in simple terms.

Current explanation:

"This is 'agentic' because the AI acts as an intelligent agent with agency to make context-aware decisions"

Problem: Uses the word to define itself. A beginner doesn't know what "agency" means in this context.

Better explanation:

Agentic means the AI can think and decide for itself. Unlike traditional automation that follows exact steps you program, agentic workflows give the AI a goal and let it figure out how to achieve it—like asking a smart assistant to "organize my issues" instead of writing code for every possible scenario.

---

2. Frontmatter Term Used Without Introduction

Issue: Quick Start section "Understanding Your First Workflow" mentions "Frontmatter (between --- markers)" but assumes you know what frontmatter is.

For a noob: I might think:

• Is this a GitHub thing?
• Is this specific to this tool?
• Where did this term come from?

Quick fix: Add a tooltip or footnote:

Frontmatter is a section at the top of a markdown file (between --- markers) that contains configuration in YAML format. It's commonly used in static site generators and documentation tools.

---

3. YAML Knowledge Assumed

Issue: The workflow examples show YAML configuration but never mention "this is YAML" or link to YAML documentation.

Problem: A beginner might think:

• Why are there so many colons?
• Why does indentation matter?
• What's the difference between - item and key: value?

Recommendation: Add a "YAML Basics" callout box in Quick Start:

:::tip[YAML Basics for Beginners]
The configuration section uses YAML syntax:
- `key: value` — Sets a configuration option
- `- item` — Creates a list item
- Indentation matters (use 2 spaces, not tabs)
- Learn more: [YAML Tutorial]((redacted))
:::

---

4. "Safe Outputs" Concept Unclear

Issue: The explanation says:

"Pre-approved actions (like creating discussions, issues, or comments) that the AI can request without needing write permissions during execution."

Why this is confusing:

• What does "request" mean here?
• Why is this safer than just giving write permissions?
• What happens if the AI tries to do something not pre-approved?

Better explanation:

Safe Outputs are a security feature. Your AI agent runs with minimal permissions (read-only). When it wants to make changes (like creating an issue), it "requests" that action. A separate, permission-controlled job reviews and executes these requests. This prevents the AI from accidentally or maliciously making unauthorized changes.

---

5. CLI Command Examples Could Be More Beginner-Friendly

Issue: CLI Commands page shows a lot of commands without explaining when a beginner would actually use each one.

Example: The command reference shows:

gh aw audit (run-id)

Missing context:

• What's a run-id?
• Where do I find it?
• When would I need this command?

Recommendation: Add a "Common Workflows" section:

### Common Workflows for Beginners

**After creating a new workflow:**
```bash
gh aw compile my-workflow  # Check for errors
gh aw run my-workflow      # Test it manually
gh aw logs my-workflow     # See what happened

When something goes wrong:

gh aw status               # Check all workflows
gh aw logs my-workflow -f  # Watch live logs

---

6. Example Workflow Too Complex for "First Workflow"

Issue: The "daily team status" workflow used in Quick Start includes:

• Schedule triggers (cron syntax)
• workflow_dispatch
• Multiple permission types
• GitHub MCP tools
• Safe outputs configuration

For a beginner: This is overwhelming. I don't understand half of these settings.

Recommendation: Start with a simpler "Hello World" example:

---
on:
  workflow_dispatch:  # Manual trigger only
permissions:
  contents: read
engine: copilot
safe-outputs:
  create-discussion: {}
---

# My First Agentic Workflow

Create a discussion that says "Hello from my first agentic workflow!"

Include today's date and a fun fact about AI.

Then show the more complex daily team status example.

---

🟢 What Worked Well

1. Visual Hierarchy is Clear

The documentation uses headings, code blocks, and callouts effectively. Easy to scan and find sections.

2. Step Numbers in Quick Start

The numbered steps (Step 1, Step 2, etc.) make it clear this is a sequential process.

3. Code Examples Are Copy-Pasteable

All bash commands and YAML examples are in proper code blocks with syntax highlighting.

4. "RESEARCH PROTOTYPE" Badge

The prominent warning on the homepage sets appropriate expectations that this is experimental.

5. Links to Related Documentation

Each page includes "Related Documentation" links at the bottom, helping with navigation.

---

📋 Recommendations

Quick Wins (high impact, low effort)

1. Add Prerequisites section to Quick Start — 15 minutes
2. Add "What is GitHub Agentic Workflows?" to homepage — 10 minutes
3. Create glossary tooltips for jargon (agentic, frontmatter, YAML) — 30 minutes
4. Add verification step after token setup — 10 minutes
5. Include "Common Workflows" section in CLI Commands — 20 minutes

Medium-Term Improvements

1. Create a simpler "Hello World" first workflow — 1 hour
2. Add interactive glossary with definitions — 2 hours
3. Create troubleshooting decision tree ("Token not working? → Check these 5 things") — 2 hours
4. Add video walkthrough for Quick Start — 4 hours
5. Create "Common Mistakes" page with real examples — 3 hours

Longer-Term Enhancements

1. Interactive tutorial that validates each step — 1 week
2. "Explain this workflow" tool that breaks down complex examples — 1 week
3. Visual workflow builder for non-technical users — 2-3 weeks
4. AI-powered documentation assistant (chatbot) — 2-3 weeks

---

🎯 Priority Focus Areas

If you can only fix 3 things this week:

1. ✅ Add clear prerequisites to Quick Start (blocking issue)
2. ✅ Simplify the first workflow example (reduces cognitive load)
3. ✅ Add glossary tooltips for jargon (removes confusion points)

These three changes would dramatically improve the new user experience.

---

📊 Testing Methodology

This report was generated by:

1. Building the documentation site locally
2. Analyzing HTML content of key pages (Home, Quick Start, CLI Commands)
3. Evaluating from the perspective of a complete beginner with no prior knowledge of:
   • GitHub Actions
   • AI workflows
   • CLI extensions
   • YAML syntax
4. Identifying pain points that would slow down or block getting started

Labels: documentation, user-experience, automated-testing

AI generated by Documentation Noob Tester

[pelikhan]

/plan

[github-actions[bot]]

🎉 Yo ho ho! Plan Command found the treasure and completed successfully! ⚓💰

#6510

[github-actions[bot]]

This discussion was automatically closed because it was created by an agentic workflow more than 3 days ago.