📚 Documentation Noob Test Report - February 2, 2026

[github-actions[bot]]

I tested the gh-aw documentation as a complete beginner attempting to get started with GitHub Agentic Workflows for the first time. This report identifies critical issues, confusing areas, and recommendations for improving the new user experience.

Test Methodology

• Date: February 2, 2026
• Documentation Version: Built from main branch
• Approach: Analyzed documentation structure and content as a complete beginner
• Focus: Getting started experience, first-time user journey

Pages Analyzed

1. Home Page (/gh-aw/)
2. Quick Start (/gh-aw/setup/quick-start/)
3. CLI Commands (/gh-aw/setup/cli/)
4. Agentic Authoring (/gh-aw/setup/agentic-authoring/)
5. Examples Section (multiple pages)

---

🔴 Critical Issues (Blockers for New Users)

1. Missing Prerequisites Section

Location: Quick Start page

Issue: The documentation doesn't clearly state upfront what users need before starting:

• Do I need Go installed?
• Do I need the GitHub CLI (gh) installed?
• What version of GitHub CLI is required?
• Do I need GitHub Actions enabled on my repository?

Impact: Users will hit errors immediately and may not understand why.

Recommendation: Add a prominent "Prerequisites" section at the top of Quick Start:

## Prerequisites

Before you begin, ensure you have:
- [ ] GitHub CLI (`gh`) version 2.0 or higher installed
- [ ] A GitHub repository with Actions enabled
- [ ] Basic familiarity with Git and GitHub
- [ ] (Optional) Docker for advanced features

2. "gh aw" vs "gh-aw" Confusion

Location: Throughout documentation

Issue: The tool is sometimes called gh-aw (with hyphen) and sometimes gh aw (space). As a beginner, I don't know:

• Is this a GitHub CLI extension?
• Do I type gh-aw or gh aw?
• Are these two different tools?

Impact: Command execution failures, confusion about installation.

Recommendation:

• Add a callout explaining: "gh-aw is a GitHub CLI extension. You'll run it as gh aw (command)"
• Be consistent: use gh aw for commands, gh-aw only when referring to the extension name

3. No "What Happens Next" After Installation

Location: Quick Start, CLI pages

Issue: After installation instructions, there's no clear "Now what?" section. I don't know:

• How do I verify installation worked?
• What's my first command to try?
• What should I expect to see?

Impact: Users successfully install but then get stuck.

Recommendation: Add a "Verify Installation" section:

## Verify Installation

Run the following to confirm everything is working:

```bash
gh aw --version
```

You should see output like:
```
gh-aw version 0.x.x
```

Next, try listing your workflows:
```bash
gh aw list
```

---

🟡 Confusing Areas (Slow Down Learning)

1. "Agentic" Terminology Not Explained

Location: Home page, throughout

Issue: The term "agentic workflows" is used extensively without explanation. As a beginner:

• What does "agentic" mean?
• How is this different from regular GitHub Actions?
• Why would I use this vs writing YAML directly?

Impact: Users don't understand the value proposition.

Recommendation: Add a glossary tooltip or sidebar definition when "agentic" first appears. Consider a "Why gh-aw?" section explaining benefits over plain GitHub Actions.

2. Frontmatter vs YAML Confusion

Location: Agentic Authoring page

Issue: The documentation shows frontmatter configuration but doesn't explain:

• Is this YAML syntax?
• How is this different from GitHub Actions YAML?
• Why use markdown with frontmatter instead of pure YAML?

Impact: Users with GitHub Actions experience may be confused about why there's a different format.

Recommendation: Add a comparison section showing the difference between traditional GitHub Actions and Agentic Workflows.

3. No Error Message Examples

Location: Troubleshooting sections

Issue: When commands fail, beginners don't know:

• What error messages are normal vs serious?
• How to interpret compiler errors?
• Where to find logs?

Impact: Users give up when hitting first error.

Recommendation: Add a "Common Error Messages" section with real examples and solutions.

4. Examples Without Context

Location: Examples section

Issue: Examples show code but don't explain:

• When would I use this pattern?
• What problem does this solve?
• What are the prerequisites for this example?

Impact: Users copy-paste without understanding, can't adapt to their needs.

Recommendation: Each example should have:

• Use Case: "Use this when you need to..."
• Prerequisites: "Requires: GitHub token, Docker, etc."
• Expected Outcome: "After running, you should see..."

5. No Local Testing Guidance

Location: Throughout documentation

Issue: Beginners don't know:

• Can I test workflows locally before pushing?
• How do I debug workflow issues?
• Is there a "dry run" mode?

Impact: Users push broken workflows, wait for Actions to run, get frustrated by cycle time.

Recommendation: Add "Local Development" section explaining validation and testing options.

---

🟢 What Works Well

1. Clean Visual Design: The Starlight template makes the docs easy to read
2. Code Syntax Highlighting: Examples are well-formatted and readable
3. Navigation Structure: Sidebar organization is logical (Setup → Examples → Reference)
4. Search Functionality: Pagefind search appears to be implemented

---

Recommendations (Prioritized)

Quick Wins (Implement Immediately)

1. ✅ Add Prerequisites section to Quick Start
2. ✅ Add "Verify Installation" section with expected output
3. ✅ Add gh-aw vs "gh aw" callout explaining it's a CLI extension
4. ✅ Add glossary definition for "agentic"

Medium Priority (Next Week)

5. 📝 Add "Common Errors" section with real error messages
6. 📝 Add "Why gh-aw?" comparison vs GitHub Actions
7. 📝 Enhance examples with use cases and prerequisites
8. 📝 Add local testing/debugging guidance

Long-term Improvements (Next Month)

9. 🔄 Create interactive tutorial or wizard
10. 🔄 Add video walkthrough for visual learners
11. 🔄 Create troubleshooting flowchart
12. 🔄 Add beginner vs advanced documentation paths

---

Conclusion

The gh-aw documentation is well-structured and visually appealing, but lacks critical context for complete beginners. The main issues are:

1. Missing prerequisites - users will fail immediately without knowing why
2. Unclear value proposition - "agentic" terminology is never defined
3. No verification steps - users don't know if installation worked
4. Insufficient error handling guidance - users give up when hitting first error

Bottom line for new users:

"I can see the documentation exists and looks professional, but I'm not confident I could successfully install and run my first workflow without getting stuck. I'd need to search external resources or ask for help."

Recommended first step: Focus on the "Quick Wins" above to unblock the most critical user journey (installation → first workflow).

AI generated by Documentation Noob Tester

• expires on Feb 9, 2026, 7:55 AM UTC