📚 Documentation Noob Test Report - January 31, 2026

[github-actions[bot]]

Documentation Noob Testing Report

Date of Test: January 31, 2026
Testing Method: Manual source file analysis (Playwright browser unavailable)
Pages Analyzed:

• Home page (/)
• Quick Start guide (/setup/quick-start/)
• CLI Commands (/setup/cli/)
• Agentic Authoring guide (/setup/agentic-authoring/)

Overall Impression: The documentation shows strong potential but has several critical pain points that would confuse a complete beginner. The "Quick Start" guide assumes significant technical knowledge and jumps into actions before explaining core concepts.

---

🔴 Critical Issues (Block Getting Started)

1. "Quick Start" isn't actually quick for beginners

Location: /setup/quick-start/

Issue: The guide title says "Quick Start" but Step 2 immediately asks users to run gh aw add githubnext/agentics/daily-repo-status without explaining:

• What "agentics" is
• Why we're pulling from a different repository
• What the workflow actually does before installing it
• How to verify it worked

Impact: A beginner would think "Wait, what's githubnext/agentics? Is that a typo? Why isn't this in the gh-aw documentation?"

Recommendation:

• Add a "What You'll Build" section at the top explaining the workflow
• Include a "What happens when I run this?" explanation before Step 2
• Add verification steps after Step 2 ("You'll know it worked when...")

2. Missing "What is an agentic workflow?" on the home page

Location: / (home page)

Issue: The home page jumps straight to "How they work" without first explaining what they are in beginner-friendly terms. The current explanation uses jargon: "repository-level AI agent descriptions in plain markdown."

Impact: A complete beginner wouldn't understand "repository-level AI agent descriptions." They'd need to understand:

• What's a workflow vs. a workflow file vs. GitHub Actions
• What makes something "agentic" vs. a regular GitHub Action
• Why markdown matters

Recommendation:

• Add a "New to GitHub Agentic Workflows?" callout box at the top with a 2-sentence explanation in plain language
• Example: "GitHub Agentic Workflows let you write automation tasks in plain English (like 'create a daily status report') instead of complex code. An AI agent reads your instructions and does the work automatically."

3. Prerequisites assume knowledge of GitHub Actions

Location: /setup/quick-start/ - Prerequisites section

Issue: Lists "GitHub Actions enabled in your repository" but doesn't explain:

• How to check if it's enabled
• How to enable it if it's not
• What happens if you skip this step

Impact: Beginners might:

• Not know how to verify GitHub Actions is enabled
• Skip this step and face confusing errors later
• Not understand why the workflow isn't running

Recommendation:

• Add expandable details: "How to check if GitHub Actions is enabled"
• Link to GitHub's official documentation
• Add troubleshooting note: "If your workflow doesn't run, check that Actions are enabled in Settings > Actions"

4. No explanation of .md vs .lock.yml files

Location: Appears in Quick Start Step 2 and CLI guide

Issue: The guide mentions editing .github/workflows/daily-repo-status.md and running compile to create .lock.yml but never explains:

• Why there are two files
• What .lock.yml is or does
• Whether users should edit .lock.yml (they shouldn't)
• What happens if the files get out of sync

Impact: Beginners would wonder:

• "Do I need both files?"
• "Which one should I edit?"
• "Can I delete one of them?"
• "What if I accidentally edit .lock.yml?"

Recommendation:

• Add a callout box explaining the two-file system early in Quick Start
• Example: "You write in .md (markdown), gh-aw compiles to .lock.yml (YAML). Only edit .md files."

---

🟡 Confusing Areas (Slow Down Learning)

5. "Compile" is confusing terminology for beginners

Location: Quick Start, CLI guide, Agentic Authoring guide

Issue: The term "compile" is used throughout without explanation. For non-developers, "compile" might sound intimidating or unclear.

Impact: Beginners might think:

• "I thought this was plain English, why do I need to compile?"
• "What does compile mean in this context?"
• "Is this going to break something?"

Recommendation:

• First time "compile" appears, add: "compile (convert your markdown instructions into GitHub Actions format)"
• Consider using friendlier language: "translate" or "convert"

6. Unclear what "safe outputs" means

Location: Home page - "Security Built-In" section

Issue: Mentions "sanitized safe outputs" and links to glossary, but doesn't explain in context what it means or why it matters.

Impact: Beginners would read "safe outputs" and think:

• "Outputs of what?"
• "Safe from what?"
• "Do I need to configure this?"

Recommendation:

• Add parenthetical explanation: "safe outputs (pre-approved actions like creating issues or comments)"
• Add a beginner-friendly example: "For example, the AI can create an issue, but it can't delete your repository"

7. Too many engine options without guidance

Location: Quick Start Step 2, Prerequisites section

Issue: Lists three AI engine options (Copilot, Claude, Codex) but doesn't help users choose:

• Which is fastest/cheapest/most powerful?
• Which should beginners start with?
• Can they switch later?

Impact: Analysis paralysis - beginners will stop and research each option instead of continuing the tutorial.

Recommendation:

• Add a recommendation: "🎯 Recommended for beginners: Start with GitHub Copilot if you have a subscription"
• Add comparison table or link to comparison
• Note: "You can change engines later"

8. "Agentic" terminology never defined

Location: Throughout documentation

Issue: The word "agentic" appears everywhere but is never explicitly defined for beginners.

Impact: Beginners would wonder:

• "What does 'agentic' even mean?"
• "Is it related to 'agent'?"
• "Why not just call them 'AI workflows'?"

Recommendation:

• Add a glossary term or first-use definition
• Example: "Agentic workflows are workflows where an AI agent makes intelligent decisions based on context, rather than following fixed rules."

9. Missing "What can go wrong?" section

Location: Quick Start guide

Issue: No troubleshooting section for common beginner mistakes:

• Workflow doesn't run
• Permission errors
• API key issues
• Compilation errors

Impact: When things go wrong (and they will), beginners have nowhere to look and will give up.

Recommendation:

• Add "Common Issues" section at bottom of Quick Start
• Include symptoms and fixes for top 3-5 problems
• Link to full troubleshooting guide

10. CLI commands page is overwhelming

Location: /setup/cli/

Issue: The page starts with "🚀 Most Common Commands" but lists 6 commands. The full reference below is very long.

Impact: Beginners will feel overwhelmed by the number of commands and options.

Recommendation:

• Reduce "Most Common" to 3 commands: init, add, run
• Add progressive disclosure: Beginner section → Intermediate section → Advanced section
• Add "You only need to learn 3 commands today" message

---

🟢 What Worked Well

11. Clear step numbering in Quick Start

The numbered steps (Step 1, Step 2) provide clear progression and make it easy to follow along.

12. Code examples are copy-pasteable

All bash commands have the wrap class for easy copying, which is excellent UX.

13. "What's Next?" section provides clear next steps

At the end of Quick Start, the "What's next?" section gives good direction for continued learning.

14. TIP callouts are helpful

The tips for Codespaces and engine selection provide useful context without overwhelming the main flow.

15. Home page hero is clear and action-oriented

The "Getting Started" button is prominent and the tagline clearly states what the tool does.

---

📋 Recommendations Summary

Quick Wins (Can implement immediately):

1. Add "What You'll Build" section to Quick Start (5 minutes)
2. Define "agentic" on first use on home page (2 minutes)
3. Add ".md vs .lock.yml" callout to Quick Start (10 minutes)
4. Add engine recommendation to Quick Start prerequisites (5 minutes)
5. Add "compile" definition on first use (2 minutes)

Short-term Improvements (1-2 hours):

6. Create "Common Issues" section in Quick Start
7. Add "How to check GitHub Actions is enabled" expandable details
8. Reorganize CLI page into Beginner/Intermediate/Advanced sections
9. Add verification steps after Step 2 of Quick Start
10. Define "safe outputs" with example in home page security section

Longer-term Improvements (Half-day projects):

11. Create a "Concepts" page explaining core terminology (workflow, agentic, compile, etc.)
12. Add interactive workflow builder or wizard for common use cases
13. Create video walkthrough of Quick Start for visual learners
14. Add FAQ section to Quick Start addressing "Why two files?", "Which engine?", etc.

---

Methodology Note

Testing Constraints: Due to network isolation preventing Playwright browser access, this analysis was conducted through direct source file review of:

• /docs/src/content/docs/index.mdx (home page)
• /docs/src/content/docs/setup/quick-start.md
• /docs/src/content/docs/setup/cli.md
• /docs/src/content/docs/setup/agentic-authoring.mdx

A future test with live browser access would allow:

• Testing navigation flow and link accessibility
• Evaluating page load times and visual layout
• Verifying search functionality
• Testing on mobile devices
• Capturing actual screenshots of confusing sections

---

Labels: documentation, user-experience, automated-testing

Priority: These issues significantly impact the new user experience. Addressing the "Quick Wins" would immediately improve onboarding success rates.

AI generated by Documentation Noob Tester

• expires on Feb 7, 2026, 7:46 AM UTC