diff --git a/.github/workflows/daily-firewall-report.lock.yml b/.github/workflows/daily-firewall-report.lock.yml index 228399be42..9fc01e2a9a 100644 --- a/.github/workflows/daily-firewall-report.lock.yml +++ b/.github/workflows/daily-firewall-report.lock.yml @@ -873,6 +873,58 @@ jobs: --- + ## 📝 Report Formatting Guidelines + + **CRITICAL**: Follow these formatting guidelines to create well-structured, readable reports: + + ### 1. Header Levels + **Use h3 (###) or lower for all headers in your report to maintain proper document hierarchy.** + + The discussion title serves as h1, so all content headers should start at h3: + - Use `###` for main sections (e.g., "### Executive Summary", "### Top Blocked Domains") + - Use `####` for subsections (e.g., "#### Blocked Domains by Workflow") + - Never use `##` (h2) or `#` (h1) in the report body + + ### 2. Progressive Disclosure + **Wrap detailed request logs and domain lists in `
Section Name` tags to improve readability and reduce scrolling.** + + Use collapsible sections for: + - Detailed request patterns and logs + - Per-workflow domain breakdowns (Section 3 below) + - Complete blocked domains list (Section 4 below) + - Verbose firewall data and statistics + + Example: + ```markdown +
+ Full Request Log Details + + [Long detailed content here...] + +
+ ``` + + ### 3. Report Structure Pattern + + Your report should follow this structure for optimal readability: + + 1. **Brief Summary** (always visible): 1-2 paragraph overview of firewall activity + 2. **Key Metrics** (always visible): Total requests, blocks, trends, block rate + 3. **Top Blocked Domains** (always visible): Top 20 most frequently blocked domains in a table + 4. **Detailed Request Patterns** (in `
` tags): Per-workflow breakdowns with domain tables + 5. **Complete Blocked Domains List** (in `
` tags): Alphabetically sorted full list + 6. **Security Recommendations** (always visible): Actionable insights and suggestions + + ### Design Principles + + Create reports that: + - **Build trust through clarity**: Most important info (metrics, top domains, recommendations) immediately visible + - **Exceed expectations**: Add helpful context, trends, and security insights + - **Create delight**: Use progressive disclosure to reduce overwhelm for detailed data + - **Maintain consistency**: Follow the same patterns as other reporting workflows + + --- + ## Objective Generate a comprehensive daily report of all rejected domains across all agentic workflows that use the firewall feature. This helps identify: @@ -1022,12 +1074,17 @@ jobs: ### Step 5: Generate Report - Create a comprehensive markdown report with the following sections: + Create a comprehensive markdown report following the formatting guidelines above. Structure your report as follows: - #### 1. Executive Summary + #### Section 1: Executive Summary (Always Visible) + A brief 1-2 paragraph overview including: - Date of report (today's date) - Total workflows analyzed (`workflow_runs_analyzed`) - - Total runs analyzed + - Total runs analyzed + - Overall firewall activity snapshot (key highlights, trends, concerns) + + #### Section 2: Key Metrics (Always Visible) + Present the core statistics: - Total network requests monitored (`firewall_requests_total`) - ✅ **Allowed** (`firewall_requests_allowed`): Count of successful requests - 🚫 **Blocked** (`firewall_requests_blocked`): Count of blocked requests @@ -1040,46 +1097,61 @@ jobs: > - A 0% block rate with listed blocked domains indicates domains that would > be blocked if accessed, but weren't actually accessed during this period - #### 2. Top Blocked Domains + #### Section 3: Top Blocked Domains (Always Visible) A table showing the most frequently blocked domains: - Domain name - Number of times blocked - Workflows that blocked it - - Example URLs (if available) + PROMPT_EOF + cat << 'PROMPT_EOF' >> "$GH_AW_PROMPT" + - Domain category (Development Services, Social Media, Analytics/Tracking, CDN, Other) Sort by frequency (most blocked first), show top 20. - #### 3. Blocked Domains by Workflow (Detailed) + #### Section 4: Detailed Request Patterns (In `
` Tags) + **IMPORTANT**: Wrap this entire section in a collapsible `
` block: + + ```markdown +
+ View Detailed Request Patterns by Workflow + For each workflow that had blocked domains, provide a detailed breakdown: - **Workflow: [workflow-name]** (X runs analyzed) + #### Workflow: [workflow-name] (X runs analyzed) + | Domain | Blocked Count | Allowed Count | Block Rate | Category | |--------|---------------|---------------|------------|----------| | example.com | 15 | 5 | 75% | Social Media | | api.example.org | 10 | 0 | 100% | Development | - For each workflow include: - - Workflow name and number of runs analyzed - - **Detailed domain table** with: - - Domain name - - Number of blocked requests to that domain - - Number of allowed requests to that domain (if any) - - Block rate percentage for that domain - - Domain category (Development Services, Social Media, Analytics/Tracking, CDN, Other) - - Total blocked requests for this workflow - - Total unique blocked domains - - Most frequently blocked domain - - **Repeat this detailed breakdown for every workflow** that had blocked domains during the analysis period. This provides visibility into which specific workflows are generating blocks and for which exact domains. - - #### 4. Complete Blocked Domains List - An alphabetically sorted list of all unique blocked domains with: - - Domain name - - Total occurrences across all workflows - - First seen date (from run timestamps) + - Total blocked requests: [count] + - Total unique blocked domains: [count] + - Most frequently blocked domain: [domain] - #### 5. Recommendations - Based on the analysis, provide: + [Repeat for all workflows with blocked domains] + +
+ ``` + + #### Section 5: Complete Blocked Domains List (In `
` Tags) + **IMPORTANT**: Wrap this entire section in a collapsible `
` block: + + ```markdown +
+ View Complete Blocked Domains List + + An alphabetically sorted list of all unique blocked domains: + + | Domain | Total Blocks | First Seen | Workflows | + |--------|--------------|------------|-----------| + | [domain] | [count] | [date] | [workflow-list] | + | ... | ... | ... | ... | + +
+ ``` + + #### Section 6: Security Recommendations (Always Visible) + Based on the analysis, provide actionable insights: - Domains that appear to be legitimate services that should be allowlisted - Potential security concerns (e.g., suspicious domains) - Suggestions for network permission improvements @@ -1090,7 +1162,13 @@ jobs: Create a new GitHub discussion with: - **Title**: "Daily Firewall Report - [Today's Date]" - **Category**: audits - - **Body**: The complete markdown report generated in Step 5 + - **Body**: The complete markdown report following the formatting guidelines and structure defined in Step 5 + + Ensure the discussion body: + - Uses h3 (###) for main section headers + - Uses h4 (####) for subsection headers + - Wraps detailed data (per-workflow breakdowns, complete domain list) in `
` tags + - Keeps critical information visible (summary, key metrics, top domains, recommendations) ## Notes diff --git a/.github/workflows/daily-firewall-report.md b/.github/workflows/daily-firewall-report.md index f49ba1d136..7c99ffd9bf 100644 --- a/.github/workflows/daily-firewall-report.md +++ b/.github/workflows/daily-firewall-report.md @@ -144,6 +144,58 @@ If insufficient data is available (less than 7 days): --- +## 📝 Report Formatting Guidelines + +**CRITICAL**: Follow these formatting guidelines to create well-structured, readable reports: + +### 1. Header Levels +**Use h3 (###) or lower for all headers in your report to maintain proper document hierarchy.** + +The discussion title serves as h1, so all content headers should start at h3: +- Use `###` for main sections (e.g., "### Executive Summary", "### Top Blocked Domains") +- Use `####` for subsections (e.g., "#### Blocked Domains by Workflow") +- Never use `##` (h2) or `#` (h1) in the report body + +### 2. Progressive Disclosure +**Wrap detailed request logs and domain lists in `
Section Name` tags to improve readability and reduce scrolling.** + +Use collapsible sections for: +- Detailed request patterns and logs +- Per-workflow domain breakdowns (Section 3 below) +- Complete blocked domains list (Section 4 below) +- Verbose firewall data and statistics + +Example: +```markdown +
+Full Request Log Details + +[Long detailed content here...] + +
+``` + +### 3. Report Structure Pattern + +Your report should follow this structure for optimal readability: + +1. **Brief Summary** (always visible): 1-2 paragraph overview of firewall activity +2. **Key Metrics** (always visible): Total requests, blocks, trends, block rate +3. **Top Blocked Domains** (always visible): Top 20 most frequently blocked domains in a table +4. **Detailed Request Patterns** (in `
` tags): Per-workflow breakdowns with domain tables +5. **Complete Blocked Domains List** (in `
` tags): Alphabetically sorted full list +6. **Security Recommendations** (always visible): Actionable insights and suggestions + +### Design Principles + +Create reports that: +- **Build trust through clarity**: Most important info (metrics, top domains, recommendations) immediately visible +- **Exceed expectations**: Add helpful context, trends, and security insights +- **Create delight**: Use progressive disclosure to reduce overwhelm for detailed data +- **Maintain consistency**: Follow the same patterns as other reporting workflows + +--- + ## Objective Generate a comprehensive daily report of all rejected domains across all agentic workflows that use the firewall feature. This helps identify: @@ -293,12 +345,17 @@ Combine data from all workflows (using standardized metric names): ### Step 5: Generate Report -Create a comprehensive markdown report with the following sections: +Create a comprehensive markdown report following the formatting guidelines above. Structure your report as follows: -#### 1. Executive Summary +#### Section 1: Executive Summary (Always Visible) +A brief 1-2 paragraph overview including: - Date of report (today's date) - Total workflows analyzed (`workflow_runs_analyzed`) -- Total runs analyzed +- Total runs analyzed +- Overall firewall activity snapshot (key highlights, trends, concerns) + +#### Section 2: Key Metrics (Always Visible) +Present the core statistics: - Total network requests monitored (`firewall_requests_total`) - ✅ **Allowed** (`firewall_requests_allowed`): Count of successful requests - 🚫 **Blocked** (`firewall_requests_blocked`): Count of blocked requests @@ -311,46 +368,59 @@ Create a comprehensive markdown report with the following sections: > - A 0% block rate with listed blocked domains indicates domains that would > be blocked if accessed, but weren't actually accessed during this period -#### 2. Top Blocked Domains +#### Section 3: Top Blocked Domains (Always Visible) A table showing the most frequently blocked domains: - Domain name - Number of times blocked - Workflows that blocked it -- Example URLs (if available) +- Domain category (Development Services, Social Media, Analytics/Tracking, CDN, Other) Sort by frequency (most blocked first), show top 20. -#### 3. Blocked Domains by Workflow (Detailed) +#### Section 4: Detailed Request Patterns (In `
` Tags) +**IMPORTANT**: Wrap this entire section in a collapsible `
` block: + +```markdown +
+View Detailed Request Patterns by Workflow + For each workflow that had blocked domains, provide a detailed breakdown: -**Workflow: [workflow-name]** (X runs analyzed) +#### Workflow: [workflow-name] (X runs analyzed) + | Domain | Blocked Count | Allowed Count | Block Rate | Category | |--------|---------------|---------------|------------|----------| | example.com | 15 | 5 | 75% | Social Media | | api.example.org | 10 | 0 | 100% | Development | -For each workflow include: -- Workflow name and number of runs analyzed -- **Detailed domain table** with: - - Domain name - - Number of blocked requests to that domain - - Number of allowed requests to that domain (if any) - - Block rate percentage for that domain - - Domain category (Development Services, Social Media, Analytics/Tracking, CDN, Other) -- Total blocked requests for this workflow -- Total unique blocked domains -- Most frequently blocked domain - -**Repeat this detailed breakdown for every workflow** that had blocked domains during the analysis period. This provides visibility into which specific workflows are generating blocks and for which exact domains. - -#### 4. Complete Blocked Domains List -An alphabetically sorted list of all unique blocked domains with: -- Domain name -- Total occurrences across all workflows -- First seen date (from run timestamps) +- Total blocked requests: [count] +- Total unique blocked domains: [count] +- Most frequently blocked domain: [domain] + +[Repeat for all workflows with blocked domains] + +
+``` -#### 5. Recommendations -Based on the analysis, provide: +#### Section 5: Complete Blocked Domains List (In `
` Tags) +**IMPORTANT**: Wrap this entire section in a collapsible `
` block: + +```markdown +
+View Complete Blocked Domains List + +An alphabetically sorted list of all unique blocked domains: + +| Domain | Total Blocks | First Seen | Workflows | +|--------|--------------|------------|-----------| +| [domain] | [count] | [date] | [workflow-list] | +| ... | ... | ... | ... | + +
+``` + +#### Section 6: Security Recommendations (Always Visible) +Based on the analysis, provide actionable insights: - Domains that appear to be legitimate services that should be allowlisted - Potential security concerns (e.g., suspicious domains) - Suggestions for network permission improvements @@ -361,7 +431,13 @@ Based on the analysis, provide: Create a new GitHub discussion with: - **Title**: "Daily Firewall Report - [Today's Date]" - **Category**: audits -- **Body**: The complete markdown report generated in Step 5 +- **Body**: The complete markdown report following the formatting guidelines and structure defined in Step 5 + +Ensure the discussion body: +- Uses h3 (###) for main section headers +- Uses h4 (####) for subsection headers +- Wraps detailed data (per-workflow breakdowns, complete domain list) in `
` tags +- Keeps critical information visible (summary, key metrics, top domains, recommendations) ## Notes