Add agentic workflow to test large payload storage and agent access

[Copilot]

Tests the jqschema middleware's ability to intercept large MCP responses (>500 chars), store them to disk at /tmp/jq-payloads/{sessionID}/{queryID}/payload.json, and enable agent retrieval via mounted volumes.

Test Protocol

Secret-based verification:

1. Setup generates UUID secret, embeds in large JSON file (~500KB)
2. Agent reads file via filesystem MCP → triggers gateway interception
3. Gateway stores payload, returns metadata with payloadPath, schema, preview, truncated
4. Agent translates path (/tmp/jq-payloads → /workspace/mcp-payloads) and reads payload
5. Agent extracts secret from payload and validates match

Test File Details

• Size: ~500KB (2000 array items + 400KB padding)
• Structure: JSON with embedded secret, large array of items each referencing the secret, and padding data
• Generation: Uses jq for proper JSON generation ensuring no shell injection risks

Volume Mounts

/tmp/mcp-test-fs → /workspace/test-data:ro         # test files
/tmp/jq-payloads → /workspace/mcp-payloads:ro      # agent reads payloads
/tmp/jq-payloads → /tmp/jq-payloads:rw             # gateway writes payloads

Files

• .github/workflows/large-payload-tester.md - workflow definition, bash setup
• .github/agentics/large-payload-tester.md - agent test protocol
• .github/workflows/large-payload-tester-README.md - architecture documentation

Security

• Filesystem MCP restricted to /workspace (not /workspace,/tmp)
• Agent mounts read-only
• Session isolation via directory structure
• Proper JSON generation with jq (no shell injection)

Scheduling

Runs daily via scattered schedule. Manual trigger via workflow_dispatch.

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

[lpcox]

@copilot make the programmatically created large file even larger, like 500kb

[Copilot]

@copilot make the programmatically created large file even larger, like 500kb

Done. Increased test file to ~500KB (2000 items + 400KB padding). Changes in commit 3528e12.

[Copilot]

Pull request overview

This PR adds an agentic workflow to test the MCP Gateway's jqschema middleware, which intercepts MCP tool responses, stores them to disk, and returns metadata to enable agent access to large payloads. The workflow uses a secret-based verification approach where a UUID is embedded in a ~500KB test file, then the agent reads the file through the filesystem MCP server, retrieves the payload via gateway metadata, and verifies the secret matches.

Changes:

• Adds .github/workflows/large-payload-tester.md with workflow definition, bash setup script, and volume mount configuration
• Adds .github/agentics/large-payload-tester.md with agent test protocol and instructions
• Adds .github/workflows/large-payload-tester-README.md documenting the test architecture and implementation details
• Generates .github/workflows/large-payload-tester.lock.yml compiled GitHub Actions workflow

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 5 comments.

File	Description
.github/workflows/large-payload-tester.md	Workflow definition with frontmatter, bash setup to generate test files with embedded secrets, and volume mount configuration for test data and payload access
.github/workflows/large-payload-tester.lock.yml	Compiled GitHub Actions workflow with agent, detection, safe_outputs, and conclusion jobs
.github/workflows/large-payload-tester-README.md	Architecture documentation explaining test protocol, volume mounts, path translation, and expected behavior
.github/agentics/large-payload-tester.md	Agent prompt with step-by-step test instructions for reading secrets, triggering large payloads, accessing stored files, and reporting results

Comments suppressed due to low confidence (1)

.github/workflows/large-payload-tester-README.md:6

• The description states the middleware intercepts "large responses (>500 chars)" but this is misleading. Based on the actual implementation in internal/middleware/jqschema.go (lines 130-232), the middleware is applied to ALL successful tool responses regardless of size. The 500 character threshold only determines whether the preview is truncated, not whether the middleware is applied.

All responses are:

1. Stored to disk (regardless of size)
2. Transformed to include metadata (payloadPath, schema, preview, etc.)
3. Have their preview truncated to 500 chars IF they exceed that length

The description should clarify that the middleware applies to all responses, not just large ones, and that 500 chars is the truncation threshold for the preview field.

1. **Payload Storage**: Verifies that large responses (>500 chars) are automatically stored to disk

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Line 120 describes step 2 as "Trigger large payload response (>1KB)" but this is inconsistent with the actual test design. The test file is ~500KB (not 1KB), and the middleware applies to all responses regardless of size. The ">1KB" threshold is not relevant to the actual implementation or test behavior.

This should either be removed or changed to reflect that the test file is ~500KB and the middleware processes all responses (with preview truncation at 500 characters).

Suggested change

	2. ✅/❌ Trigger large payload response (>1KB)
	2. ✅/❌ Trigger large payload response (~500KB; middleware processes all responses, preview truncated to 500 chars)

[Copilot]

The comment states "Middleware that intercepts large responses" but this is inaccurate. Based on the implementation in internal/middleware/jqschema.go, the middleware intercepts and processes ALL successful tool responses regardless of size. It always stores payloads to disk and returns metadata. The 500 character threshold only determines whether the preview is truncated, not whether the middleware is applied.

The comment should be clarified to say "Middleware that intercepts all responses" or "Middleware that processes tool responses and stores payloads".

Suggested change

	- `internal/middleware/jqschema.go` - Middleware that intercepts large responses
	- `internal/middleware/jqschema.go` - Middleware that processes tool responses and stores payloads

[Copilot]

The comment says "Generating 2000 items + 400KB padding to create ~500KB file" but this calculation is imprecise. The 400KB padding alone is 400,000 bytes, and 2000 array items with the structure shown would add approximately 100-300KB more, resulting in a file that's likely 500-700KB in size, not exactly ~500KB.

While the validation at line 101 correctly checks for at least 512000 bytes, the comment could be more accurate. Consider updating to "Generating 2000 items + 400KB padding to create ~500-700KB file" or just "Generating 2000 items + 400KB padding to create a large test file (target >500KB)".

Suggested change

	# Generating 2000 items + 400KB padding to create ~500KB file
	# Generating 2000 items + 400KB padding to create a large test file (target >500KB)

[Copilot]

The path in line 93 contains an extra "session/" component that doesn't match the actual payload storage structure. According to lines 88-91 and the rest of the documentation, the path structure is /tmp/jq-payloads/{sessionID}/{queryID}/payload.json where {sessionID} is the actual session identifier. Line 93 incorrectly shows /workspace/mcp-payloads/session/{queryID}/payload.json which adds an extra "session" directory that doesn't exist.

The path should be: /workspace/mcp-payloads/{sessionID}/{queryID}/payload.json (where {sessionID} is replaced with the actual session ID value)

Suggested change

	4. Agent reads payload from mounted path: `/workspace/mcp-payloads/session/{queryID}/payload.json`
	4. Agent reads payload from mounted path: `/workspace/mcp-payloads/{sessionID}/{queryID}/payload.json`

[Copilot]

The description states the secret file "contains ONLY the secret UUID" (line 28), but the actual implementation in the workflow file shows the secret is prefixed with "test-secret-" (lines 54, 56 in large-payload-tester.md). For example, the actual format is "test-secret-{uuid}" not just "{uuid}".

The example format "abc123-def456-ghi789" in line 28 also doesn't match the actual UUID format which would be something like "test-secret-550e8400-e29b-41d4-a716-446655440000" (if using uuidgen) or "test-secret-1708963200123456789-12345" (if using timestamp fallback).

This discrepancy could confuse the agent during testing since it needs to know the exact format to properly extract and compare the secret.