Generate Implementation Dashboard

[laurentsenta]

Base Bricks

• feat: add version flag #125
• feat: output workflow url with test metadata #145 | 1 day
• feat: test implementations with reusable jsons #146
  • Update workflows to expose their JSON outputs & retrieve them from gateway-conformance | 1 day
  • Try in experiment repositories
  • Implement the artifact-retrieval script
  • Update client repositories
    • ci: upload conformance logs kubo#10092
    • ci: upload conformance logs boxo#442
    • ci: upload conformance logs ipfs-inactive/bifrost-gateway#195

Phase 1: Example output

Generate Dashboard - Base Structure

We'll generate a dashboard in gh-pages, under a sub-path like project.io/dashboard.

• Define the Dashboard sitemap | 1 day
• feat: generate placeholder dashboard & deploy to github pages #150
  • Output to gh-pages and setup a single URL | 1 day
  • Generate the dashboard daily using outputs from other repos | 1 day
• feat: generate the Dashboard - base structure #152 | 3 days
  • /(index)
  • /dashboard
    • render gateway version
    • render implementation
    • render results
• feat: add user-oriented taxonomy #156 to be used in gateway checker | 1 day

Generate Dashboard - Drill Down

We'll let the user drill down test results, and link to test outputs / github action results

• Create pages to drill-down tests results | 2 day
  • per version
  • per implementation
  • per test / sub test / etc
• Render the execution logs in tests | 1 day
• fix: escape names that might collide with other operations #166 (comment) | 1 day

Generate Dashboard - Specs Relationships

• feat: add specs and user taxonomies #159
• feat: with specs dashboard #163
  • Add https:// to the specs URL + add Specs field for multiple specs
  • Create spec -> tests & tests -> spec indexes | 2 days

Implementation Dashboard - Going Public | 5 days

Generate Dashboard - 💅

• Export Hints in the test outputs as well
• If possible, add links to the test definition sources (it should be possible to extract the test line number and file, etc and create permalink to github's conformance source)
• Nicer Design | 1 day
• Edit Content | 2 day

Follow Ups

• fix the redirects for /tests and /results (not critical if we host at the root)
• Deal with matrix in client jobs (like https://github.com/ipfs/bifrost-gateway/actions/runs/5925256845)
• Validate specs (check the page and anchors exists)
• Generate a spec coverage table (detect how many specs pages + anchors we have and how much we test)

Related Discussions

#116
#116 (review)

[BigLep]

Awesome project breakdown @laurentsenta !

[laurentsenta]

Updated with notes from @BigLep review's and discussion with @lidel + reorganized to support ipfs/public-gateway-checker#442 more quickly.