[Tracking] w3up phase 3

[heyjay44]

DAG House – w3up MVP for March 17

Milestones:

1. w3up demo series for PL
2. associated blog posts highlighting the latest and greatest
3. ability to start user migration

🔴 P0 🟡 P1 🟢 P2

🔴 w3access + w3provider + w3session

• w3provider (MVP free tier provider)

  • w3protocol#347 Email template improvements - @travis
    ^ MOVED here: Tracking w3up work 🆙 #650 Finish implementing
  • w3protocol#348 GET -> POST on email validation links - @travis
  • w3protocol#341 Postmark daghouse domain move to web3.storage - @travis

• Finish implementing w3access

  • (Sharing delegations across agents/devices)
  • w3protocol#393 access/delegate - @gobengo
  • w3protocol#394 access/claim - @gobengo
  • w3protocol#395 (w3access & w3session in client) - @Gozala, @alanshaw, @travis, @gobengo

• Finish implementing w3session @Gozala

  • (Verification via email and did:mailto usage)feat!: update session API ucanto#227
  • update session spec to reflect 227 update session spec to reflect recent changes ucanto#228
  • update ucanto
  • w3protocol#293 Fix the delegation over websocket slowness - @travis

• w3protocol#384 Finish single URL for web3.storage Ucanto - @gobengo

• access-client Agent.url defaults to HOST even when custom connection is used #344 (blocking w3ui testing)

• Upgrade upload-api to ucanto5, make sure space verification works still @Gozala

• Make space verification better, space/info returns provider info @gobengo

• Propagate access/account changes to access.agent -> w3up-client -> w3cli and w3ui and w3console

• create separate email verification flows for web3.storage and nft.storage #520
  ^ MOVED here: Tracking w3up work 🆙 #650

• accounts/email for account recovery

🔴 Turn off clusters

• backup#1 Migrate data
  • make (new) list of CIDs to migrate for web3.storage – @alanshaw
  • complete web3.storage cluster migration
    ^ moved here: Tracking w3up work 🆙 #650
  • complete nft.storage cluster migration
    ^ moved here: Tracking w3up work 🆙 #650
• pickup#68 Rollout pickup
  • migrate web3.storage pinning services to pickup (current: 100% traffic) - @olizilla
  • migrate nft.storage pinning services to pickup (current: 100% traffic) - @olizilla
• Inform netops when safe to decommission
  ^ moved here: Tracking w3up work 🆙 #650

🟡 w3s.link + nftstorage.link gateways

• decrease R2 operations - @vasco-santos web3-storage/freeway#24, nftstorage/nft.storage#2331, web3-storage/web3.storage#2227
• remove upstream CF gateway (save $) - @vasco-santos feat: add configurable redirect reads#142
• remove upstream pinata gateway (save $) - @vasco-santos feat: add configurable redirect reads#142
• redirect back to dweb.link for content not available in freeway^ - @vasco-santos feat: add configurable redirect reads#142
• Q: Can we turn on gzip and save some bandwidth? - @olizilla + @vasco-santos
  ^ moved here: Tracking w3up work 🆙 #650

Notes

This would reduce proxied traffic significantly... CF is charging us for egress of content we're not storing.

🟡 Elastic IPFS costs

• Verify blake2b fix is cause of increased egress - @alanshaw
• Research solutions for reducing costs - @vasco-santos
  • document low hanging fruit and implement changes to reduce costs
    • fix: stop writing to blocks v1 table elastic-ipfs/indexer-lambda#72
• Reduce write operations to dynamoDB - @vasco-santos fix: stop writing to blocks v1 table elastic-ipfs/indexer-lambda#72
• Implement bitswap cancel handling - @vasco-santos feat: handle cancel messages elastic-ipfs/bitswap-peer#193

Notes

Seeing sustained increase since jan

• egress costs
• dyanmo costs from indexing per block

🟡 w3up Metrics

• Kinesis consumers for tracking UCAN Log stream consumers w3infra#117 - @vasco-santos:

• User facing aggregations
  • Space upload/add counter - feat: ucan stream consumer count uploads in space w3infra#118
  • Space store/add counter - feat: ucan stream consumer space store add and remove count w3infra#160
  • Space store/remove counter - feat: ucan stream consumer space store add and remove count w3infra#160
  • Space store/add size sum - feat: ucan stream consumer space size track w3infra#164
  • Space store/remove size sum - feat: ucan stream consumer space size track w3infra#164
  • Space upload/remove counter - feat: ucan stream consumer space upload remove count w3infra#163
• System wide metrics
  • Total storage committed => store/add size feat: ucan stream consumer w3 accum store/add size w3infra#146
  • Total number of CARs stored => store/add count feat: ucan stream consumer store/add and store/remove count w3infra#151
  • Total number of CARs removed => store/remove count feat: ucan stream consumer store/add and store/remove count w3infra#151
  • Total number of uploads removed => upload/remove count feat: ucan stream consumer upload remove w3infra#157
  • Total number of uploads => upload/add count feat: ucan stream consumer system upload add count w3infra#162
  • Total storage uncommited => store/remove + receipt from system ? feat: ucan stream consumer store remove size w3infra#158
• Metrics visualization
  • HTTP Endpoint feat: metrics endpoint w3infra#156
  • Prometheus feat: add w3up prod scraping web3-storage/telemetry-grafana-agent#3
  • Grafana dashboard

• UCAN LOG Stream Kinesis gateway for other services (access-api, ...) - @vasco-santos
• integrate Redshift as Kinesis consumer - @vasco-santos
• Kinesis to emit events when there are 32 GB deals (as a placeholder, for now, call a function that does nothing)

Once above is completed, then tackle:

• Integrate Prometheus like API + Grafana - @vasco-santos Add metrics endpoint for system metrics in prometheus format w3infra#147
  • decide what Grafana instance to use
  • usage over time??
    ^ MOVED here: Tracking w3up work 🆙 #650
• Figure out how to extract NFT.Storage specific metrics
  ^ MOVED here: Tracking w3up work 🆙 #650

Notes

Docs:

• https://www.notion.so/DS9-Q1-metrics-goals-41b617591d8e4f7fb5bae94f655d6c82
• 2023-02-02 meeting notes
• Draft of product requirements

🟡 Specifications

• Tweaks on specification for store/upload protocol @Gozala
  • Reads like blog/how-to - needs more formal
• Specify expected invocation responses

🟢 w3ui/console

• as a w3console user, I can upload data and view my uploads #494
• as a w3console user, I can login with an email address #487
• identify bugs/quirks in current iteration of console
• w3ui#142 w3console design - @plwileverts
  • Update: we are reducing the scope of work by removing the customisable components for now. This makes the initial design
    easier and overall maintenance less burdensome.
  • requirements here
• w3ui#378 upload progress reporting (fix upload-client, w3up-client, headless components) - @travis
  ^ MOVED here: Tracking w3up work 🆙 #650
• consider what is essential for deploying beta console site
• fix: ensure that paging forward and backward yields the same first page w3infra#144
• deploy console for nft.storage and web3.storage to separate domains

🟢 w3filecoin

• Align on dagcargo for previous system - @vasco-santos
• w3protocol#353 CommP in client - @Gozala @vasco-santos
• Consumer stack - add CAR CIDs to w3filecoin pipe after store/add w3filecoin#12 - @vasco-santos
• Wire dead letter queues w3filecoin#10 - @vasco-santos

Notes

Filecoin integration is a big missing piece!

These tasks are Blocked on spade:

• spade write integration - Infra for aggregate api to submit offers to Spade w3filecoin-infra#14
• spade deal success check / retry?
• (not) spade read integration - find deals my CID is in. API to get deal information from spade API w3filecoin-infra#15
• metrics - Metrics w3filecoin-infra#13

🟢 Demos

• Demo series advocating for the w3up stack @dchoi27 @alanshaw

🟢 Sharding

• Add UnixFS directory sharding w3up-client#54
  • ensure that other implementations can read it but don't worry about overall compliance yet
• Add support for HAMT directories ipld/js-unixfs#37 (this is not on the board as @heyjay44 doesn't have permissions to add to a project)

🟢 Checkup

• checkup#3 Update checkup to consider data in new S3 buckets - @alanshaw @vasco-santos

🟢 differentiating uploads

• differentiate between nft and web3 dot storage uploads

---

A future phase!

Receipts (deprioritized)

We are invested in this. It's WIP. We should land it. it's:

• on critical path for billing.
• on critical for metrics (currently, we could calculate in other way)

what is the minimal receipts flow?

we need to tightly define what success of this endevour looks like

• finish off work to land it ucanto
• do work work to integrate it into store/add and s3 ObjectCreated

w3provider (deprioritized)

• w3protocol#349 provider/get consumer/add
• w3protocol#350 provider/add

Others

• Publicly verifiable timestamps for UCAN invocations web3-storage/specs#27
• CAR retrieval by CAR CID w3protocol#274

Notes

^gateway: total cost of serving old content may be higher if we bounce the user to dweb.link and it then has to fetch it from e-ipfs, but we should just go for it and monitor. if freeway doesn't have it then currently we pay twice as ipfs.io serves it to w3s.link.

migration off old api: ???

Tech debt

• validate CIDs for blocks in CARs - previously cluster was our validator. we don't have a replacement yet.

Questions

what's costing us the most right now?

• CDN data transfer in Cloudflare (not clear, but everything that leaves CF's network.)