[PoC] libdweb experiment: protocol handler API #533

lidel · 2018-07-17T20:04:24Z

`ipfs://` and `ipns://` sitting on a tree

Background

This PR aims to implement ipfs:// and ipns:// using browser.protocol.registerProtocol API from mozilla/libdweb. The goal is to ensure proposed API delivers everything IPFS needs, and squash some bugs in the process.

Developer notes on working with libdweb can be found in docs/libdweb.md

ref mozilla/libdweb#2, #343, #164, ipfs/in-web-browsers#6, ipfs/in-web-browsers#14, ipfs/specs#152
cc @Gozala @kyledrake

4 min demo (August 2018)

Tasks

Known Issues

HTML issues

content scripts such as linkify experiment do not get injected on ipfs:// pages
- upstream issue: Content scripts for custom protocols mozilla/libdweb#26
SVG does not render when used in <img src=
- ustream issue: Protocol Handler: SVG does not render when used in img HTML tag mozilla/libdweb#54
webm does not play when used in <video> if it was not present in browser cache before
- upstream issue: Protocol Handler: WEBM does not load when used in video HTML tag mozilla/libdweb#57
cross-origin assets within the same protocol do not load
- upstream issue: Resources from different origin but same protocol should load mozilla/libdweb#55
cross-protocol assets (ipns:// loading ipfs://) do not load
- upstream issue: Protocol Handler: Cross-protocol Content mozilla/libdweb#52

Missing API / Spec Issues

plain text files such as this README.md are returned as downloads instead of displayed in the browser
how to make redirects?
- upstream issue: support redirects in protocol handlers mozilla/libdweb#34
how to indicate errors (response status codes) ?
- upstream for status codes: Response (HTTP) status codes for protocol handlers mozilla/libdweb#35
headers? metadata? cache controls?
- upstream for headers in Request / Response: Request headers support for protocols mozilla/libdweb#36 & Response Headers support for protocols mozilla/libdweb#37

IPFS-specific, UX Issues, Other

Jenkins build fails because addon-linter shipping with web-ext needs to be updated (Generator function causing JS_SYNTAX_ERROR mozilla/addons-linter#2108 (comment))
Legacy CIDv0 (eg. ipfs://QmbWqxBEKC3P8tqsKc98xmWNzrzDtRLMiMPL8wBuTGsMnR) are lowercasesed and produce an error. We should provide conversion tool for user on the error screen, but we need go-ipfs and js-ipfs to support lookups for CIDv0 converted to CIDv1 first.
remove duplicated code and reuse ipfs-http-response for directory listings
- blocked by the lack of support for CIDv1: Add resolver support for CIDv1 and Base32 js-ipfs-http-response#8
Solve UX for "Not Found" scenarios, such as:
- ipfs://bafybeigwa5rlpq42cj3arbw27aprhjezhimhqkvhbb2kztjtdxyhjalr3q/big-buck-bunny_trailer.mp4
Solve UX for "Not Found" scenarios, such as:
- ipns://tr.wikipedia-on-ipfs.org/wiki/G%C3%BCne%C5%9F_r%C3%BCzg%C3%A2r%C4%B1.html (lack of unescaping)
If native protocol handler is available, use it it in all redirects instead of HTTP
- redirect via onBeforeRequest blocked by upstream issue: Allow webRequest.onBeforeRequest to redirect to libdweb protocol handler mozilla/libdweb#74

Test Resources

IPFS

picture:
- ipfs://bafybeie5gq4jxvzmsym6hjlwxej4rwdoxt7wadqvmmwbqi7r27fclha2va (CID codec: dag-pb)
- ipfs://bafkreigh2akiscaildcqabsyg3dfr6chu3fgpregiymsck7e7aqa4s52zy/(CID coded: raw)
html page: ipfs://bafybeidvtwx54qr44kidymvhfzefzxhgkieigwth6oswk75zhlzjdmunoy/linkify-demo.html
directory tree: ipfs://babybeiccrv3uc3hjipdnwf4nnntbxuwvt4pn5dsgelvvyueucracbevtha
plain text file: ipfs://babybeiccrv3uc3hjipdnwf4nnntbxuwvt4pn5dsgelvvyueucracbevtha/go-proofs/README.md
video
- small video (2MB): ipfs://bafybeigwa5rlpq42cj3arbw27aprhjezhimhqkvhbb2kztjtdxyhjalr3q/big-buck-bunny_trailer.webm
- standalone trickle-dag webm video (22MB): ipfs://bafybeigxjv2o4jse2lajbd5c7xxl5rluhyqg5yupln42252e5tcao7hbge
- same video but used in <video src="ipfs://cidv1b32" /> ipfs://bafkreihosso7nxtyhjtpvjmrdoflyup6ex6b2xgzezibavuonobt2l4dai
invalid CID (lowercased CIDv0): ipfs://qmbwqxbekc3p8tqskc98xmwnzrzdtrlmimpl8wbutgsmnr/
A snapshot of Eng. Wikipedia: ipfs://QmXoypizjW3WknFiJnKLwHCnL72vedxjQkDDP1mXWo6uco

IPNS

ipns://docs.ipfs.io
ipns://cid-utils.ipfs.team
ipns://tr.wikipedia-on-ipfs.org
ipns://explore.ipld.io
ipns://explorecatena.com

Screenshots

This is a first step on the road to native protocol handler and more. mozilla/libdweb is included as a git submodule, which enables us to develop against latest API prototypes with as little overhead as possible. `libdweb-build` will produce extension bundle with access to experimental APIs `firefox:libdweb` will start Firefox Nightly with mentioned bundle. Make sure firefox-nightly is on your PATH

Just a basic handler that reads images and returns them to the browser. TODO: - support directory listing - support streaming - support mime-sniffing

+ cleanup of libdweb scripts

- port directory detection from streaming branch in Muon-Brave - detect streaming responses and return them as async iterator - trivia: switch from uglify-es to terser to support for-await

lidel · 2018-07-22T17:32:17Z

Added:

streaming
- 👍 standalone trickle-dag webm video works fine: ipfs://bafybeigxjv2o4jse2lajbd5c7xxl5rluhyqg5yupln42252e5tcao7hbge
- 👎 use in <video src="ipfs://cidv1b32" /> does not seem to work yet. Opening ipfs://bafkreihosso7nxtyhjtpvjmrdoflyup6ex6b2xgzezibavuonobt2l4dai produces errors, probably due to lack of content-range semantics.
error handling
- basic error message on invalid CID (lowercased CIDv0): ipfs://qmbwqxbekc3p8tqskc98xmwnzrzdtrlmimpl8wbutgsmnr/

lidel · 2018-08-02T21:28:47Z

Small update: got IPNS working during a short workshop at Dweb Summit :-)

Also, simplified docs/libdweb.md it is now 2 commands that require only firefox-nightly, git and npm on your PATH

js-ipfs does not support IPNS yet, but it has ipfs.dns API for dnslink lookup, which actually all we need

We want npm run libdweb work on MS Windows as well.

lidel · 2018-08-15T23:59:08Z

Ack. We are using async handler for multiple reasons, setting arbitrary (future: ipld/legacy-unixfs-v2#11) content type being only one of them (dnslink lookup and other apis are/will be async as well).

Also, some updates:

Cross-origin assets within the same protocol now load fine (Allow cross-origin resoure loading / embedding mozilla/libdweb#69)
Smoke-tested on ipfs://bafkreihosso7nxtyhjtpvjmrdoflyup6ex6b2xgzezibavuonobt2l4dai
(unrelated to libdweb) I've found a bug in js-ipfs related to directory sharding. Loading ipns://tr.wikipedia-on-ipfs.org while using embedded js-ipfs node does not work at the moment because of ipfs.get is not working with sharded directories js-ipfs#1279 (comment)
Started thinking about doing transport fallback to public gateways if no node available (libdweb/protocol handler: fallback to public gateways if no node available #554)
CI build is green again.

mozilla/libdweb#71

Context: mozilla/libdweb#82

We mix different module types for now, this restores branch to buildable state.

lidel · 2020-09-22T12:25:44Z

This PR was important step towards demonstrating value of IPFS in the browser context, and how future could look like, and unlocked collaborations with Brave, Opera and more to come.

Sadly Mozilla dropped the ball on libdweb in 2019 and APIs used in this PR never landed in Firefox Nightly, and after a year of no activity it is time to close it together with #553

lidel added 5 commits July 17, 2018 12:40

feat: PoC for browser.protocol.registerProtocol

43bdb5f

Just a basic handler that reads images and returns them to the browser. TODO: - support directory listing - support streaming - support mime-sniffing

docs: notes on eveloping with libdweb

08ef6f3

fix: crashes caused by omitted contentType in protocol handler

a9f2b4d

+ cleanup of libdweb scripts

feat: PoC of browser.protocol with directory listing

33cf210

lidel added the area/cidv1b32 Moving to CIDv1/Base32 by default label Jul 17, 2018

lidel added this to the 2018-Q3 milestone Jul 17, 2018

lidel self-assigned this Jul 17, 2018

This was referenced Jul 18, 2018

Redirect ipfs://example.net to ipns:// #534

Closed

Create WebExtensions Experiments to Prototype Missing APIs #343

Open

lidel added 2 commits July 20, 2018 19:13

feat: streaming in ipfs://

101966e

- port directory detection from streaming branch in Muon-Brave - detect streaming responses and return them as async iterator - trivia: switch from uglify-es to terser to support for-await

fix: basic error handling for libdweb protocol

dd870ee

lidel added 3 commits August 2, 2018 11:19

Merge remote-tracking branch 'origin/master' into libdweb

0a305c8

PoC IPNS support

1d814f7

docs: simplified libdweb dev notes

2ddf9ea

lidel added 6 commits August 2, 2018 15:35

fix: make ipns:// work with embedded js-ipfs

252776a

js-ipfs does not support IPNS yet, but it has ipfs.dns API for dnslink lookup, which actually all we need

fix: meaningful error when global toggle is off

3378c1a

feat(libdweb): download nightly if missing

c488120

fix(libdweb): udpate web-ext to fix addon-lint

f4486de

feat(libdweb): run libdweb:build on win as a part of CI

8beea27

We want npm run libdweb work on MS Windows as well.

feat(libdweb): run libdweb:build on mac during CI

1d18fc5

lidel force-pushed the libdweb branch from fabfc24 to 28f7e24 Compare August 3, 2018 18:26

fix(libdweb/ci): switch to dev yarn build

3133ad0

lidel force-pushed the libdweb branch 3 times, most recently from 18e78bf to 89543d1 Compare August 3, 2018 18:54

fix(libdweb/ci): network lock and submodule sync

34eafa4

lidel force-pushed the libdweb branch from 89543d1 to 34eafa4 Compare August 3, 2018 19:08

fix(libdweb): allow loading of cross-origin assets

87ef474

fix(libdweb): non-unwrappable cross-compartment wrapper promise error

8c87835

mozilla/libdweb#71

lidel removed the area/libdweb Work related to new APIs from Mozilla/libdweb label Aug 17, 2018

lidel modified the milestones: 2018-Q3, libdweb demos Aug 17, 2018

lidel added the area/libdweb Work related to new APIs from Mozilla/libdweb label Aug 17, 2018

lidel mentioned this pull request Aug 24, 2018

IPFS Integration - Roadmap and discussion brave/brave-browser#819

Closed

This was referenced Sep 8, 2018

Hostnames for localhost HTTP gateway ipfs/in-web-browsers#109

Closed

Support Custom Protocols in WebExtension #164

Open

TypeError: Cc['@mozilla.org/network/content-sniffer;1'] is undefined with Nightly 64.0a1 mozilla/libdweb#82

Closed

lidel added 2 commits September 14, 2018 20:07

fix: missing content-sniffer

1bf413f

Context: mozilla/libdweb#82

Merge branch 'master' into libdweb

eb399fd

lidel added the status/in-progress In progress label Sep 22, 2018

lidel mentioned this pull request Sep 22, 2018

Support for js-ipfs as a replacement for HTTP gateway #248

Closed

lidel added 3 commits October 11, 2018 10:36

fix: workaround for mused module types

39b2d0e

We mix different module types for now, this restores branch to buildable state.

Merge remote-tracking branch 'origin/master' into libdweb

72cb11a

chore: switch to ipfs-http-response v0.2.0

d6dfc0c

lidel mentioned this pull request Dec 9, 2018

FlyWeb ipfs/js-ipfs#599

Closed

lidel mentioned this pull request Jan 15, 2019

Expose HTTP Gateway when running in browser (eg. Service Worker) ipfs/js-ipfs#1820

Closed

lidel mentioned this pull request Jan 23, 2019

Redirect of CID-in-subdomain and DNSLink websites #667

Closed

djdv mentioned this pull request Feb 13, 2019

Add input cleaner ipfs/ipget#51

Merged

lidel mentioned this pull request Mar 23, 2019

Improve onboarding on Firefox for Android #690

Open

lidel added status/inactive No significant work in the previous month and removed status/in-progress In progress labels May 28, 2020

lidel marked this pull request as draft September 22, 2020 12:20

lidel changed the title ~~[WIP] libdweb experiment: protocol handler API~~ [PoC] libdweb experiment: protocol handler API Sep 22, 2020

lidel closed this Sep 22, 2020

lidel mentioned this pull request Sep 22, 2020

[PoC] libdweb discovery and transport #553

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[PoC] libdweb experiment: protocol handler API #533

[PoC] libdweb experiment: protocol handler API #533

lidel commented Jul 17, 2018 •

edited

Loading

lidel commented Jul 22, 2018

lidel commented Aug 2, 2018

lidel commented Aug 15, 2018 •

edited

Loading

lidel commented Sep 22, 2020

[PoC] libdweb experiment: protocol handler API #533

[PoC] libdweb experiment: protocol handler API #533

Conversation

lidel commented Jul 17, 2018 • edited Loading

ipfs:// and ipns:// sitting on a tree

Background

4 min demo (August 2018)

Tasks

Known Issues

HTML issues

Missing API / Spec Issues

IPFS-specific, UX Issues, Other

Test Resources

IPFS

IPNS

Screenshots

lidel commented Jul 22, 2018

lidel commented Aug 2, 2018

lidel commented Aug 15, 2018 • edited Loading

lidel commented Sep 22, 2020

lidel commented Jul 17, 2018 •

edited

Loading

`ipfs://` and `ipns://` sitting on a tree

lidel commented Aug 15, 2018 •

edited

Loading