Add ability to publish a webpage to IPFS #14770

bbondy · 2021-03-17T00:57:29Z

This is a placeholder to add the ability to recursively publish a website to IPFS.

Perhaps this will pin content of the current page, subresources, and some depth of links automatically.

bbondy · 2021-04-06T19:19:30Z

@lidel do you have an idea on how this could be implemented?

lidel · 2021-04-06T22:47:46Z

Depends on if you want to publish an immutable snapshot (ipfs://), or want to have an address that can track updates over time (ipns://).

Former is easy, you ipfs add --cid-version 1 -rw <dir> to recursively add a directory to IPFS and you get CID of the root directory in return. It can be loaded via ipfs://{cid}

The latter is the same, but additionally you need to have a mutable pointer to an immutable snapshot at /ipfs/{cid} that you can update over time.

Right now we have two ways of creating such pointers:

IPNS - built-in cryptographic solution provided by IPFS node (independent of DNS).
Broad strokes of potential integration:
- Brave could provide UI for generating unique publishing keys via ipfs key commands. Each website would have own key and can be loaded via ipns://{libp2p-key} like ipns://k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r
- Publishing a CID under a key is ipfs name publish --key={key} --ipns-base=base36 /ipfs/{cid}
- Sidenote:
  - Most likely want to enable ipns over pubsub for improved publishing/resolution performance
DNSLink – requires publishing DNS TXT record with IPFS a content path under some DNS name
- Regular users won't have own domains, so this could work only if Brave takes care of publishing and updating TXT record for subdomains under your control, like ipns://<somename>.brave-sites.com
- Sidenote:
  - content path can be /ipfs/{cid} OR /ipns/{libp2p-key}, so one can publish DNS TXT with /ipns/ path and then use ipfs name publish for posting updates over IPNS

spylogsster · 2021-04-20T14:09:18Z

Created PR without p2p-key management because we need to clarify how it should look for users from the product side

stephendonner · 2021-05-18T19:23:34Z

I started testing & verifying this using the testplan at brave/brave-core#8561 (where 3/5 cases are fine), but ran into #15634 - I'll wait for that to be resolved before coming back to fully verify this.

LaurenWags · 2021-05-18T21:00:41Z

@stephendonner is this issue blocked by #15634? if so, we should have this as QA/Blocked, but if it can continue to be tested, might be a good idea to shake out anything else.

stephendonner · 2021-05-18T22:04:08Z

@stephendonner is this issue blocked by #15634? if so, we should have this as QA/Blocked, but if it can continue to be tested, might be a good idea to shake out anything else.

Not totally blocked by that behavior (which I've since been told is expected via Chromium's, which we inherit), thanks for asking for clarification; the real problem, I think, is #15935, which @spylogsster is looking into. I'll mark it QA/Blocked for that reason, and revisit soon 👍

stephendonner · 2021-05-21T17:03:41Z

Removed QA/Blocked - #15935 stemmed from a regression in #15422, so we'll get #16020 tested in nightly and then uplifted to 1.26.x - nothing for 1.25.x should be needed from any of those.

Running through the tests for this now 👍

stephendonner · 2021-05-21T18:13:04Z

Verified PASSED using the testplan from brave/brave-core#8561 on build

Brave	1.25.65 Chromium: 91.0.4472.57 (Official Build) (x86_64)
Revision	e3443317fa07f1e9997e4a9c738eddfefc3c0292-refs/branch-heads/4472_54@{#6}
OS	macOS Version 11.3.1 (Build 20E241)

Check various cases when you have ongoing downloads and you do the import for a web page, it should not appear on the shelf with others

Steps:

loaded https://www.leaseweb.com/platform/network and downloaded a 10,000 mb file (http://mirror.sfo12.us.leaseweb.net/speedtest/10000mb.bin), so I could test sharing while downloads were still ongoing
loaded cnn.com
chose File | Save Page As...
clicked Save
clicked on the "hamburger" menu and chose IPFS -> Share Local Folder Using IPFS
browsed to the CNN directory and clicked OK

Confirmed I only saw the original CNN page being saved to Downloads, not when IPFS sharing

Check that downloaded pages are not visible in the download history

Steps:

loaded https://en.wikipedia.org/wiki/Vincent_van_Gogh
chose File | Save Page As...
clicked Save
clicked on the "hamburger" menu and chose IPFS -> Share Local File Using IPFS
browsed to the Vincent_van_Gosh - Wikipedia.htmlfile and clickedOK`

Confirmed that only the Wikipedia page I saved locally to share later via IPFS showed up in the Downloads window

s>

Check that the tab does not crash if downloading is in progress

Confirmed no crashes while testing

Check imported files have same content as originals

Steps:

loaded https://ia801003.us.archive.org/26/items/1941themaltesefalconelhalconmaltesjohnhuston/1941%20-%20The%20Maltese%20Falcon%20-%20El%20halc%C3%B3n%20malt%C3%A9s%20-%20John%20Huston%20-%20VOSE.ogv
chose File | Save Page As... and saved to Desktop
clicked on the "hamburger" menu and chose Share Local File using IPFS
waited
confirmed I got a shareable link copied to my clipboard, and a system notification - link is https://dweb.link/ipfs/QmXxxrEnnqNgXMyAqNUZ4vhqugXmgfbSHJo3yp8tzReTvF
confirmed after clicking the folder name, I was able to watch The Maltese Falcon

Check imported pages open correctly by shareable link

Steps:

loaded https://en.wikipedia.org/wiki/Vincent_van_Gogh
chose File | Save Page As... and clicked Save
clicked on the "hamburger" menu on the browser toolbar and chose Share Local File Using IPFS
browsed to Vincent_van_Gogh - Wikipedia.html and clicked Open
confirmed I got a shareable IPFS link copied to my clipboard, and a system notification - link is https://dweb.link/ipfs/QmX9YryCrJjvP4xBqVv5yEVexjm9CHpuhtrVqWapxHz6iR?filename=Vincent+van+Gogh+-+Wikipedia.html
copied and pasted the link in a new tab and pressed return to open
compared the output of the IPFS-shared file to the locally-saved one
confirmed the content and layout is mostly the same (IPFS version looks better, actually)

@spylogsster @bbondy @lidel do you think this needs to be tested on all platforms, or is this abstracted/modular enough to be expected to work on macOS, Windows, and Linux? Thanks!

Verification passed on

Brave | 1.25.66 Chromium: 91.0.4472.70&nbsp;(Official Build)&nbsp;(64-bit)
-- | --
Revision | fe095368270a32c92959403754bf6fd357dd9953-refs/branch-heads/4472@{#1172}
OS | Windows&nbsp;10 OS Version 2004 (Build 19041.985)

Verified the test plan from testplan from Import web page to ipfs brave-core#8561

Verification passed on

Brave	1.25.68 Chromium: 91.0.4472.77 (Official Build) (64-bit)
Revision	1cecd5c8a856bc2a5adda436e7b84d8d21b339b6-refs/branch-heads/4472@{#1246}
OS	Ubuntu 20

Verified the test plan from testplan from Import web page to ipfs brave-core#8561

Made sure folders can be imported

Verification PASSED on Win 10 x64 using the following build:

Brave | 1.25.68 Chromium: 91.0.4472.77 (Official Build) (64-bit)
--- | ---
Revision | 1cecd5c8a856bc2a5adda436e7b84d8d21b339b6-refs/branch-heads/4472@{#1246}
OS | Windows 10 OS Version 2009 (Build 19042.964)

Check various cases when you have ongoing downloads and you do the import for a web page, it should not appear on the shelf with others

loaded https://www.leaseweb.com/platform/network and started download the 10,000 mb file
- http://mirror.sfo12.us.leaseweb.net/speedtest/10000mb.bin (download ongoing while running through cases)
loaded cnn.com
saved the file via CTRL + S into the Download directory
clicked on the "hamburger" menu and chose IPFS -> Share Local Folder Using IPFS
uploaded the previously saved CNN directory via Downloads folder

Confirmed that only the original CNN page was being displayed via brave://downloads and not when IPFS sharing

`Example`	`Example`

Check that downloaded pages are not visible in the download history

loaded https://en.wikipedia.org/wiki/Vincent_van_Gogh
saved the file via CTRL + S into the Download directory
clicked on the "hamburger" menu and chose IPFS -> Share Local File Using IPFS
uploaded the previously saved .html page via Downloads folder

Confirmed that only the Wikipedia page I saved locally to share later via IPFS showed up in the Downloads window

`Example`	`Example`	`Example`

Check imported files have same content as originals

loaded https://ia801003.us.archive.org/26/items/1941themaltesefalconelhalconmaltesjohnhuston/1941%20-%20The%20Maltese%20Falcon%20-%20El%20halc%C3%B3n%20malt%C3%A9s%20-%20John%20Huston%20-%20VOSE.ogv
saved the file via CTRL + S into the Download directory
clicked on the "hamburger" menu and chose Share Local File using IPFS
confirmed I got a shareable link copied to my clipboard, and a system notification - link is ipfs://bafybeiadxe37uewwbbgjv6soy3v6u7yvxkxdked7a5d2ky2geza24ulhdu

`Example`	`Example`

Check imported pages open correctly by shareable link

Steps:

loaded https://en.wikipedia.org/wiki/Vincent_van_Gogh
saved the file via CTRL + S into the Download directory
clicked on the "hamburger" menu on the browser toolbar and chose Share Local File Using IPFS
confirmed I got a shareable IPFS link copied to my clipboard, and a system notification - link is https://dweb.link/ipfs/QmZqUgvG2ZZznedmLnJmTiBxokRsd5kTmmYr4s4raHrc6S?filename=Vincent%20van%20Gogh%20-%20Wikipedia.html
copied and pasted the link in a new tab and pressed return to open
compared the output of the IPFS-shared file to the locally-saved one
confirmed the content and layout is mostly the same

lidel · 2021-05-22T00:20:19Z

@stephendonner I am not familiar with Chromium internals to tell if this is abstracted away, but there is a potential diff around each OS having code paths for opening the file/dir picker GUI. Everything else should be the same.

bbondy added feature/web3/ipfs OS/Desktop labels Mar 17, 2021

lidel mentioned this issue Apr 6, 2021

Add ability to prove IPNS content ownership with libp2p-key #14774

Closed

spylogsster assigned spylogsster and unassigned spylogsster Apr 13, 2021

spylogsster mentioned this issue Apr 19, 2021

Import web page to ipfs brave/brave-core#8561

Merged

24 tasks

spylogsster mentioned this issue Apr 22, 2021

Allow users to use p2p-keys to publish content #15422

Closed

spylogsster closed this as completed in brave/brave-core#8561 Apr 23, 2021

spylogsster added this to the 1.25.x - Nightly milestone Apr 23, 2021

spylogsster added QA/Yes release-notes/include labels Apr 23, 2021

stephendonner mentioned this issue May 5, 2021

IPFS-imported Wikipedia page is missing its images #15634

Closed

LaurenWags added the QA/Test-All-Platforms label May 10, 2021

stephendonner added QA/In-Progress Indicates that QA is currently in progress for that particular issue and removed QA/In-Progress Indicates that QA is currently in progress for that particular issue labels May 17, 2021

stephendonner added QA/Blocked and removed QA/Blocked labels May 18, 2021

stephendonner added the QA Pass-macOS label May 21, 2021

stephendonner mentioned this issue May 21, 2021

Share local file/dir using IPFS does not work on desktop #16020

Closed

GeetaSarvadnya added the QA Pass-Win64 label May 26, 2021

btlechowski added the QA Pass-Linux label May 26, 2021

LaurenWags mentioned this issue May 26, 2021

[Desktop] Release Notes for 1.25.x Release #1 #16089

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add ability to publish a webpage to IPFS #14770

Add ability to publish a webpage to IPFS #14770

bbondy commented Mar 17, 2021

bbondy commented Apr 6, 2021

lidel commented Apr 6, 2021 •

edited

Loading

spylogsster commented Apr 20, 2021

stephendonner commented May 18, 2021

LaurenWags commented May 18, 2021

stephendonner commented May 18, 2021

stephendonner commented May 21, 2021

stephendonner commented May 21, 2021 •

edited by kjozwiak

Loading

lidel commented May 22, 2021

Add ability to publish a webpage to IPFS #14770

Add ability to publish a webpage to IPFS #14770

Comments

bbondy commented Mar 17, 2021

bbondy commented Apr 6, 2021

lidel commented Apr 6, 2021 • edited Loading

spylogsster commented Apr 20, 2021

stephendonner commented May 18, 2021

LaurenWags commented May 18, 2021

stephendonner commented May 18, 2021

stephendonner commented May 21, 2021

stephendonner commented May 21, 2021 • edited by kjozwiak Loading

Check various cases when you have ongoing downloads and you do the import for a web page, it should not appear on the shelf with others

Check that downloaded pages are not visible in the download history

Check that the tab does not crash if downloading is in progress

Check imported files have same content as originals

Check imported pages open correctly by shareable link

Check various cases when you have ongoing downloads and you do the import for a web page, it should not appear on the shelf with others

Check that downloaded pages are not visible in the download history

Check imported files have same content as originals

Check imported pages open correctly by shareable link

lidel commented May 22, 2021

lidel commented Apr 6, 2021 •

edited

Loading

stephendonner commented May 21, 2021 •

edited by kjozwiak

Loading