webcrypto in Firefox

This release enables webcrypto in Firefox and includes many fixes to issues found by Solène Rapenne. Thank you! The result of this is 10x fewer connections used, much less memory used and more configurations work.

Features:

• add bookmark button to html, md and notes viewers
• -domain arg renamed to -listen-host

Optimisations:

• Enable webcrypto in firefox

Fixes:

• gc on single server S3 hosting
• block repeated failed p2p block gets
• support insecure webdav auth on localhost, required by rclone
• fix kademlia connection leak
• fix kademlia lookups which broke with new multiaddrs we can't parse
• p2p puts of large blocks to S3
• fix memory leak for incoming p2p http requests
• custom peergos path set via env
• enforce filename restrictions in both directions
• gallery keyboard navigation bug

Mirrored at https://peergos.net/public/peergos/releases

Release commit:
web-ui/d9106a115c082bd15c27d4f9cc5d76f580fcba2d

Instant moves, 100x faster small block writes with S3

The common cases of moving a file or directory within your own space are now instant (independent of the size of the file or directory). We now have a webdav bridge to Peergos so you can manage your Peergos files with any webdav client. We now support openbsd for running Peergos.

Features:

• Webdav bridge
• QR codes for secret links
• Don't persist blocks retrieved from non local-users (but still cache)
• Annual billing support
• Add email api and associated permission
• New shell commands - lls, lcd
• Better shell command flags (before the arguments)
• crossOriginIsolated=true in sandboxed app iframes now

Optimisations:

• Make moving a file or dir O(1) rather than O(file size)
  • Add the option of preserving existing access and links
• Offline login and access through a local proxy daemon is 300x faster

Fixes:

• Deleting a write shared folder with a sub-folder
• Check space usage before reducing account storage
• Fix navigation in markdown browser
• Fix opening images in new tabs in Chrome
• Fix shell progress bar for zero byte files
• Handle partial char in mimetype detection
• Support low bandwidth clients
• Support escaping spaces in file names in shell commands
• Make ipfs swarm addresses configurable
• Fix div by 0 in shell.put of empty file
• Fix duplicate app launches on a slow connection

Mirrored at https://peergos.net/public/peergos/releases

Release commit:
web-ui/8d4da5616964fb893203b636ff19e552da26fb27

New secret links design and huge speed ups

A brand new and much better design for secret links. Previous secret links were pure capabilities in the URL. This meant revoking them required rotating all the keys which is expensive. They are also very long, e.g. https://peergos.net/#%7B%22secretLink%22:true%2c%22link%22:%22#6MDZhRRPT4ugkJuUfcWtaZodN5QYzkZKJtHpDHomFJrVhNSZysiFYimpgtcA2F/6MDZhRRPT4ugkJuUfcRzRbPpFimcBNJx2N9TJDnL4W3ETYhwdsWdvgCkXkwipF/FCYSFhpQ1xD2cydr6CFQ6UwFkgB82pWReAUzKVDxe4KA/5Pf7SvCG1mMtui2aPd9F3SH2wdwsPro1GxTa7VfxkWrj9XQGAUB%22%2c%22open%22:true%2c%22path%22:%22/demo/%22%7D

The new secret links require the assistance of your home server (or a mirror). The capability that was in the URL before is now encrypted and stored in a new champ on the identity writer data. There is a new server API to lookup a secret link and return the ciphertext. This is then decrypted locally using the hash in the URL and optionally an additional password supplied by the user.

This means the URLs can be much shorter (4x), e.g. https://your-domain/secret/z59vuwzfFDoy1Mya69m7j1nZHL7hH4yLpAN7qKQusEUREveNVJifs9R/1368969#cWNWXq9jPahK

The new champ includes the mirror bat in its root block so the champ itself is not public, meaning the number of secret links you have is not revealed to anyone but your server and mirrors.

Because we are using the server, we can get it to enforce expiry times or n-use links. Revoking these links is instant because you just need to delete the mapping from the champ. We also allow writable secret links, with a suitable warning that holders of the link could fill your space quota (until we implement subquotas). Links can also had additional password protection.

If you migrate to another server, your links come with you and all pre-existing secret links will continue to work. There is no limit to the number of links you can create, because they are just normal blocks that contribute to your space usage.

The password based encryption on each link is designed to take 100 years to brute force if you have 1,000,000 GPUs, each of which can do 1M scrypt hashes per second.

Uploading small files has been optimised by uploading cbor blocks and raw blocks concurrently, rather than in 2 steps. Also by grouping small raw blocks and writing them directly to Peergos server, rather than S3 in direct S3 case.

The reason this is so beneficial is direct S3 uploads from browser are constrained by http1.1. This means we can only upload 5 blocks concurrently over the 5 connections provided by the browser.

Features:

• new secret links design with:
  • expiring secret links
  • n-use secret links
  • password protected secret links
  • writable secret links
  • instant revocation
  • supported when migrating server
• mkdir in shell CLI
• fullscreen support in html browser
• support large file streaming in apps
• double click to open in any installed app
• open json files in text editor
• move built-in tasks app to an installable app
• ebook reader app
• support downloading all your data in a single zip file
• ease installation of recommended apps supporting a given file type

Optimisations:

• Local S3 small block buffer (50x faster small writes)
• 10x faster signing and signature verification (use webcrypto Ed25519 when present)
• 2-3x faster bulk small file/folder upload
• 20% faster large file upload
• cache profile pictures in newsfeed

Fixes:

• support wasm mimetype
• fix failed uploads in a different writing space
• fix an edge case in revoking write access
• Support general mimetypes in the gateway for public websites

Mirrored at https://peergos.net/public/peergos/releases

Release commit:
web-ui/3ef5e383e9fe01ee2710b13ab15b7e733bae5e52

Local peer discovery

Peergos can now discover peers on the local network (LAN) using mdns. This means two peers on the same LAN can friend each other and share files etc entirely offline.

Features:

• local peer discovery using mdns
• launch recommended-app for secret links to supported formats (docx, clsx etc.)
• make server configurable in writable-link generator cli

Optimisations:

• cache pointer targets
• improve newsfeed performance

Fixes:

• bulk mirror of raw blocks
• offline login
• fix stay logged in with MFA enabled
• remove any cached login data if MFA is enabled

Mirrored at https://peergos.net/public/peergos/releases

Release commit:
web-ui/f6788b5a3366148e21dcf70d56a5a735e4c6243b

60x faster mirroring, chat API

The initial pki sync is now 60x faster (down from 10min to 10s). The pki sync also happens in the background now. There is a new chat api enabling fully fledged 3rd party chat apps.

Features:

• Extended chat API for apps
• Background pki sync
• Show profile picture in drive header
• Improved newsfeed design
• Thumbnails displayed in picker dialogue
• Directly launch specific apps from search results
• Additional app permission to retrieve profile photos

Optimisations:

• Mirroring data is about 60X faster now

Fixes:

• Use direct S3 blockstore when available
• Fix bulk chat invites
• Improve S3 error handling

Mirrored at https://peergos.net/public/peergos/releases

Release commit:
web-ui/04354df8a322502916e792a496b1e7604c84a9a0

Server identity rotation and internationalisation

Peergos v0.16.0

Servers can now rotate their identity keypair independently of the users on that server. Clients verify the rotation using signed IPNS records if they update their host, which are periodically republished. Internationalisation support is included with simplified Chinese supported already.

Features

• remove 2 GB limit for size of zipped folder that can be downloaded
• server identity rotation (and CLI)
• select all/none

Optimisations

• make GC about 100x more scalable by storing reachability and block list on disk instead of ram

Fixes

• fix native image builds
• handle OPFS returning empty blocks
• remove excess space at bottom of login page
• fix race condition with repeated updateSocial calls during login
• fix multi-select in UI
• fix occasional failures to resolve a peerid in dht
• miscellaneous web ui improvements
• context menu dark mode

Mirrored at https://peergos.net/public/peergos/releases

Release commit:
web-ui/512532b3a2e280f5d15e0b89bc8a050f050b76f4

Latex support, 15x less bandwidth

Peergos v0.15.0

Large performance improvements after switch to Nabu. 15x reduction in baseline bandwidth usage. Now ~12KB/s is typical. Also 60% less CPU usage. The size of peergos.js has also been halved by a bug fix in jpack!

Features:

• Latex support in notes app
• Latex support in markdown app
• Support UTF-8 in text mimetype detection
• Usage CLI for server operators
• Authed usage call

Optimisations:

• Don't query blockstore when calculating usage for users that haven't changed
• Add local cbor block cache for S3 to make champ gets 10x faster
• 2x less RAM and bandwidth usage in Kademlia
• Huge reduction to kademlia routing table size by not adding non-server peers
• Don't call identify or kademlia on incoming connections for peers we already know
• Reduce baseline bandwidth usage from 1.5Mb/s to 0.1Mb/s by switching to a new bitswap protocol ID
• Fix Jpack to halve size of compiled peergos.js (2.1 MB to 1.1 MB)

Fixes:

• Handle S3 internal errors as rate limits
• Correct usage update during migration to new node
• Improve information for GC errors
• Make GC tolerant to non existent blocks
• Handle .apk mimetype
• Treat http connection refused as a rate limit
• Darkmode fix for sub pages in md viewer
• Fix opening depth 2 sub-pages in md viewer

Published at https://peergos.net/public/peergos/releases

Release commit:
web-ui/ac4bba146e5faad4f98ab9ca1b5aebd330823667

Notes app + bug fixes

A new app and a few bug fixes from the previous major rewrite in v0.14.0.

New apps:

• notes

Features:

• ipfs metrics for S3 calls, bitswap and kademlia bandwidth
• decimal storage sizes

Optimisations:

• cache peer id in AuthedStorage
• inline block request authoriser (no separate allow listener)
• block aggressive bitswap peers
• don't retrieve blocks to auth a bitswap request for them

Fixes:

• timezone for block auth generation
• use bare peer id in bat auth
• always persist generated ipfs identity
• make sure block metadata db is populted for file blockstores also
• handle empty block files
• filter blocks of wrong type in listing blockstore

Release commit:
web-ui/163f59fcbbff6e3cf2814de985b7bba5c0585e97

Java IPFS implementation (Nabu), block metadata db, versioned S3 buckets.

This is an exciting new release that means we finally control our whole stack as we switch to a Java implementation of IPFS (that we wrote). We introduce a blockmetadata store that stores for each cid the size, links and BATs. This massively speeds up GC, space usage updates, and read authorizations.

Features:

• Switch from Golang based ipfs-nucleus to Java based Nabu
• Support versioned S3 buckets
• Introduce block metadata db to massively speed up many operations
• infini filter (infinitely expandable bloom filter)
• only use authWrites for raw blocks, and send the BATs for these
• store BATs in block metadata store

Privacy improvements

• don't broadcast block retrievals to DHT or bitswap
• don't provide blocks to DHT unless we are running a mirror (also reduces CPU and bandwidth)

Optimisations:

• 10x less bandwidth usage when idle
• faster peer dials to get blocks
• less CPU and RAM usage
• inline p2p block get authorization
• block repeated block get attempts with invalid auth
• huge speed up to cacheless delete (e.g. in incognito mode)
• use owner to lookup peer to retrieve block from in the common case
• improve sign up oplog from 10 pointer updates and 29 blocks to, 2 pointer updates and 10 blocks
• upgrade HTTP client in JavaPoster

Fixes:

• timezone issue in BAT verification
• fix handling of writable secret links
• Bulk delete on B2 (which is not S3 compliant)
• CAS exception making social post
• support Ed25519 PKI peer id
• event loading in calendar

Release commit:
web-ui/59c4fa5820dd7964a70d992c7bec97ff252e98ea

v0.13.0

Two factor auth (2FA), Multi-file-select, new apps

Features:

• 2FA including TOTP and webauthn
• select multiple files/folders to {delete, download, cut, copy, paste}
• enable quic, noise
• require token to signup if not only listening on localhost
• support HEIC and JPEGXL image mimetypes
• require Java 17
• build docker image
• add delete chat and setTitle to chat API
• human readable file sizes
• new and improved gallery with media controls and prefetch
• app grid on launcher page
• file picker api for apps
• export to PDF in paintz, ck-editor and tui-markdown editor apps
• app to convert html to docx
• new draw.io app

Optimisations:

• download chunks in parallel (much faster downloads)
• make bloom adds async
• bulk block writes for small blocks (speed up deletes, uploads)
• parallelise block puts to S3 in bulk write
• bulk delete
• add OPFS (Origin Private File System) cache for blocks in browser

Fixes:

• fix svg mimetype case
• allow wildcard listen addresses (fixes docker)
• fix immediate login after changing password in browser
• many miscellaneous fixes