README.md

Rivet

Rivet is both a command-line tool and a Golang package for archiving webpages to IPFS.

Supported Golang version: See .github/workflows/testing.yml

Installation

The simplest, cross-platform way is to download from GitHub Releases and place the executable file in your PATH.

From source:

go get -u github.com/wabarc/rivet/cmd/rivet

From GitHub Releases:

sh <(wget https://github.com/wabarc/rivet/raw/main/install.sh -O-)

Using Snapcraft (on GNU/Linux)

sudo snap install rivet

Usage

Command line

A toolkit makes it easier to archive webpages to IPFS.

Usage:

  rivet [options] [url1] ... [urlN]

  -host string
        IPFS node address (default "localhost")
  -m string
        Pin mode, supports mode: local, remote, archive (default "remote")
  -p string
        Pinner sceret or password.
  -port int
        IPFS node port (default 5001)
  -t string
        IPFS pinner, supports pinners: infura, pinata, nftstorage, web3storage. (default "infura")
  -timeout uint
        Timeout for every input URL (default 30)
  -u string
        Pinner apikey or username.

Examples

Stores data on local IPFS node.

rivet -m local https://example.com https://example.org

Stores data to remote pinning services.

rivet https://example.com

Or, specify a pinning service.

rivet -t pinata -k your-apikey -s your-secret https://example.com

Or, stores file locally without any IPFS node.

rivet -m archive https://example.com

Go package

package main

import (
	"context"
	"fmt"
	"net/url"

	"github.com/wabarc/ipfs-pinner"
	"github.com/wabarc/rivet"
	"github.com/wabarc/rivet/ipfs"
)

func main() {
	opts := []ipfs.PinningOption{
		ipfs.Mode(ipfs.Remote),
		ipfs.Uses(pinner.Infura),
	}
	p := ipfs.Options(opts...)
	r := &rivet.Shaft{Hold: p}
	l := "https://example.com"
	input, err := url.Parse(l)
	if err != nil {
		panic(err)
	}

	dst, err := r.Wayback(context.TODO(), input)
	if err != nil {
		panic(err)
	}
	fmt.Println(dst)
}

F.A.Q

Optional to disable JavaScript for some URI?

If you prefer to disable JavaScript on saving webpage, you could add environmental variables DISABLEJS_URIS and set the values with the following formats:

export DISABLEJS_URIS=wikipedia.org|eff.org/tags

It will disable JavaScript for domain of the wikipedia.org and path of the eff.org/tags if matching it.

Credit

Special thanks to @RadhiFadlillah for making obelisk, under which the crawling of the web is based.

Contributing

We encourage all contributions to this repository! Open an issue! Or open a Pull Request!

License

This software is released under the terms of the MIT. See the LICENSE file for details.