IPLD Prime In IPFS: Target Merge Branch

.circleci/config.yml

@@ -225,7 +225,7 @@ jobs:
       - run:
           name: Cloning
           command: |
-            git clone https://github.com/ipfs/go-ipfs-api.git
+            git clone -b ipld-in-ipfs https://github.com/ipfs/go-ipfs-api.git
             git -C go-ipfs-api log -1
       - run:
           name: Starting the daemon

CHANGELOG.md

@@ -1,5 +1,27 @@
 # go-ipfs changelog
 
+## v0.10.0 TBD
+
+**IPLD Levels Up**
+
+The handling of data serialization as well as many aspects of DAG traversal and pathing have been migrated from older libraries, including [go-merkledag](https://github.com/ipfs/go-merkledag) and [go-ipld-format](https://github.com/ipfs/go-ipld-format) to the new **[go-ipld-prime](https://github.com/ipld/go-ipld-prime)** library and its components. This allows us to use many of the newer tools afforded by go-ipld-prime, stricter and more uniform codec implementations, support for additional (pluggable) codecs, and some minor performance improvements.
+
+This is significant refactor of a core component that touches many parts of IPFS, and does come with some **breaking changes**:
+
+* **IPLD plugins** (those implementing the `PluginIPLD` interface) are not registered into go-ipld-prime. Such plugins should instead register their encoder and decoder functions in their initialization functions against the multicodec table. We demonstrate the change with the [bundled git plugin](./plugin/plugins/git/).
+* **The semantics of `dag put` and `dag get` change**:
+  * `dag get` now takes the `format` option which accepts a multicodec name used to encode the output. By default this is `dag-json`. Users may notice differences from the previously plain Go JSON output, particularly where bytes are concerned which are now encoded using a form similar to CIDs: `{"/":{"bytes":"unpadded-base64-bytes"}}` rather than the previously Go-specific plain padded base64 string. See the [dag-json specification](https://ipld.io/specs/codecs/dag-json/spec/) for an explanation of these forms.
+  * `dag get` no longer prints an additional new-line character at the end of the encoded block output. This means that the output as presented by `dag get` are the exact bytes of the requested node. A round-trip of such bytes back in through `dag put` using the same codec should result in the same CID.
+  * `dag put` uses the `input-enc` option to specify the multicodec name of the format data is being provided in, and the `format` option to specify the multicodec multicodec name of the format the data should be stored in. These formerly defaulted to `json` and `cbor` respectively. They now default to `dag-json` and `dag-cbor` respectively but may be changed to any supported codec (bundled or loaded via plugin) by its [multicodec name](https://github.com/multiformats/multicodec/blob/master/table.csv).
+  * The `json` and `cbor` multicodec names (as used by `input-enc` and `format` options) are now no longer aliases for `dag-json` and `dag-cbor` respectively. Instead, they now refer to their proper [multicodec](https://github.com/multiformats/multicodec/blob/master/table.csv) types. `cbor` refers to a plain CBOR format, which will not encode CIDs and does not have strict deterministic encoding rules. `json` is a plain JSON format, which also won't encode CIDs and will encode bytes in the Go-specific padded base64 string format rather than the dag-json method of byte encoding. See https://ipld.io/specs/codecs/ for more information on IPLD codecs.
+* The **dag-pb codec**, which is used to encode UnixFS data for IPFS, is now represented in a form via the `dag` API that mirrors the protobuf schema used to define the binary format and unifies the implementations and specification of dag-pb across the IPLD and IPFS stacks. Previously, additional layers of code within IPFS between protobuf serialization and UnixFS handling for file and directory data, obscured the forms that are described by the protobuf representation. Much of this code has now been replaced and there are fewer layers of transformation. This means that interacting with dag-pb data via the `dag` API will use different forms:
+  * Previously, using `dag get` on a dag-pb block would present the block serialized as JSON as `{"data":"padded-base64-bytes","links":[{"Name":"foo","Size":100,"Cid":{"/":"Qm..."}},...]}`.
+  * Using the dag-pb data model specification and the new default dag-json codec for output, this will now be serialized as: `{"Data":{"/":{"bytes":"unpadded-base64-bytes"}},"Links":[{"Name":"foo","Tsize":100,"Hash":{"/":"Qm..."}},...]}`. Aside from the change in byte formatting, most field names have changed: `data` → `Data`, `links` → `Links`, `Size` → `Tsize`, `Cid` → `Hash`. Note that this output can be changed now using the `--format` option to specify an alternative codec.
+  * Using `dag put` and a `format` option of `dag-pb` now requires that the input conform to this dag-pb specified form. Previously, input using `{"data":"...","links":[...]}` was accepted, now it must be `{"Data":"...","Links":[...]}`.
+  * Previously it was not possible to use paths to navigate to any of these properties of a dag-pb node, the only possible paths were named links, e.g. `dag get QmFoo/NamedLink` where `NamedLink` was one of the links whose name was `NamedLink`. This functionality remains the same, but by prefixing the path with `/ipld/` we enter data model pathing semantics and can `dag get /ipld/QmFoo/Links/0/Hash` to navigate to links or `/ipld/QmFoo/Data` to simply retrieve the data section of the node, for example.
+  * See the [dag-pb specification](https://ipld.io/specs/codecs/dag-pb/) for details on the codec and its data model representation.
+  * See this [detailed write-up](https://github.com/ipld/ipld/blob/master/design/tricky-choices/dag-pb-forms-impl-and-use.md) for further background on these changes.
+
 ## v0.9.1 2021-07-20
 
 This is a small bug fix release resolving the following issues:

core/commands/dag/dag.go

@@ -77,10 +77,10 @@ into an object of the specified format.
 		cmds.FileArg("object data", true, true, "The object to put").EnableStdin(),
 	},
 	Options: []cmds.Option{
-		cmds.StringOption("format", "f", "Format that the object will be added as.").WithDefault("cbor"),
-		cmds.StringOption("input-enc", "Format that the input object will be.").WithDefault("json"),
+		cmds.StringOption("format", "f", "Format that the object will be added as.").WithDefault("dag-cbor"),
+		cmds.StringOption("input-enc", "Format that the input object will be.").WithDefault("dag-json"),
 		cmds.BoolOption("pin", "Pin this object when adding."),
-		cmds.StringOption("hash", "Hash function to use").WithDefault(""),
+		cmds.StringOption("hash", "Hash function to use").WithDefault("sha2-256"),
 	},
 	Run:  dagPut,
 	Type: OutputObject{},
@@ -108,6 +108,9 @@ format.
 	Arguments: []cmds.Argument{
 		cmds.StringArg("ref", true, false, "The object to get").EnableStdin(),
 	},
+	Options: []cmds.Option{
+		cmds.StringOption("format", "f", "Format that the object will be serialized as.").WithDefault("dag-json"),
+	},
 	Run: dagGet,
 }
 

core/commands/dag/get.go

@@ -1,11 +1,18 @@
 package dagcmd
 
 import (
-	"strings"
+	"fmt"
+	"io"
 
 	"github.com/ipfs/go-ipfs/core/commands/cmdenv"
+	ipldlegacy "github.com/ipfs/go-ipld-legacy"
 	"github.com/ipfs/interface-go-ipfs-core/path"
 
+	"github.com/ipld/go-ipld-prime"
+	"github.com/ipld/go-ipld-prime/multicodec"
+	"github.com/ipld/go-ipld-prime/traversal"
+	mc "github.com/multiformats/go-multicodec"
+
 	cmds "github.com/ipfs/go-ipfs-cmds"
 )
 
@@ -15,6 +22,12 @@ func dagGet(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment) e
 		return err
 	}
 
+	format, _ := req.Options["format"].(string)
+	var fCodec mc.Code
+	if err := fCodec.Set(format); err != nil {
+		return err
+	}
+
 	rp, err := api.ResolvePath(req.Context, path.New(req.Arguments[0]))
 	if err != nil {
 		return err
@@ -25,14 +38,34 @@ func dagGet(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment) e
 		return err
 	}
 
-	var out interface{} = obj
+	universal, ok := obj.(ipldlegacy.UniversalNode)
+	if !ok {
+		return fmt.Errorf("%T is not a valid IPLD node", obj)
+	}
+
+	finalNode := universal.(ipld.Node)
+
 	if len(rp.Remainder()) > 0 {
-		rem := strings.Split(rp.Remainder(), "/")
-		final, _, err := obj.Resolve(rem)
+		remainderPath := ipld.ParsePath(rp.Remainder())
+
+		finalNode, err = traversal.Get(finalNode, remainderPath)
 		if err != nil {
 			return err
 		}
-		out = final
 	}
-	return cmds.EmitOnce(res, &out)
+
+	encoder, err := multicodec.LookupEncoder(uint64(fCodec))
+	if err != nil {
+		return fmt.Errorf("invalid encoding: %s - %s", format, err)
+	}
+
+	r, w := io.Pipe()
+	go func() {
+		defer w.Close()
+		if err := encoder(finalNode, w); err != nil {
+			_ = res.CloseWithError(err)
+		}
+	}()
+
+	return res.Emit(r)
 }

core/commands/dag/put.go

@@ -1,16 +1,28 @@
 package dagcmd
 
 import (
+	"bytes"
 	"fmt"
-	"math"
 
+	blocks "github.com/ipfs/go-block-format"
+	"github.com/ipfs/go-cid"
 	"github.com/ipfs/go-ipfs/core/commands/cmdenv"
-	"github.com/ipfs/go-ipfs/core/coredag"
+	ipldlegacy "github.com/ipfs/go-ipld-legacy"
+	"github.com/ipld/go-ipld-prime/multicodec"
+	basicnode "github.com/ipld/go-ipld-prime/node/basic"
 
 	cmds "github.com/ipfs/go-ipfs-cmds"
 	files "github.com/ipfs/go-ipfs-files"
 	ipld "github.com/ipfs/go-ipld-format"
-	mh "github.com/multiformats/go-multihash"
+	mc "github.com/multiformats/go-multicodec"
+
+	// Expected minimal set of available format/ienc codecs.
+	_ "github.com/ipld/go-codec-dagpb"
+	_ "github.com/ipld/go-ipld-prime/codec/cbor"
+	_ "github.com/ipld/go-ipld-prime/codec/dagcbor"
+	_ "github.com/ipld/go-ipld-prime/codec/dagjson"
+	_ "github.com/ipld/go-ipld-prime/codec/json"
+	_ "github.com/ipld/go-ipld-prime/codec/raw"
 )
 
 func dagPut(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment) error {
@@ -24,16 +36,33 @@ func dagPut(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment) e
 	hash, _ := req.Options["hash"].(string)
 	dopin, _ := req.Options["pin"].(bool)
 
-	// mhType tells inputParser which hash should be used. MaxUint64 means 'use
-	// default hash' (sha256 for cbor, sha1 for git..)
-	mhType := uint64(math.MaxUint64)
+	var icodec mc.Code
+	if err := icodec.Set(ienc); err != nil {
+		return err
+	}
+	var fcodec mc.Code
+	if err := fcodec.Set(format); err != nil {
+		return err
+	}
+	var mhType mc.Code
+	if err := mhType.Set(hash); err != nil {
+		return err
+	}
 
-	if hash != "" {
-		var ok bool
-		mhType, ok = mh.Names[hash]
-		if !ok {
-			return fmt.Errorf("%s in not a valid multihash name", hash)
-		}
+	cidPrefix := cid.Prefix{
+		Version:  1,
+		Codec:    uint64(fcodec),
+		MhType:   uint64(mhType),
+		MhLength: -1,
+	}
+
+	decoder, err := multicodec.LookupDecoder(uint64(icodec))
+	if err != nil {
+		return err
+	}
+	encoder, err := multicodec.LookupEncoder(uint64(fcodec))
+	if err != nil {
+		return err
 	}
 
 	var adder ipld.NodeAdder = api.Dag()
@@ -48,22 +77,36 @@ func dagPut(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment) e
 		if file == nil {
 			return fmt.Errorf("expected a regular file")
 		}
-		nds, err := coredag.ParseInputs(ienc, format, file, mhType, -1)
+
+		node := basicnode.Prototype.Any.NewBuilder()
+		if err := decoder(node, file); err != nil {
+			return err
+		}
+		n := node.Build()
+
+		bd := bytes.NewBuffer([]byte{})
+		if err := encoder(n, bd); err != nil {
+			return err
+		}
+
+		blockCid, err := cidPrefix.Sum(bd.Bytes())
+		if err != nil {
+			return err
+		}
+		blk, err := blocks.NewBlockWithCid(bd.Bytes(), blockCid)
 		if err != nil {
 			return err
 		}
-		if len(nds) == 0 {
-			return fmt.Errorf("no node returned from ParseInputs")
+		ln := ipldlegacy.LegacyNode{
+			Block: blk,
+			Node:  n,
 		}
 
-		for _, nd := range nds {
-			err := b.Add(req.Context, nd)
-			if err != nil {
-				return err
-			}
+		if err := b.Add(req.Context, &ln); err != nil {
+			return err
 		}
 
-		cid := nds[0].Cid()
+		cid := ln.Cid()
 		if err := res.Emit(&OutputObject{Cid: cid}); err != nil {
 			return err
 		}

core/core.go

@@ -17,6 +17,7 @@ import (
 	"github.com/ipfs/go-ipfs-pinner"
 
 	bserv "github.com/ipfs/go-blockservice"
+	"github.com/ipfs/go-fetcher"
 	"github.com/ipfs/go-graphsync"
 	bstore "github.com/ipfs/go-ipfs-blockstore"
 	exchange "github.com/ipfs/go-ipfs-exchange-interface"
@@ -78,6 +79,7 @@ type IpfsNode struct {
 	Blocks          bserv.BlockService        // the block service, get/add blocks.
 	DAG             ipld.DAGService           // the merkle dag service, get/add objects.
 	Resolver        *resolver.Resolver        // the path resolution system
+	FetcherFactory  fetcher.Factory           // an implementation of the fetcher
 	Reporter        *metrics.BandwidthCounter `optional:"true"`
 	Discovery       discovery.Service         `optional:"true"`
 	FilesRoot       *mfs.Root

core/coreapi/coreapi.go

@@ -19,14 +19,18 @@ import (
 	"fmt"
 
 	bserv "github.com/ipfs/go-blockservice"
-	"github.com/ipfs/go-ipfs-blockstore"
-	"github.com/ipfs/go-ipfs-exchange-interface"
+	"github.com/ipfs/go-dagwriter"
+	bsdagwriter "github.com/ipfs/go-dagwriter/impl/blockservice"
+	"github.com/ipfs/go-fetcher"
+	blockstore "github.com/ipfs/go-ipfs-blockstore"
+	exchange "github.com/ipfs/go-ipfs-exchange-interface"
 	offlinexch "github.com/ipfs/go-ipfs-exchange-offline"
-	"github.com/ipfs/go-ipfs-pinner"
-	"github.com/ipfs/go-ipfs-provider"
+	pin "github.com/ipfs/go-ipfs-pinner"
+	provider "github.com/ipfs/go-ipfs-provider"
 	offlineroute "github.com/ipfs/go-ipfs-routing/offline"
 	ipld "github.com/ipfs/go-ipld-format"
 	dag "github.com/ipfs/go-merkledag"
+	"github.com/ipfs/go-path/resolver"
 	coreiface "github.com/ipfs/interface-go-ipfs-core"
 	"github.com/ipfs/interface-go-ipfs-core/options"
 	ci "github.com/libp2p/go-libp2p-core/crypto"
@@ -55,9 +59,11 @@ type CoreAPI struct {
 	baseBlocks blockstore.Blockstore
 	pinning    pin.Pinner
 
-	blocks bserv.BlockService
-	dag    ipld.DAGService
-
+	blocks          bserv.BlockService
+	dag             ipld.DAGService
+	fetcherFactory  fetcher.Factory
+	dagWriter       dagwriter.DagWritingService
+	resolver        *resolver.Resolver
 	peerstore       pstore.Peerstore
 	peerHost        p2phost.Host
 	recordValidator record.Validator
@@ -107,6 +113,19 @@ func (api *CoreAPI) Dag() coreiface.APIDagService {
 	}
 }
 
+type nodeAPI struct {
+	fetcher.Factory
+	dagwriter.DagWritingService
+}
+
+// Node returns the Node interface implementation backed by the go-ipfs node
+func (api *CoreAPI) Node() coreiface.NodeAPI {
+	return &nodeAPI{
+		api.fetcherFactory,
+		api.dagWriter,
+	}
+}
+
 // Name returns the NameAPI interface implementation backed by the go-ipfs node
 func (api *CoreAPI) Name() coreiface.NameAPI {
 	return (*NameAPI)(api)
@@ -167,8 +186,10 @@ func (api *CoreAPI) WithOptions(opts ...options.ApiOption) (coreiface.CoreAPI, e
 		baseBlocks: n.BaseBlocks,
 		pinning:    n.Pinning,
 
-		blocks: n.Blocks,
-		dag:    n.DAG,
+		blocks:         n.Blocks,
+		dag:            n.DAG,
+		resolver:       n.Resolver,
+		fetcherFactory: n.FetcherFactory,
 
 		peerstore:       n.Peerstore,
 		peerHost:        n.PeerHost,
@@ -186,6 +207,8 @@ func (api *CoreAPI) WithOptions(opts ...options.ApiOption) (coreiface.CoreAPI, e
 		parentOpts: settings,
 	}
 
+	subApi.dagWriter = bsdagwriter.NewDagWriter(subApi.blocks)
+
 	subApi.checkOnline = func(allowOffline bool) error {
 		if !n.IsOnline && !allowOffline {
 			return coreiface.ErrOffline

core/coreapi/path.go

@@ -8,12 +8,12 @@ import (
 	"github.com/ipfs/go-namesys/resolve"
 
 	"github.com/ipfs/go-cid"
+	"github.com/ipfs/go-fetcher"
 	ipld "github.com/ipfs/go-ipld-format"
 	ipfspath "github.com/ipfs/go-path"
-	"github.com/ipfs/go-path/resolver"
-	uio "github.com/ipfs/go-unixfs/io"
 	coreiface "github.com/ipfs/interface-go-ipfs-core"
 	path "github.com/ipfs/interface-go-ipfs-core/path"
+	ipldprime "github.com/ipld/go-ipld-prime"
 )
 
 // ResolveNode resolves the path `p` using Unixfs resolver, gets and returns the
@@ -49,23 +49,20 @@ func (api *CoreAPI) ResolvePath(ctx context.Context, p path.Path) (path.Resolved
 		return nil, err
 	}
 
-	var resolveOnce resolver.ResolveOnce
-
-	switch ipath.Segments()[0] {
-	case "ipfs":
-		resolveOnce = uio.ResolveUnixfsOnce
-	case "ipld":
-		resolveOnce = resolver.ResolveSingle
-	default:
+	if ipath.Segments()[0] != "ipfs" && ipath.Segments()[0] != "ipld" {
 		return nil, fmt.Errorf("unsupported path namespace: %s", p.Namespace())
 	}
-
-	r := &resolver.Resolver{
-		DAG:         api.dag,
-		ResolveOnce: resolveOnce,
+	resolver := *api.resolver
+	if ipath.Segments()[0] == "ipld" {
+		type reifiable interface {
+			WithReifier(nr ipldprime.NodeReifier) fetcher.Factory
+		}
+		if bsf, ok := resolver.FetcherFactory.(reifiable); ok {
+			resolver.FetcherFactory = bsf.WithReifier(nil)
+		}
 	}
 
-	node, rest, err := r.ResolveToLastNode(ctx, ipath)
+	node, rest, err := resolver.ResolveToLastNode(ctx, ipath)
 	if err != nil {
 		return nil, err
 	}