Make js-ipfs capable of working with cidv1b32

[kyledrake]

js-ipfs needs to be able to work with cidv1b32 addresses, for both creation and retrieval.

• Ability to set option in new IPFS({}) or as option to commands like ipfs add to output base32 cidv1
• Refactor the current blockstore to use multihashes for keys

Eventually we want to do a synchronized release with go-ipfs and default to cidv1b32, but for now let's start with getting the basic support in

https://github.com/ipfs/ipfs/issues/337

@alanshaw

[alanshaw]

My rough plan for this is:

1. ASAP
   • Allow add to IPFS and get base32 CIDv1 back via an option to specify base encoding (defaults do not change)
   • Retrieve content by CIDv1 key also checks for content by CIDv0 key and vice versa (compatibility)
   • Default string encoding for v1 CIDs is base32
2. Next
   • Migration of Blockstore to use multihash instead of CID as key
   • DHT provider records use multihash instead of CID
   • Change defaults for adding content to CIDv1
3. Future
   • Ad-hoc migration - request with CIDv0 key causes copy to multihash key

For the first part of 1 there will need to be changes to multiple APIs to add a --cid-base option (or similar name - I've had no resistance to this so far) that allows the output to be encoded using base32. e.g. files.add, files.stat, files.ls, dag.put...I will post a definitive list when I have gone through them.

For the second part of 1 the local checks might be easy enough to implement. In terms of finding content that isn't local, I'd assume we'd wantlist both versions of the CID, and race them. When we get a response we'd unwant the other...? Something like that.

I will write interop tests for the second part of 1 to ensure js-ipfs and go-ipfs can get content from each other using different CID versions. I'll also PR interface-ipfs-core to document the --cid-base option in the relevant commands.

Release sync we need to sync up 2 js-ipfs/go-ipfs releases for steps 1 and 2 so that when IPFS gets these new super powers we can all still fetch content from each other. Happyness ensues and double rainbows appear.

Does this sound reasonable? Please leave your thoughts, feedback or just +1 if all good!

@whyrusleeping @Stebalien @kevina @magik6k @diasdavid @achingbrain

[richardschneider]

I don't like cid-base because its the first query string argument name (HTTP-API) with a dash (I think). There is already mhtype for multihash type. How about cbtype for cid base type,

[achingbrain]

I think --cid-base is fine, it's descriptive and fits with the --cid-version arg we already have in a few places.

[kevina]

For the go-ipfs side of (1) see ipfs/kubo#5285.

[kyledrake]

@alanshaw Is there a different meta issue for this other than this issue? Was wondering if there were any updates here this issue is missing.

[alanshaw]

No this is it. Updates are:

• https://github.com/ipfs-shipyard/js-cid-tool exists, can be installed as a standalone tool. PR for integrating with js-ipfs is here feat: add cid command #1560
• --cid-base option PR is now here feat: cid base option #1552 - is currently still WIP

I haven't been able to work on anything more 😢

[daviddias]

Congratz on #1560 and releasing it! :)

[alanshaw]

Update:

The CID tool was shipped with JS IPFS 0.33

JS IPFS 0.34 will include the two items from (1) ASAP:

• Allow add to IPFS and get base32 CIDv1 back via an option to specify base encoding (defaults do not change) feat: cid base option #1552
• Retrieve content by CIDv1 key also checks for content by CIDv0 key and vice versa (compatibility) feat: cid agnostic blockstore .get and .has js-ipfs-repo#184

[alanshaw]

Tracking PRs for (2):

• refactor: base32 encode CIDs by default multiformats/js-cid#73
• refactor: default to CID v1 and encode with base32 ipld/js-ipld#193
• refactor: default to CID v1 and encode with base32 ipld/js-ipld-bitcoin#43
• [WIP] refactor: use CIDv1 by default ipld/js-ipld-dag-pb#115
• refactor: default to CID v1 and encode with base32 ipld/js-ipld-dag-cbor#96
• refactor: default to CID v1 and encode with base32 ipld/js-ipld-ethereum#47
• refactor: default to CID v1 and encode with base32 ipld/js-ipld-git#42
• refactor: default to CID v1 and encode with base32 ipld/js-ipld-raw#28
• refactor: default to CID v1 and encode with base32 ipld/js-ipld-zcash#35
• [WIP] refactor: use CID v1 with base32 encoding by default ipfs-inactive/js-ipfs-unixfs-importer#21
• [WIP] refactor: base32 encoded CID v1 by default ipfs-inactive/js-ipfs-unixfs-engine#235
• [WIP] refactor: use CIDv1 and encode with base32 by default ipfs-inactive/js-ipfs-mfs#38
• [WIP] refactor: default to CID v1 and encode with base32 #1868
• refactor: default to CID v1 and encode with base32 multiformats/js-cid-tool#7
• refactor: default to CID v1 and encode with base32 js-ipfs-bitswap#188
• refactor: default to CID v1 and encode with base32 ipld/js-ipld-block#44
• [WIP] refactor: default to CID v1 and encode with base32 js-ipfs-http-response#17
• refactor: default to CID v1 and encode with base32 js-ipfs-repo#192
• refactor: tests for CIDv1 base32 ipfs-inactive/interface-js-ipfs-core#444
• refactor: default to CID v1 and encode with base32 ipfs-inactive/js-ipfs-http-client#949
• refactor: default to CID v1 and encode with base32 ipfs-shipyard/is-ipfs#29
• refactor: default to CID v1 and encode with base32 libp2p/js-libp2p-kad-dht#84