A fully offline, CAR-centric js-ipfs node

[ikreymer]

Description:

I'm wondering if it'd be possible to support a fully offline, no-network, in-memory instance of js-ipfs that is designed to be used to only import, export CAR files, or add data directly, while supporting existing apis (or at least the workable subset).

I guess this functionality is sort of provided by web3-storage/ipfs-car, but the interface there is fairly different.

The goal would be to provide similar functionality, but with the standard ipfs apis, as well as ability to specify a custom blockstore from web3-storage/ipfs-car blockstores.

Could look something like:

const ipfs = await IPFS.create({"offline": true, blockstore: new MemoryBlockStore()})
...
ipfs.add(...)
...
ipfs.dag.put(...)
...
// export
const CAR = await ipfs.dag.export(...)

Probably at least the ipfs.files, ipfs.dag, ipfs.object apis could be supported?

This would allow for switching between an offline, CAR-only environment, and a 'live' environment, w/o having to use two different sets of APIs / systems.

Is this something that might be feasible in some way (or not?), or perhaps alternatively, something that could be built on top of web3-storage/ipfs-car itself. (If that repo is better place for this, feel free to move there!)

[achingbrain]

This should be possible, yes.

Instead of specifying a blockstore argument as above, you should check out the js-ipfs-custom-repo example. Instead of using FsDatastore everywhere for the root, blocks etc datastore as per the example, use MemoryDatastore from datastore-core instead.

[github-actions]

Oops, seems like we needed more information for this issue, please comment with more details or this issue will be closed in 7 days.

[ikreymer]

Thanks @achingbrain !

It worked! Based on that example I was able to create the following:

import { createRepo } from "ipfs-repo";

import { MemoryLock } from "ipfs-repo/locks/memory";
import { MemoryDatastore } from "datastore-core";
import { MemoryBlockstore } from "blockstore-core";

import * as dagPb from "@ipld/dag-pb";
import * as dagCbor from "@ipld/dag-cbor";
import * as raw from "multiformats/codecs/raw";

// multiformat codecs to support
const codecs = [dagPb, dagCbor, raw].reduce((acc, curr) => {
  acc[curr.name] = curr;
  acc[curr.code] = curr;
  return acc;
}, {});

export async function createInMemoryRepo() {
  const loadCodec = (nameOrCode) => {
    if (codecs[nameOrCode]) {
      return codecs[nameOrCode];
    }

    throw new Error(`Could not load codec for ${nameOrCode}`);
  };

  const repo = createRepo(
    "",
    loadCodec,
    {
      root: new MemoryDatastore(),

      blocks: new MemoryBlockstore(),

      keys: new MemoryDatastore(),
      datastore: new MemoryDatastore(),
      pins: new MemoryDatastore(),
    },
    {
      lock: MemoryLock,
      autoMigrate: false,
      repoOwner: true,
    }
  );

  return {
    repo,
    init: {
      emptyRepo: true
    },

    offline: true,
    start: false,

    config: {
      Bootstrap: [],
      Addresses: {},
      Discovery: {},
    },

    libp2p: {
      nat: {
        enabled: false,
      },
    },
  };
}

which can then be passed in to await IPFS.create(await createInMemoryRepo());

I think it'd be really neat if js-ipfs offered createInMemoryRepo() as a utility function, as it is a lot of boilerplate, but it seems to working well!

[BigLep]

@ikreymer: great to hear that you were successful here. This isn't functionality we want to expose in js-ipfs itself, but we're glad you were able to find utility. Feel free to post/share on discuss.ipfs.tech for others to be aware of your solution.

[ikreymer]

Actually, after testing in the browser, it seems the repo (in the above example) is not fully 'offline', eg. it tries to make connection to the preload nodes (https://node0.preload.ipfs.io/api/v0/refs...), despite also adding offline: true, and

config: {
      Bootstrap: [],
      Addresses: {},
      Discovery: {},
 }

Is there anything else that's missing? perhaps something in libp2p to make it truly offline and not connect anywhere?

[ikreymer]

It looks like this is also in some ways a duplicate of:
#2751 (comment) (though w/o custom codecs).

It looks like that issue was in relation to https://github.com/mikeal/ipfs-for-car from @mikeal, which appears to no longer be used.

Also don't want to be reinventing the wheel, what I was hoping for was basically ipfs-for-car, but wonder if it was abandoned for certain reasons?