Ipfs repo created by cli neither compatible with go-ipfs nor js-ipfs

[maxkerp]

What I'm trying to accomplish:

I want to examine single ipfs objects created by orbit-db-cli with go-ipfs or js-ipfs. Given a <hash> from adding to a feed-store with orbitdb add <address> 'Hello World!', then I want to ipfs object get <hash> or jsipfs object get <hash> respectively.

This didn't work at all in the last couple of days, but now I'm at the point that I can make sense of the error messages.

What I think is wrong

The way orbit-db-cli sets up the ipfs repo is somewhat off, especially the /blocks subdirectory .

Take a look at my global ~/.ipfs repo created by go-ipfs init

~/.ipfs on machine
➜ ls blocks
000018.ldb  BE  CQ       L5                 QR       RY        
000020.ldb  BG  CR       L7                 QU       S7        
000021.log  BI  CS       LA                 QV       SA        
23          BJ  CURRENT  LD                 QW       SC        
24          BN  CV       LG                 QX       SD        
25          BO  CY       LH                 R2       SE        
26          BQ  CZ       LJ                 R3       SG        
27          BS  D2       LK                 R4       SH        
2D          BT  D4       LN                 RB       SHARDING  
2E          BV  D6       LO                 RC       SI        
2F          BZ  D7       LOCK               _README  SJ        
2G          C7  DD       LOG                RF       SK        
2H          CA  DE       LOG.old            RG       SM        
2I          CC  DF       LP                 RI       SN        
2L          CF  DH       LS                 RJ       SO        
2M          CG  DL       LT                 RN       SR        
2P          CH  DM       LV                 RP       SS        
2W          CI  DO       LZ                 RQ       SU        
2Z          CJ  DP       M2                 RS       SV        
32          CL  DS       M4                 RV       SW        
35          CP  DU       MANIFEST-000019    RW       SY        

Now take a look at the ipfs repo created by orbit-cb-cli:

➜ ls ./orbitdb/ipfs/blocks
000005.ldb  000008.ldb  000009.log  CURRENT  LOCK  LOG  LOG.old  MANIFEST-000007  _README  SHARDING

And the following was created by a node script using the orbit-db module.

➜ ls ipfs_repo_by_node/blocks
7J  D2  I7  IL  K5  LG  M4  OO  PJ  PU  QP  R3  _README  S2  SHARDING  TD  X3  ZS

How I came to that conclusion

Examining the orbit-db-cli ipfs repo yields following results:

IPFS/experiments/local_orbitdb on machine
➜ IPFS_PATH=./orbitdb/ipfs ipfs object get QmRgyjgbNgDfxt3KQw4b1cM7ZZ9EgLdfmLd2h7m2gN5SVW
Error: required Datastore.Spec entry missing form config file

and

➜ IPFS_PATH=./orbitdb/ipfs jsipfs object get QmRgyjgbNgDfxt3KQw4b1cM7ZZ9EgLdfmLd2h7m2gN5SVW
/home/make/.config/yarn/global/node_modules/ipfs/src/cli/commands/object/get.js:15
        throw err
                ^

                Error: ENOENT: no such file or directory, open '/home/make/College/IPFS/experiments/local_orbitdb/orbitdb/ipfs/blocks/HC/CIQDDSBQ5HQ2AGXT3KOQVS5E3MN3LWYIBYP2F4BENYHJ2PWFKKBFHCI.data'

The js-ipfs implementation tells me, that there need to be the sharded directories and files under the /blocks directory, which is not the case when using orbit-db-cli.
If I use a node script to create an orbitdb database it creates the ipfs repo correctly with the sharded directories under ./<ipfs repo>/blocks and examining the ipfs objects works perfectly fine with jsipfs object get <hash>.

Notes

It seems that the go-ipfs implementation:

1. Expects some config parameter that the current js implementation doesn't care about.
2. Creates a directory called /keystore instead of /keys under ./<ipfs repo>. (Has nothing to do with this issue, but maybe that'll be important latter? 😅 )

Whatever module creates the ipfs repo for orbit-db-cli it does not create it how it is expected.

If you use the tool ldb and look at the /blocks directory of orbit-db-cli ipfs repo you will find keys in leveldb what the /blocks directory should have looked like.

IPFS/experiments/local_orbitdb on machine
➜ cd orbitdb/ipfs/blocks

orbitdb/ipfs/blocks on machine
➜ ldb ./
>l
LIMIT set to 1000
>ls
/6T/CIQNNDI62MKDAM5ZVTK2TEINMYYU4HSC74RS3XZLBS2...           /7J/CIQKKLBWAIBQZOIS5X7E32LQAL6236OUKZTMHPQSFIX...
/AL/CIQKXYLYLJ7WHZXF6NX6V45FWAS5U2LQ2K5CU2JWHXO...           /FJ/CIQPWOVV72FZ65H6T6E4SONMKZB4NVBPNHNCHYB2U2Q...
/FR/CIQNTIIUTCQT6OSZS7OKCWEGWJLZW3KOWATDAID3PG4...           /HC/CIQDDSBQ5HQ2AGXT3KOQVS5E3MN3LWYIBYP2F4BENYH...
/IC/CIQIU7KROHOI7JWOGFON3GBBT6PQDJI5AZP7BJG54M7...           /IL/CIQJFGRQHQ45VCQLM7AJNF2GF5UHUAGGHC6LLAH6VYD...
/IV/CIQOWERFZ2KLTU2RN2D2CQKWCCLL3M3BNFRTL3UEENY...           /K5/CIQPW4MAGTUNEBGZCEFZU7XAJL2BMIHGVB5ZR2IOKOS...
/LG/CIQJBQD2O6K4CGJVCCTJNUP57QHR4SKHZ74OIITBBGL...           /M4/CIQOLBQZSZAODJGGH6RYYVBUXHTS3SM5EORZDU63LYP...
/OC/CIQOTRIFYF2I2WQAF6MLWIH3ASSUOU3PI644TXSTA5T...           /OO/CIQBT4N7PS5IZ5IG2ZOUGKFK27IE33WKGJNDW2TY3LS...
/PD/CIQHWF2DR43OINVHEUB5NZ2BVZMCSFOPNPLP4XMATMT...           /PJ/CIQB4F7VKKQDXHMXX6WYQZTRR5QVLP7VBQYAYW2Y5BA...
/PU/CIQJF2E6OPOEYYEVHYML4UD5A3R4QRAVBI7NVH3PL64...           /QP/CIQNLGENZXNRUMUHZYGPPLZNZOMHHZVIU76LCD5GF5D...
/R3/CIQBED3K6YA5I3QQWLJOCHWXDRK5EXZQILBCKAPEDUJ...           /S2/CIQPF3CHDB5GQ5ZBISOV2GWVMLAJPVEUMDMFKJZE7CM...
/SHARDING                                                    /X3/CIQFTFEEHEDF6KLBT32BFAGLXEZL4UWFNWM4LFTLMXQ...
/_README
>

Solution

So far I don't know what really causes this and how to fix it, but I might be able to find the cause of the problem on Saturday. I hope this helps anyone playing around with orbitdb. A workaround is to use node scripts to create orbitdbs and use js-ipfs to examine the ipfs objects.

[maxkerp]

Okay long story short, I think since orbit-db-cli is only used with node and not in the browser datastore-fs should be used here instead of datastore-level:

https://github.com/orbitdb/orbit-db-cli/blob/a08b98c5e392c62ac03a597caab31feb93b67ed3/src/start-ipfs.js#L5-L11

I modified my fork to do this and it works just as expected with js-ipfs. go-ipfs still complains about the Datastore.Spec missing in the config of the ipfs repo, but this is probably an issue for ipfs/js-ipfs-repo.

Is there a specific reason why datastore-level is used @haadcode ?
I base my assumptions on https://github.com/ipfs/js-ipfs-repo/blob/master/README.md#background
Deleting lines 5-11 and omitting the config object when calling new IPFSRepo causes ipfs-repo to use datastore-fs by default and makes it possible to jsipfs object get <manifest hash>.