Make badger-ds the default datastore

[Stebalien]

This is the master issue to centralize all other issues/PRs related to the Badger transition.

The priorities to check before the transition are:

• DB integrity. That means, besides minimizing data loss in case of errors such as a system crash or running out of disk space, to always keep a consistent database for Badger (and IPFS) to be able to start. Even though some truncation for example might be needed, the scenario to avoid is for Badger to encounter a DB it can't work with (e.g., a failed assertion that Badger doesn't know how to recover from) and refusing to start (which would mean IPFS would not work) without some manual interaction (which can't be expected from the normal end-user).

• Performance in worst case scenarios. We are transitioning from a flat file-system storage (one key, one file) which in most cases has a (much) lower performance than Badger, but there are some scenarios (e.g., GC or some search cases) where a flat architecture may outperform Badger (or any other LSM architecture for that matter), that should be minimized as much as possible so the end-user won't notice the transition.

The active issues (mostly the ones tagged with badger) are:

• Testing:

  • Re-run all sharness tests with badger: Add datastore tests #4549

• Integrity:

  • Expectations of ACID properties from the datastore for different error scenarios #4921: general discussion of ACID in Badger.
  • Error: reading varint: buffer too small #4511: (possible) corrupted hash in DB.
  • Limit manifest's change set size dgraph-io/badger#490: Manifest maximum changeset size.
  • os: O_SYNC not utilized in os.OpenFile() on Windows golang/go#35358: sync writes are broken on linux. Otherwise, fix Asynchronous Datastores go-datastore#137 to sidestep this issue.

• Performance:

  • Datastore benchmarks #4870: general benchmarks.
  • GC is slower with badger (with syncwrites enabled) #4298: GC - will be fixed by Asynchronous Datastores go-datastore#137 (async writes).
  • Memory usage while garbage collecting (Memory balloons when compacting go-ds-badger#86)
  • GC doesn't actually work in v1: GC: Data may remain after garbage collecting everything go-ds-badger#54 (comment)

• Usability:

  • #4824: what to expect from GC (small amounts of data -single value log file- won't be GC'ed).
  • repo fsck doesn't remove badger lockfile #4301: Windows lock file management.
  • ipfs block rm doesn't notify of a non-existing key #5040
  • Periodically garbage collect go-ds-badger#51 - We still need to run the datastore level GC even if block GC is disabled.

• Nice to have:

  • Avoid (re)converting to the same datastore type ipfs-inactive/ipfs-ds-convert#9: conversion to badger.

[schomatis]

@Stebalien Do you mind if I hijack this issue to keep track of all the other issues related to the Badger transition?

[Stebalien]

@schomatis go right ahead!

[ajbouh]

Is this still on track to happen sometime soon?

[Stebalien]

In addition to the issues listed in the description, we're still working through some recovery issues (not a bug) and memory usage is pretty bad (we may be able to tune this a bit but I'm getting some really weird behavior on Linux) (can't reproduce anymore).

Basically, we can't roll this out until:

1. We can always recover after a crash.
2. It doesn't eat ram needlessly.

[schomatis]

Thanks for the reference, I should add those.

[djdv]

I doubt this effects most users but I'm linking it anyway.
My own instance of IPFS runs with flatfs hosted on an SMB/CIFS share.
badger doesn't currently handle this: dgraph-io/badger#699
although it can.

For full context, I do this because my local disks are small. And I can't run IPFS on the remote machine because components of libp2p don't build on Solaris yet.
(when trying to port it I encountered an oddity where the Go standard library says something is implemented but it isn't)

[Kubuxu]

@djdv that is why we provide other datastore implementations and simple switches to initialise repo with different configurations.

[Stebalien]

@magik6k IMO, we should be able to graduate badger from experimental even if we don't go ahead and make it the default. However, we may want to land ipfs/go-ds-badger#51 first.

[ZerxXxes]

Hey, whats the status here? I see that ipfs/go-ds-badger#51 is closed, does that mean that the badger datastore is to be considered pretty mature now?

[Kubuxu]

We should probably update to badger v2 before using it as default.

[Stebalien]

Yes, but I don't actually think we have to. Once we fix the final memory issue (shouldn't be too hard), low power nodes should be just fine.

[dokterbob]

Within this context I would like to point out the following issue, a huge problem regarding garbage not being collected: ipfs/go-ds-badger#54 (comment)

[Stebalien]

Yep, I've added that to the list. Unfortunately, IIRC, badger v2 had its own issues so we're on to badger v3 now.

[dokterbob]

May I inform on the progress on this one?

[godcong]

GC doesn't actually work in v1: ipfs/go-ds-badger#54 (comment)

v1 seems to have stopped updating, and no one will probably ever address it in this entry.
Isn't it better to try another plan, like using v2 or v3 or something.

[guseggert]

We will not make v1 the default as it is unmaintained, and v2 has issues as @Stebalien pointed out. DGraph, the company that maintains BadgerDB, is undergoing a leadership shake-up, so we're hesitant to make v3 the default until we are confident that v3 will be maintained in the long run.

[dokterbob]

@guseggert Thanks for the update. Any news on this, by now?
Are you looking into alternative datastores?

[Jorropo]

Are you looking into alternative datastores?

flatfs

I want to write an LSM datastore with reflinking, but I wont work on this before #8201 is a thing.