Txn is too big to fit into one request

[rof20004]

I am trying to insert one million register but I got this error: Txn is too big to fit into one request

How can I solve this?

This is my code.

package main

import (
	"log"
	"strconv"

	"github.com/dgraph-io/badger"
)

var database = &badger.DB{}

func initDB() {
	// Open the Badger database located in the /tmp/badger directory.
	// It will be created if it doesn't exist.
	opts := badger.DefaultOptions
	opts.Dir = "./badger"
	opts.ValueDir = "./badger"
	db, err := badger.Open(opts)
	if err != nil {
		log.Fatal(err)
	}
	database = db
}

func main() {
	initDB()

	log.Println("Writing to database...")

	database.Update(func(txn *badger.Txn) error {
		for i := 0; i < 1000000; i++ {
			key := "Character" + strconv.Itoa(i)
			value := strconv.Itoa(i)
			err := txn.Set([]byte(key), []byte(value))
			if err != nil {
				log.Fatalln(err)
			}
		}
		return nil
	})
}

[manishrjain]

You can stop adding more keys and just commit the transaction. A single txn can't be too big, which is what the error is indicating.

[rof20004]

@manishrjain Thanks and sorry.

[manishrjain]

No worries, mate!

[AaronVasquez]

I'm running into this issue too. What is the recommended txn size?

[manishrjain]

Badger would tell you when the txn size is too big by returning that error. In general, a txn should not exceed the size of a single memtable.

[leighmcculloch]

What's the right way to determine that error has occurred? From what I can see the error returned is errors.fundamental which isn't exported, so there's no way of doing if err == badger.ErrTxnTooBig to make a programatic decision to try again.

[manishrjain]

badger/errors.go

Line 38 in a6f0866

ErrTxnTooBig = errors.New("Txn is too big to fit into one request")

This is the error that gets returned. It is exported. Here's the documentation around retrying:

https://github.com/dgraph-io/badger#read-write-transactions

[leighmcculloch]

Ah, thank you!!!