[R4R]txindex/kv: Fsync data to disk immediately after receiving it

[unclezoro]

Why this pr:
When restarting chain node, sometimes we lost tx index about recent(around 80)blocks, and some client complains that they can't find the tx by RPC call(tx_search) when the tx do exist in the block.

I try to partially fix this issue in a simple way by writing the index data in a sync way.
There is no performance difference under 1K TPS according to our test.

It is still possible that lost index data after restarting the node, but only 2 block data will lost at most.
I try to totally fix this in https://github.com/tendermint/tendermint/pull/3847/files, but this one is simple and can solve most part of the issue. Please review first, thks.

[melekes]

Suggested change

	- [index] [\#4104](https://github.com/tendermint/tendermint/pull/4104) Fix tx index lag too much
	- [state] [\#4104](https://github.com/tendermint/tendermint/pull/4104) txindex/kv: Fsync data to disk immediately after receiving it (@guagualvcha)

[melekes]

BEFORE:

BenchmarkTxIndex1-2               100000             12434 ns/op
BenchmarkTxIndex500-2                300           5151564 ns/op
BenchmarkTxIndex1000-2               100          15053910 ns/op
BenchmarkTxIndex2000-2               100          18238892 ns/op
BenchmarkTxIndex10000-2               20         124287930 ns/op

AFTER:

BenchmarkTxIndex1-2                 2000            795431 ns/op
BenchmarkTxIndex500-2                200           6385124 ns/op
BenchmarkTxIndex1000-2               100          11388219 ns/op
BenchmarkTxIndex2000-2               100          20514873 ns/op
BenchmarkTxIndex10000-2               20         107456004 ns/op

Performance drop is pretty steep, but I think it's the right thing to do UNTIL we have a WAL.

[ebuchman]

@melekes what is that benchmark on ? Individual Index calls or AddBatch calls? The former we don't seem to actually use other than in tests, the later I would expect much lesser performance hit from.

Regardless this is an important fix - thanks @guagualvcha . Seems also independent from #3211 which we'll need to address as well.

[melekes]

@melekes what is that benchmark on ? Individual Index calls or AddBatch calls?

AddBatch calls

tendermint/state/txindex/kv/kv_test.go

Lines 314 to 353 in 97222eb

	func benchmarkTxIndex(txsCount int64, b *testing.B) {
	dir, err := ioutil.TempDir("", "tx_index_db")
	if err != nil {
	b.Fatal(err)
	}
	defer os.RemoveAll(dir) // nolint: errcheck
	store := db.NewDB("tx_index", "goleveldb", dir)
	indexer := NewTxIndex(store)
	batch := txindex.NewBatch(txsCount)
	txIndex := uint32(0)
	for i := int64(0); i < txsCount; i++ {
	tx := cmn.RandBytes(250)
	txResult := &types.TxResult{
	Height: 1,
	Index: txIndex,
	Tx: tx,
	Result: abci.ResponseDeliverTx{
	Data: []byte{0},
	Code: abci.CodeTypeOK,
	Log: "",
	Events: []abci.Event{},
	},
	}
	if err := batch.Add(txResult); err != nil {
	b.Fatal(err)
	}
	txIndex++
	}
	b.ResetTimer()
	for n := 0; n < b.N; n++ {
	err = indexer.AddBatch(batch)
	}
	if err != nil {
	b.Fatal(err)
	}
	}

[ebuchman]

Hmm. Is the first number in BenchmarkTxIndex2000-2 the number of txs? The cost is pretty high for the smaller nums but seems it's amortized for larger ones so maybe it's ok. Crazy that the cost is like 70x for the first one and otherwise only around ~1.2x or so for the others

[melekes]

Is the first number in BenchmarkTxIndex2000-2 the number of txs?

Yes