Seeding compaction commit and split fate transactions is slow

[keith-turner]

Describe the bug

While testing splitting lots of tablets it was observed that seeding the split fate transactions was the bottleneck and not running the actual fate operation. Looking at the code there is a lot of back and forth to the seed a fate transaction

1. Do conditional mutation reserve
2. Do a scan to see if there is a top on the stack
3. Do a conditional mutation to set the transaction info
4. Do a conditional mutation to set the transaction info
5. Do a conditional mutation to set the status

All of this could likely be done in a single conditional mutation. This would avoid the scan and doing multiple conditional mutations. Observed this while looking into splits, but improving this would speed up compaction commit and split.

Expected behavior
If possible a single conditional mutation is run to seed a fate transaction.

Additional context

There may be other places in the code that are submitting multiple conditional mutations for fate operations when one would suffice. This is caused by the code evolving from initially having a zookeeper persistent store where data was stored in multiple zk nodes for a single fate operation. Multiple ZK nodes require multiple write. Implementing #4905 could allow zookeeper and accumuo table impls to have less writes for fate updates by improving the code for both to support doing more single writes.

[keith-turner]

When experimenting with this was seeing that a single thread could seed around 1.3 fate operations per second. Making this a single conditional mutation should increase that rate significantly.

[kevinrr888]

Interesting, yeah the number of conditional mutations could definitely be decreased here and maybe elsewhere too...
I can take a look at this when I get the chance (unless someone else wants to pick this up before then, which is also fine).

[keith-turner]

Interesting, yeah the number of conditional mutations could definitely be decreased here and maybe elsewhere too...
I can take a look at this when I get the chance (unless someone else wants to pick this up before then, which is also fine).

I just saw this comment and I had started working on this earlier today. I hope to have a PR up tomorrow. Also I have realized there are follow on changes that could further speed some things up, plan to open follow on issues up after posting that PR.

[kevinrr888]

Sounds good, I can take a look at the PR