db: add experimental DB.RegisterFlushCompletedCallback

[sumeerbhola]

This will be used in CockroachDB's loosely coupled raft log
truncation (see
https://github.com/cockroachdb/cockroach/blob/c5d146b536aaf70f0b91aa38f88d62b967ac8411/pkg/kv/kvserver/raft_log_queue.go#L959-L967
in a WIP PR).

This is done via a registration function instead of adding
to the Options passed to Open since it simplifies integration
in CockroachDB -- kvserver.NewStore is given an already open
DB.

This interface is subject to change, hence it is marked
experimental with a warning regarding usage.

[cockroach-teamcity]

This change is 

[nicktrav]

Reviewed 3 of 3 files at r1, all commit messages.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on @sumeerbhola)

[nicktrav]

Reviewed 1 of 1 files at r2, all commit messages.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on @sumeerbhola)

[sumeerbhola]

TFTR!

[petermattis]

Reviewable status: all files reviewed, 1 unresolved discussion

---

compaction.go, line 1353 at r2 (raw file):

//
// Only a single callback func can be registered. Repeated calls will replace
// the previous registered callback.

Rather than a new callback, could EventListener.FlushEnd provide the same signal?

[sumeerbhola]

Reviewable status: all files reviewed, 1 unresolved discussion

---

compaction.go, line 1353 at r2 (raw file):

Previously, petermattis (Peter Mattis) wrote…

Rather than a new callback, could EventListener.FlushEnd provide the same signal?

I missed this comment earlier. I think we could with an extra level of indirection in CockroachDB -- we won't have the real callback when creating the DB, so we will need to have the real callback be later registered in the Pebble struct.
It will be called while holding locks, but I don't think that is a big deal.

[sumeerbhola]

Reviewable status: all files reviewed, 1 unresolved discussion

---

compaction.go, line 1353 at r2 (raw file):

Previously, sumeerbhola wrote…

I missed this comment earlier. I think we could with an extra level of indirection in CockroachDB -- we won't have the real callback when creating the DB, so we will need to have the real callback be later registered in the Pebble struct.
It will be called while holding locks, but I don't think that is a big deal.

I'll use FlushEnd in the forthcoming CockroachDB change, so I can rollback this change. Thanks for the suggestion.