Local storage revamp phase 1

[andrwng]

The high level goal of this first part is to have a fully encapsulated interface between storage and everything else that's usable as a Raft log implementation (no housekeeping yet, but at least usable by Raft and archival).

RFC: https://docs.google.com/document/d/1b5UCT5DenW0TOTARgxATYhjVDzYoxq_lhSvjqrAVWpQ/edit?usp=sharing

Interface changes to existing log

Preview Give feedback

1. Implement the fill_gaps option to have the storage layer own ghost batches
2. Move the offset translator state into disk_log_impl #16812
   +0
   
3. Implement the translate_offsets option of the log_reader to have the storage layer own offset translation

Write IO path

Preview Give feedback

1. Segment appender append batches #17556
   +0
2. Segment appender roll onto the next segment' #18255
   +0
3. Segment appender truncate entries #18256
   +0
4. Close segment #18265
   +0

Read IO path

Preview Give feedback

1. Skipping file stream implementation #17256
   +0
   
2. Segment entry reader as a building block for record_batch_reader #17255
   +0
   
3. Reader segment set / reader snapshot implementation #18257
   +0
4. Reader option to collect the size #18260
   +0

Write path in-memory indexing

Preview Give feedback

1. Truncation: add to per-segment gap list and truncate the segment's positional index #18258
   +0
2. Terms indexing: add a list of new terms within a given segment #18261
   +0

Read path in-memory indexing

Preview Give feedback

1. Gap collection from index from a given segment reader and snapshot #18259
   +0
2. Terms indexing: be able to serve term-based queries from Kafka and Raft #18262
   +0

End-to-end testing

Preview Give feedback

1. Storage-only fuzz test #18263
   +0
2. Initial benchmarking against disk_log_impl #18264
   +0

JIRA Link: CORE-1838