Rework on KAFKA-3968: fsync the parent directory of a segment file when the file is created

[ccding]

(reverted #10405). #10405 has several issues, for example:

• It fails to create a topic with 9000 partitions.
• It flushes in several unnecessary places.
• If multiple segments of the same partition are flushed at roughly the same time, we may end up doing multiple unnecessary flushes: the logic of handling the flush in LogSegments.scala is weird.

Kafka does not call fsync() on the directory when a new log segment is created and flushed to disk.

The problem is that following sequence of calls doesn't guarantee file durability:

fd = open("log", O_RDWR | O_CREATE); // suppose open creates "log"
write(fd);
fsync(fd);

If the system crashes after fsync() but before the parent directory has been flushed to disk, the log file can disappear.

This PR is to flush the directory when flush() is called for the first time.

Did performance test which shows this PR has a minimal performance impact on Kafka clusters.

Committer Checklist (excluded from commit message)

• Verify design and implementation
• Verify test coverage and CI build status
• Verify documentation (including upgrade notes)

[ccding]

I finished performance tests and this PR has minimal impact on performance. It can successfully create a topic with 9000 partitions. @junrao please take a look

cc @ijuma

[ccding]

The two failed tests are for Scala 2.13, but not Scala 2.12.

Both passed on my local run with

./gradlew -PscalaVersion=2.13 core:test --tests kafka.server.RaftClusterTest.testCreateClusterAndCreateAndManyTopics
./gradlew -PscalaVersion=2.13 core:test --tests kafka.server.RaftClusterTest.testCreateClusterAndCreateListDeleteTopic

[junrao]

@ccding : Thanks for the PR. LGTM