Closing streams cost a lot during shutdown

[Chillax-0v0]

part of AutoMQ/automq#586

normally:

[2023-12-22 08:46:06,603] INFO start to upload delta WAL, blockId=100 (com.automq.stream.s3.S3Storage)
[2023-12-22 08:46:06,603] INFO backgroundExecutor: uploadDeltaWAL start, blockId=100 (com.automq.stream.s3.S3Storage)
[2023-12-22 08:46:06,610] INFO backgroundExecutor: uploadDeltaWAL end, blockId=100 (com.automq.stream.s3.S3Storage)
[2023-12-22 08:46:06,611] INFO backgroundExecutor: prepareDeltaWALUpload start, blockId=100 (com.automq.stream.s3.S3Storage)
[2023-12-22 08:46:06,615] INFO backgroundExecutor: prepareDeltaWALUpload end, blockId=100 (com.automq.stream.s3.S3Storage)
[2023-12-22 08:46:14,035] INFO Upload delta WAL CommitStreamSetObjectRequest{objectId=99, orderId=99, objectSize=537515395, streamRanges=[...], streamObjects=null, compactedObjectIds=null}, cost 7426ms, rate limiter 1.073877112E8bytes/s (com.automq.stream.s3.DeltaWALUploadTask)
[2023-12-22 08:46:14,039] INFO backgroundExecutor: commitDeltaWALUpload start, blockId=100 (com.automq.stream.s3.S3Storage)
[2023-12-22 08:46:14,039] INFO upload delta WAL done, blockId=100, cost=7436ms (com.automq.stream.s3.S3Storage)
[2023-12-22 08:46:14,039] INFO backgroundExecutor: commitDeltaWALUpload end, blockId=100 (com.automq.stream.s3.S3Storage)

the first call of forceUpload:

[2023-12-22 08:51:17,001] INFO start to upload delta WAL, blockId=140 (com.automq.stream.s3.S3Storage)
[2023-12-22 08:51:17,002] INFO backgroundExecutor: uploadDeltaWAL start, blockId=140 (com.automq.stream.s3.S3Storage)
[2023-12-22 08:51:17,003] INFO backgroundExecutor: uploadDeltaWAL end, blockId=140 (com.automq.stream.s3.S3Storage)
[2023-12-22 08:51:17,005] INFO backgroundExecutor: prepareDeltaWALUpload start, blockId=140 (com.automq.stream.s3.S3Storage)
[2023-12-22 08:51:17,006] INFO backgroundExecutor: prepareDeltaWALUpload end, blockId=140 (com.automq.stream.s3.S3Storage)
[2023-12-22 08:51:25,282] INFO Upload delta WAL CommitStreamSetObjectRequest{objectId=139, orderId=139, objectSize=537571989, streamRanges=[...], streamObjects=null, compactedObjectIds=null}, cost 8278ms, rate limiter 1.073882622E8bytes/s (com.automq.stream.s3.DeltaWALUploadTask)
[2023-12-22 08:51:25,285] INFO backgroundExecutor: commitDeltaWALUpload start, blockId=140 (com.automq.stream.s3.S3Storage)
[2023-12-22 08:51:27,797] INFO upload delta WAL done, blockId=140, cost=10795ms (com.automq.stream.s3.S3Storage)
[2023-12-22 08:51:27,798] INFO backgroundExecutor: commitDeltaWALUpload end, blockId=140 (com.automq.stream.s3.S3Storage)

one of following calls of forceUpload:

[2023-12-22 08:52:11,433] INFO start to upload delta WAL, blockId=171 (com.automq.stream.s3.S3Storage)
[2023-12-22 08:52:13,588] INFO backgroundExecutor: uploadDeltaWAL start, blockId=171 (com.automq.stream.s3.S3Storage)
[2023-12-22 08:52:13,589] INFO backgroundExecutor: uploadDeltaWAL end, blockId=171 (com.automq.stream.s3.S3Storage)
[2023-12-22 08:52:13,635] INFO backgroundExecutor: prepareDeltaWALUpload start, blockId=171 (com.automq.stream.s3.S3Storage)
[2023-12-22 08:52:13,638] INFO backgroundExecutor: prepareDeltaWALUpload end, blockId=171 (com.automq.stream.s3.S3Storage)
[2023-12-22 08:52:14,164] INFO Upload delta WAL CommitStreamSetObjectRequest{objectId=170, orderId=170, objectSize=269524, streamRanges=[...], streamObjects=null, compactedObjectIds=null}, cost 575ms, rate limiter 1.073882622E8bytes/s (com.automq.stream.s3.DeltaWALUploadTask)
[2023-12-22 08:52:14,164] INFO backgroundExecutor: commitDeltaWALUpload start, blockId=171 (com.automq.stream.s3.S3Storage)
[2023-12-22 08:52:15,712] INFO upload delta WAL done, blockId=171, cost=4279ms (com.automq.stream.s3.S3Storage)
[2023-12-22 08:52:15,712] INFO backgroundExecutor: commitDeltaWALUpload end, blockId=171 (com.automq.stream.s3.S3Storage)

[Chillax-0v0]

After this issue, it took 17s to close 5000 partitions, down from 45s.