You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I have confirmed that this should be classified as an enhancement rather than a bug/feature.
Summary
改进和修复分级存储模块相关的几个小问题
Motivation
改进和修复分级存储模块
Describe the Solution You'd Like
File Deletion Policy: Shortly after a broker starts, the data retention time in tiered storage will be shorter than that in local storage. A scheduled task will delete the data in tiered storage and retransmit it. While this does not pose a correctness issue, the retransmission wastes computing resources and bandwidth.
File Rolling Policy: The system supports configuring the minimum size for retained files. For the rolling of flat files, the default requirement is that the time exceeds one day and the file size is greater than 16 MB.
Index Module: The index supports forced uploading of the last file, and the init logic has been moved out of the constructor.
Remove invalid configuration items which only exist in old versions.
During pop consumption, the revive process generates a large number of random reads, reducing the amount of message count retrieved through prefetch.
Before Creating the Enhancement Request
Summary
改进和修复分级存储模块相关的几个小问题
Motivation
改进和修复分级存储模块
Describe the Solution You'd Like
File Deletion Policy: Shortly after a broker starts, the data retention time in tiered storage will be shorter than that in local storage. A scheduled task will delete the data in tiered storage and retransmit it. While this does not pose a correctness issue, the retransmission wastes computing resources and bandwidth.
File Rolling Policy: The system supports configuring the minimum size for retained files. For the rolling of flat files, the default requirement is that the time exceeds one day and the file size is greater than 16 MB.
Index Module: The index supports forced uploading of the last file, and the init logic has been moved out of the constructor.
Remove invalid configuration items which only exist in old versions.
During pop consumption, the revive process generates a large number of random reads, reducing the amount of message count retrieved through prefetch.
改进和修复分级存储模块:
文件删除策略,在节点启动不久时,分级存储中数据保留时间会小于本地存储。
定时任务会删除分级存储中的数据并重传,此时没有正确性问题,但重传浪费了计算资源和带宽。
文件滚动策略,支持配置文件保留的最小大小,对于 flat file 文件的滚动,默认需要满足时间大于 1 天并且文件大于 16M。
索引部分,Index 支持强制上传最后一个文件,并将初始化逻辑从构造函数中移出。
删除低版本中的无效配置项。
在 pop 消费时,revive 流程会产生大量对于分级存储的随机读,此时减少了预读取回的数据量。
Describe Alternatives You've Considered
None
Additional Context
No response
The text was updated successfully, but these errors were encountered: