Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(s3stream): support config MAJOR_V1 skip compact small object #1584

Merged
merged 3 commits into from
Jul 20, 2024

Conversation

lifepuzzlefun
Copy link
Contributor

@lifepuzzlefun lifepuzzlefun commented Jul 17, 2024

  1. support MAJOR_V1 compaction skip object which size is below the MINOR_V1_COMPACTION_SIZE.
  2. default skip logic is disabled.

@lifepuzzlefun
Copy link
Contributor Author

lifepuzzlefun commented Jul 17, 2024

Why this pr?

I found that in single bucket the object number is very large 1230517 object for 1200 s3Object recored in metadata.

82G total data for 1230517 object in one bucket = 71.58KB / object

I think the major_v1 may compose both the small object and the large object together.

for MINIO the object storage metadata may prefer more bigger object with less total object number.

Changes

when the major_v1 compaction started the small object should be compaction at least 4MB.
then it can be composed into small composite object.

.objectStorage(objectStorage)
.maxStreamObjectSize(config.streamObjectCompactionMaxSizeBytes())
.minorV1CompactionThreshold(MINOR_V1_COMPACTION_SIZE)
.majorV1CompactionSkipSmallObject(MAJOR_V1_COMPACTION_SKIP_SMALL_OBJECT)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can skip the small object logic for objectsCount < MAJOR_V1_COMPACTION_MAX_OBJECT_THRESHOLD instead of by a switch.

@CLAassistant
Copy link

CLAassistant commented Jul 19, 2024

CLA assistant check
All committers have signed the CLA.

@superhx superhx merged commit fd9d5dd into AutoMQ:main Jul 20, 2024
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants