[ManagedLedger] Compress managed ledger info #11490

gaoran10 · 2021-07-28T19:39:34Z

Motivation

Currently, the ManagedLedgerInfo contains offload context info, if there are too many ledgers in one ManagedLedger, a Zookeeper ZNode data will increase, it's hard for Zookeeper to manage these data. We could compress the ManagedLedgerInfo data, this will decrease the data size.

For example, if one ManagedLedgerInfo contains 30000 ledgers and each LedgerInfo contains offload context, the uncompressed data size is about 6MB, after compress with ZSTD, the compression data size could be decreased to about 1.3MB.

Modifications

Add a ManagedLedgerInfoMetadata before ManagedLedgerInfo, the data structure as below.

[MAGIC_NUMBER] (2) + [METADATA_SIZE] (4) + [METADATA_PAYLOAD] + [MANAGED_LEDGER_INFO_PAYLOAD]

Add a new configuration managedLedgerInfoCompressionType to control ManagedLedgerInfo compression type, if not set this configuration, then don't compress ManagedLedgerInfo data.

Migration

When reading data from the Zookeeper, check the magic number of the data, if the head data match the magic number then try to parse metadata and uncompress data, if encounter errors or not match fall back to parse ManagedLedgerInfo directly.

Verifying this change

Verify compress and uncompress ManagedLedgerInfo data.

Does this pull request potentially affect one of the following parts:

If yes was chosen, please highlight the changes

Dependencies (does it add or upgrade a dependency): (no)
The public API: (no)
The schema: (no)
The default values of configurations: (no)
The wire protocol: (no)
The rest endpoints: (no)
The admin cli options: (no)
Anything that affects deployment: (no)

Documentation

For committer

For this PR, do we need to update docs?

If yes,
- if you update docs in this PR, label this PR with the doc label.
- if you plan to update docs later, label this PR with the doc-required label.
- if you need help on updating docs, create a follow-up issue with the doc-required label.
If no, label this PR with the no-need-doc label and explain why.

codelipenghui · 2021-07-29T01:05:06Z

pulsar-common/src/main/proto/PulsarApi.proto

@@ -186,6 +186,11 @@ message BrokerEntryMetadata {
    optional uint64 index = 2;
 }

+message ManagedLedgerInfoMetadata {


@gaoran10 We should move it to MLDataFormats.proto? The PulsarApi.proto is used for the broker and client interaction.

Ok, I'll move it.

codelipenghui · 2021-07-29T01:05:53Z

managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/MetaStoreImpl.java

        String path = PREFIX + ledgerName;
-        store.put(path, serializedMlInfo, Optional.of(stat.getVersion()))
+        store.put(path, compressLedgerInfo(mlInfo, CompressionType.ZSTD), Optional.of(stat.getVersion()))


The compression type should be configurable

codelipenghui · 2021-07-29T01:06:38Z

managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/MetaStoreImpl.java

@@ -47,6 +54,15 @@
    private final MetadataStore store;
    private final OrderedExecutor executor;

+    public static final short magicManagedLedgerInfoMetadata = 0x0b9c;


Suggested change

public static final short magicManagedLedgerInfoMetadata = 0x0b9c;

public static final short MAGIC_MANAGED_LEDGER_INFO_METADATA = 0x0b9c;

I'll fix this.

2. Add configuration managedLedgerInfoCompressionType to control ManagedLedgerInfo compression type.

315157973 · 2021-07-29T13:56:50Z

managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/MetaStoreImpl.java

+
+    public ManagedLedgerInfo parseManagedLedgerInfo(byte[] data) throws InvalidProtocolBufferException {
+        ByteBuf byteBuf = PulsarByteBufAllocator.DEFAULT.buffer(data.length, data.length);
+        byteBuf.writeBytes(data);


It seems we can use wrappedBuffer to reduce one copy

Yes, I'll fix this.

315157973 · 2021-07-29T14:05:21Z

managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/MetaStoreImpl.java

+        metadataByteBuf.writeBytes(mlInfoMetadata.toByteArray());
+
+        ByteBuf originalByteBuf = PulsarByteBufAllocator.DEFAULT.buffer(originalBytes.length, originalBytes.length);
+        originalByteBuf.writeBytes(originalBytes);


It seems we can use wrappedBuffer to reduce one copy

Thanks, I'll fix this.

315157973 · 2021-07-29T14:11:52Z

managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/MetaStoreImpl.java

+
+        byte[] dataBytes = new byte[compositeByteBuf.readableBytes()];
+        compositeByteBuf.readBytes(dataBytes);
+        return dataBytes;


Do these ByteBufs need to be released？

Release the compositeByteBuf will encounter error o.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1.

315157973 · 2021-07-29T14:14:50Z

managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/MetaStoreImpl.java

+                        MLDataFormats.ManagedLedgerInfoMetadata.parseFrom(metadataBytes);
+
+                long unpressedSize = metadata.getUnpressedSize();
+                ByteBuf decodeByteBuf = CompressionCodecProvider.getCompressionCodec(


Do we need a release here ?

/** * Decompress a buffer. * * <p>The buffer needs to have been compressed with the matching Encoder. * * @param encoded * the compressed content * @param uncompressedSize * the size of the original content * @return a ByteBuf with the compressed content. The buffer needs to be released by the receiver * @throws IOException * if the decompression fails */ ByteBuf decode(ByteBuf encoded, int uncompressedSize) throws IOException;

we need to release the ByteBuf

I'll fix this.

hangc0276 · 2021-07-29T14:39:06Z

managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/MetaStoreImpl.java

@@ -47,11 +53,27 @@
    private final MetadataStore store;
    private final OrderedExecutor executor;

+    public static final short MAGIC_MANAGED_LEDGER_INFO_METADATA = 0x0b9c;
+    private CompressionType compressionType = null;


initiate to be None instead of null?

That's better.

hangc0276 · 2021-07-29T14:41:23Z

managed-ledger/src/main/proto/MLDataFormats.proto

@@ -124,3 +124,8 @@ message ManagedCursorInfo {
    // Store which index in the batch message has been deleted
    repeated BatchedEntryDeletionIndexInfo batchedEntryDeletionIndexInfo = 7;
 }
+
+message ManagedLedgerInfoMetadata {
+    required string compressionType = 1;


Use compression enum type instead of String？

The CompressionType in the PulsarApi.proto, if we want to use the enum CompressionType, we need import PulsarApi.proto, maybe we could use string type here?

I think we could just copy the same enum type here, or even declare it as int and use the other enum type

Looks we'd better to copy the enum type to the MLDataFormats.proto because the managed ledger module can be a independent component.

hangc0276 · 2021-07-29T14:42:36Z

pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java

@@ -1601,6 +1601,11 @@
    private String managedLedgerDataReadPriority = OffloadedReadPriority.TIERED_STORAGE_FIRST
            .getValue();

+    @FieldContext(category = CATEGORY_STORAGE_ML,
+            doc = "ManagedLedgerInfo compression type, option values (LZ4, ZLIB, ZSTD, SNAPPY). \n"


add NONE compression value type to doc, and default value set to be NONE instead of null?

Ok, I'll fix this.

hangc0276 · 2021-07-29T14:43:33Z

managed-ledger/src/main/java/org/apache/bookkeeper/mledger/ManagedLedgerFactoryConfig.java

+    /**
+     * ManagedLedgerInfo compression type. If the compression type is null or invalid, don't compress data.
+     */
+    private String managedLedgerInfoCompressionType = null;


We'd better use enum type to strict compression type info specific type instead of string. I'm not sure whether it's easy to implement.

Using the configuration managedLedgerInfoCompressionType of the ServiceConfiguration to set this field, maybe we could convert String to ComprssionType in MetaStoreImpl.

hangc0276 · 2021-07-29T14:54:57Z

managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/MetaStoreImpl.java

+                        MLDataFormats.ManagedLedgerInfoMetadata.parseFrom(metadataBytes);
+
+                long unpressedSize = metadata.getUnpressedSize();
+                ByteBuf decodeByteBuf = CompressionCodecProvider.getCompressionCodec(


/** * Decompress a buffer. * * <p>The buffer needs to have been compressed with the matching Encoder. * * @param encoded * the compressed content * @param uncompressedSize * the size of the original content * @return a ByteBuf with the compressed content. The buffer needs to be released by the receiver * @throws IOException * if the decompression fails */ ByteBuf decode(ByteBuf encoded, int uncompressedSize) throws IOException;

we need to release the ByteBuf

hangc0276 · 2021-07-29T14:56:00Z

managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/MetaStoreImpl.java

+
+        ByteBuf originalByteBuf = PulsarByteBufAllocator.DEFAULT.buffer(originalBytes.length, originalBytes.length);
+        originalByteBuf.writeBytes(originalBytes);
+        ByteBuf encodeByteBuf = CompressionCodecProvider.getCompressionCodec(compressionType).encode(originalByteBuf);


/** * Compress a buffer. * * @param raw * a buffer with the uncompressed content. The reader/writer indexes will not be modified * @return a new buffer with the compressed content. The buffer needs to be released by the receiver */ ByteBuf encode(ByteBuf raw);

Need to be released.

I'll fix this.

eolivelli

Overall looks good.

What is the upgrade story?

eolivelli · 2021-07-29T07:04:10Z

managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/MetaStoreImpl.java

@@ -47,11 +53,27 @@
    private final MetadataStore store;
    private final OrderedExecutor executor;

+    public static final short MAGIC_MANAGED_LEDGER_INFO_METADATA = 0x0b9c;
+    private CompressionType compressionType = null;


That's better.

eolivelli · 2021-07-30T05:21:47Z

@codelipenghui this is a new feature, with a significant impact, as you cannot rollback if you enable this feature.

I believe that we cannot cherry pick this feature to 2.8.x release line.

We should cherry pick to released versions only bug fixes and small features that have a low impact.

does this make sense to you ?

sijie · 2021-07-30T06:07:28Z

@eolivelli: the upgrade requires two steps:

upgrade the binary without enabling this feature.
after all brokers are upgraded, enable this feature.

Because it is a 2-steps upgrade story, we can add the code to 2.8.1 because this feature would be turned off. With the code available in 2.8.1, it provides a smooth path to enable this feature.

eolivelli · 2021-07-30T06:13:01Z

With the code available in 2.8.1, it provides a smooth path to enable this feature.
makes sense to me

we have to document it carefully, because it will ease switching from 2.8.1 to 2.9.0,
but then the problem goes down to the upgrade from 2.8.0 to 2.8.1: you have to take care of enabling this feature by upgrading to 2.8.1 all the brokers and then enable the flag.

if you feel strong then I am not against cherry picking this into 2.8.x.
I just wanted to point out that we should start porting less "new features" to released branches, in order to preserve stability as much as possible, but this is not the right thread for discussing this point.

eolivelli

LGTM as soon as all the existing comments have been addressed

sijie · 2021-07-30T06:30:03Z

we have to document it carefully, because it will ease switching from 2.8.1 to 2.9.0,
but then the problem goes down to the upgrade from 2.8.0 to 2.8.1: you have to take care of enabling this feature by upgrading to 2.8.1 all the brokers and then enable the flag.

Marked it for doc-required and releasenote-required.

315157973

Should we change their positions so as to avoid NPE ?

metadata.getCompressionType().equals(CompressionType.ZLIB.name())

change to

CompressionType.ZLIB.name().equals(...)

gaoran10 · 2021-08-03T02:25:15Z

Should we change their positions so as to avoid NPE ?
metadata.getCompressionType().equals(CompressionType.ZLIB.name())
change to
CompressionType.ZLIB.name().equals(...)

Good catch, the parameter compressionType of the metadata is required, but using CompressionType.ZLIB.name() is better, I'll change it.

hangc0276 · 2021-08-03T03:53:58Z

managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/MetaStoreImpl.java

@@ -47,9 +54,31 @@
    private final MetadataStore store;
    private final OrderedExecutor executor;

+    public static final short MAGIC_MANAGED_LEDGER_INFO_METADATA = 0x0b9c;


We should try to use a magic number with the most significant bit set to 1 because that should be guaranteed to be a non-valid protobuf sequence

Ok, I'll change this.

merlimat · 2021-08-03T04:01:12Z

managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/MetaStoreImpl.java

+            try {
+                finalCompressionType = CompressionType.valueOf(compressionType);
+            } catch (Exception e) {
+                log.warn("Failed to get compression type {}, disable managedLedgerInfo compression, error msg: {}.",


If the compression type is not valid we should just throw exception instead of trying to handle and disable compression. This could lead to situation in which someone thinks compression is enabled but it really is failing.

Ok, I'll throw the exception.

merlimat · 2021-08-03T04:04:25Z

managed-ledger/src/main/proto/MLDataFormats.proto

@@ -124,3 +124,8 @@ message ManagedCursorInfo {
    // Store which index in the batch message has been deleted
    repeated BatchedEntryDeletionIndexInfo batchedEntryDeletionIndexInfo = 7;
 }
+
+message ManagedLedgerInfoMetadata {
+    required string compressionType = 1;


I think we could just copy the same enum type here, or even declare it as int and use the other enum type

merlimat · 2021-08-03T04:05:06Z

managed-ledger/src/main/proto/MLDataFormats.proto

+
+message ManagedLedgerInfoMetadata {
+    required string compressionType = 1;
+    required int32 unpressedSize = 2;


Suggested change

required int32 unpressedSize = 2;

required int32 uncompressedSize = 2;

merlimat · 2021-08-03T04:07:26Z

managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/MetaStoreImpl.java

@@ -47,9 +54,31 @@
    private final MetadataStore store;
    private final OrderedExecutor executor;

+    public static final short MAGIC_MANAGED_LEDGER_INFO_METADATA = 0x0b9c;


We should try to use a magic number with the most significant bit set to 1 because that should be guaranteed to be a non-valid protobuf sequence

2. change compression magic number. 3. add a unit test to verify compression could work well.

gaoran10 · 2021-08-03T09:45:41Z

We should try to use a magic number with the most significant bit set to 1 because that should be guaranteed to be a non-valid protobuf sequence

@merlimat I'm not sure about how to generate a non-valid protobuf sequence. I take a look at some docs, the basic protobuf encode format is field_number + wire_type + value, currently, the wire type range is 0-5, so I use a code 111. Because I want to use a short value to present the magic number, so the MSB couldn't be 1, could you provide some suggestions? Or we could use a magic number > 111 1111 1111 1111, an int value?

* compress managed ledger info * 1. Move the `ManagedLedgerInfoMetadata` to `MLDataFormats.proto`. 2. Add configuration managedLedgerInfoCompressionType to control ManagedLedgerInfo compression type. * use ByteBuf wrap bytes array, release ByteBuf if needed. * make the compressionType as a final field * fix comment * 1. throw exception if using a invalid compression type. 2. change compression magic number. 3. add a unit test to verify compression could work well. * change compression magic number. * fix test (cherry picked from commit 4361b6d)

…configuration doc (#11563) ### Modifications Add a new configuration managedLedgerInfoCompressionType in broker configuration doc. Related to #11490

* compress managed ledger info * 1. Move the `ManagedLedgerInfoMetadata` to `MLDataFormats.proto`. 2. Add configuration managedLedgerInfoCompressionType to control ManagedLedgerInfo compression type. * use ByteBuf wrap bytes array, release ByteBuf if needed. * make the compressionType as a final field * fix comment * 1. throw exception if using a invalid compression type. 2. change compression magic number. 3. add a unit test to verify compression could work well. * change compression magic number. * fix test

…configuration doc (apache#11563) ### Modifications Add a new configuration managedLedgerInfoCompressionType in broker configuration doc. Related to apache#11490

* compress managed ledger info * 1. Move the `ManagedLedgerInfoMetadata` to `MLDataFormats.proto`. 2. Add configuration managedLedgerInfoCompressionType to control ManagedLedgerInfo compression type. * use ByteBuf wrap bytes array, release ByteBuf if needed. * make the compressionType as a final field * fix comment * 1. throw exception if using a invalid compression type. 2. change compression magic number. 3. add a unit test to verify compression could work well. * change compression magic number. * fix test

…configuration doc (apache#11563) ### Modifications Add a new configuration managedLedgerInfoCompressionType in broker configuration doc. Related to apache#11490

compress managed ledger info

66b87b3

gaoran10 self-assigned this Jul 28, 2021

gaoran10 requested review from merlimat, 315157973, codelipenghui, congbobo184, eolivelli, sijie and zymap July 28, 2021 19:40

sijie approved these changes Jul 28, 2021

View reviewed changes

sijie added this to the 2.9.0 milestone Jul 28, 2021

sijie added component/storage type/enhancement The enhancements for the existing features or docs. e.g. reduce memory usage of the delayed messages labels Jul 28, 2021

codelipenghui requested changes Jul 29, 2021

View reviewed changes

1. Move the ManagedLedgerInfoMetadata to MLDataFormats.proto.

d1d5ed6

2. Add configuration managedLedgerInfoCompressionType to control ManagedLedgerInfo compression type.

gaoran10 changed the title ~~[WIP] [ManagedLedger] Compress managed ledger info~~ [ManagedLedger] Compress managed ledger info Jul 29, 2021

codelipenghui approved these changes Jul 29, 2021

View reviewed changes

codelipenghui added the release/2.8.1 label Jul 29, 2021

315157973 reviewed Jul 29, 2021

View reviewed changes

hangc0276 requested changes Jul 29, 2021

View reviewed changes

eolivelli requested changes Jul 29, 2021

View reviewed changes

eolivelli approved these changes Jul 30, 2021

View reviewed changes

sijie added doc-required Your PR changes impact docs and you will update later. release/note-required labels Jul 30, 2021

gaoran10 added 2 commits July 31, 2021 03:19

use ByteBuf wrap bytes array, release ByteBuf if needed.

108b2c2

make the compressionType as a final field

816a65c

codelipenghui requested a review from 315157973 August 2, 2021 15:54

315157973 approved these changes Aug 2, 2021

View reviewed changes

fix comment

e565b97

sijie added the release/2.7.4 label Aug 3, 2021

hangc0276 reviewed Aug 3, 2021

View reviewed changes

hangc0276 approved these changes Aug 3, 2021

View reviewed changes

merlimat reviewed Aug 3, 2021

View reviewed changes

gaoran10 added 2 commits August 3, 2021 17:17

1. throw exception if using a invalid compression type.

52cf031

2. change compression magic number. 3. add a unit test to verify compression could work well.

change compression magic number.

ffb8163

fix test

8bec01b

merlimat approved these changes Aug 4, 2021

View reviewed changes

merlimat merged commit 4361b6d into apache:master Aug 4, 2021

gaoran10 deleted the ledger-metadata-compression branch August 5, 2021 00:38

codelipenghui added the cherry-picked/branch-2.8 Archived: 2.8 is end of life label Aug 5, 2021

Anonymitaet mentioned this pull request Aug 5, 2021

[Doc] Add a new configuration managedLedgerInfoCompressionType #11563

Merged

codelipenghui added the cherry-picked/branch-2.7 Archived: 2.7 is end of life label Aug 5, 2021

codelipenghui mentioned this pull request Aug 8, 2021

java.lang.NoSuchMethodError: java.nio.ByteBuffer.position(I)Ljava/nio/ByteBuffer when enabling topic metadata compression #11593

Closed

sijie mentioned this pull request Aug 8, 2021

ISSUE-11593: java.lang.NoSuchMethodError: java.nio.ByteBuffer.position(I)Ljava/nio/ByteBuffer when enabling topic metadata compression streamnative/pulsar-archived#2884

Closed

Anonymitaet removed the doc-required Your PR changes impact docs and you will update later. label Aug 17, 2021

lifepuzzlefun mentioned this pull request May 11, 2023

PIP-270 Add config to set metadata size threshold for compression. #20307

Closed

lhotari mentioned this pull request Sep 23, 2024

[improve] PIP-381: Handle PositionInfo that's too large to serialize as a single entry #22799

Open

15 tasks

	public static final short magicManagedLedgerInfoMetadata = 0x0b9c;
	public static final short MAGIC_MANAGED_LEDGER_INFO_METADATA = 0x0b9c;

	required int32 unpressedSize = 2;
	required int32 uncompressedSize = 2;

[ManagedLedger] Compress managed ledger info #11490

[ManagedLedger] Compress managed ledger info #11490

Conversation

gaoran10 commented Jul 28, 2021 • edited Loading

Motivation

Modifications

Migration

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

For committer

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

315157973 Jul 29, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eolivelli left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eolivelli commented Jul 30, 2021

sijie commented Jul 30, 2021

eolivelli commented Jul 30, 2021

eolivelli left a comment

Choose a reason for hiding this comment

sijie commented Jul 30, 2021

315157973 left a comment

Choose a reason for hiding this comment

gaoran10 commented Aug 3, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gaoran10 commented Aug 3, 2021

gaoran10 commented Jul 28, 2021 •

edited

Loading

315157973 Jul 29, 2021 •

edited

Loading

gaoran10 commented Aug 3, 2021 •

edited

Loading