Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added support for nested messages in protobuf output format #4493

Conversation

vitlibar
Copy link
Member

@vitlibar vitlibar commented Feb 24, 2019

I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en

For changelog. Remove if this is non-significant change.

Category (leave one):

  • Improvement
  • Performance Improvement

This PR adds support for nested messages in the format schema, making possible to output protobuf messages with schema like this:

message MessageType {
  message XType {
    message YType {
      int32 z;
    };
    repeated YType y;
  };
  XType x;
};

This PR also improve performance of writing protobuf messages.

@vitlibar vitlibar force-pushed the support-for-nested-messages-in-protobuf-output branch 8 times, most recently from a40ffb0 to 07a864f Compare February 26, 2019 18:26
@vitlibar vitlibar changed the title [WIP] Support for nested messages in protobuf output Added support for nested messages in protobuf output format Feb 26, 2019
@vitlibar vitlibar marked this pull request as ready for review February 26, 2019 18:27
@vitlibar
Copy link
Member Author

I've measured the performance:

SELECT * FROM datasets.hits_v1 FORMAT TSV            33.16 real    28.67 user     4.05 sys
SELECT * FROM datasets.hits_v1 FORMAT Protobuf       38.18 real    34.59 user     3.32 sys
SELECT * FROM datasets.hits_v1 FORMAT JSONEachRow    58.02 real    47.56 user     8.97 sys

@vitlibar vitlibar force-pushed the support-for-nested-messages-in-protobuf-output branch from b92dc1c to e4e5275 Compare February 26, 2019 19:54
@vitlibar vitlibar force-pushed the support-for-nested-messages-in-protobuf-output branch 3 times, most recently from 0e1f514 to 245bbb5 Compare February 27, 2019 09:00
@vitlibar vitlibar force-pushed the support-for-nested-messages-in-protobuf-output branch from 245bbb5 to 147a7a1 Compare February 27, 2019 11:37
@vitlibar vitlibar merged commit 6fbcc12 into ClickHouse:master Feb 27, 2019
@vitlibar vitlibar deleted the support-for-nested-messages-in-protobuf-output branch February 27, 2019 15:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant