Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Disable flush after every DefaultObjectMapper write #3748

Merged
merged 1 commit into from
Dec 7, 2016

Conversation

jon-wei
Copy link
Contributor

@jon-wei jon-wei commented Dec 6, 2016

This PR disables the FLUSH_AFTER_WRITE_VALUE setting on DefaultObjectMapper:

http://wiki.fasterxml.com/JacksonFeaturesSerialization
https://fasterxml.github.io/jackson-databind/javadoc/2.3.0/com/fasterxml/jackson/databind/SerializationFeature.html#FLUSH_AFTER_WRITE_VALUE

Disabling this setting resulted in better network performance between the historical and broker when transferring query results. When serializing results from a yielder, a flush was occurring for every yielder result. This had significant impact on a test groupby query with compression disabled at the HttpClient level.

Using iftop, a transfer rate of ~350Mb/s was observed between the historical (i2.8xlarge node) and broker (r3.8xlarge node) with flush after write disabled, compared to a rate of ~170Mb/s with the setting enabled.

@jon-wei jon-wei added this to the 0.9.3 milestone Dec 6, 2016
@drcrallen
Copy link
Contributor

https://fasterxml.github.io/jackson-databind/javadoc/2.6/com/fasterxml/jackson/databind/SerializationFeature.html#FLUSH_AFTER_WRITE_VALUE explicitly states

for example when underlying stream is compressing and flush() causes compression state to be flushed (which occurs with some compression codecs).

Can you verify or refute if you are using compressed or uncompressed streams between the broker and historical?

@jon-wei
Copy link
Contributor Author

jon-wei commented Dec 6, 2016

The numbers in the original post were for uncompressed streams, using a modified version of http-client that allows for compression codec selection:

metamx/http-client#26

@fjy
Copy link
Contributor

fjy commented Dec 7, 2016

👍

1 similar comment
@cheddar
Copy link
Contributor

cheddar commented Dec 7, 2016

👍

@fjy fjy merged commit d1896a2 into apache:master Dec 7, 2016
dgolitsyn pushed a commit to metamx/druid that referenced this pull request Feb 14, 2017
@jon-wei jon-wei deleted the mapperflush branch October 6, 2017 22:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants