Fixes some oddities dealing with ClickHouse #278
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bnaecker
requested review from
davepacheco
and removed request for
davepacheco
bnaecker
force-pushed
the
fix/clickhouse-client-oddities
branch
2 times, most recently
from
75de035
to
3a7fadd
Compare
davepacheco
approved these changes
Sep 30, 2021
This commit fixes two bugs dealing with ClickHouse, and adds regressions for them. The first is about deserialization of structs which have a `&str` field. `serde_json` is generally fine about zero-copy, except if the string requires escaping. In that case, it can no longer deserialize the data into a `&str`, since it must take ownership and manipulate the returned buffer. There are a few ways around this, but this commit opts for the simplest -- using a owned `String` instead of a `&str`. The string is immediately moved into a `Measurement`, which takes ownership anyway, so this is the most straightfoward choice. See serde-rs/serde#1413 for more details. Note that we also parse all string datum from the DB as `String` directly, rather than as `&str`, again to avoid this. The second bug is handling a ClickHouse oddity. By default, 64-bit integer types are returned as _quoted_ strings. As in, `"1"` instead of `1`. This is apparently to support JavaScript's JSON implementation, but in any case it's not the behavior we expect or want. There is thankfully a setting to turn this off, which the database client object now sets when it makes requests returning JSON data. See ClickHouse/ClickHouse#2375 for more details.
bnaecker
force-pushed
the
fix/clickhouse-client-oddities
branch
from
3a7fadd
to
21d753d
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This commit fixes two bugs dealing with ClickHouse, and adds regressions
for them.
The first is about deserialization of structs which have a
&strfield.serde_jsonis generally fine about zero-copy, except if the stringrequires escaping. In that case, it can no longer deserialize the data
into a
&str, since it must take ownership and manipulate the returnedbuffer. This requires replacing
&'a strwithCow<'a, str>in the DBmodel types, specifically for the timeseries key fields. See
serde-rs/serde#1413 for more details. Note
that we also parse all string datum from the DB as
Stringdirectly,rather than as
&str, again to avoid this (and because we'll beconverting it to an owned representation soon anyway).
The second bug is handling a ClickHouse oddity. By default, 64-bit
integer types are returned as quoted strings. As in,
"1"instead of1. This is apparently to support JavaScript's JSON implementation, butin any case it's not the behavior we expect or want. There is thankfully
a setting to turn this off, which the database client object now sets
when it makes requests returning JSON data. See
ClickHouse/ClickHouse#2375 for more details.