Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Binary encoding of JSON? #390

Open
DennisHeimbigner opened this issue Jan 15, 2019 · 3 comments
Open

Binary encoding of JSON? #390

DennisHeimbigner opened this issue Jan 15, 2019 · 3 comments
Labels
enhancement New features or improvements

Comments

@DennisHeimbigner
Copy link

Is there a previous discussion about storing the Json in a binary format
such as protobuf? Unidata experience with this for the OPeNDAP protocol
shows that parsing text is a performance bottleneck and that using e.g.
protobuf is much faster.

@alimanfoo
Copy link
Member

alimanfoo commented Jan 15, 2019 via email

@jakirkham
Copy link
Member

This blog post about what Uber did with JSON is kind of interesting and potentially relevant.

We could consider adding a JSON filter chain much like the on we have for objects. The default could be to simply use json.loads (possibly wrapped in some kind of Codec class). That way users could change this as needed to handle things like binary encoding and compression. This should be pretty flexible for users trying to explore what is best for them. A bonus of this approach might be to provide users a way to handle things like fill_values that are not JSON serializable. ( #216 ) ( #244 ) ( #354 )

@DennisHeimbigner
Copy link
Author

Thanks for the pointer. I was actually proposing something different than
they did. I am proposing we write an e.g. protobuf scheme for JSON
and then use that to encode/decode ascii-json <-> binary-encoded-json
The test that would be interesting would be comparing a compression of
ascii-json to the compression of the binary-encoded-json.

@dstansby dstansby added the enhancement New features or improvements label Dec 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New features or improvements
Projects
None yet
Development

No branches or pull requests

4 participants