Skip to content
This repository has been archived by the owner on Feb 18, 2024. It is now read-only.

Refactored JSON read to split IO-bounded from CPU-bounded tasks #706

Merged
merged 1 commit into from
Dec 24, 2021

Conversation

jorgecarleitao
Copy link
Owner

This PR replaces the JSON reader by functions that offer the flexibility to read JSON asynchronously, via futures' AsyncBufRead or Tokios' AsyncBufReadExt.

The core idea of this PR, as it is for other parts of this crate, is that it separates functionality that is IO-bounded (read 1000 lines from a stream) from functionality that is CPU-bounded (parse 1000 strings into an arrow struct of 1000 rows).

The PR also refactors schema inference to make it easier to follow, use generics, etc.

@jorgecarleitao jorgecarleitao added the feature A new feature label Dec 24, 2021
@codecov
Copy link

codecov bot commented Dec 24, 2021

Codecov Report

Merging #706 (2b0fe14) into main (39695e9) will decrease coverage by 0.11%.
The diff coverage is 76.78%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #706      +/-   ##
==========================================
- Coverage   70.28%   70.17%   -0.12%     
==========================================
  Files         309      309              
  Lines       16873    16800      -73     
==========================================
- Hits        11860    11790      -70     
+ Misses       5013     5010       -3     
Impacted Files Coverage Δ
src/io/json/mod.rs 100.00% <ø> (ø)
src/io/json/read/iterator.rs 63.15% <63.15%> (ø)
src/io/json/read/mod.rs 66.66% <66.66%> (ø)
src/io/json/read/deserialize.rs 68.42% <77.77%> (-5.27%) ⬇️
src/io/json/read/infer_schema.rs 80.48% <81.25%> (-0.71%) ⬇️
src/datatypes/schema.rs 32.00% <0.00%> (-12.00%) ⬇️
src/io/avro/read/schema.rs 53.68% <0.00%> (+1.05%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 39695e9...2b0fe14. Read the comment docs.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
feature A new feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant