Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding assertion to check for regular JSON inputs of size greater than INT_MAX bytes #17057

Merged
merged 6 commits into from
Oct 14, 2024
+11 −6
Merged

Adding assertion to check for regular JSON inputs of size greater than INT_MAX bytes #17057

Changes from 2 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
70e4699
we cannot parse json inputs of size greater than intmax
shrshi Oct 10, 2024
22e9514
Merge branch 'branch-24.12' into json-reading-fix
shrshi Oct 11, 2024
a212b1c
update check
shrshi Oct 11, 2024
b538c5f
Merge branch 'json-reading-fix' of github.com:shrshi/cudf into json-r…
shrshi Oct 11, 2024
e7996e1
formatting
shrshi Oct 11, 2024
06fcfa8
addressing PR review
shrshi Oct 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 11 additions & 4 deletions cpp/src/io/json/read_json.cu
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -351,10 +351,17 @@ table_with_metadata read_json(host_span<std::unique_ptr<datasource>> sources,
* JSON inputs.
*/
std::size_t const total_source_size = sources_size(sources, 0, 0);
std::size_t chunk_offset = reader_opts.get_byte_range_offset();
std::size_t chunk_size = reader_opts.get_byte_range_size();
chunk_size = !chunk_size ? total_source_size - chunk_offset
: std::min(chunk_size, total_source_size - chunk_offset);

// Batching is enabled only for JSONL inputs, not regular JSON files
if (!reader_opts.is_enabled_lines()) {
CUDF_EXPECTS(total_source_size < std::numeric_limits<int32_t>::max(),
"Parsing Regular JSON inputs of size greater than INT_MAX bytes is not supported");
}
shrshi marked this conversation as resolved.
Show resolved Hide resolved

std::size_t chunk_offset = reader_opts.get_byte_range_offset();
std::size_t chunk_size = reader_opts.get_byte_range_size();
chunk_size = !chunk_size ? total_source_size - chunk_offset
: std::min(chunk_size, total_source_size - chunk_offset);

std::size_t const size_per_subchunk = estimate_size_per_subchunk(chunk_size);
std::size_t const batch_size_upper_bound = get_batch_size_upper_bound();
Expand Down
Loading