Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding assertion to check for regular JSON inputs of size greater than INT_MAX bytes #17057

Merged
merged 6 commits into from
Oct 14, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions cpp/src/io/json/nested_json_gpu.cu
Original file line number Diff line number Diff line change
Expand Up @@ -83,8 +83,7 @@ struct tree_node {
void check_input_size(std::size_t input_size)
{
// Transduce() writes symbol offsets that may be as large input_size-1
CUDF_EXPECTS(input_size == 0 ||
(input_size - 1) <= std::numeric_limits<cudf::io::json::SymbolOffsetT>::max(),
CUDF_EXPECTS(input_size == 0 || (input_size - 1) <= std::numeric_limits<int32_t>::max(),
"Given JSON input is too large");
}
} // namespace
Expand Down
14 changes: 10 additions & 4 deletions cpp/src/io/json/read_json.cu
Original file line number Diff line number Diff line change
Expand Up @@ -351,10 +351,16 @@ table_with_metadata read_json(host_span<std::unique_ptr<datasource>> sources,
* JSON inputs.
*/
std::size_t const total_source_size = sources_size(sources, 0, 0);
std::size_t chunk_offset = reader_opts.get_byte_range_offset();
std::size_t chunk_size = reader_opts.get_byte_range_size();
chunk_size = !chunk_size ? total_source_size - chunk_offset
: std::min(chunk_size, total_source_size - chunk_offset);

// Batching is enabled only for JSONL inputs, not regular JSON files
CUDF_EXPECTS(
reader_opts.is_enabled_lines() || total_source_size < std::numeric_limits<int32_t>::max(),
"Parsing Regular JSON inputs of size greater than INT_MAX bytes is not supported");

std::size_t chunk_offset = reader_opts.get_byte_range_offset();
std::size_t chunk_size = reader_opts.get_byte_range_size();
chunk_size = !chunk_size ? total_source_size - chunk_offset
: std::min(chunk_size, total_source_size - chunk_offset);

std::size_t const size_per_subchunk = estimate_size_per_subchunk(chunk_size);
std::size_t const batch_size_upper_bound = get_batch_size_upper_bound();
Expand Down
Loading