[opt](parquet-reader)Implement late materialization of parquet complex types. #44098
Conversation
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| * A row of complex type may be stored across two(or more) pages, and the parameter `align_rows` indicates that | ||
| * whether the reader should read the remaining value of the last row in previous page. | ||
| */ | ||
| Status ScalarColumnReader::_read_nested_column(ColumnPtr& doris_column, DataTypePtr& type, |
There was a problem hiding this comment.
warning: function '_read_nested_column' has cognitive complexity of 77 (threshold 50) [readability-function-cognitive-complexity]
Status ScalarColumnReader::_read_nested_column(ColumnPtr& doris_column, DataTypePtr& type,
^Additional context
be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:330: +1, including nesting penalty of 0, nesting level increased to 1
if (align_rows) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:335: +1, nesting level increased to 1
} else {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:344: +1, including nesting penalty of 0, nesting level increased to 1
if (has_rep_level) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:346: +2, including nesting penalty of 1, nesting level increased to 2
while (parsed_rows <= batch_size && remaining_values > 0) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:346: +1
while (parsed_rows <= batch_size && remaining_values > 0) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:348: +3, including nesting penalty of 2, nesting level increased to 3
if (rep_level == 0) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:349: +4, including nesting penalty of 3, nesting level increased to 4
if (parsed_rows == batch_size) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:359: +2, including nesting penalty of 1, nesting level increased to 2
if (filter_map.has_filter()) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:363: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(filter_map.generate_nested_filter_map(
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:363: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(filter_map.generate_nested_filter_map(
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:368: +1, nesting level increased to 1
} else if (!align_rows) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:378: +1, including nesting penalty of 0, nesting level increased to 1
if (has_def_level) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:380: +1, nesting level increased to 1
} else {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:387: +1, including nesting penalty of 0, nesting level increased to 1
if (doris_column->is_nullable()) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:393: +1, nesting level increased to 1
} else {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:394: +2, including nesting penalty of 1, nesting level increased to 2
if (_field_schema->is_nullable) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:404: +1, including nesting penalty of 0, nesting level increased to 1
while (has_read < origin_size + parsed_values) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:407: +2, including nesting penalty of 1, nesting level increased to 2
while (has_read < origin_size + parsed_values && _def_levels[has_read] == def_level) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:411: +2, including nesting penalty of 1, nesting level increased to 2
if (def_level < _field_schema->repeated_parent_def_level) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:418: +2, including nesting penalty of 1, nesting level increased to 2
if (prev_is_null == is_null && (USHRT_MAX - null_map.back() >= loop_read)) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:418: +1
if (prev_is_null == is_null && (USHRT_MAX - null_map.back() >= loop_read)) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:422: +1, nesting level increased to 2
} else {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:423: +3, including nesting penalty of 2, nesting level increased to 3
if (!(prev_is_null ^ is_null)) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:427: +3, including nesting penalty of 2, nesting level increased to 3
while (remaining > USHRT_MAX) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:442: +1, including nesting penalty of 0, nesting level increased to 1
RETURN_IF_ERROR(
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:442: +2, including nesting penalty of 1, nesting level increased to 2
RETURN_IF_ERROR(
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:447: +1, including nesting penalty of 0, nesting level increased to 1
RETURN_IF_ERROR(_chunk_reader->decode_values(data_column, type, select_vector, is_dict_filter));
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:447: +2, including nesting penalty of 1, nesting level increased to 2
RETURN_IF_ERROR(_chunk_reader->decode_values(data_column, type, select_vector, is_dict_filter));
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:448: +1, including nesting penalty of 0, nesting level increased to 1
if (ancestor_nulls != 0) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:449: +2, including nesting penalty of 1, nesting level increased to 2
RETURN_IF_ERROR(_chunk_reader->skip_values(ancestor_nulls, false));
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:449: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(_chunk_reader->skip_values(ancestor_nulls, false));
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:452: +1, including nesting penalty of 0, nesting level increased to 1
if (!align_rows) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:456: +1, including nesting penalty of 0, nesting level increased to 1
if (_chunk_reader->remaining_num_values() == 0) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:457: +2, including nesting penalty of 1, nesting level increased to 2
if (_chunk_reader->has_next_page()) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:458: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(_chunk_reader->next_page());
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:458: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(_chunk_reader->next_page());
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:459: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(_chunk_reader->load_page_data());
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:459: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(_chunk_reader->load_page_data());
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:462: +1, nesting level increased to 2
} else {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:467: +1, including nesting penalty of 0, nesting level increased to 1
if (current_filter_map->has_filter()) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:488: +1, including nesting penalty of 0, nesting level increased to 1
if (_rep_levels.size() > 0) {
^| return Status::OK(); | ||
| } | ||
|
|
||
| Status ScalarColumnReader::read_column_data(ColumnPtr& doris_column, DataTypePtr& type, |
There was a problem hiding this comment.
warning: function 'read_column_data' has cognitive complexity of 81 (threshold 50) [readability-function-cognitive-complexity]
Status ScalarColumnReader::read_column_data(ColumnPtr& doris_column, DataTypePtr& type,
^Additional context
be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:534: +1, including nesting penalty of 0, nesting level increased to 1
if (_converter == nullptr) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:537: +2, including nesting penalty of 1, nesting level increased to 2
if (!_converter->support()) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:546: +1, including nesting penalty of 0, nesting level increased to 1
do {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:547: +2, including nesting penalty of 1, nesting level increased to 2
if (_chunk_reader->remaining_num_values() == 0) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:548: +3, including nesting penalty of 2, nesting level increased to 3
if (!_chunk_reader->has_next_page()) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:553: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(_chunk_reader->next_page());
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:553: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(_chunk_reader->next_page());
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:555: +2, including nesting penalty of 1, nesting level increased to 2
if (_nested_column) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:556: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(_chunk_reader->load_page_data_idempotent());
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:556: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(_chunk_reader->load_page_data_idempotent());
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:557: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(_read_nested_column(resolved_column, resolved_type, filter_map,
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:557: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(_read_nested_column(resolved_column, resolved_type, filter_map,
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:567: +2, including nesting penalty of 1, nesting level increased to 2
if (read_ranges.size() == 0) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:570: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(_chunk_reader->skip_page());
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:570: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(_chunk_reader->skip_page());
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:572: +1, nesting level increased to 2
} else {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:576: +3, including nesting penalty of 2, nesting level increased to 3
if (filter_map.has_filter() && filter_map.filter_ratio() > 0.6) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:576: +1
if (filter_map.has_filter() && filter_map.filter_ratio() > 0.6) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:582: +4, including nesting penalty of 3, nesting level increased to 4
if (batch_size >= remaining_num_values &&
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:582: +1
if (batch_size >= remaining_num_values &&
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:587: +5, including nesting penalty of 4, nesting level increased to 5
RETURN_IF_ERROR(_chunk_reader->skip_page());
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:587: +6, including nesting penalty of 5, nesting level increased to 6
RETURN_IF_ERROR(_chunk_reader->skip_page());
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:589: +5, including nesting penalty of 4, nesting level increased to 5
if (!_chunk_reader->has_next_page()) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:594: +1
skip_whole_batch = batch_size <= remaining_num_values &&
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:596: +4, including nesting penalty of 3, nesting level increased to 4
if (skip_whole_batch) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:601: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(_chunk_reader->load_page_data_idempotent());
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:601: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(_chunk_reader->load_page_data_idempotent());
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:626: +2, including nesting penalty of 1, nesting level increased to 2
if (_chunk_reader->remaining_num_values() == 0 && !_chunk_reader->has_next_page()) {
^| return Status::OK(); | ||
| } | ||
|
|
||
| Status RowGroupReader::_do_lazy_read(Block* block, size_t batch_size, size_t* read_rows, |
There was a problem hiding this comment.
warning: function '_do_lazy_read' has cognitive complexity of 109 (threshold 50) [readability-function-cognitive-complexity]
Status RowGroupReader::_do_lazy_read(Block* block, size_t batch_size, size_t* read_rows,
^Additional context
be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:454: +1, including nesting penalty of 0, nesting level increased to 1
for (uint32_t i = 0; i < origin_column_num; ++i) {
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:459: +1, including nesting penalty of 0, nesting level increased to 1
while (!_state->is_cancelled()) {
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:464: +2, including nesting penalty of 1, nesting level increased to 2
RETURN_IF_ERROR(_read_column_data(block, _lazy_read_ctx.predicate_columns.first, batch_size,
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:464: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(_read_column_data(block, _lazy_read_ctx.predicate_columns.first, batch_size,
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:466: +2, including nesting penalty of 1, nesting level increased to 2
if (pre_read_rows == 0) {
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:471: +2, including nesting penalty of 1, nesting level increased to 2
RETURN_IF_ERROR(_fill_partition_columns(block, pre_read_rows,
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:471: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(_fill_partition_columns(block, pre_read_rows,
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:473: +2, including nesting penalty of 1, nesting level increased to 2
RETURN_IF_ERROR(_fill_missing_columns(block, pre_read_rows,
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:473: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(_fill_missing_columns(block, pre_read_rows,
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:476: +2, including nesting penalty of 1, nesting level increased to 2
RETURN_IF_ERROR(_build_pos_delete_filter(pre_read_rows));
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:476: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(_build_pos_delete_filter(pre_read_rows));
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:479: +2, including nesting penalty of 1, nesting level increased to 2
if (_lazy_read_ctx.resize_first_column) {
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:487: +2, including nesting penalty of 1, nesting level increased to 2
if (_position_delete_ctx.has_filter) {
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:498: +2, including nesting penalty of 1, nesting level increased to 2
RETURN_IF_ERROR(VExprContext::execute_conjuncts(filter_contexts, &filters, block,
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:498: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(VExprContext::execute_conjuncts(filter_contexts, &filters, block,
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:502: +2, including nesting penalty of 1, nesting level increased to 2
if (_lazy_read_ctx.resize_first_column) {
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:509: +2, including nesting penalty of 1, nesting level increased to 2
RETURN_IF_ERROR(filter_map_ptr->init(filter_map_data, pre_read_rows, can_filter_all));
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:509: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(filter_map_ptr->init(filter_map_data, pre_read_rows, can_filter_all));
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:510: +2, including nesting penalty of 1, nesting level increased to 2
if (filter_map_ptr->filter_all()) {
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:523: +3, including nesting penalty of 2, nesting level increased to 3
if (!pre_eof) {
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:524: +4, including nesting penalty of 3, nesting level increased to 4
if (pre_raw_read_rows >= config::doris_scanner_row_num) {
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:529: +1, nesting level increased to 3
} else { // pre_eof
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:537: +1, nesting level increased to 2
} else {
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:541: +1, including nesting penalty of 0, nesting level increased to 1
if (_state->is_cancelled()) {
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:545: +1, including nesting penalty of 0, nesting level increased to 1
if (filter_map_ptr == nullptr) {
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:554: +1, including nesting penalty of 0, nesting level increased to 1
if (_cached_filtered_rows != 0) {
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:555: +2, including nesting penalty of 1, nesting level increased to 2
RETURN_IF_ERROR(_rebuild_filter_map(filter_map, rebuild_filter_map, pre_read_rows));
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:555: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(_rebuild_filter_map(filter_map, rebuild_filter_map, pre_read_rows));
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:563: +1, including nesting penalty of 0, nesting level increased to 1
RETURN_IF_ERROR(_read_column_data(block, _lazy_read_ctx.lazy_read_columns, pre_read_rows,
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:563: +2, including nesting penalty of 1, nesting level increased to 2
RETURN_IF_ERROR(_read_column_data(block, _lazy_read_ctx.lazy_read_columns, pre_read_rows,
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:566: +1, including nesting penalty of 0, nesting level increased to 1
if (pre_read_rows != lazy_read_rows) {
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:573: +1, including nesting penalty of 0, nesting level increased to 1
if (filter_map.has_filter()) {
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:574: +2, including nesting penalty of 1, nesting level increased to 2
if (block->columns() == origin_column_num) {
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:578: +1, nesting level increased to 2
} else {
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:579: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_CATCH_EXCEPTION(Block::filter_block_internal(
^be/src/common/exception.h:79: expanded from macro 'RETURN_IF_CATCH_EXCEPTION'
do { \
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:579: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_CATCH_EXCEPTION(Block::filter_block_internal(
^be/src/common/exception.h:84: expanded from macro 'RETURN_IF_CATCH_EXCEPTION'
} catch (const doris::Exception& e) { \
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:579: +5, including nesting penalty of 4, nesting level increased to 5
RETURN_IF_CATCH_EXCEPTION(Block::filter_block_internal(
^be/src/common/exception.h:85: expanded from macro 'RETURN_IF_CATCH_EXCEPTION'
if (e.code() == doris::ErrorCode::MEM_ALLOC_FAILED) { \
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:583: +1, nesting level increased to 1
} else {
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:591: +1, including nesting penalty of 0, nesting level increased to 1
for (int i = 0; i < column_num; ++i) {
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:593: +2, including nesting penalty of 1, nesting level increased to 2
if (column_size != 0 && cz != 0) {
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:593: +1
if (column_size != 0 && cz != 0) {
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:596: +2, including nesting penalty of 1, nesting level increased to 2
if (cz != 0) {
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:604: +1, including nesting penalty of 0, nesting level increased to 1
RETURN_IF_ERROR(_fill_partition_columns(block, column_size, _lazy_read_ctx.partition_columns));
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:604: +2, including nesting penalty of 1, nesting level increased to 2
RETURN_IF_ERROR(_fill_partition_columns(block, column_size, _lazy_read_ctx.partition_columns));
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:605: +1, including nesting penalty of 0, nesting level increased to 1
RETURN_IF_ERROR(_fill_missing_columns(block, column_size, _lazy_read_ctx.missing_columns));
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:605: +2, including nesting penalty of 1, nesting level increased to 2
RETURN_IF_ERROR(_fill_missing_columns(block, column_size, _lazy_read_ctx.missing_columns));
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:606: +1, including nesting penalty of 0, nesting level increased to 1
if (!_not_single_slot_filter_conjuncts.empty()) {
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:609: +2, including nesting penalty of 1, nesting level increased to 2
RETURN_IF_CATCH_EXCEPTION(
^be/src/common/exception.h:79: expanded from macro 'RETURN_IF_CATCH_EXCEPTION'
do { \
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:610: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(VExprContext::execute_conjuncts_and_filter_block(
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:610: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(VExprContext::execute_conjuncts_and_filter_block(
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:609: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_CATCH_EXCEPTION(
^be/src/common/exception.h:84: expanded from macro 'RETURN_IF_CATCH_EXCEPTION'
} catch (const doris::Exception& e) { \
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:609: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_CATCH_EXCEPTION(
^be/src/common/exception.h:85: expanded from macro 'RETURN_IF_CATCH_EXCEPTION'
if (e.code() == doris::ErrorCode::MEM_ALLOC_FAILED) { \
^
kaka11chen
force-pushed
the
parquet_complex_type_late_materialization
branch
from
24c9cf6 to
30bf98b
Compare
|
run buildall |
| return Status::OK(); | ||
| } | ||
|
|
||
| Status ScalarColumnReader::read_column_data(ColumnPtr& doris_column, DataTypePtr& type, |
There was a problem hiding this comment.
warning: function 'read_column_data' has cognitive complexity of 81 (threshold 50) [readability-function-cognitive-complexity]
Status ScalarColumnReader::read_column_data(ColumnPtr& doris_column, DataTypePtr& type,
^Additional context
be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:529: +1, including nesting penalty of 0, nesting level increased to 1
if (_converter == nullptr) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:532: +2, including nesting penalty of 1, nesting level increased to 2
if (!_converter->support()) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:541: +1, including nesting penalty of 0, nesting level increased to 1
do {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:542: +2, including nesting penalty of 1, nesting level increased to 2
if (_chunk_reader->remaining_num_values() == 0) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:543: +3, including nesting penalty of 2, nesting level increased to 3
if (!_chunk_reader->has_next_page()) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:548: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(_chunk_reader->next_page());
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:548: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(_chunk_reader->next_page());
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:550: +2, including nesting penalty of 1, nesting level increased to 2
if (_nested_column) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:551: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(_chunk_reader->load_page_data_idempotent());
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:551: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(_chunk_reader->load_page_data_idempotent());
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:552: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(_read_nested_column(resolved_column, resolved_type, filter_map,
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:552: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(_read_nested_column(resolved_column, resolved_type, filter_map,
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:562: +2, including nesting penalty of 1, nesting level increased to 2
if (read_ranges.size() == 0) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:565: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(_chunk_reader->skip_page());
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:565: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(_chunk_reader->skip_page());
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:567: +1, nesting level increased to 2
} else {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:571: +3, including nesting penalty of 2, nesting level increased to 3
if (filter_map.has_filter() && filter_map.filter_ratio() > 0.6) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:571: +1
if (filter_map.has_filter() && filter_map.filter_ratio() > 0.6) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:577: +4, including nesting penalty of 3, nesting level increased to 4
if (batch_size >= remaining_num_values &&
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:577: +1
if (batch_size >= remaining_num_values &&
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:582: +5, including nesting penalty of 4, nesting level increased to 5
RETURN_IF_ERROR(_chunk_reader->skip_page());
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:582: +6, including nesting penalty of 5, nesting level increased to 6
RETURN_IF_ERROR(_chunk_reader->skip_page());
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:584: +5, including nesting penalty of 4, nesting level increased to 5
if (!_chunk_reader->has_next_page()) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:589: +1
skip_whole_batch = batch_size <= remaining_num_values &&
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:591: +4, including nesting penalty of 3, nesting level increased to 4
if (skip_whole_batch) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:596: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(_chunk_reader->load_page_data_idempotent());
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:596: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(_chunk_reader->load_page_data_idempotent());
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:621: +2, including nesting penalty of 1, nesting level increased to 2
if (_chunk_reader->remaining_num_values() == 0 && !_chunk_reader->has_next_page()) {
^
kaka11chen
force-pushed
the
parquet_complex_type_late_materialization
branch
from
30bf98b to
5ef9e17
Compare
|
run buildall |
| * A row of complex type may be stored across two(or more) pages, and the parameter `align_rows` indicates that | ||
| * whether the reader should read the remaining value of the last row in previous page. | ||
| */ | ||
| Status ScalarColumnReader::_read_nested_column(ColumnPtr& doris_column, DataTypePtr& type, |
There was a problem hiding this comment.
warning: function '_read_nested_column' has cognitive complexity of 77 (threshold 50) [readability-function-cognitive-complexity]
Status ScalarColumnReader::_read_nested_column(ColumnPtr& doris_column, DataTypePtr& type,
^Additional context
be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:331: +1, including nesting penalty of 0, nesting level increased to 1
if (align_rows) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:336: +1, nesting level increased to 1
} else {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:345: +1, including nesting penalty of 0, nesting level increased to 1
if (has_rep_level) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:347: +2, including nesting penalty of 1, nesting level increased to 2
while (parsed_rows <= batch_size && remaining_values > 0) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:347: +1
while (parsed_rows <= batch_size && remaining_values > 0) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:349: +3, including nesting penalty of 2, nesting level increased to 3
if (rep_level == 0) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:350: +4, including nesting penalty of 3, nesting level increased to 4
if (parsed_rows == batch_size) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:360: +2, including nesting penalty of 1, nesting level increased to 2
if (filter_map.has_filter()) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:364: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(filter_map.generate_nested_filter_map(
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:364: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(filter_map.generate_nested_filter_map(
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:369: +1, nesting level increased to 1
} else if (!align_rows) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:379: +1, including nesting penalty of 0, nesting level increased to 1
if (has_def_level) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:381: +1, nesting level increased to 1
} else {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:388: +1, including nesting penalty of 0, nesting level increased to 1
if (doris_column->is_nullable()) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:394: +1, nesting level increased to 1
} else {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:395: +2, including nesting penalty of 1, nesting level increased to 2
if (_field_schema->is_nullable) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:405: +1, including nesting penalty of 0, nesting level increased to 1
while (has_read < origin_size + parsed_values) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:408: +2, including nesting penalty of 1, nesting level increased to 2
while (has_read < origin_size + parsed_values && _def_levels[has_read] == def_level) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:412: +2, including nesting penalty of 1, nesting level increased to 2
if (def_level < _field_schema->repeated_parent_def_level) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:419: +2, including nesting penalty of 1, nesting level increased to 2
if (prev_is_null == is_null && (USHRT_MAX - null_map.back() >= loop_read)) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:419: +1
if (prev_is_null == is_null && (USHRT_MAX - null_map.back() >= loop_read)) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:423: +1, nesting level increased to 2
} else {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:424: +3, including nesting penalty of 2, nesting level increased to 3
if (!(prev_is_null ^ is_null)) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:428: +3, including nesting penalty of 2, nesting level increased to 3
while (remaining > USHRT_MAX) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:443: +1, including nesting penalty of 0, nesting level increased to 1
RETURN_IF_ERROR(
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:443: +2, including nesting penalty of 1, nesting level increased to 2
RETURN_IF_ERROR(
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:448: +1, including nesting penalty of 0, nesting level increased to 1
RETURN_IF_ERROR(_chunk_reader->decode_values(data_column, type, select_vector, is_dict_filter));
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:448: +2, including nesting penalty of 1, nesting level increased to 2
RETURN_IF_ERROR(_chunk_reader->decode_values(data_column, type, select_vector, is_dict_filter));
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:449: +1, including nesting penalty of 0, nesting level increased to 1
if (ancestor_nulls != 0) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:450: +2, including nesting penalty of 1, nesting level increased to 2
RETURN_IF_ERROR(_chunk_reader->skip_values(ancestor_nulls, false));
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:450: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(_chunk_reader->skip_values(ancestor_nulls, false));
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:453: +1, including nesting penalty of 0, nesting level increased to 1
if (!align_rows) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:457: +1, including nesting penalty of 0, nesting level increased to 1
if (_chunk_reader->remaining_num_values() == 0) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:458: +2, including nesting penalty of 1, nesting level increased to 2
if (_chunk_reader->has_next_page()) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:459: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(_chunk_reader->next_page());
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:459: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(_chunk_reader->next_page());
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:460: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(_chunk_reader->load_page_data());
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:460: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(_chunk_reader->load_page_data());
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:463: +1, nesting level increased to 2
} else {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:468: +1, including nesting penalty of 0, nesting level increased to 1
if (current_filter_map->has_filter()) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:489: +1, including nesting penalty of 0, nesting level increased to 1
if (_rep_levels.size() > 0) {
^| return Status::OK(); | ||
| } | ||
|
|
||
| Status ScalarColumnReader::read_column_data(ColumnPtr& doris_column, DataTypePtr& type, |
There was a problem hiding this comment.
warning: function 'read_column_data' has cognitive complexity of 81 (threshold 50) [readability-function-cognitive-complexity]
Status ScalarColumnReader::read_column_data(ColumnPtr& doris_column, DataTypePtr& type,
^Additional context
be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:530: +1, including nesting penalty of 0, nesting level increased to 1
if (_converter == nullptr) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:533: +2, including nesting penalty of 1, nesting level increased to 2
if (!_converter->support()) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:542: +1, including nesting penalty of 0, nesting level increased to 1
do {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:543: +2, including nesting penalty of 1, nesting level increased to 2
if (_chunk_reader->remaining_num_values() == 0) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:544: +3, including nesting penalty of 2, nesting level increased to 3
if (!_chunk_reader->has_next_page()) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:549: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(_chunk_reader->next_page());
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:549: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(_chunk_reader->next_page());
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:551: +2, including nesting penalty of 1, nesting level increased to 2
if (_nested_column) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:552: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(_chunk_reader->load_page_data_idempotent());
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:552: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(_chunk_reader->load_page_data_idempotent());
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:553: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(_read_nested_column(resolved_column, resolved_type, filter_map,
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:553: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(_read_nested_column(resolved_column, resolved_type, filter_map,
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:563: +2, including nesting penalty of 1, nesting level increased to 2
if (read_ranges.size() == 0) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:566: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(_chunk_reader->skip_page());
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:566: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(_chunk_reader->skip_page());
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:568: +1, nesting level increased to 2
} else {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:572: +3, including nesting penalty of 2, nesting level increased to 3
if (filter_map.has_filter() && filter_map.filter_ratio() > 0.6) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:572: +1
if (filter_map.has_filter() && filter_map.filter_ratio() > 0.6) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:578: +4, including nesting penalty of 3, nesting level increased to 4
if (batch_size >= remaining_num_values &&
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:578: +1
if (batch_size >= remaining_num_values &&
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:583: +5, including nesting penalty of 4, nesting level increased to 5
RETURN_IF_ERROR(_chunk_reader->skip_page());
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:583: +6, including nesting penalty of 5, nesting level increased to 6
RETURN_IF_ERROR(_chunk_reader->skip_page());
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:585: +5, including nesting penalty of 4, nesting level increased to 5
if (!_chunk_reader->has_next_page()) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:590: +1
skip_whole_batch = batch_size <= remaining_num_values &&
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:592: +4, including nesting penalty of 3, nesting level increased to 4
if (skip_whole_batch) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:597: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(_chunk_reader->load_page_data_idempotent());
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:597: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(_chunk_reader->load_page_data_idempotent());
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:622: +2, including nesting penalty of 1, nesting level increased to 2
if (_chunk_reader->remaining_num_values() == 0 && !_chunk_reader->has_next_page()) {
^
kaka11chen
force-pushed
the
parquet_complex_type_late_materialization
branch
from
5ef9e17 to
046228a
Compare
|
run buildall |
| * A row of complex type may be stored across two(or more) pages, and the parameter `align_rows` indicates that | ||
| * whether the reader should read the remaining value of the last row in previous page. | ||
| */ | ||
| Status ScalarColumnReader::_read_nested_column(ColumnPtr& doris_column, DataTypePtr& type, |
There was a problem hiding this comment.
warning: function '_read_nested_column' has cognitive complexity of 119 (threshold 50) [readability-function-cognitive-complexity]
Status ScalarColumnReader::_read_nested_column(ColumnPtr& doris_column, DataTypePtr& type,
^Additional context
be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:331: +1, including nesting penalty of 0, nesting level increased to 1
if (align_rows) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:336: +1, nesting level increased to 1
} else {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:345: +1, including nesting penalty of 0, nesting level increased to 1
if (has_rep_level) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:347: +2, including nesting penalty of 1, nesting level increased to 2
while (parsed_rows <= batch_size && remaining_values > 0) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:347: +1
while (parsed_rows <= batch_size && remaining_values > 0) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:349: +3, including nesting penalty of 2, nesting level increased to 3
if (rep_level == 0) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:350: +4, including nesting penalty of 3, nesting level increased to 4
if (parsed_rows == batch_size) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:360: +2, including nesting penalty of 1, nesting level increased to 2
if (filter_map.has_filter()) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:364: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(filter_map.generate_nested_filter_map(
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:364: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(filter_map.generate_nested_filter_map(
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:369: +1, nesting level increased to 1
} else if (!align_rows) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:379: +1, including nesting penalty of 0, nesting level increased to 1
if (has_def_level) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:381: +1, nesting level increased to 1
} else {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:388: +1, including nesting penalty of 0, nesting level increased to 1
if (doris_column->is_nullable()) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:394: +1, nesting level increased to 1
} else {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:395: +2, including nesting penalty of 1, nesting level increased to 2
if (_field_schema->is_nullable) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:408: +1, including nesting penalty of 0, nesting level increased to 1
while (has_read < origin_size + parsed_values) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:411: +2, including nesting penalty of 1, nesting level increased to 2
while (has_read < origin_size + parsed_values && _def_levels[has_read] == def_level) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:416: +2, including nesting penalty of 1, nesting level increased to 2
if (def_level < _field_schema->repeated_parent_def_level) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:422: +2, including nesting penalty of 1, nesting level increased to 2
if (is_null) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:424: +1, nesting level increased to 2
} else {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:428: +2, including nesting penalty of 1, nesting level increased to 2
if (prev_is_null == is_null && (USHRT_MAX - null_map.back() >= loop_read)) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:428: +1
if (prev_is_null == is_null && (USHRT_MAX - null_map.back() >= loop_read)) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:430: +1, nesting level increased to 2
} else {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:431: +3, including nesting penalty of 2, nesting level increased to 3
if (!(prev_is_null ^ is_null)) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:435: +3, including nesting penalty of 2, nesting level increased to 3
while (remaining > USHRT_MAX) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:447: +1, including nesting penalty of 0, nesting level increased to 1
if (current_filter_map->filter_all()) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:448: +2, including nesting penalty of 1, nesting level increased to 2
if (null_size > 0) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:449: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(_chunk_reader->skip_values(null_size, false));
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:449: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(_chunk_reader->skip_values(null_size, false));
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:451: +2, including nesting penalty of 1, nesting level increased to 2
if (nonnull_size > 0) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:452: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(_chunk_reader->skip_values(nonnull_size, true));
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:452: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(_chunk_reader->skip_values(nonnull_size, true));
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:454: +2, including nesting penalty of 1, nesting level increased to 2
if (ancestor_nulls != 0) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:455: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(_chunk_reader->skip_values(ancestor_nulls, false));
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:455: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(_chunk_reader->skip_values(ancestor_nulls, false));
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:457: +1, nesting level increased to 1
} else {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:461: +2, including nesting penalty of 1, nesting level increased to 2
RETURN_IF_ERROR(
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:461: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:466: +2, including nesting penalty of 1, nesting level increased to 2
RETURN_IF_ERROR(
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:466: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:468: +2, including nesting penalty of 1, nesting level increased to 2
if (ancestor_nulls != 0) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:469: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(_chunk_reader->skip_values(ancestor_nulls, false));
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:469: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(_chunk_reader->skip_values(ancestor_nulls, false));
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:473: +1, including nesting penalty of 0, nesting level increased to 1
if (!align_rows) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:477: +1, including nesting penalty of 0, nesting level increased to 1
if (_chunk_reader->remaining_num_values() == 0) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:478: +2, including nesting penalty of 1, nesting level increased to 2
if (_chunk_reader->has_next_page()) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:479: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(_chunk_reader->next_page());
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:479: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(_chunk_reader->next_page());
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:480: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(_chunk_reader->load_page_data());
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:480: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(_chunk_reader->load_page_data());
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:483: +1, nesting level increased to 2
} else {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:488: +1, including nesting penalty of 0, nesting level increased to 1
if (current_filter_map->has_filter()) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:489: +2, including nesting penalty of 1, nesting level increased to 2
if (current_filter_map->filter_all()) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:492: +1, nesting level increased to 2
} else {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:514: +1, including nesting penalty of 0, nesting level increased to 1
if (_rep_levels.size() > 0) {
^| return Status::OK(); | ||
| } | ||
|
|
||
| Status ScalarColumnReader::read_column_data(ColumnPtr& doris_column, DataTypePtr& type, |
There was a problem hiding this comment.
warning: function 'read_column_data' has cognitive complexity of 81 (threshold 50) [readability-function-cognitive-complexity]
Status ScalarColumnReader::read_column_data(ColumnPtr& doris_column, DataTypePtr& type,
^Additional context
be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:555: +1, including nesting penalty of 0, nesting level increased to 1
if (_converter == nullptr) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:558: +2, including nesting penalty of 1, nesting level increased to 2
if (!_converter->support()) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:567: +1, including nesting penalty of 0, nesting level increased to 1
do {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:568: +2, including nesting penalty of 1, nesting level increased to 2
if (_chunk_reader->remaining_num_values() == 0) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:569: +3, including nesting penalty of 2, nesting level increased to 3
if (!_chunk_reader->has_next_page()) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:574: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(_chunk_reader->next_page());
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:574: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(_chunk_reader->next_page());
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:576: +2, including nesting penalty of 1, nesting level increased to 2
if (_nested_column) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:577: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(_chunk_reader->load_page_data_idempotent());
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:577: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(_chunk_reader->load_page_data_idempotent());
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:578: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(_read_nested_column(resolved_column, resolved_type, filter_map,
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:578: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(_read_nested_column(resolved_column, resolved_type, filter_map,
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:588: +2, including nesting penalty of 1, nesting level increased to 2
if (read_ranges.size() == 0) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:591: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(_chunk_reader->skip_page());
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:591: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(_chunk_reader->skip_page());
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:593: +1, nesting level increased to 2
} else {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:597: +3, including nesting penalty of 2, nesting level increased to 3
if (filter_map.has_filter() && filter_map.filter_ratio() > 0.6) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:597: +1
if (filter_map.has_filter() && filter_map.filter_ratio() > 0.6) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:603: +4, including nesting penalty of 3, nesting level increased to 4
if (batch_size >= remaining_num_values &&
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:603: +1
if (batch_size >= remaining_num_values &&
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:608: +5, including nesting penalty of 4, nesting level increased to 5
RETURN_IF_ERROR(_chunk_reader->skip_page());
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:608: +6, including nesting penalty of 5, nesting level increased to 6
RETURN_IF_ERROR(_chunk_reader->skip_page());
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:610: +5, including nesting penalty of 4, nesting level increased to 5
if (!_chunk_reader->has_next_page()) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:615: +1
skip_whole_batch = batch_size <= remaining_num_values &&
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:617: +4, including nesting penalty of 3, nesting level increased to 4
if (skip_whole_batch) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:622: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(_chunk_reader->load_page_data_idempotent());
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:622: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(_chunk_reader->load_page_data_idempotent());
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:647: +2, including nesting penalty of 1, nesting level increased to 2
if (_chunk_reader->remaining_num_values() == 0 && !_chunk_reader->has_next_page()) {
^
|
run buildall |
| return Status::OK(); | ||
| } | ||
|
|
||
| Status RowGroupReader::_do_lazy_read(Block* block, size_t batch_size, size_t* read_rows, |
There was a problem hiding this comment.
warning: function '_do_lazy_read' has cognitive complexity of 92 (threshold 50) [readability-function-cognitive-complexity]
Status RowGroupReader::_do_lazy_read(Block* block, size_t batch_size, size_t* read_rows,
^Additional context
be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:445: +1, including nesting penalty of 0, nesting level increased to 1
for (uint32_t i = 0; i < origin_column_num; ++i) {
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:450: +1, including nesting penalty of 0, nesting level increased to 1
while (!_state->is_cancelled()) {
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:455: +2, including nesting penalty of 1, nesting level increased to 2
RETURN_IF_ERROR(_read_column_data(block, _lazy_read_ctx.predicate_columns.first, batch_size,
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:455: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(_read_column_data(block, _lazy_read_ctx.predicate_columns.first, batch_size,
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:457: +2, including nesting penalty of 1, nesting level increased to 2
if (pre_read_rows == 0) {
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:462: +2, including nesting penalty of 1, nesting level increased to 2
RETURN_IF_ERROR(_fill_partition_columns(block, pre_read_rows,
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:462: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(_fill_partition_columns(block, pre_read_rows,
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:464: +2, including nesting penalty of 1, nesting level increased to 2
RETURN_IF_ERROR(_fill_missing_columns(block, pre_read_rows,
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:464: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(_fill_missing_columns(block, pre_read_rows,
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:467: +2, including nesting penalty of 1, nesting level increased to 2
RETURN_IF_ERROR(_build_pos_delete_filter(pre_read_rows));
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:467: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(_build_pos_delete_filter(pre_read_rows));
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:470: +2, including nesting penalty of 1, nesting level increased to 2
if (_lazy_read_ctx.resize_first_column) {
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:478: +2, including nesting penalty of 1, nesting level increased to 2
if (_position_delete_ctx.has_filter) {
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:489: +2, including nesting penalty of 1, nesting level increased to 2
RETURN_IF_ERROR(VExprContext::execute_conjuncts(filter_contexts, &filters, block,
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:489: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(VExprContext::execute_conjuncts(filter_contexts, &filters, block,
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:493: +2, including nesting penalty of 1, nesting level increased to 2
if (_lazy_read_ctx.resize_first_column) {
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:500: +2, including nesting penalty of 1, nesting level increased to 2
RETURN_IF_ERROR(filter_map_ptr->init(filter_map_data, pre_read_rows, can_filter_all));
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:500: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(filter_map_ptr->init(filter_map_data, pre_read_rows, can_filter_all));
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:501: +2, including nesting penalty of 1, nesting level increased to 2
if (filter_map_ptr->filter_all()) {
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:514: +3, including nesting penalty of 2, nesting level increased to 3
if (!pre_eof) {
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:515: +4, including nesting penalty of 3, nesting level increased to 4
if (pre_raw_read_rows >= config::doris_scanner_row_num) {
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:520: +1, nesting level increased to 3
} else { // pre_eof
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:528: +1, nesting level increased to 2
} else {
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:532: +1, including nesting penalty of 0, nesting level increased to 1
if (_state->is_cancelled()) {
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:536: +1, including nesting penalty of 0, nesting level increased to 1
if (filter_map_ptr == nullptr) {
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:545: +1, including nesting penalty of 0, nesting level increased to 1
if (_cached_filtered_rows != 0) {
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:546: +2, including nesting penalty of 1, nesting level increased to 2
RETURN_IF_ERROR(_rebuild_filter_map(filter_map, rebuild_filter_map, pre_read_rows));
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:546: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(_rebuild_filter_map(filter_map, rebuild_filter_map, pre_read_rows));
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:554: +1, including nesting penalty of 0, nesting level increased to 1
RETURN_IF_ERROR(_read_column_data(block, _lazy_read_ctx.lazy_read_columns, pre_read_rows,
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:554: +2, including nesting penalty of 1, nesting level increased to 2
RETURN_IF_ERROR(_read_column_data(block, _lazy_read_ctx.lazy_read_columns, pre_read_rows,
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:557: +1, including nesting penalty of 0, nesting level increased to 1
if (pre_read_rows != lazy_read_rows) {
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:564: +1, including nesting penalty of 0, nesting level increased to 1
if (filter_map.has_filter()) {
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:565: +2, including nesting penalty of 1, nesting level increased to 2
if (block->columns() == origin_column_num) {
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:569: +1, nesting level increased to 2
} else {
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:570: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_CATCH_EXCEPTION(Block::filter_block_internal(
^be/src/common/exception.h:79: expanded from macro 'RETURN_IF_CATCH_EXCEPTION'
do { \
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:570: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_CATCH_EXCEPTION(Block::filter_block_internal(
^be/src/common/exception.h:84: expanded from macro 'RETURN_IF_CATCH_EXCEPTION'
} catch (const doris::Exception& e) { \
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:570: +5, including nesting penalty of 4, nesting level increased to 5
RETURN_IF_CATCH_EXCEPTION(Block::filter_block_internal(
^be/src/common/exception.h:85: expanded from macro 'RETURN_IF_CATCH_EXCEPTION'
if (e.code() == doris::ErrorCode::MEM_ALLOC_FAILED) { \
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:574: +1, nesting level increased to 1
} else {
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:582: +1, including nesting penalty of 0, nesting level increased to 1
for (int i = 0; i < column_num; ++i) {
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:584: +2, including nesting penalty of 1, nesting level increased to 2
if (column_size != 0 && cz != 0) {
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:584: +1
if (column_size != 0 && cz != 0) {
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:587: +2, including nesting penalty of 1, nesting level increased to 2
if (cz != 0) {
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:595: +1, including nesting penalty of 0, nesting level increased to 1
RETURN_IF_ERROR(_fill_partition_columns(block, column_size, _lazy_read_ctx.partition_columns));
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:595: +2, including nesting penalty of 1, nesting level increased to 2
RETURN_IF_ERROR(_fill_partition_columns(block, column_size, _lazy_read_ctx.partition_columns));
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:596: +1, including nesting penalty of 0, nesting level increased to 1
RETURN_IF_ERROR(_fill_missing_columns(block, column_size, _lazy_read_ctx.missing_columns));
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:596: +2, including nesting penalty of 1, nesting level increased to 2
RETURN_IF_ERROR(_fill_missing_columns(block, column_size, _lazy_read_ctx.missing_columns));
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^
kaka11chen
force-pushed
the
parquet_complex_type_late_materialization
branch
from
210df93 to
343c348
Compare
| * A row of complex type may be stored across two(or more) pages, and the parameter `align_rows` indicates that | ||
| * whether the reader should read the remaining value of the last row in previous page. | ||
| */ | ||
| Status ScalarColumnReader::_read_nested_column(ColumnPtr& doris_column, DataTypePtr& type, |
There was a problem hiding this comment.
warning: function '_read_nested_column' has cognitive complexity of 124 (threshold 50) [readability-function-cognitive-complexity]
Status ScalarColumnReader::_read_nested_column(ColumnPtr& doris_column, DataTypePtr& type,
^Additional context
be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:328: +1, including nesting penalty of 0, nesting level increased to 1
if (align_rows) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:333: +1, nesting level increased to 1
} else {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:336: +2, including nesting penalty of 1, nesting level increased to 2
if (_nested_filter_map_data) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:346: +1, including nesting penalty of 0, nesting level increased to 1
if (has_rep_level) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:349: +2, including nesting penalty of 1, nesting level increased to 2
while (parsed_rows <= batch_size && remaining_values > 0) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:349: +1
while (parsed_rows <= batch_size && remaining_values > 0) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:351: +3, including nesting penalty of 2, nesting level increased to 3
if (rep_level == 0) { // rep_level 0 indicates start of new row
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:352: +4, including nesting penalty of 3, nesting level increased to 4
if (parsed_rows == batch_size) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:363: +2, including nesting penalty of 1, nesting level increased to 2
if (filter_map.has_filter() && (!filter_map.filter_all())) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:363: +1
if (filter_map.has_filter() && (!filter_map.filter_all())) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:364: +3, including nesting penalty of 2, nesting level increased to 3
if (_nested_filter_map_data == nullptr) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:367: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(filter_map.generate_nested_filter_map(
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:367: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(filter_map.generate_nested_filter_map(
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:373: +1, nesting level increased to 1
} else if (!align_rows) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:383: +1, including nesting penalty of 0, nesting level increased to 1
if (has_def_level) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:385: +1, nesting level increased to 1
} else {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:393: +1, including nesting penalty of 0, nesting level increased to 1
if (doris_column->is_nullable()) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:399: +1, nesting level increased to 1
} else {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:400: +2, including nesting penalty of 1, nesting level increased to 2
if (_field_schema->is_nullable) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:414: +1, including nesting penalty of 0, nesting level increased to 1
while (has_read < origin_size + parsed_values) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:417: +2, including nesting penalty of 1, nesting level increased to 2
while (has_read < origin_size + parsed_values && _def_levels[has_read] == def_level) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:422: +2, including nesting penalty of 1, nesting level increased to 2
if (def_level < _field_schema->repeated_parent_def_level) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:428: +2, including nesting penalty of 1, nesting level increased to 2
if (is_null) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:430: +1, nesting level increased to 2
} else {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:434: +2, including nesting penalty of 1, nesting level increased to 2
if (prev_is_null == is_null && (USHRT_MAX - null_map.back() >= loop_read)) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:434: +1
if (prev_is_null == is_null && (USHRT_MAX - null_map.back() >= loop_read)) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:436: +1, nesting level increased to 2
} else {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:437: +3, including nesting penalty of 2, nesting level increased to 3
if (!(prev_is_null ^ is_null)) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:441: +3, including nesting penalty of 2, nesting level increased to 3
while (remaining > USHRT_MAX) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:454: +1, including nesting penalty of 0, nesting level increased to 1
if (current_filter_map->filter_all()) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:456: +2, including nesting penalty of 1, nesting level increased to 2
if (null_size > 0) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:457: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(_chunk_reader->skip_values(null_size, false));
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:457: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(_chunk_reader->skip_values(null_size, false));
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:459: +2, including nesting penalty of 1, nesting level increased to 2
if (nonnull_size > 0) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:460: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(_chunk_reader->skip_values(nonnull_size, true));
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:460: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(_chunk_reader->skip_values(nonnull_size, true));
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:462: +2, including nesting penalty of 1, nesting level increased to 2
if (ancestor_nulls != 0) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:463: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(_chunk_reader->skip_values(ancestor_nulls, false));
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:463: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(_chunk_reader->skip_values(ancestor_nulls, false));
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:465: +1, nesting level increased to 1
} else {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:469: +2, including nesting penalty of 1, nesting level increased to 2
RETURN_IF_ERROR(
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:469: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:474: +2, including nesting penalty of 1, nesting level increased to 2
RETURN_IF_ERROR(
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:474: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:476: +2, including nesting penalty of 1, nesting level increased to 2
if (ancestor_nulls != 0) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:477: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(_chunk_reader->skip_values(ancestor_nulls, false));
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:477: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(_chunk_reader->skip_values(ancestor_nulls, false));
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:484: +1, including nesting penalty of 0, nesting level increased to 1
if (_chunk_reader->remaining_num_values() == 0) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:485: +2, including nesting penalty of 1, nesting level increased to 2
if (_chunk_reader->has_next_page()) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:486: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(_chunk_reader->next_page());
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:486: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(_chunk_reader->next_page());
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:487: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(_chunk_reader->load_page_data());
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:487: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(_chunk_reader->load_page_data());
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:490: +1, nesting level increased to 2
} else {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:496: +1, including nesting penalty of 0, nesting level increased to 1
if (current_filter_map->has_filter()) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:497: +2, including nesting penalty of 1, nesting level increased to 2
if (current_filter_map->filter_all()) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:500: +1, nesting level increased to 2
} else {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:523: +1, including nesting penalty of 0, nesting level increased to 1
if (_rep_levels.size() > 0) {
^| return Status::OK(); | ||
| } | ||
|
|
||
| Status ScalarColumnReader::read_column_data(ColumnPtr& doris_column, DataTypePtr& type, |
There was a problem hiding this comment.
warning: function 'read_column_data' has cognitive complexity of 81 (threshold 50) [readability-function-cognitive-complexity]
Status ScalarColumnReader::read_column_data(ColumnPtr& doris_column, DataTypePtr& type,
^Additional context
be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:564: +1, including nesting penalty of 0, nesting level increased to 1
if (_converter == nullptr) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:567: +2, including nesting penalty of 1, nesting level increased to 2
if (!_converter->support()) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:576: +1, including nesting penalty of 0, nesting level increased to 1
do {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:577: +2, including nesting penalty of 1, nesting level increased to 2
if (_chunk_reader->remaining_num_values() == 0) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:578: +3, including nesting penalty of 2, nesting level increased to 3
if (!_chunk_reader->has_next_page()) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:583: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(_chunk_reader->next_page());
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:583: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(_chunk_reader->next_page());
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:585: +2, including nesting penalty of 1, nesting level increased to 2
if (_nested_column) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:586: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(_chunk_reader->load_page_data_idempotent());
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:586: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(_chunk_reader->load_page_data_idempotent());
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:587: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(_read_nested_column(resolved_column, resolved_type, filter_map,
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:587: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(_read_nested_column(resolved_column, resolved_type, filter_map,
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:597: +2, including nesting penalty of 1, nesting level increased to 2
if (read_ranges.size() == 0) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:600: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(_chunk_reader->skip_page());
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:600: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(_chunk_reader->skip_page());
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:602: +1, nesting level increased to 2
} else {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:606: +3, including nesting penalty of 2, nesting level increased to 3
if (filter_map.has_filter() && filter_map.filter_ratio() > 0.6) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:606: +1
if (filter_map.has_filter() && filter_map.filter_ratio() > 0.6) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:612: +4, including nesting penalty of 3, nesting level increased to 4
if (batch_size >= remaining_num_values &&
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:612: +1
if (batch_size >= remaining_num_values &&
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:617: +5, including nesting penalty of 4, nesting level increased to 5
RETURN_IF_ERROR(_chunk_reader->skip_page());
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:617: +6, including nesting penalty of 5, nesting level increased to 6
RETURN_IF_ERROR(_chunk_reader->skip_page());
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:619: +5, including nesting penalty of 4, nesting level increased to 5
if (!_chunk_reader->has_next_page()) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:624: +1
skip_whole_batch = batch_size <= remaining_num_values &&
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:626: +4, including nesting penalty of 3, nesting level increased to 4
if (skip_whole_batch) {
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:631: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(_chunk_reader->load_page_data_idempotent());
^be/src/common/status.h:631: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:631: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(_chunk_reader->load_page_data_idempotent());
^be/src/common/status.h:633: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:656: +2, including nesting penalty of 1, nesting level increased to 2
if (_chunk_reader->remaining_num_values() == 0 && !_chunk_reader->has_next_page()) {
^| // specific language governing permissions and limitations | ||
| // under the License. | ||
|
|
||
| #include <gtest/gtest.h> |
There was a problem hiding this comment.
warning: 'gtest/gtest.h' file not found [clang-diagnostic-error]
#include <gtest/gtest.h>
^
|
run buildall |
kaka11chen
force-pushed
the
parquet_complex_type_late_materialization
branch
from
343c348 to
14f5bfc
Compare
|
run buildall |
kaka11chen
force-pushed
the
parquet_complex_type_late_materialization
branch
from
14f5bfc to
a82a456
Compare
|
run buildall |
kaka11chen
force-pushed
the
parquet_complex_type_late_materialization
branch
from
a82a456 to
c635719
Compare
kaka11chen
changed the title
|
run buildall |
kaka11chen
force-pushed
the
parquet_complex_type_late_materialization
branch
from
c635719 to
e4bad17
Compare
|
run buildall |
|
TeamCity be ut coverage result: |
|
run buildall |
|
TeamCity be ut coverage result: |
morningman
force-pushed
the
parquet_complex_type_late_materialization
branch
from
815442d to
20b44a6
Compare
|
run buildall |
TPC-H: Total hot run time: 32676 ms |
TPC-DS: Total hot run time: 190737 ms |
|
TeamCity be ut coverage result: |
ClickBench: Total hot run time: 31.58 s |
|
run external |
|
PR approved by at least one committer and no changes requested. |
github-actions
bot
added
the
approved
…x types. (#44098) ### What problem does this PR solve? Problem Summary: Late materialization is not supported when querying fields with complex types. ### Release note [opt](parquet-reader)Implement late materialization of parquet complex types.
…x types. (apache#232) Cherry-pick apache#44098
…x types. (#44098) ### What problem does this PR solve? Problem Summary: Late materialization is not supported when querying fields with complex types. ### Release note [opt](parquet-reader)Implement late materialization of parquet complex types.
4 participants
What problem does this PR solve?
Problem Summary:
Late materialization is not supported when querying fields with complex types.
Release note
Split
ColumnSelectVectorinto two components:FilterMapandColumnSelectVector。FilterMapbecomes stateless, withfilter_map_indexnow being tracked by individualColumnReaders. This separation provides better encapsulation and more flexible filter management.During reading operations, generates new nested filter maps dynamically. Implements dual tracking mechanism:
Uses
filter_map_indexfor current filter map tracking. Maintains_orig_filter_map_indexto preserve original filter map tracking.Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)