v2 segment support string encode(#1766) #1816

wangbo · 2019-09-17T05:49:39Z

#1766
major change

change data format of binary dict page, appending (dict page data) and (dict page offset) to binary dict page;
add new decoding method for new binary dict page format
add ut for segment test
set the elements of initial array to 0 ,when calling arena.AllocateNewBlock
hard code way to choose dict coding for string

0919 commit major change

change dict file format:when saving binary dict page, separate dict page from dict page,one dict page may have multi data pages;when reading a binary dict page,one ColumnReader keeps one dict page
loading dict when calling column_reader._read_page
3.rollback BinaryDictPage
no longer using memset(0) to inital column_zonemap.max_value

0926 17 commit major change

init column_zone_map min value column_zone_map slice's data array;
set char/varchar column_zone_map'max value size to 0
add ut for char column zone map query hit/miss

0929 10 commit major change

allocate mem for column_zone_map 's max and min value
direct copy content to column_zone_map's max and min value

be/src/olap/rowset/segment_v2/binary_dict_page.cpp

be/src/util/arena.cpp

be/src/olap/rowset/segment_v2/column_reader.cpp

be/src/olap/rowset/segment_v2/column_reader.h

be/src/olap/rowset/segment_v2/binary_dict_page.h

be/src/olap/rowset/segment_v2/column_reader.cpp

be/src/olap/rowset/segment_v2/column_reader.h

be/src/olap/rowset/segment_v2/column_reader.cpp

be/src/olap/rowset/segment_v2/segment_writer.cpp

keep dict page handle in FileColumnIterator;

be/src/olap/rowset/segment_v2/binary_dict_page.h

be/src/olap/rowset/segment_v2/column_reader.cpp

be/src/olap/rowset/segment_v2/column_reader.h

be/src/olap/rowset/segment_v2/segment_writer.cpp

be/test/olap/rowset/segment_v2/binary_dict_page_test.cpp

be/src/olap/rowset/segment_v2/column_reader.h

be/test/olap/rowset/segment_v2/segment_test.cpp

gaodayue · 2019-09-25T08:45:42Z

be/test/olap/rowset/segment_v2/segment_test.cpp

+        for (int j = 0; j < 4; ++j) {
+            auto cell = row.cell(j);
+            cell.set_not_null();
+            set_column_value_by_type(tablet_schema->_cols[j]._type, i * 10 + j, (char*)cell.mutable_cell_ptr(), tablet_schema->_cols[j]._length);


I'm confused, how can you access private field _cols and _type?

me too;not FRIEND_TEST,not "#define private public"

be/test/olap/rowset/segment_v2/segment_test.cpp

be/src/olap/rowset/segment_v2/binary_dict_page.h

be/src/olap/rowset/segment_v2/column_zone_map.cpp

be/src/olap/rowset/segment_v2/encoding_info.cpp

be/src/olap/rowset/segment_v2/binary_dict_page.h

be/src/olap/rowset/segment_v2/column_reader.cpp

kangpinghuang · 2019-09-29T01:58:10Z

be/src/olap/rowset/segment_v2/column_zone_map.cpp

+    } else if (_type_info->type() == OLAP_FIELD_TYPE_CHAR) {
+        Slice *min_value = (Slice *)_zone_map.min_value;
+        min_value->data = _max_char_value;
+        min_value->size = OLAP_CHAR_MAX_LENGTH;


why set char type's max length to 255? I think it should be equal to OLAP_STRING_MAX_LENGTH.

Fe defines max char and varchar length,see ScalarType;this is a temporary define,later I will use columns real length

yes, for char type, it is a bit complex

gaodayue

LGTM generally. One thing I'm not sure is the change to varchar's set_to_max since it's been used by segment v1's zonemap. Would it break backward compatibility? @kangpinghuang Could you verify it?

be/src/olap/types.h

gaodayue · 2019-09-29T09:56:54Z

be/src/olap/types.h

@@ -587,13 +604,19 @@ struct FieldTypeTraits<OLAP_FIELD_TYPE_VARCHAR> : public FieldTypeTraits<OLAP_FI
    }
    static void set_to_max(void* buf) {
        auto slice = reinterpret_cast<Slice*>(buf);
-        slice->size = 1;
-        memset(slice->data, 0xFF, 1);
+        memset(slice->data, 0xFF, slice->size);


should first reset slice->size to OLAP_STRING_MAX_LENGTH

be/src/olap/types.h

be/src/olap/field.h

be/src/olap/rowset/segment_v2/column_zone_map.h

2. interpret not cast 3.remove useless var 4. set_to_min/max consist with master

kangpinghuang · 2019-09-30T02:11:20Z

I have check set_to_max in olap, I think original code has bug and this pr's code has fix it.
I check set_to_max used in two place of v1:

in zone map building. But for char/varchar type, there are no zone map for them, so there is no influence.
in split_range for query, I think the new max is larger than the old one, will not cause problems.
LGTM

kangpinghuang

LGTM

gaodayue

LGTM

wangbo · 2019-09-30T02:31:08Z

current commit is old version:in varchar's set_to_max,slice size set to 1
because set to max_string_value will cause delete_handler_test core dump,I will see this case today

be/src/olap/rowset/segment_v2/column_reader.cpp

be/src/olap/types.h

wangbo · 2019-09-30T04:38:04Z

current commit is old version:in varchar's set_to_max,slice size set to 1
because set to max_string_value will cause delete_handler_test core dump,I will see this case today

Core dump stack

#0 SLL_Next (t=0xffffffffffffffff) at src/linked_list.h:45
#1 SLL_TryPop (rv=, list=0x436dce0) at src/linked_list.h:69
#2 TryPop (rv=, this=0x436dce0) at src/thread_cache.h:220
#3 Allocate (oom_handler=0x25e5700 <tcmalloc::cpp_throw_oom(unsigned long)>, cl=5, size=64, this=) at src/thread_cache.h:379
#4 malloc_fast_pathtcmalloc::cpp_throw_oom (size=64) at src/tcmalloc.cc:1855
#5 tc_new (size=64) at src/tcmalloc.cc:1976
#6 0x0000000000c21672 in doris::WrapperField::WrapperField (this=0x4e07d40, rep=, variable_len=,
is_string_type=) at /home/wangbo/incubator-doris/be/src/olap/wrapper_field.cpp:84
#7 0x0000000000c21b9e in doris::WrapperField::create (column=..., len=, len@entry=0)
at /home/wangbo/incubator-doris/be/src/olap/wrapper_field.cpp:51
#8 0x0000000000cb88e1 in doris::ColumnDataWriter::init (this=0x4bc4780) at /home/wangbo/incubator-doris/be/src/olap/rowset/column_data_writer.cpp:72
#9 0x0000000000ca273c in doris::AlphaRowsetWriter::_init (this=this@entry=0x4bf83c0)
at /home/wangbo/incubator-doris/be/src/olap/rowset/alpha_rowset_writer.cpp:275
#10 0x0000000000ca3694 in doris::AlphaRowsetWriter::init (this=0x4bf83c0, rowset_writer_context=...)
at /home/wangbo/incubator-doris/be/src/olap/rowset/alpha_rowset_writer.cpp:73
#11 0x0000000000c91b03 in doris::RowsetFactory::create_rowset_writer (context=..., output=output@entry=0x7fffcdf75318)
at /home/wangbo/incubator-doris/be/src/olap/rowset/rowset_factory.cpp:50
#12 0x0000000000bc9b3f in doris::TabletManager::_create_inital_rowset (this=this@entry=0x4d5a180, tablet=..., request=...)
at /home/wangbo/incubator-doris/be/src/olap/tablet_manager.cpp:1250
#13 0x0000000000bcc919 in doris::TabletManager::_internal_create_tablet (this=this@entry=0x4d5a180, alter_type=alter_type@entry=doris::SCHEMA_CHANGE,
request=..., is_schema_change_tablet=is_schema_change_tablet@entry=false, ref_tablet=..., data_dirs=...)
at /home/wangbo/incubator-doris/be/src/olap/tablet_manager.cpp:371
#14 0x0000000000bcdd72 in doris::TabletManager::create_tablet (this=this@entry=0x4d5a180, request=..., stores=...)
at /home/wangbo/incubator-doris/be/src/olap/tablet_manager.cpp:318
#15 0x0000000000b82642 in doris::StorageEngine::create_tablet (this=, request=...)
at /home/wangbo/incubator-doris/be/src/olap/storage_engine.cpp:760
#16 0x0000000000b4fac3 in doris::TestDeleteConditionHandler::SetUp (this=0x4e08460)
at /home/wangbo/incubator-doris/be/test/olap/delete_handler_test.cpp:173
#17 0x0000000002530f37 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::)(), char const) ()
#18 0x000000000252ba9c in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::)(), char const) ()
#19 0x0000000002510df7 in testing::Test::Run() ()
#20 0x00000000025116ce in testing::TestInfo::Run() ()
#21 0x0000000002511d25 in testing::TestCase::Run() ()
#22 0x0000000002518899 in testing::internal::UnitTestImpl::RunAllTests() ()
#23 0x0000000002531eb3 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::)(), char const) ()
#24 0x000000000252c7a2 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::)(), char const) ()
#25 0x0000000002517570 in testing::UnitTest::Run() ()
#26 0x00000000009fb316 in RUN_ALL_TESTS () at /home/wangbo/incubator-doris/thirdparty/installed/include/gtest/gtest.h:2233
#27 main (argc=, argv=0x7fffcdf75f88) at /home/wangbo/incubator-doris/be/test/olap/delete_handler_test.cpp:951

core dump happends in wrapper_field.cpp

line:80 _string_content.reset(new char[slice->size]);
the slice->size is 16,it seems reasonable
the code logic here is that we want to create a varchar wrapper_field

before that in column_data_writer.cpp

line:68 WrapperField::create // create varchar wrapper_field first
line:70 _zone_maps[i].first->set_to_max();
...
line:72 WrapperField::create # core dump here,when we want to create a another varchar wrapper_field

in line:68,we get a char[64]
in line:70, varchar-> set_to_max, memset(char[64],0xff,65536)
it seems that unreasonable,correct is:"memset(char[64],0xff,strlen(char[64])])"
So I think that we show set_to_max with the correct column length.

I prepare to solve it with a new pr which allocate ColumnZoneMapBuilder max/min value with real column length

wangbo · 2019-09-30T07:56:34Z

@imay please review again

add comment for dict lazy init
2.merge master branch to resolve conflict

imay

LGTM

wangbo added 3 commits September 17, 2019 13:45

v2 segment support string encode(apache#1766)

0835bc1

Merge remote-tracking branch 'upstream/master'

b7e5bb7

segment test using uint64

2380732

imay requested changes Sep 17, 2019

View reviewed changes

be/src/olap/rowset/segment_v2/binary_dict_page.cpp Outdated Show resolved Hide resolved

be/src/olap/rowset/segment_v2/binary_dict_page.cpp Outdated Show resolved Hide resolved

be/src/util/arena.cpp Outdated Show resolved Hide resolved

wangbo added 2 commits September 19, 2019 19:21

update commit

0a40fa8

update zone map

61f2868

imay reviewed Sep 21, 2019

View reviewed changes

be/src/olap/rowset/segment_v2/column_reader.cpp Outdated Show resolved Hide resolved

be/src/olap/rowset/segment_v2/column_reader.h Outdated Show resolved Hide resolved

gaodayue requested changes Sep 22, 2019

View reviewed changes

wangbo added 2 commits September 24, 2019 12:07

update commit 0924:

12e31ef

keep dict page handle in FileColumnIterator;

Merge remote-tracking branch 'upstream/master' into master_test

c191220

gaodayue requested changes Sep 24, 2019

View reviewed changes

update commit 0925 10

52195ad

gaodayue requested changes Sep 25, 2019

View reviewed changes

wangbo added 2 commits September 25, 2019 20:31

update commit 0925 8

2d519fc

Merge remote-tracking branch 'upstream/master'

694b1be

kangpinghuang requested changes Sep 25, 2019

View reviewed changes

kangpinghuang reviewed Sep 26, 2019

View reviewed changes

be/src/olap/rowset/segment_v2/binary_dict_page.h Outdated Show resolved Hide resolved

kangpinghuang reviewed Sep 26, 2019

View reviewed changes

be/src/olap/rowset/segment_v2/column_reader.cpp Show resolved Hide resolved

update commit 0926:17

349b251

wangbo force-pushed the master branch from 56d951a to 349b251 Compare September 27, 2019 07:19

re fresh commit

ec186d8

wangbo force-pushed the master branch from d740714 to ec186d8 Compare September 27, 2019 07:28

wangbo added 4 commits September 27, 2019 17:58

reuse column zone map char array

118bce2

reuse column zone map char array

c09aabc

add get_type_value_with_arena

2489045

Merge branch 'upstream_master' into master_test

839bbd8

kangpinghuang requested changes Sep 29, 2019

View reviewed changes

Merge branch 'master_test'

40a498a

gaodayue requested changes Sep 29, 2019

View reviewed changes

1. use allocate from

429d372

2. interpret not cast 3.remove useless var 4. set_to_min/max consist with master

wangbo force-pushed the master branch from 9754aeb to 429d372 Compare September 29, 2019 14:48

refresh commit

98fdf6f

kangpinghuang previously approved these changes Sep 30, 2019

View reviewed changes

gaodayue previously approved these changes Sep 30, 2019

View reviewed changes

wangbo closed this Sep 30, 2019

wangbo reopened this Sep 30, 2019

imay reviewed Sep 30, 2019

View reviewed changes

be/src/olap/rowset/segment_v2/column_reader.cpp Show resolved Hide resolved

be/src/olap/types.h Show resolved Hide resolved

add commit for lazy init dict

e0d6ca0

wangbo dismissed stale reviews from gaodayue and kangpinghuang via e0d6ca0 September 30, 2019 05:03

Merge remote-tracking branch 'remotes/upstream/master' into master_test

abcf356

imay approved these changes Sep 30, 2019

View reviewed changes

imay merged commit 8aa8e08 into apache:master Sep 30, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

v2 segment support string encode(#1766) #1816

v2 segment support string encode(#1766) #1816

wangbo commented Sep 17, 2019 •

edited

Loading

gaodayue Sep 25, 2019

wangbo Sep 25, 2019

kangpinghuang Sep 29, 2019

wangbo Sep 29, 2019

kangpinghuang Sep 30, 2019

gaodayue left a comment

gaodayue Sep 29, 2019

wangbo Sep 29, 2019

kangpinghuang commented Sep 30, 2019

kangpinghuang left a comment

gaodayue left a comment

wangbo commented Sep 30, 2019

wangbo commented Sep 30, 2019 •

edited

Loading

wangbo commented Sep 30, 2019

imay left a comment

v2 segment support string encode(#1766) #1816

v2 segment support string encode(#1766) #1816

Conversation

wangbo commented Sep 17, 2019 • edited Loading

gaodayue Sep 25, 2019

Choose a reason for hiding this comment

wangbo Sep 25, 2019

Choose a reason for hiding this comment

kangpinghuang Sep 29, 2019

Choose a reason for hiding this comment

wangbo Sep 29, 2019

Choose a reason for hiding this comment

kangpinghuang Sep 30, 2019

Choose a reason for hiding this comment

gaodayue left a comment

Choose a reason for hiding this comment

gaodayue Sep 29, 2019

Choose a reason for hiding this comment

wangbo Sep 29, 2019

Choose a reason for hiding this comment

kangpinghuang commented Sep 30, 2019

kangpinghuang left a comment

Choose a reason for hiding this comment

gaodayue left a comment

Choose a reason for hiding this comment

wangbo commented Sep 30, 2019

wangbo commented Sep 30, 2019 • edited Loading

wangbo commented Sep 30, 2019

imay left a comment

Choose a reason for hiding this comment

wangbo commented Sep 17, 2019 •

edited

Loading

wangbo commented Sep 30, 2019 •

edited

Loading