Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Urgent]parquet-cpp write a row done without column type exception ,but cause a segfault #13339

Closed
helloqingbing opened this issue Jun 8, 2022 · 1 comment
Closed

[Urgent]parquet-cpp write a row done without column type exception ,but cause a segfault #13339

helloqingbing opened this issue Jun 8, 2022 · 1 comment

Comments

@helloqingbing
Copy link

Program terminated with signal 11, Segmentation fault.
#0 0x00007fef761c475f in apache::thrift::transport::TMemoryBuffer::borrowSlow(unsigned char*, unsigned int*) () from /data/smash-data/kv-1/yunzhe/redkvreplicator/deps/thrift/lib64/libthrift-0.9.1.so
Missing separate debuginfos, use: debuginfo-install glibc-2.17-307.el7.1.x86_64 libevent-2.0.21-4.el7.x86_64 snappy-1.1.0-3.el7.x86_64
(gdb) bt
#0 0x00007fef761c475f in apache::thrift::transport::TMemoryBuffer::borrowSlow(unsigned char*, unsigned int*) () from /data/smash-data/kv-1/yunzhe/redkvreplicator/deps/thrift/lib64/libthrift-0.9.1.so
#1 0x00007fef7057de7f in apache::thrift::protocol::TVirtualProtocol<apache::thrift::protocol::TCompactProtocolTapache::thrift::transport::TMemoryBuffer, apache::thrift::protocol::TProtocolDefaults>::writeBinary_virt(std::string const&) () from /data/smash-data/kv-1/yunzhe/redkvreplicator/deps/libs/libparquet.so.700
#2 0x00007fef7063019d in parquet::format::Statistics::write(apache::thrift::protocol::TProtocol*) const () from /data/smash-data/kv-1/yunzhe/redkvreplicator/deps/libs/libparquet.so.700
#3 0x00007fef70633eda in parquet::format::ColumnMetaData::write(apache::thrift::protocol::TProtocol*) const () from /data/smash-data/kv-1/yunzhe/redkvreplicator/deps/libs/libparquet.so.700
#4 0x00007fef706341e9 in parquet::format::ColumnChunk::write(apache::thrift::protocol::TProtocol*) const () from /data/smash-data/kv-1/yunzhe/redkvreplicator/deps/libs/libparquet.so.700
#5 0x00007fef706332bc in parquet::format::RowGroup::write(apache::thrift::protocol::TProtocol*) const () from /data/smash-data/kv-1/yunzhe/redkvreplicator/deps/libs/libparquet.so.700
#6 0x00007fef7063372c in parquet::format::FileMetaData::write(apache::thrift::protocol::TProtocol*) const () from /data/smash-data/kv-1/yunzhe/redkvreplicator/deps/libs/libparquet.so.700
#7 0x00007fef70625e22 in void parquet::ThriftSerializer::SerializeObjectparquet::format::FileMetaData(parquet::format::FileMetaData const*) ()
from /data/smash-data/kv-1/yunzhe/redkvreplicator/deps/libs/libparquet.so.700
#8 0x00007fef706291ad in parquet::FileMetaData::FileMetaDataImpl::WriteTo(arrow::io::OutputStream*, std::shared_ptrparquet::Encryptor const&) const ()
from /data/smash-data/kv-1/yunzhe/redkvreplicator/deps/libs/libparquet.so.700
#9 0x00007fef70612a9d in parquet::WriteFileMetaData(parquet::FileMetaData const&, arrow::io::OutputStream*) () from /data/smash-data/kv-1/yunzhe/redkvreplicator/deps/libs/libparquet.so.700
#10 0x00007fef70616521 in parquet::FileSerializer::Close() () from /data/smash-data/kv-1/yunzhe/redkvreplicator/deps/libs/libparquet.so.700
#11 0x00007fef7061276f in parquet::ParquetFileWriter::Close() () from /data/smash-data/kv-1/yunzhe/redkvreplicator/deps/libs/libparquet.so.700
#12 0x00007fef7061287c in parquet::ParquetFileWriter::~ParquetFileWriter() () from /data/smash-data/kv-1/yunzhe/redkvreplicator/deps/libs/libparquet.so.700
#13 0x00000000005cfa5c in operator() (this=, __ptr=0x7fef62505e80) at /opt/rh/devtoolset-7/root/usr/include/c++/7/bits/unique_ptr.h:78
#14 ~unique_ptr (this=0x7fef603fdf70, __in_chrg=) at /opt/rh/devtoolset-7/root/usr/include/c++/7/bits/unique_ptr.h:268
#15 ~StreamWriter (this=0x7fef603fdf50, __in_chrg=) at /data/smash-data/kv-1/yunzhe/redkvreplicator/deps/include/parquet/stream_writer.h:74
#16 ParquetInputfile::BuildParquetInputfile (this=this@entry=0x7fef603fe370, type=type@entry=ROCKSDB_REDIS_HASH, table_name=..., columns=...)
at /data/smash-data/kv-1/yunzhe/redkvreplicator/rocksdb_replicator/parquet_inputfile.cpp:264
#17 0x00000000005afa91 in replicator::RocksDBReplicator::scanTable (this=, db_addr=..., table_info=...)
at /data/smash-data/kv-1/yunzhe/redkvreplicator/rocksdb_replicator/rocksdb_replicator.cpp:293
#18 0x0000000000584fc3 in ScanTable (params=..., table_name=...) at /data/smash-data/kv-1/yunzhe/redkvreplicator/rocksdb_replicator/ReplicatorAdmin.cpp:68
#19 ScanTableCommandHandler::Run (this=, args=..., output=0x7fef603febc0) at /data/smash-data/kv-1/yunzhe/redkvreplicator/rocksdb_replicator/ReplicatorAdmin.cpp:147
#20 0x00007fef741be28d in brpc::policy::ConsumeCommand (ctx=ctx@entry=0x7fef5ce8e070, args=..., flush_batched=flush_batched@entry=true, appender=appender@entry=0x7fef603fed00)
at src/brpc/policy/redis_protocol.cpp:99
#21 0x00007fef741bee0c in brpc::policy::ParseRedisMessage (source=0x7fef5cee56c0, socket=0x7fef5cee5640, read_eof=, arg=) at src/brpc/policy/redis_protocol.cpp:181
#22 0x00007fef740eac9e in brpc::InputMessenger::CutInputMessage (this=this@entry=0x7fef6c253c00, m=m@entry=0x7fef5cee5640, index=index@entry=0x7fef603feec0, read_eof=read_eof@entry=false)
at src/brpc/input_messenger.cpp:71
#23 0x00007fef740eb5c4 in brpc::InputMessenger::OnNewMessages (m=0x7fef5cee5640) at src/brpc/input_messenger.cpp:234
#24 0x00007fef7408b61d in brpc::Socket::ProcessEvent (arg=0x7fef5cee5640) at src/brpc/socket.cpp:1017
#25 0x00007fef73fdbfd7 in bthread::TaskGroup::task_runner (skip_remained=) at src/bthread/task_group.cpp:295
#26 0x00007fef73fe05e1 in bthread_make_fcontext () from /data/smash-data/kv-1/yunzhe/redkvreplicator/deps/incubator-brpc/lib/libbrpc.so
#27 0x0000000000000000 in ?? ()
@helloqingbing helloqingbing changed the title parquet-cpp write a row done without column type exception ,but cause a segfault [Urgent]parquet-cpp write a row done without column type exception ,but cause a segfault Jun 8, 2022
@helloqingbing
Copy link
Author

  1. You can only write 6 columns for thrift-0.9.1
  2. To fix the issue, you need thrift-0.12.1+, then it can support more columns in parquet

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@helloqingbing