Ref page lifetime management is broken

[jiaqizho]

Bug Report

Please answer these questions before submitting your issue. Thanks!

[2022/02/14 13:52:07.854 +08:00] [ERROR] [DAGDriver.cpp:184] ["DAGDriver:void DB::DAGDriver<batch>::execute() [with bool batch = true]: DB Exception: Cannot open file /xxxxxxxx/data/t_65/stable/dmf_2/%2D1.mrk, errno: 2, strerror: No such file or directory
0. bin/tiflash/tiflash(StackTrace::StackTrace()+0x16) [0x36d32b6]
1. bin/tiflash/tiflash(DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int)+0x26) [0x36c6e36]
2. bin/tiflash/tiflash(DB::throwFromErrno(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, int)+0x1ad) [0x36ccc1d]
3. bin/tiflash/tiflash(DB::PosixRandomAccessFile::PosixRandomAccessFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, std::shared_ptr<DB::ReadLimiter> const&)+0x1d8) [0x7460ae8]
4. bin/tiflash/tiflash(DB::FileProvider::newRandomAccessFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, DB::EncryptionPath const&, std::shared_ptr<DB::ReadLimiter> const&, int) const+0x68) [0x745c458]
5. bin/tiflash/tiflash() [0x76b66fb]
6. bin/tiflash/tiflash(DB::DM::DMFileReader::Stream::Stream(DB::DM::DMFileReader&, long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, unsigned long, Poco::Logger*, std::shared_ptr<DB::ReadLimiter> const&)+0x10f8) [0x76b7ea8]
7. bin/tiflash/tiflash() [0x76b87c0]
8. bin/tiflash/tiflash(DB::DM::DMFileReader::DMFileReader(std::shared_ptr<DB::DM::DMFile> const&, std::vector<DB::DM::ColumnDefine, std::allocator<DB::DM::ColumnDefine> > const&, bool, unsigned long, std::vector<DB::DM::RowKeyRange, std::allocator<DB::DM::RowKeyRange> > const&, std::shared_ptr<DB::DM::RSOperator> const&, std::shared_ptr<std::set<unsigned long, std::less<unsigned long>, std::allocator<unsigned long> > > const&, unsigned long, std::shared_ptr<DB::MarkCache> const&, std::shared_ptr<DB::DM::MinMaxIndexCache> const&, bool, std::shared_ptr<DB::DM::ColumnCache> const&, unsigned long, unsigned long, std::shared_ptr<DB::FileProvider> const&, std::shared_ptr<DB::ReadLimiter> const&, unsigned long, bool)+0x990) [0x76baea0]
9. bin/tiflash/tiflash(DB::DM::ColumnBigFileReader::initStream()+0x380) [0x76c6880]
10. bin/tiflash/tiflash(DB::DM::ColumnBigFileReader::readRows(std::vector<COWPtr<DB::IColumn>::mutable_ptr<DB::IColumn>, std::allocator<COWPtr<DB::IColumn>::mutable_ptr<DB::IColumn> > >&, unsigned long, unsigned long, DB::DM::RowKeyRange const*)+0x24) [0x76c6f64]
11. bin/tiflash/tiflash(DB::DM::DeltaValueReader::readRows(std::vector<COWPtr<DB::IColumn>::mutable_ptr<DB::IColumn>, std::allocator<COWPtr<DB::IColumn>::mutable_ptr<DB::IColumn> > >&, unsigned long, unsigned long, DB::DM::RowKeyRange const*)+0xcf) [0x76df72f]
12. bin/tiflash/tiflash(DB::DM::DeltaValueReader::readPKVersion(unsigned long, unsigned long)+0xc7) [0x76e0e67]
13. bin/tiflash/tiflash(DB::DM::DeltaValueReader::getPlaceItems(unsigned long, unsigned long, unsigned long, unsigned long)+0x390) [0x76e2540]
14. bin/tiflash/tiflash(DB::DM::Segment::ensurePlace(DB::DM::DMContext const&, std::shared_ptr<DB::DM::StableValueSpace::Snapshot> const&, std::shared_ptr<DB::DM::DeltaValueReader> const&, std::vector<DB::DM::RowKeyRange, std::allocator<DB::DM::RowKeyRange> > const&, unsigned long) const+0x3d7) [0x764d2f7]
15. bin/tiflash/tiflash(DB::DM::Segment::getReadInfo(DB::DM::DMContext const&, std::vector<DB::DM::ColumnDefine, std::allocator<DB::DM::ColumnDefine> > const&, std::shared_ptr<DB::DM::SegmentSnapshot> const&, std::vector<DB::DM::RowKeyRange, std::allocator<DB::DM::RowKeyRange> > const&, unsigned long) const+0x236) [0x764e716]
16. bin/tiflash/tiflash(DB::DM::Segment::getInputStream(DB::DM::DMContext const&, std::vector<DB::DM::ColumnDefine, std::allocator<DB::DM::ColumnDefine> > const&, std::shared_ptr<DB::DM::SegmentSnapshot> const&, std::vector<DB::DM::RowKeyRange, std::allocator<DB::DM::RowKeyRange> > const&, std::shared_ptr<DB::DM::RSOperator> const&, unsigned long, unsigned long)+0x97) [0x764eff7]
17. bin/tiflash/tiflash(DB::DM::DMSegmentThreadInputStream::readImpl(DB::PODArray<unsigned char, 4096ul, Allocator<false>, 15ul, 16ul>*&, bool)+0x65e) [0x7631a4e]
18. bin/tiflash/tiflash(DB::DM::DMSegmentThreadInputStream::readImpl()+0x1b) [0x762b95b]
19. bin/tiflash/tiflash(DB::IProfilingBlockInputStream::read(DB::PODArray<unsigned char, 4096ul, Allocator<false>, 15ul, 16ul>*&, bool)+0x3bd) [0x6d816fd]
20. bin/tiflash/tiflash(DB::IProfilingBlockInputStream::read()+0x17) [0x6d818e7]
21. bin/tiflash/tiflash(DB::ExpressionBlockInputStream::readImpl()+0x1b) [0x7ab214b]
22. bin/tiflash/tiflash(DB::IProfilingBlockInputStream::read(DB::PODArray<unsigned char, 4096ul, Allocator<false>, 15ul, 16ul>*&, bool)+0x3bd) [0x6d816fd]
23. bin/tiflash/tiflash(DB::ConcatBlockInputStream::readImpl(DB::PODArray<unsigned char, 4096ul, Allocator<false>, 15ul, 16ul>*&, bool)+0xe5) [0x760b685]
24. bin/tiflash/tiflash(DB::ConcatBlockInputStream::readImpl()+0x1b) [0x760ae0b]
25. bin/tiflash/tiflash(DB::IProfilingBlockInputStream::read(DB::PODArray<unsigned char, 4096ul, Allocator<false>, 15ul, 16ul>*&, bool)+0x3bd) [0x6d816fd]
26. bin/tiflash/tiflash(DB::IProfilingBlockInputStream::read()+0x17) [0x6d818e7]
27. bin/tiflash/tiflash(DB::Aggregator::execute(std::shared_ptr<DB::IBlockInputStream> const&, DB::AggregatedDataVariants&, std::shared_ptr<DB::FileProvider> const&)+0x23a) [0x7b16cba]
28. bin/tiflash/tiflash(DB::AggregatingBlockInputStream::readImpl()+0x40a) [0x7a8099a]
29. bin/tiflash/tiflash(DB::IProfilingBlockInputStream::read(DB::PODArray<unsigned char, 4096ul, Allocator<false>, 15ul, 16ul>*&, bool)+0x3bd) [0x6d816fd]
30. bin/tiflash/tiflash(DB::IProfilingBlockInputStream::read()+0x17) [0x6d818e7]
31. bin/tiflash/tiflash(DB::ExpressionBlockInputStream::readImpl()+0x1b) [0x7ab214b]
"] [thread_id=60]

1. Minimal reproduce step (Required)

1. tiup cluster clean tidb --all
2. tiup bench tpcc prepare --warehouses 10 --threads 100 --host xxxxxx --port xxxxx -U root --db tpcc
3. ALTER TABLE all table in tpcc SET TIFLASH REPLICA 1;
4. select count(*) from tpcc.xxxx;

2. What did you expect to see? (Required)

3. What did you see instead (Required)

4. What is your TiFlash version? (Required)

TiDB && TiFlash is master version.

[JaySon-Huang]

[2022/02/14 14:14:44.822 +08:00] [DEBUG] [DeltaMergeStore.cpp:296] ["DeltaMergeStore[db_63.t_81]:GC removed useless dmfile: /xxx/data/t_81/stable/.del.dmf_2"] [thread_id=29]

It is cause by broken lifetime management of "external page". #4000 will fix this.
https://github.com/pingcap/tics/blob/f3304b76370e2648bc933ff265deed57d98a2678/dbms/src/Storages/Page/V3/tests/gtest_page_storage.cpp#L347

[JaySon-Huang]

Only the developing PageStorage V3 is affected by this bug.