Query failed by "failed to initialize storage reader, res=-230"

[gaodayue]

When load frequency is high and query planning took several seconds, FE may query an already compacted tablet version which leads to query failure.

The user facing error is like "failed to initialize storage reader. tablet=223602817.1265196163.de411743660db983-e1cfcb59e3e86abd, res=-230, backend=xx.xx.xx.xx".

We can replay related events from logs, listed below.

1. 11:51:37 BE updated the latest version of tablet 223602817 to 18647

I0325 11:51:37.129848 33983 engine_publish_version_task.cpp:116] publish version successfully on tablet. tablet=223602817.1265196163.de411743660db983-e1cfcb59e3e86abd, transaction_id=848571860, version=18647, res=0

2. 11:51:37 FE updated the read version for table's partition 223602812 to 18647

2020-03-25 11:51:37,176 INFO 33 [GlobalTransactionMgr.updateCatalogAfterVisible():1086] transaction state TransactionState. transaction id: 848571860, label: 0_table_name_1168497, db id: 126584253, coordinator: BE: xx.xx.xx.xx:56373, transaction status: VISIBLE, error replicas num: 0, replica ids: , prepare time: 1585108296144, commit time: 1585108297049, finish time: 1585108297175, reason: set partition 223602812's version to [18647] and version hash to [6755194206027783188]

3. 11:51:41 User executing a query which acquired version 18647 for tablet 223602817

4. 11:51:44 BE updated the latest version of tablet 223602817 to 18648

I0325 11:51:44.921618 33984 engine_publish_version_task.cpp:116] publish version successfully on tablet. tablet=223602817.1265196163.de411743660db983-e1cfcb59e3e86abd, transaction_id=848571890, version=18648, res=0

5. 11:51:45 the query in step 3 finished analyzing, begin to schedule
   2020-03-25 11:51:45,428 INFO 943132 [QeProcessorImpl.registerQuery():68] register query id = f31d984bca9d40ba-9cf7e7544747b31a, job: -1

6. 11:51:52 BE compacted version 18546-18648

I0325 11:51:51.573143 33777 compaction.cpp:36] start cumulative compaction. tablet=223602817.1265196163.de411743660db983-e1cfcb59e3e86abd
I0325 11:51:52.198297 33777 compaction.cpp:77] succeed to do cumulative compaction. tablet=223602817.1265196163.de411743660db983-e1cfcb59e3e86abd, output_version=18546-18648. elapsed time=6.25136s.

7. 11:51:52 BE received the fragment for query in step 3. The request failed because the queried version 18647 has been compacted

W0325 11:51:52.464841 46274 tablet.cpp:479] tablet:223602817.1265196163.de411743660db983-e1cfcb59e3e86abd, version already has been merged. spec_version: 0-18647
W0325 11:51:52.479046 46274 olap_scanner.cpp:122] fail to init reader.res=-230
W0325 11:51:52.491998 46274 olap_scanner.cpp:63] OlapScanner preapre failed, status:failed to initialize storage reader. tablet=223602817.1265196163.de411743660db983-e1cfcb59e3e86abd, res=-230, backend=xx.xx.xx.xx
W0325 11:51:52.541859 46290 rowset_graph.cpp:194] fail to find path in version_graph. spec_version: 0-18647

[HangyuanLiu]

We had the same problem #2962 ，also because be load too much

[imay]

Good analysis. Do you have a good idea to solve this problem?
By the way, it is too long to analysis a query in 4 seconds, do you know why?

[gaodayue]

We use a straightforward approach to avoid compacting recently added version, which works well in practice.

By the way, it is too long to analysis a query in 4 seconds, do you know why?

In our case, the query in trouble is a fairly complicated "insert into" query which scans dozens of tables, some of them without partition pruning.

[imay]

We use a straightforward approach to avoid compacting recently added version, which works well in practice.

This sounds good for me. And actually, Doris keeps a hard link for recent loaded delta to do an incremental clone. However, these delta is not used when doing scan, it will be better if Doris can use them.

In our case, the query in trouble is a fairly complicated "insert into" query which scans dozens of tables, some of them without partition pruning.

Anyway, 4 seconds seems too long. You can create another issue to track it if it is convenient to provide the insert SQL.