-
-
Notifications
You must be signed in to change notification settings - Fork 1.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
MDEV-34703 LOAD DATA INFILE using Innodb bulk load aborts #3751
Conversation
|
509f16c
to
5b90845
Compare
5b90845
to
59895a4
Compare
I gave it a try using the test case from the bug report https://jira.mariadb.org/browse/MDEV-34703. The results are good - Now the optimization actually works, and cuts load time by factor 2, i.e
|
storage/innobase/include/trx0trx.h
Outdated
/* Avoid using bulk buffer for load statement */ | ||
if (index->is_clust() && it->second.load_stmt()) | ||
return nullptr; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The load
in the name of the member function or data member is not descriptive. It’s really about skipping the merge sort, and possibly also skipping the building of the index one page at a time. The name needs to reflect that.
Based on the changes of row_merge_bulk_t::bulk_insert_buffered()
it seems that skip_sort
would be an apppropriate name.
Shouldn’t it suffice to test this flag only, and to add ut_ad(index->is_primary())
? In that way, we’d execute fewer conditions in a release build.
storage/innobase/row/row0merge.cc
Outdated
trx->error_info= index; | ||
else if (ind.is_primary() && index->table->persistent_autoinc) | ||
btr_write_autoinc(index, 1); | ||
err= btr_bulk.finish(err); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should this be avoided if we got err!=DB_SUCCESS
from the previous call?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
BtrBulk::finish() is called even when error exists. This internally calls pageAbort(page_bulk);
59895a4
to
33a3704
Compare
33a3704
to
71af51b
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you. This is rather simple. I would suggest some clarification to the comments.
storage/innobase/row/row0merge.cc
Outdated
/** During load bulk, InnoDB does build the clustered index | ||
by one record at a time and doesn't use bulk buffer. | ||
So skip the clustered index while applying the buffered | ||
bulk operation */ | ||
if (i) | ||
index= UT_LIST_GET_NEXT(indexes, index); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The comment should start with /*
and not /**
. It would be good to mention load_one_row()
here, as well as row_ins_index_entry()
, which will insert subsequent rows by invoking row_ins_clust_index_entry()
.
While I was figuring out the logic from the code, I noticed that row_ins_index_entry()
is not ideal:
if (index->is_btree()) {
if (auto t= trx->check_bulk_buffer(index->table)) {
/* MDEV-25036 FIXME:
row_ins_check_foreign_constraint() check
should be done before buffering the insert
operation. */
ut_ad(index->table->skip_alter_undo
|| !trx->check_foreigns);
return t->bulk_insert_buffered(*entry, *index, trx);
}
}
if (index->is_primary()) {
return row_ins_clust_index_entry(index, entry, thr, 0);
is_clust()
should be equivalent to is_primary()
here, because the code is not being invoked for ibuf.index
. Also, is_primary()
should imply is_btree()
. We seem to be unnecessarily traversing trx->mod_tables
for each index. The check had better be done in row_ins()
, once for all indexes, and then for the primary clustered index separately. However, refactoring this is outside the scope of this ticket. Can you please file a separate MDEV for this, and only revise the comment here?
problem: ======= - During load statement, InnoDB bulk operation relies on temporary directory and it got crash when tmpdir is exhausted. Solution: ======== During bulk insert, LOAD statement is building the clustered index one record at a time instead of page. By doing this, InnoDB does the following 1) Avoids creation of temporary file for clustered index. 2) Writes the undo log for first insert operation alone
71af51b
to
2d42e9f
Compare
Description
Problem:
Solution:
How can this PR be tested?
./mtr innodb.bulk_load
Basing the PR against the correct MariaDB version
main
branch.PR quality check