Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Eliminate the table lock between reading, writing and DDL operators #1162

Closed
flowbehappy opened this issue Oct 21, 2020 · 3 comments · Fixed by #1736
Closed

Eliminate the table lock between reading, writing and DDL operators #1162

flowbehappy opened this issue Oct 21, 2020 · 3 comments · Fixed by #1736
Assignees
Labels
type/enhancement The issue or PR belongs to an enhancement.

Comments

@flowbehappy
Copy link
Contributor

Currently in DAGQueryBlockInterpreter::executeTS, a table structure read lock is created by table_lock = storage->lockStructure(false, __PRETTY_FUNCTION__) and kept holding in pipeline.transform([&](auto & stream) { stream->addTableLock(table_lock); });. And SchemaBuilder<Getter, NameMapper>::applyAlterPhysicalTable need to obtain a table structure write lock. The current design could block parallel reads towards a table if the table's schema got updated.

Actually by the design of DT, after the read() return an InputStream, we already have a snapshot of the storage. So we don't need to hold the table structure read lock during the life time of the InputStream.

@flowbehappy flowbehappy added this to the v5.0-alpha milestone Oct 21, 2020
@flowbehappy flowbehappy added the type/enhancement The issue or PR belongs to an enhancement. label Mar 23, 2021
@JaySon-Huang
Copy link
Contributor

There is a situation that one coprocessor request and one DDL operation will block all other coprocessor requests:

  1. A coprocessor request create streams and hold a read lock on table t's structure lock (Thread X)
  2. Users apply some DDL operation on table t, TiFlash sync schema with TiDB and try to get a write lock on table t, (Thread Y is blocked by thread X, with schema_mutex in TiDBSchemaSyncer held)
  3. All other coprocessor requests try to sync schema with TiDB, but all of them will be blocked by thread Y

@JaySon-Huang
Copy link
Contributor

JaySon-Huang commented Mar 30, 2021

DeltaTree support online-ddl.
In other words, once the caller gets BlockInputStreams from StorageDeltaMerge::read, the caller can release the structure lock of that table. DT ensures the call can get a consistent snapshot for reading from the streams even if any ddl operations happen.
So we can consider to limit the lifetime for structure lock from the whole coprocessor life time to be generating streams inside DAGQueryBlockInterpreter::executeTS

Now comes another new question:
How to prevent the storage from being dropped when we have streams on it?

@JaySon-Huang
Copy link
Contributor

The proposal for eliminating the blocking between reading and applying DDL operators for TiFlash: https://docs.google.com/document/d/1LDo3MLU5H3SranepCiDIskbsAU7tCG3dB5Jml_dD7Es/edit#
@flowbehappy @zanmato1984

@JaySon-Huang JaySon-Huang self-assigned this Apr 8, 2021
@JaySon-Huang JaySon-Huang modified the milestones: v5.0-alpha, v5.0.1 Apr 9, 2021
@JaySon-Huang JaySon-Huang removed this from the v5.0.1 milestone Apr 23, 2021
@JaySon-Huang JaySon-Huang changed the title Structure lock of IStorage is kept holding during the life time of InputStream Eliminate the table lock between reading, writing and DDL operators Apr 27, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/enhancement The issue or PR belongs to an enhancement.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants