Documentation Quest

[nrc]

TiKV needs better documentation. The transaction SIG will do its part by documenting the concepts and code behind the TiKV/TiDB transaction system.

If the quest is successful, it should be easier for

• users to understand the isolation properties and performance characteristics of TiKV/TiDB
• users to reason about how to configure their transaction usage
• new contributors to understand TiKV transactions and write code to change their implementation
• outsiders to survey and understand TiKV's transaction implementation, and compare it to other databases

See the leader board for a ranking of documentation questers.

Mentors

If you want advice, help, or review, the following people are available and will prioritize quest issues:

• @nrc
• @sticnarf
• @youjiali1995

Resources

Existing documentation and other resources are listed in the doc directory.

Tasks

To claim a task:

• file an issue for it (either in this repository or TiKV
• include a reference to this issue, and ask a mentor (listed above) to add the issue link here
• a mentor will be assigned to that issue and they will keep this issue up to date with the task's status
• let the mentor know if you need help or if someone else should take on the task

Tasks may or may not have a related issue. Tasks with names are being worked on by that user and should have an open issue for discussion. Boxes get ticked when the documentation task is complete.

High level documentation

Describe the underlying concepts of transactions (mostly be linking to other material, but we should organize and document that material and ensure it covers everything, filling in gaps where necessary). Describe at an algorithmic level the way transactions are implemented in TiKV.

• Transactions overview, data format, latches, MVCC (TiDB docs, TinyKV docs, Xuelian's blog, VLDB paper)
• Optimistic locking
• Pessimistic locking (blog post)
• Isolation/consistency properties (Jepsen report)
• Lock management/waiter mangement, deadlock detection
• GC, green GC
• Pipelined pessimistic locking
• Large transactions (blog post)
• Parallel commit (repo)
• 1pc
• Replica read (aka follower read)
• Timestamps and HLC
• Alternative implementations

Module documentation

Describe modules of the TiKV implementation. This should how the code works, why that design was chosen, and how it fits in with the larger transaction system. We should have documentation per-concept, which should be roughly a Rust module, but is unlikely to exactly match.

MongoDB has lots of great examples of module-level docs, e.g., storage.

• - storage - storage: add some top-level docs tikv#8684 @nrc
• - kv
• - reader
• - latches
• - process
• - sched_pool
• - scheduler There is already some documentation here, but it is more suited for a higher level module and here we should explain what the scheduler does and how it works.
• - store
• - lock manager
• - read pool

Code comments

Not every type and function needs a comment, but it would be great to document the more important and more complex code.

TiKV:

• - Engine kv - Document Engine sub-type in kv module as a part of doc quest tikv#8546 - @palash25
• - Snapshot kv
• - Iterator kv
• - tls engine kv
• - MvccTxn txn
• - test infrastructure mvcc
• - Store store
• - Scanner store
• - sched_txn_command storage

Rust Client:

TiDB:

TODO more tasks

[nrc]

Leader board

User	Tasks complete
@nrc	0

[pramodbiligiri]

Noting down my thoughts on what the existing TiDB docs look like. I'm keeping in mind two types of users (Users and Developers) and the kind of documentation they'll need:

1. Users → User Guide: a step by step guide on how to use TiKV Transactions.
2. Developers → Design docs, Module level and code level docs.
3. Both Users and Developers → TiDB Transactions Reference, which is exhaustive but may not have detailed explanations.

Existing TiDB docs look like this:

Target audience	Type of docs	Current location
Users	User Guide	docs.pingcap.com -> Reference -> SQL -> Transactions
Users, Developers	Reference	docs.pingcap.com -> Reference -> SQL -> Transactions
Developers	Developer docs	sig-transaction
Existing and prospective customers	Blog posts	https://pingcap.com/blog/pessimistic-locking-better-mysql-compatibility-fewer-rollbacks-under-high-load and https://pingcap.com/blog/large-transactions-in-tidb

Blog posts are good to announce new features in a timely manner, but they are hard to discover when reading other official documentation. So any information in blogs should also be available in one of the other sources. Ideally, blog posts should contain links to more official documentation but never the other way round.

[pramodbiligiri]

Based on this, I'm planning to write up some docs within sig-transaction, because Developer docs can do with small improvements and won't affect the end user directly. Any further suggestion on which part of sig-transaction I can expand on will be welcome. For my first task, I'd prefer something than can be done in a handful of days of work.

[nrc]

Based on this, I'm planning to write up some docs within sig-transaction, because Developer docs can do with small improvements and won't affect the end user directly.

Awsome, thank you!

Any further suggestion on which part of sig-transaction I can expand on will be welcome

I think a good thing to do is to pick a module and add some comments on types - it is mostly possible to do that by searching for uses.