unifying OLAP and OLTP modes

[frouioui]

Over time we have enhanced OLAP support to be at parity with OLTP support.
Related PRs: #6077 #6205 #6655 #6691 #6692 #6949 #7752 #8018 #8139 #8722 #9115 #9253 #9484 #10946 #11090 #11517

However, we still have duplication at the code level. At some point we want to refactor the code to eliminate as much of the duplication as possible.

[techbelle]

I have a quick question for the group; I've been reading the Vitess documentation and even though it says that the default workload type is OLTP, and that there is a general execution timeout set as a result of having the workload type = OLTP, I haven't found anything that says what the execution timeout settings are for OLAP, OLTP, or DBA. Can someone please guide me to wherever this is documented OR the actual place in the code? I found an old PDF of the documentation (https://vitess.io/files/version-pdfs/Vitess-Docs-8.0-10-27-2020.pdf) that says

• there's a 10k rows limit per query in OLTP mode, in OLAP there's no limit, DBA doesn't say
• "Vitess does have strict query timeouts for OLTP workloads"... and the PDF says "see below" (this is true both in current documentation and this PDF I've linked) and never actually clarifies what that timeout is or the value of the parameter. After searching the entire document for every use of the word timeout, I think this is -queryserver-config-querytimeout and the default is 30 seconds... can anyone confirm this is accurate?

Anyway based on the above, I can't tell if this issue of unifying the modes would mean that the execution timeout value becomes configurable or if it means that all modes become n seconds? also can't tell what impact would be in terms of rows scanned?

[deepthi]

Hi @techbelle, thanks for the feedback. This issue was probably confusing because it had only a title and no description. I've added a description that documents the scope of work.

Re documentation, here's what I see in the MySQL Compatibility page on the website

Killing running queries

Vitess does not yet support killing running queries via the KILL command through VTGate.
Vitess does have strict query timeouts for OLTP workloads (see below). If you need to kill a query, you can connect to the underlying MySQL shard instance and run KILL from there.

Workload

By default, Vitess sets some intentional restrictions on the execution time and number of rows that a query can return. This default workload mode is called OLTP. This can be disabled by setting the workload to OLAP:

So when it says "see below", it is just referring to the next section, which pretty much just says "intentional restrictions blah blah".

What you have discovered is accurate, the queryserver-config-query-timeout flag which defaults to 30 seconds controls how long OLTP queries can run. There is no timeout on OLAP read queries.
In addition to this, there are flags that control how long transactions are allowed to stay open in OLTP and OLAP modes. Both of those default to 30 seconds as well.

With DBA mode, no timeout is enforced on transactions and they are allowed to stay open forever (subject to MySQL level idle connection timeouts / vttablet shutdown). There is some debate about whether this is actually a good idea and whether we should stop allowing these in future.

Hope this helps. I started off trying to update the website docs to answer your questions, but couldn't find a good place for this. Maybe we need to write a new page!

Default flag values in code:

vitess/go/vt/vttablet/tabletserver/tabletenv/config.go

Line 500 in 467972a

var defaultConfig = TabletConfig{

Transaction timeouts (0 means no timeout):

vitess/go/vt/vttablet/tabletserver/tabletenv/config.go

Line 437 in 800c6b0

func (c *TabletConfig) TxTimeoutForWorkload(workload querypb.ExecuteOptions_Workload) time.Duration {

[techbelle]

wow, thank you for this incredibly thorough reply ! 💯 makes total sense. FWIW, i only even heard of this feature because another engineer was telling me how great it is. That's an anecdote, so I can't speculate on the usage or the lovability of the feature overall (nor do I have any opinion) but it does seem very useful and plan to tinker with it.

thanks again!