Add support for query payload limit

[spark4]

Overview

Currently, there is no way to enforce a limit on the query payload for an insert operations. This PR introduces a new comment directive, which, will enable us to configure threshold for insert query payloads.

Implementation

1. Define a new comment directive DirectiveMaxPayloadSize for maximum payload size in bytes.
2. Introduce validation functions which will determine whether a given insert query is within the configured max payload size threshold, if set
3. At the plan building step, add the above validation check before returning a valid insert plan.

Testing

Testing was done primarily via unit testing and covers the following cases:

• unsharded inserts, within the MAX_PAYLOAD_SIZE threshold
• unsharded inserts, above the MAX_PAYLOAD_SIZE threshold
• sharded inserts with multiple rows, within the MAX_PAYLOAD_SIZE threshold
• sharded inserts with multiple rows, above the MAX_PAYLOAD_SIZE threshold

[sougou]

Haven't looked deep in the code. Some high level considerations:

• This feature overlaps a bit with the grpc max packet size.
• We can think about what this means for the rowcount limit. People have previously expressed concern that rowcount is not an accurate measure of payload size.

[rafael]

@sougou - This is how I’m thinking about the considerations you point out:

1. I would see grpc limit as a last line of defense.
2. Similarly, the queryserver-config-max-result-size limit is a hard stop but for select queries.
3. This feature, adds on these capabilities to do this on a per query basis for inserts. In the future, I think we could also expand the same directive to enforce these limits but for selects.

Although there is some overlap, I think this kind of granularity is a powerful lever to have in the toolkit.

For context, our motivation for this, is that it is difficult for us to enforce the hard limits right now. We are going to towards that state, but we still have many places in our application where we can’t enforce this yet. With this new feature, we are going to be able to start enforcing limits with more granularity.

I agree with the point around the concern around rowcount not being an accurate measure of payload size. That’s why we leaned more towards payload size. We have run into situations where we the rowcount is not that big, but the payload is.

[zmagg]

Agreed with everything Rafael said. Expanding on that a little:

The gRPC limit is also one global limit for both SELECTs and INSERTs. We're not yet in a place where we're able to bring that down globally across the board.

We've also seen a few incidents internally where the query payload size is large enough to start causing replication lag.

Having this granularity of control around limits lets us fail those queries before they even hit the database. We also have a large number of tables that have started putting large protobufs in LONGBLOB columns. We'd like to be able to protect ourselves from upstream application mistakes where we suddenly start sending all 4GB in that insert. 😁

[spark4]

Summary of changes

I have incorporated provided feedback and have made the following updates to the initial implementation:

• Flags over query comments: the threshold can now be configured as a flag in the vtgate package instead of a query comment directive.
• Payload validation occurs in query execution layer: we no longer perform the check within the plan builder.
• Apply threshold to all operations, not just inserts: the primary motivator for this shift was to be able to use this flag to catch queries with large IN lists across all operations, in addition to large insert payloads.
• Add warn threshold: this will allow us to tweak the max_payload_size limit
• Add query comment override: this will allow us to override the max_payload_size limit for specific queries
• Set 0 as default value for the flag to indicate no restrictions on payload size
• Use vterror over error

Testing

In addition to unit tests, I performed the following manual checks on a vtgate host:

• Ensured vtgate was able to execute queries with the default max_payload_size value
• Ensured vtgate returned the expected error if the query payload exceeded the configured limit
• Validated that a query comment will bypass the max_payload_size limit
• Confirmed that the PayloadSizeExceeded counter incremented as expected when the warn_payload_size was set and exceeded

[spark4]

Should this be Code_FAILED_PRECONDITION instead?

[zmagg]

I think ResourceExhausted is fine, but it'd be nice to use a MySQL error code that indicates it's not retryable. In the upstream community Slack, we were discussing using MySQL error code 1153 for things like this and gRPC packet size exceeded going forward. I'm not sure where that MySQL error code is set and where the gRPC error code is set.

[deepthi]

A few questions and some style / coding standard type changes.