Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Detect and signal schema changes on vttablets #8005

Merged
merged 15 commits into from
May 7, 2021

Conversation

systay
Copy link
Collaborator

@systay systay commented Apr 30, 2021

Detect schema changes on the tablet.

With this PR, a vttablet that finds itself as a master, will periodically check if schema has changed on the underlying mysqld. If any schema changes are found, the changed table names will be sent to any listening vtgates using the pre existing health check mechanism.

If this feature is enabled, a database will be created on the underlying MySQL instance named _vt, where meta-information about the schema will be stored.

Related to #7994

Signed-off-by: Andres Taylor <andres@planetscale.com>
@systay systay added Type: Enhancement Logical improvement (somewhere between a bug and feature) Component: Query Serving labels Apr 30, 2021
harshit-gangal and others added 6 commits April 30, 2021 19:01
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
Signed-off-by: Andres Taylor <andres@planetscale.com>
Signed-off-by: Andres Taylor <andres@planetscale.com>
Signed-off-by: Andres Taylor <andres@planetscale.com>
@systay systay marked this pull request as ready for review May 5, 2021 06:13
Signed-off-by: Andres Taylor <andres@planetscale.com>
@systay systay changed the title add queries and tests for detecting schema changes Detect and signal schema changes on vttablets May 5, 2021
@shlomi-noach
Copy link
Contributor

If this feature is enabled, a database will be created on the underlying MySQL instance named _vt

I think _vt is always created, if only for the heartbeat mechanism.

@shlomi-noach
Copy link
Contributor

Elaborating on my previous comment: _vt schema is always created, within a few seconds of tablet startup. There's multiple parties that would create it, but onlineddl/Executor is one I'm most familiar with.

Creation:

if err := e.initSchema(ctx); err != nil {

Called by tablet state manager, Open(), right after tablet is ready.

Which runs these queries:

var applyDDL = []string{
sqlCreateSidecarDB,
sqlCreateSchemaMigrationsTable,
alterSchemaMigrationsTableRetries,
alterSchemaMigrationsTableTablet,
alterSchemaMigrationsTableArtifacts,
alterSchemaMigrationsTableTabletFailure,
alterSchemaMigrationsTableTabletFailureIndex,
alterSchemaMigrationsTableProgress,
alterSchemaMigrationsTableContext,
alterSchemaMigrationsTableDDLAction,
alterSchemaMigrationsTableMessage,
alterSchemaMigrationsTableTableCompleteIndex,
alterSchemaMigrationsTableETASeconds,
}

Which, in particular, include:

sqlCreateSidecarDB = "create database if not exists _vt"

Copy link
Contributor

@shlomi-noach shlomi-noach left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

one bug found.

Fear mongering alert: I guess my reservation is that every X minutes (30 in this code) we're going to query information_schema for all tables in a schema, which can be tens of thousands of tables in some environments. I actually never tried querying I_S for that many tables, and I fear this might be a risky operation.

go/mysql/schema.go Show resolved Hide resolved
go/mysql/schema.go Show resolved Hide resolved
go/mysql/schema.go Show resolved Hide resolved
go/vt/vttablet/tabletserver/health_streamer.go Outdated Show resolved Hide resolved
@@ -107,6 +107,7 @@ func init() {
flag.Int64Var(&currentConfig.QueryCacheMemory, "queryserver-config-query-cache-memory", defaultConfig.QueryCacheMemory, "query server query cache size in bytes, maximum amount of memory to be used for caching. vttablet analyzes every incoming query and generate a query plan, these plans are being cached in a lru cache. This config controls the capacity of the lru cache.")
flag.BoolVar(&currentConfig.QueryCacheLFU, "queryserver-config-query-cache-lfu", defaultConfig.QueryCacheLFU, "query server cache algorithm. when set to true, a new cache algorithm based on a TinyLFU admission policy will be used to improve cache behavior and prevent pollution from sparse queries")
SecondsVar(&currentConfig.SchemaReloadIntervalSeconds, "queryserver-config-schema-reload-time", defaultConfig.SchemaReloadIntervalSeconds, "query server schema reload time, how often vttablet reloads schemas from underlying MySQL instance in seconds. vttablet keeps table schemas in its own memory and periodically refreshes it from MySQL. This config controls the reload time.")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess this code already exists. But it's unclear from the name queryserver-config-schema-reload-time what the units are. I'd assume it was a Duration, but it's in fact seconds. I guess too later to change the name,

… signal

Signed-off-by: Harshit Gangal <harshit@planetscale.com>
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
@systay
Copy link
Collaborator Author

systay commented May 5, 2021

Fear mongering alert: I guess my reservation is that every X minutes (30 in this code) we're going to query information_schema for all tables in a schema, which can be tens of thousands of tables in some environments. I actually never tried querying I_S for that many tables, and I fear this might be a risky operation.

This one of the reasons we are adding a flag for this feature. We don't think that it will be something everyone uses. The use case you mention sounds like a situation where you might not want to keep track of schema like this.

Signed-off-by: Andres Taylor <andres@planetscale.com>
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
@@ -114,7 +114,7 @@ func StartServer(connParams, connAppDebugParams mysql.ConnParams, dbName string)
config.HotRowProtection.Mode = tabletenv.Enable
config.TrackSchemaVersions = true
config.GracePeriods.ShutdownSeconds = 2
config.SchemaReloadIntervalSeconds = tabletenv.Seconds(3)
config.SchemaReloadIntervalSeconds = tabletenv.Seconds(2.1)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just curious why 2.1 😄

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 seconds check related to no schema change was added by @vmg to avoid query cache test to fail. To make the test faster made it to 2.1, i wanted to make it 2.0000001 but that would be too ambitious

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then I think 2.1 might also be too ambitious. Might introduce flakyness?

Signed-off-by: Harshit Gangal <harshit@planetscale.com>
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
@systay systay merged commit 28cf98b into vitessio:master May 7, 2021
@systay systay deleted the schema-fetch branch May 7, 2021 08:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Component: Query Serving Type: Enhancement Logical improvement (somewhere between a bug and feature)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants