Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Detect and signal schema changes on vttablets #8005

Merged
merged 15 commits into from
May 7, 2021
Merged

Detect and signal schema changes on vttablets #8005

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
7228320
add queries and tests for detecting schema changes
systay Apr 30, 2021
9e1ff35
added schema change detection and schema copy to health streamer
harshit-gangal Apr 30, 2021
67dcf8e
proto changed to send table name having changed schema
harshit-gangal Apr 30, 2021
d3fda2a
send table names in health stream for changed schema
harshit-gangal Apr 30, 2021
8e1f210
wrote test for healthstreamer.Stream and schema changes
systay May 3, 2021
ce022a0
Merge remote-tracking branch 'upstream/master' into schema-fetch
systay May 5, 2021
22337a4
make it possible to turn schema change signaling on and off
systay May 5, 2021
d34acd8
make sure to create the database for the schemacopy
systay May 5, 2021
c0eb6ba
added e2e test for health stream to test the schema changes detection…
harshit-gangal May 5, 2021
d22c4aa
change the limit from 5 rows to 10K rows in detect schema change query
harshit-gangal May 5, 2021
ed0487a
close the conn pool
harshit-gangal May 5, 2021
b739db5
set the schema reload time to a more reasonable value
systay May 5, 2021
33a16d2
change schema reload time and refactor test
harshit-gangal May 6, 2021
a903ca5
publish the schema changed tables after the schema copy table is updated
harshit-gangal May 6, 2021
6a52cf5
not to error on initial schema change check
harshit-gangal May 6, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion go/vt/vttablet/endtoend/framework/server.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,7 +114,7 @@ func StartServer(connParams, connAppDebugParams mysql.ConnParams, dbName string)
config.HotRowProtection.Mode = tabletenv.Enable
config.TrackSchemaVersions = true
config.GracePeriods.ShutdownSeconds = 2
config.SchemaReloadIntervalSeconds = tabletenv.Seconds(3)
config.SchemaReloadIntervalSeconds = tabletenv.Seconds(2.1)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just curious why 2.1 😄

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 seconds check related to no schema change was added by @vmg to avoid query cache test to fail. To make the test faster made it to 2.1, i wanted to make it 2.0000001 but that would be too ambitious

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then I think 2.1 might also be too ambitious. Might introduce flakyness?

config.SignalWhenSchemaChange = true
gotBytes, _ := yaml2.Marshal(config)
log.Infof("Config:\n%s", gotBytes)
Expand Down
48 changes: 14 additions & 34 deletions go/vt/vttablet/endtoend/healthstream_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,6 @@ limitations under the License.
package endtoend

import (
"strconv"
"sync"
"testing"
"time"

Expand All @@ -38,10 +36,6 @@ func TestSchemaChange(t *testing.T) {
ddl string
}{
{
"default",
[]string{"upsert_test", "vitess_a", "vitess_acl_admin", "vitess_acl_all_user_read_only", "vitess_acl_no_access", "vitess_acl_read_only", "vitess_acl_read_write", "vitess_acl_unmatched", "vitess_autoinc_seq", "vitess_b", "vitess_big", "vitess_bit_default", "vitess_bool", "vitess_c", "vitess_d", "vitess_e", "vitess_f", "vitess_fracts", "vitess_ints", "vitess_misc", "vitess_mixed_case", "vitess_part", "vitess_reset_seq", "vitess_seq", "vitess_stress", "vitess_strings", "vitess_test", "vitess_test_debuguser"},
"",
}, {
"create table 1",
[]string{"vitess_sc1"},
"create table vitess_sc1(id bigint primary key)",
Expand Down Expand Up @@ -69,8 +63,6 @@ func TestSchemaChange(t *testing.T) {
}

ch := make(chan []string)
var wg sync.WaitGroup
wg.Add(1)
go func(ch chan []string) {
client.StreamHealth(func(response *querypb.StreamHealthResponse) error {
if response.RealtimeStats.TableSchemaChanged != nil {
Expand All @@ -80,35 +72,23 @@ func TestSchemaChange(t *testing.T) {
})
}(ch)

go func(client *framework.QueryClient, ch chan []string) {
index := 0
for {
t.Run(strconv.Itoa(index), func(t *testing.T) {
res := <-ch
utils.MustMatch(t, tcs[index].response, res, "")
})
index++
if index == len(tcs) {
close(ch)
break
}
if tcs[index].ddl != "" {
_, err := client.Execute(tcs[index].ddl, nil)
require.NoError(t, err)
}
}
wg.Done()
}(client, ch)

c := make(chan struct{})
go func() {
defer close(c)
wg.Wait()
}()
select {
case <-c:
case <-ch: // get the schema notification
case <-time.After(5 * time.Second):
t.Errorf("timed out")
}

for _, tc := range tcs {
t.Run(tc.tName, func(t *testing.T) {
_, err := client.Execute(tc.ddl, nil)
require.NoError(t, err)
select {
case res := <-ch: // get the schema notification
utils.MustMatch(t, tc.response, res, "")
case <-time.After(5 * time.Second):
t.Errorf("timed out")
return
}
})
}
}