Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

*(all): update dependencies on tidb and pd. #8126

Merged
merged 32 commits into from
Feb 8, 2023
Merged
Show file tree
Hide file tree
Changes from 26 commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
dde7691
fix get source id error
asddongmen Jan 29, 2023
f8011ce
add unit test
asddongmen Jan 29, 2023
022862a
add unit test
asddongmen Jan 29, 2023
72c0087
fix fmt
asddongmen Jan 29, 2023
053c101
fix ut
asddongmen Jan 29, 2023
34b01cd
fix ut
asddongmen Jan 29, 2023
9c47ff4
fix ut
asddongmen Jan 29, 2023
dee9636
fix ut
asddongmen Jan 30, 2023
cb0aa87
add arch check to makefile
asddongmen Jan 30, 2023
9a0d3db
format makefile
asddongmen Jan 30, 2023
7bc84de
merge upstream/master
asddongmen Feb 3, 2023
37f27f7
update dependencies
asddongmen Feb 6, 2023
4181e92
update dependencies to master
asddongmen Feb 6, 2023
9a86faa
fix ut build error
asddongmen Feb 6, 2023
c46cd35
fix some retryable MySQL pattern
lance6716 Feb 6, 2023
712b87a
fix DM IT
lance6716 Feb 6, 2023
2affa94
try fix some tests
lance6716 Feb 6, 2023
7ad35b2
revert negligent change of go.mod
asddongmen Feb 6, 2023
ae2cb54
Merge branch 'fix-get-source-id' of github.com:asddongmen/ticdc into …
asddongmen Feb 6, 2023
4cd5a82
fix DM IT
lance6716 Feb 6, 2023
6148e9b
Merge branch 'fix-get-source-id' of github.com:asddongmen/ticdc into …
lance6716 Feb 6, 2023
ccaf0cb
fix DM IT
lance6716 Feb 6, 2023
1a060a7
fix DM IT
lance6716 Feb 6, 2023
310f0fe
fix some UT and IT
lance6716 Feb 6, 2023
696fa26
fix more
lance6716 Feb 6, 2023
775f97a
fix a DM IT
lance6716 Feb 6, 2023
c42dab2
address comments
asddongmen Feb 7, 2023
4b2e19b
address comments again
asddongmen Feb 7, 2023
03e6f46
remove debug log
asddongmen Feb 7, 2023
f1628fd
address comments
asddongmen Feb 7, 2023
3fd5f58
Update Makefile
asddongmen Feb 7, 2023
b35f4e9
Merge branch 'master' into fix-get-source-id
ti-chi-bot Feb 7, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 18 additions & 6 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -40,14 +40,26 @@ ifeq (${CDC_ENABLE_VENDOR}, 1)
GOVENDORFLAG := -mod=vendor
endif

GOBUILD := CGO_ENABLED=0 $(GO) build $(BUILD_FLAG) -trimpath $(GOVENDORFLAG)
GOBUILDNOVENDOR := CGO_ENABLED=0 $(GO) build $(BUILD_FLAG) -trimpath
GOTEST := CGO_ENABLED=1 $(GO) test -p $(P) --race
GOTESTNORACE := CGO_ENABLED=1 $(GO) test -p $(P)

OS := "$(shell go env GOOS)"
ARCH := "$(shell uname -s)"
LINUX := "Linux"
MAC := "Darwin"
ifeq (${ARCH}, ${LINUX})
zhaoxinyu marked this conversation as resolved.
Show resolved Hide resolved
CGO := 0
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if someone cross compile macOS TiCDC on a linux machine?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think only GOOS matters.

Copy link
Contributor Author

@asddongmen asddongmen Feb 7, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you! You are right, only GOOS matters, but no matter set CGO_ENABLED=0 or CGO_ENABLED=1 runGOOS=darwin make cdc fails.
I revert related changes and run the following commands:

➜  ticdc git:(fix-get-source-id) uname
Linux
➜  ticdc git:(fix-get-source-id) uname -m
x86_64

➜  ticdc git:(fix-get-source-id) CGO_ENABLED=0 GOOS=darwin make cdc
GO111MODULE=on go build  -trimpath  -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v6.5.0-master-dirty" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2023-02-07 09:41:40" -X "github.com/pingcap/tiflow/pkg/version.GitHash=03e6f4650e98fe8b47788622844fc1a5f9272ead" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=fix-get-source-id" -X "github.com/pingcap/tiflow/pkg/version.GoVersion=go version go1.19.3 linux/amd64"' -o bin/cdc ./cmd/cdc/main.go
# github.com/cloudfoundry/gosigar
vendor/github.com/cloudfoundry/gosigar/concrete_sigar.go:20:12: cpuUsage.Get undefined (type Cpu has no field or method Get)
vendor/github.com/cloudfoundry/gosigar/concrete_sigar.go:30:14: cpuUsage.Get undefined (type Cpu has no field or method Get)
vendor/github.com/cloudfoundry/gosigar/concrete_sigar.go:49:11: l.Get undefined (type LoadAverage has no field or method Get)
vendor/github.com/cloudfoundry/gosigar/concrete_sigar.go:55:11: m.Get undefined (type Mem has no field or method Get)
vendor/github.com/cloudfoundry/gosigar/concrete_sigar.go:61:11: m.GetIgnoringCGroups undefined (type Mem has no field or method GetIgnoringCGroups)
vendor/github.com/cloudfoundry/gosigar/concrete_sigar.go:67:11: s.Get undefined (type Swap has no field or method Get)
vendor/github.com/cloudfoundry/gosigar/sigar_shared.go:12:21: procTime.Get undefined (type *ProcTime has no field or method Get)
^Cmake: *** [cdc] Interrupt

➜  ticdc git:(fix-get-source-id) ✗ CGO_ENABLED=1 GOOS=darwin make cdc
GO111MODULE=on go build  -trimpath  -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v6.5.0-master-dirty" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2023-02-07 09:41:32" -X "github.com/pingcap/tiflow/pkg/version.GitHash=03e6f4650e98fe8b47788622844fc1a5f9272ead" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=fix-get-source-id" -X "github.com/pingcap/tiflow/pkg/version.GoVersion=go version go1.19.3 linux/amd64"' -o bin/cdc ./cmd/cdc/main.go
# runtime/cgo
gcc: error: x86_64: No such file or directory
gcc: error: unrecognized command line option '-arch'
make: *** [cdc] Error 2

Do you have any idea to fix this problem?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pingcap/tidb#41165

use CGO_ENABLED=1 is the only way to build on darwin.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pingcap/tidb#41165

use CGO_ENABLED=1 is the only way to build on darwin.

Yes, but can't build a darwin cdc on linux machine.

else ifeq (${OS}, "linux")
Rustin170506 marked this conversation as resolved.
Show resolved Hide resolved
CGO := 0
else ifeq (${ARCH}, ${MAC})
CGO := 1
else ifeq (${OS}, "darwin")
Rustin170506 marked this conversation as resolved.
Show resolved Hide resolved
CGO := 1
endif

GOBUILD := CGO_ENABLED=$(CGO) $(GO) build $(BUILD_FLAG) -trimpath $(GOVENDORFLAG)
GOBUILDNOVENDOR := CGO_ENABLED=0 $(GO) build $(BUILD_FLAG) -trimpath
GOTEST := CGO_ENABLED=1 $(GO) test -p $(P) --race --tags=intest
GOTESTNORACE := CGO_ENABLED=1 $(GO) test -p $(P)

asddongmen marked this conversation as resolved.
Show resolved Hide resolved

CDC_PKG := github.com/pingcap/tiflow
DM_PKG := github.com/pingcap/tiflow/dm
ENGINE_PKG := github.com/pingcap/tiflow/engine
Expand Down Expand Up @@ -163,7 +175,7 @@ unit_test_in_verify_ci: check_failpoint_ctl tools/bin/gotestsum tools/bin/gocov
mkdir -p "$(TEST_DIR)"
$(FAILPOINT_ENABLE)
@export log_level=error;\
CGO_ENABLED=1 tools/bin/gotestsum --junitfile cdc-junit-report.xml -- -v -timeout 5m -p $(P) --race \
CGO_ENABLED=1 tools/bin/gotestsum --junitfile cdc-junit-report.xml -- -v -timeout 5m -p $(P) --race --tags=intest \
Rustin170506 marked this conversation as resolved.
Show resolved Hide resolved
-covermode=atomic -coverprofile="$(TEST_DIR)/cov.unit.out" $(PACKAGES_TICDC) \
|| { $(FAILPOINT_DISABLE); exit 1; }
tools/bin/gocov convert "$(TEST_DIR)/cov.unit.out" | tools/bin/gocov-xml > cdc-coverage.xml
Expand Down
2 changes: 1 addition & 1 deletion cdc/api/v2/changefeed_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -321,7 +321,7 @@ func TestUpdateChangefeed(t *testing.T) {
cp.EXPECT().IsOwner().Return(true).AnyTimes()

// case 1 invalid id
invalidID := "#Invalid_"
invalidID := "Invalid_#"
Rustin170506 marked this conversation as resolved.
Show resolved Hide resolved
w := httptest.NewRecorder()
req, _ := http.NewRequestWithContext(context.Background(), update.method,
fmt.Sprintf(update.url, invalidID), nil)
Expand Down
23 changes: 23 additions & 0 deletions cdc/entry/schema/snapshot.go
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,9 @@ import (
"github.com/pingcap/tidb/parser/mysql"
"github.com/pingcap/tidb/session"
"github.com/pingcap/tiflow/cdc/model"
"github.com/pingcap/tiflow/pkg/config"
cerror "github.com/pingcap/tiflow/pkg/errors"
"github.com/pingcap/tiflow/pkg/filter"
"go.uber.org/zap"
)

Expand Down Expand Up @@ -527,13 +529,28 @@ func (s *Snapshot) SchemaCount() (count int) {

// DumpToString dumps the snapshot to a string.
func (s *Snapshot) DumpToString() string {
// TiCDC will not use any table in system database,
// so we can ignore them.
cfg := config.GetDefaultReplicaConfig()
cfg.Filter.Rules = []string{"mysql.*"}
asddongmen marked this conversation as resolved.
Show resolved Hide resolved
asddongmen marked this conversation as resolved.
Show resolved Hide resolved
ft, err := filter.NewFilter(cfg, "")
if err != nil {
log.Panic("new filter error", zap.Error(err))
}

schemas := make([]string, 0, s.inner.schemas.Len())
s.IterSchemas(func(dbInfo *timodel.DBInfo) {
if ft.ShouldIgnoreTable(dbInfo.Name.O, "") {
return
}
schemas = append(schemas, fmt.Sprintf("%v", dbInfo))
})

tables := make([]string, 0, s.inner.tables.Len())
s.IterTables(true, func(tbInfo *model.TableInfo) {
if ft.ShouldIgnoreTable(tbInfo.TableName.Schema, tbInfo.TableName.Schema) {
asddongmen marked this conversation as resolved.
Show resolved Hide resolved
return
}
tables = append(tables, fmt.Sprintf("%v", tbInfo))
})

Expand All @@ -544,12 +561,18 @@ func (s *Snapshot) DumpToString() string {

schemaNames := make([]string, 0, s.inner.schemaNameToID.Len())
s.IterSchemaNames(func(schema string, target int64) {
if ft.ShouldIgnoreTable(schema, "") {
return
}
schemaNames = append(schemaNames, fmt.Sprintf("%s:%d", schema, target))
})

tableNames := make([]string, 0, s.inner.tableNameToID.Len())
s.IterTableNames(func(schemaID int64, table string, target int64) {
schema, _ := s.inner.schemaByID(schemaID)
if ft.ShouldIgnoreTable(schema.Name.O, table) {
return
}
tableNames = append(tableNames, fmt.Sprintf("%s.%s:%d", schema.Name.O, table, target))
})

Expand Down
2 changes: 1 addition & 1 deletion cdc/entry/schema_storage_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -734,7 +734,7 @@ func TestExplicitTables(t *testing.T) {
require.GreaterOrEqual(t, snap2.TableCount(false), 4)

require.Equal(t, snap3.TableCount(true)-snap1.TableCount(true), 5)
require.Equal(t, snap3.TableCount(false), 45)
require.Equal(t, 51, snap3.TableCount(false))
asddongmen marked this conversation as resolved.
Show resolved Hide resolved
}

/*
Expand Down
4 changes: 2 additions & 2 deletions cdc/owner/schema_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ func TestAllTables(t *testing.T) {
require.Equal(t, tableName, model.TableName{
Schema: "test",
Table: "t1",
TableID: 76,
TableID: 84,
})
// add ineligible table
require.Nil(t, schema.HandleDDL(helper.DDL2Job("create table test.t2(id int)")))
Expand All @@ -104,7 +104,7 @@ func TestAllTables(t *testing.T) {
require.Equal(t, tableName, model.TableName{
Schema: "test",
Table: "t1",
TableID: 76,
TableID: 84,
})
}

Expand Down
2 changes: 1 addition & 1 deletion cdc/server/server_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ import (
"github.com/pingcap/tiflow/pkg/security"
"github.com/pingcap/tiflow/pkg/util"
"github.com/stretchr/testify/require"
"github.com/tikv/pd/pkg/tempurl"
"github.com/tikv/pd/pkg/utils/tempurl"
clientv3 "go.etcd.io/etcd/client/v3"
"go.etcd.io/etcd/server/v3/embed"
"golang.org/x/sync/errgroup"
Expand Down
2 changes: 1 addition & 1 deletion cdc/sink/codec/maxwell/maxwell_message.go
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ import (
"github.com/pingcap/tiflow/cdc/model"
"github.com/pingcap/tiflow/cdc/sink/codec/internal"
cerror "github.com/pingcap/tiflow/pkg/errors"
"github.com/tikv/pd/pkg/tsoutil"
"github.com/tikv/pd/pkg/utils/tsoutil"
)

type maxwellMessage struct {
Expand Down
2 changes: 1 addition & 1 deletion dm/master/election_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ import (
"github.com/pingcap/tiflow/dm/pkg/etcdutil"
"github.com/pingcap/tiflow/dm/pkg/log"
"github.com/stretchr/testify/require"
"github.com/tikv/pd/pkg/tempurl"
"github.com/tikv/pd/pkg/utils/tempurl"
)

func TestFailToStartLeader(t *testing.T) {
Expand Down
2 changes: 1 addition & 1 deletion dm/master/etcd_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ import (
"github.com/pingcap/tiflow/dm/pkg/log"
"github.com/pingcap/tiflow/dm/pkg/terror"
"github.com/pingcap/tiflow/dm/pkg/utils"
"github.com/tikv/pd/pkg/tempurl"
"github.com/tikv/pd/pkg/utils/tempurl"
clientv3 "go.etcd.io/etcd/client/v3"
"go.etcd.io/etcd/server/v3/embed"
)
Expand Down
2 changes: 1 addition & 1 deletion dm/master/openapi_view_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ import (
"github.com/pingcap/tiflow/dm/pkg/utils"
"github.com/stretchr/testify/require"
"github.com/stretchr/testify/suite"
"github.com/tikv/pd/pkg/tempurl"
"github.com/tikv/pd/pkg/utils/tempurl"
)

// some data for test.
Expand Down
2 changes: 1 addition & 1 deletion dm/master/server_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ import (
"github.com/pingcap/tiflow/pkg/version"
"github.com/stretchr/testify/require"
"github.com/stretchr/testify/suite"
"github.com/tikv/pd/pkg/tempurl"
"github.com/tikv/pd/pkg/utils/tempurl"
clientv3 "go.etcd.io/etcd/client/v3"
"go.etcd.io/etcd/server/v3/verify"
"go.etcd.io/etcd/tests/v3/integration"
Expand Down
2 changes: 1 addition & 1 deletion dm/master/workerrpc/workerrpc_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ import (
"github.com/pingcap/tiflow/dm/pb"
"github.com/pingcap/tiflow/dm/pbmock"
"github.com/pingcap/tiflow/dm/pkg/terror"
"github.com/tikv/pd/pkg/tempurl"
"github.com/tikv/pd/pkg/utils/tempurl"
)

var _ = Suite(&testWorkerRPCSuite{})
Expand Down
2 changes: 1 addition & 1 deletion dm/pkg/election/election_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ import (
"github.com/pingcap/tiflow/dm/pkg/log"
"github.com/pingcap/tiflow/dm/pkg/terror"
"github.com/pingcap/tiflow/dm/pkg/utils"
"github.com/tikv/pd/pkg/tempurl"
"github.com/tikv/pd/pkg/utils/tempurl"
clientv3 "go.etcd.io/etcd/client/v3"
"go.etcd.io/etcd/server/v3/embed"
)
Expand Down
2 changes: 1 addition & 1 deletion dm/pkg/etcdutil/etcdutil_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ import (
"github.com/pingcap/tiflow/dm/pkg/log"
"github.com/pingcap/tiflow/dm/pkg/terror"
"github.com/stretchr/testify/require"
"github.com/tikv/pd/pkg/tempurl"
"github.com/tikv/pd/pkg/utils/tempurl"
"go.etcd.io/etcd/api/v3/etcdserverpb"
v3rpc "go.etcd.io/etcd/api/v3/v3rpc/rpctypes"
clientv3 "go.etcd.io/etcd/client/v3"
Expand Down
12 changes: 6 additions & 6 deletions dm/pkg/retry/errors.go
Original file line number Diff line number Diff line change
Expand Up @@ -36,17 +36,17 @@ var (
"Unsupported collation",
"Invalid default value for",
"Unsupported drop primary key",
"Error 1059: Identifier name", // Limitations on identifier length
"Error 1117: Too many columns",
"Error 1069: Too many keys specified",
"Error 1059",
"Error 1117",
"Error 1069",
}

// UnsupportedDMLMsgs list the error messages of some un-recoverable DML, which is used in task auto recovery.
UnsupportedDMLMsgs = []string{
"Error 1062: Duplicate",
"Error 1406: Data too long for column",
"Error 1062",
"Error 1406",
"Error 1366",
"Error 8025: entry too large",
"Error 8025",
}

// ReplicationErrMsgs list the error message of un-recoverable replication error.
Expand Down
2 changes: 1 addition & 1 deletion dm/tests/all_mode/run.sh
Original file line number Diff line number Diff line change
Expand Up @@ -552,7 +552,7 @@ function run() {
run_sql_source1 "create table all_mode.db_error (c int primary key);"
run_dm_ctl_with_retry $WORK_DIR "127.0.0.1:$MASTER_PORT" \
"query-status $ILLEGAL_CHAR_NAME" \
"Error 1049: Unknown database" 1
"Error 1049 (42000): Unknown database" 1

# stop task, task state should be cleaned
run_dm_ctl $WORK_DIR "127.0.0.1:$MASTER_PORT" \
Expand Down
2 changes: 1 addition & 1 deletion dm/tests/check_task/run.sh
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ function test_privileges_can_migrate() {
run_sql_source1 "create table checktask1.test_privilege(id int primary key, b varchar(10))"
run_sql_source1 "insert into checktask1.test_privilege values (1, 'a'),(2, 'b');"
run_sql_tidb "create user 'test1'@'%' identified by '123456';"
run_sql_tidb "grant select, create, insert, update, delete, alter, drop, index on *.* to 'test1'@'%';"
run_sql_tidb "grant select, create, insert, update, delete, alter, drop, index, config on *.* to 'test1'@'%';"
run_sql_tidb "flush privileges;"
run_dm_ctl $WORK_DIR "127.0.0.1:$MASTER_PORT" \
"start-task $cur/conf/task-priv.yaml --remove-meta" \
Expand Down
4 changes: 2 additions & 2 deletions dm/tests/full_mode/run.sh
Original file line number Diff line number Diff line change
Expand Up @@ -53,8 +53,8 @@ function fail_acquire_global_lock() {
run_dm_ctl_with_retry $WORK_DIR "127.0.0.1:$MASTER_PORT" \
"query-status test" \
"\"stage\": \"Paused\"" 2 \
"LOCK TABLES \`full_mode\`.\`t1\` READ: Error 1044: Access denied" 1 \
"LOCK TABLES \`full_mode\`.\`t2\` READ: Error 1044: Access denied" 1
"LOCK TABLES \`full_mode\`.\`t1\` READ: Error 1044 (42000): Access denied" 1 \
"LOCK TABLES \`full_mode\`.\`t2\` READ: Error 1044 (42000): Access denied" 1

cleanup_process $*
cleanup_data full_mode
Expand Down
2 changes: 1 addition & 1 deletion dm/tests/handle_error/run.sh
Original file line number Diff line number Diff line change
Expand Up @@ -521,7 +521,7 @@ function DM_EXEC_ERROR_SKIP_CASE() {

run_dm_ctl_with_retry $WORK_DIR "127.0.0.1:$MASTER_PORT" \
"query-status test" \
"Error 1062: Duplicate " 1
"Error 1062 (23000): Duplicate " 1

run_sql_tidb "insert into ${db}.${tb} values(5,3,3);"
run_sql_tidb "insert into ${db}.${tb} values(6,4,4);"
Expand Down
3 changes: 1 addition & 2 deletions dm/tests/lightning_mode/run.sh
Original file line number Diff line number Diff line change
Expand Up @@ -78,8 +78,7 @@ function run() {
"checksum mismatched, KV number in source files: 6, KV number in TiDB cluster: 3" 1 \
'"unit": "Load"' 2
run_dm_ctl $WORK_DIR "127.0.0.1:$MASTER_PORT" \
"resume-task test" \
"\"result\": true" 3
"resume-task test"
run_dm_ctl_with_retry $WORK_DIR "127.0.0.1:$MASTER_PORT" \
"query-status test" \
'unit": "Sync"' 2
Expand Down
2 changes: 1 addition & 1 deletion dm/tests/many_tables/run.sh
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,7 @@ function run() {
run_sql_tidb "select count(*) from dm_meta.test_syncer_checkpoint"
check_contains "count(*): $(($TABLE_NUM + 1))"

check_log_contains $WORK_DIR/worker1/log/dm-worker.log 'Error 8004: Transaction is too large'
check_log_contains $WORK_DIR/worker1/log/dm-worker.log 'Error 8004 (HY000): Transaction is too large'

# check https://github.com/pingcap/tiflow/issues/5063
check_time=100
Expand Down
2 changes: 1 addition & 1 deletion dm/tests/print_status/run.sh
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ function check_print_status() {

# check load unit print status
status_file=$WORK_DIR/worker1/log/loader_status.log
grep -oP "\[unit=lightning-load\] \[IsCanceled=false\] \[finished_bytes=59637\] \[total_bytes=59637\] \[progress=.*\]" $WORK_DIR/worker1/log/dm-worker.log >$status_file
grep -oP "\[unit=lightning-load\] \[IsCanceled=false\] \[finished_bytes=59834\] \[total_bytes=59834\] \[progress=.*\]" $WORK_DIR/worker1/log/dm-worker.log >$status_file
status_count=$(wc -l $status_file | awk '{print $1}')
[ $status_count -eq 1 ]
# must have a non-zero speed in log
Expand Down
2 changes: 1 addition & 1 deletion dm/tests/shardddl1_1/run.sh
Original file line number Diff line number Diff line change
Expand Up @@ -253,7 +253,7 @@ function DM_027_CASE() {
run_sql_source1 "insert into ${shardddl1}.${tb3} values (5,6)"
run_dm_ctl_with_retry $WORK_DIR "127.0.0.1:$MASTER_PORT" \
"query-status test" \
"Error 1054: Unknown column 'val' in 'field list'" 1
"Error 1054 (42S22): Unknown column 'val' in 'field list'" 1
}

function DM_027() {
Expand Down
8 changes: 4 additions & 4 deletions dm/tests/sync_collation/run.sh
Original file line number Diff line number Diff line change
Expand Up @@ -100,8 +100,8 @@ function run() {
echo "check full phase error"
run_dm_ctl_with_retry $WORK_DIR "127.0.0.1:$MASTER_PORT" \
"query-status ${TASK_NAME}" \
"Error 1273: Unsupported collation when new collation is enabled: 'latin1_swedish_ci'" 1 \
"Error 1273: Unsupported collation when new collation is enabled: 'utf8mb4_0900_ai_ci'" 1
"Error 1273 (HY000): Unsupported collation when new collation is enabled: 'latin1_swedish_ci'" 1 \
"Error 1273 (HY000): Unsupported collation when new collation is enabled: 'utf8mb4_0900_ai_ci'" 1

dmctl_stop_task $TASK_NAME $MASTER_PORT

Expand All @@ -118,8 +118,8 @@ function run() {
echo "check incremental phase error"
run_dm_ctl_with_retry $WORK_DIR "127.0.0.1:$MASTER_PORT" \
"query-status ${TASK_NAME}" \
"Error 1273: Unsupported collation when new collation is enabled: 'latin1_swedish_ci'" 1 \
"Error 1273: Unsupported collation when new collation is enabled: 'utf8mb4_0900_ai_ci'" 1
"Error 1273 (HY000): Unsupported collation when new collation is enabled: 'latin1_swedish_ci'" 1 \
"Error 1273 (HY000): Unsupported collation when new collation is enabled: 'utf8mb4_0900_ai_ci'" 1
}

cleanup_data $TEST_NAME
Expand Down
2 changes: 1 addition & 1 deletion dm/tests/tracker_ignored_ddl/run.sh
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ function run() {
run_sql_file $cur/data/db.increment2.sql $MYSQL_HOST1 $MYSQL_PORT1 $MYSQL_PASSWORD1
run_dm_ctl_with_retry $WORK_DIR "127.0.0.1:$MASTER_PORT" \
"query-status test" \
"Error 1054: Unknown column" 1
"Error 1054 (42S22): Unknown column" 1

# force a resume, the error is still there, but we want to check https://github.com/pingcap/tiflow/issues/5272#issuecomment-1109283279
run_dm_ctl $WORK_DIR "127.0.0.1:$MASTER_PORT" \
Expand Down
2 changes: 1 addition & 1 deletion dm/unit/unit_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ func TestIsResumableError(t *testing.T) {
{terror.ErrDBExecuteFailed.Delegate(&tmysql.SQLError{Code: 1105, Message: "unsupported modify column length 20 is less than origin 40", State: tmysql.DefaultMySQLState}, "alter table t modify col varchar(20)"), false},
{terror.ErrDBExecuteFailed.Delegate(&tmysql.SQLError{Code: 1105, Message: "unsupported drop integer primary key", State: tmysql.DefaultMySQLState}, "alter table t drop column id"), false},
{terror.ErrDBExecuteFailed.Delegate(&tmysql.SQLError{Code: 1067, Message: "Invalid default value for 'ct'", State: tmysql.DefaultMySQLState}, "CREATE TABLE `tbl` (`c1` int(11) NOT NULL,`ct` datetime NOT NULL DEFAULT '0000-00-00 00:00:00' COMMENT '创建时间',PRIMARY KEY (`c1`)) ENGINE=InnoDB DEFAULT CHARSET=latin1"), false},
{terror.ErrDBExecuteFailed.Delegate(errors.New("Error 1062: Duplicate entry '5' for key 'PRIMARY'")), false},
{terror.ErrDBExecuteFailed.Delegate(errors.New("Error 1062 (23000): Duplicate entry '5' for key 'PRIMARY'")), false},
{terror.ErrDBExecuteFailed.Delegate(errors.New("INSERT INTO `db`.`tbl` (`c1`,`c2`) VALUES (?,?);: Error 1406: Data too long for column 'c2' at row 1")), false},
// real error is generated by `Delegate` and multiple `Annotatef`, we use `New` to simplify it
{terror.ErrParserParseRelayLog.New("parse relay log file bin.000018 from offset 555 in dir /home/tidb/deploy/relay_log/d2e831df-b4ec-11e9-9237-0242ac110008.000004: parse relay log file bin.000018 from offset 0 in dir /home/tidb/deploy/relay_log/d2e831df-b4ec-11e9-9237-0242ac110008.000004: parse relay log file /home/tidb/deploy/relay_log/d2e831df-b4ec-11e9-9237-0242ac110008.000004/bin.000018: binlog checksum mismatch, data may be corrupted"), false},
Expand Down
2 changes: 1 addition & 1 deletion dm/worker/server_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ import (
"github.com/pingcap/tiflow/dm/relay"
"github.com/pingcap/tiflow/dm/unit"
"github.com/stretchr/testify/require"
"github.com/tikv/pd/pkg/tempurl"
"github.com/tikv/pd/pkg/utils/tempurl"
v3rpc "go.etcd.io/etcd/api/v3/v3rpc/rpctypes"
clientv3 "go.etcd.io/etcd/client/v3"
"go.etcd.io/etcd/server/v3/embed"
Expand Down
2 changes: 1 addition & 1 deletion dm/worker/source_worker_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ import (
"github.com/pingcap/tiflow/dm/syncer"
"github.com/pingcap/tiflow/dm/unit"
"github.com/stretchr/testify/require"
"github.com/tikv/pd/pkg/tempurl"
"github.com/tikv/pd/pkg/utils/tempurl"
clientv3 "go.etcd.io/etcd/client/v3"
)

Expand Down
2 changes: 1 addition & 1 deletion engine/test/integration_tests/dm_collation/run.sh
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ function run() {
# wait for dump and load finished

exec_with_retry --count 30 "curl \"http://127.0.0.1:10245/api/v1/jobs/$job_id/status\" | tee /dev/stderr | jq -e '.task_status.\"mysql-01\".status.unit == \"DMSyncTask\" and .task_status.\"mysql-02\".status.unit == \"DMSyncTask\"'"
curl http://127.0.0.1:10245/api/v1/jobs/$job_id/status | tee /dev/stderr | jq -e '.finished_unit_status."mysql-02"[1].Status.finishedBytes == 174'
curl http://127.0.0.1:10245/api/v1/jobs/$job_id/status | tee /dev/stderr | jq -e '.finished_unit_status."mysql-02"[1].Status.finishedBytes == 248'

# check data

Expand Down
4 changes: 2 additions & 2 deletions engine/test/integration_tests/dm_full_mode/run.sh
Original file line number Diff line number Diff line change
Expand Up @@ -55,10 +55,10 @@ function run() {

# create job & wait for job finished
job_id=$(create_job "DM" "$CUR_DIR/conf/job.yaml" "dm_full_mode")
exec_with_retry --count 30 "curl \"http://127.0.0.1:10245/api/v1/jobs/$job_id/status\" | tee /dev/stderr | grep -q 'Error 1142: ALTER command denied'"
exec_with_retry --count 30 "curl \"http://127.0.0.1:10245/api/v1/jobs/$job_id/status\" | tee /dev/stderr | grep -q 'Error 1142 (42000): ALTER command denied'"
docker restart server-executor-0 server-executor-1 server-executor-2
# check the error is not related to lightning checkpoint
exec_with_retry --count 60 "curl \"http://127.0.0.1:10245/api/v1/jobs/$job_id/status\" | tee /dev/stderr | grep -q 'Error 1142: ALTER command denied'"
exec_with_retry --count 60 "curl \"http://127.0.0.1:10245/api/v1/jobs/$job_id/status\" | tee /dev/stderr | grep -q 'Error 1142 (42000): ALTER command denied'"

run_sql --port 4000 "GRANT ALTER ON *.* TO 'dm_full'@'%';"
curl -X PUT "http://127.0.0.1:10245/api/v1/jobs/$job_id/status" -H 'Content-Type: application/json' -d '{"op": "resume"}'
Expand Down
Loading