Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ccl/backupccl: TestBackupRestoreDataDriven failed #55911

Closed
cockroach-teamcity opened this issue Oct 23, 2020 · 13 comments · Fixed by #55937
Closed

ccl/backupccl: TestBackupRestoreDataDriven failed #55911

cockroach-teamcity opened this issue Oct 23, 2020 · 13 comments · Fixed by #55937
Assignees
@solongordon
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Milestone
20.2

Comments

@cockroach-teamcity
Copy link
Member

(ccl/backupccl).TestBackupRestoreDataDriven failed on master@33e3abc57cccedd86dd1b2e1d58feb075fcb8dd4:

  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.(*restoreDataProcessor).Run()
      <autogenerated>:1 +0x57
  github.com/cockroachdb/cockroach/pkg/sql/flowinfra.(*FlowBase).Run()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/flowinfra/flow.go:392 +0x432
  github.com/cockroachdb/cockroach/pkg/sql/rowflow.(*rowBasedFlow).Run()
      <autogenerated>:1 +0x7f
  github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:422 +0x94e
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.distRestore()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_processor_planning.go:228 +0x19a0
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.restore()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_job.go:621 +0xede
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.(*restoreResumer).Resume()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_job.go:1187 +0xbcb
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1089 +0x144
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1090 +0xa8e
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).runJob()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:245 +0x46a
  github.com/cockroachdb/cockroach/pkg/jobs.(*StartableJob).Start.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/jobs.go:853 +0x2c9
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:347 +0x162

Goroutine 935 (running) created at:
  golang.org/x/sync/errgroup.(*Group).Go()
      /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x73
  github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:165 +0xda
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.runSplitAndScatter()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/split_and_scatter_processor.go:252 +0x2ef
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.(*splitAndScatterProcessor).Run.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/split_and_scatter_processor.go:182 +0x107

Goroutine 729 (running) created at:
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:342 +0x14a
  github.com/cockroachdb/cockroach/pkg/jobs.(*StartableJob).Start()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/jobs.go:852 +0x49e
  github.com/cockroachdb/cockroach/pkg/jobs.(*StartableJob).Run()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/jobs.go:871 +0x16d
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.doRestorePlan()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_planning.go:1627 +0x27dd
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.restorePlanHook.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_planning.go:1297 +0x8bc
  github.com/cockroachdb/cockroach/pkg/sql.(*hookFnNode).startExec.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/planhook.go:146 +0x10d
==================
FAIL	github.com/cockroachdb/cockroach/pkg/ccl/backupccl	48.900s
More

Parameters:

  • GOFLAGS=-json
make stressrace TESTS=TestBackupRestoreDataDriven PKG=./pkg/ccl/backupccl TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
powered by pkg/cmd/internal/issues
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Oct 23, 2020
@cockroach-teamcity cockroach-teamcity added this to the 20.2 milestone Oct 23, 2020
@rytaft
Copy link
Collaborator

rytaft commented Oct 23, 2020

This is the same data race as the recent failure in #50032.

@cockroach-teamcity
Copy link
Member Author

(ccl/backupccl).TestBackupRestoreDataDriven failed on master@33e3abc57cccedd86dd1b2e1d58feb075fcb8dd4:

  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.(*restoreDataProcessor).Run()
      <autogenerated>:1 +0x57
  github.com/cockroachdb/cockroach/pkg/sql/flowinfra.(*FlowBase).Run()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/flowinfra/flow.go:392 +0x432
  github.com/cockroachdb/cockroach/pkg/sql/rowflow.(*rowBasedFlow).Run()
      <autogenerated>:1 +0x7f
  github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:422 +0x94e
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.distRestore()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_processor_planning.go:228 +0x19a0
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.restore()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_job.go:621 +0xede
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.(*restoreResumer).Resume()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_job.go:1187 +0xbcb
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1089 +0x144
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1090 +0xa8e
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).runJob()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:245 +0x46a
  github.com/cockroachdb/cockroach/pkg/jobs.(*StartableJob).Start.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/jobs.go:853 +0x2c9
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:347 +0x162

Goroutine 124 (running) created at:
  golang.org/x/sync/errgroup.(*Group).Go()
      /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x73
  github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:165 +0xda
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.runSplitAndScatter()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/split_and_scatter_processor.go:252 +0x2ef
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.(*splitAndScatterProcessor).Run.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/split_and_scatter_processor.go:182 +0x107

Goroutine 470 (running) created at:
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:342 +0x14a
  github.com/cockroachdb/cockroach/pkg/jobs.(*StartableJob).Start()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/jobs.go:852 +0x49e
  github.com/cockroachdb/cockroach/pkg/jobs.(*StartableJob).Run()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/jobs.go:871 +0x16d
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.doRestorePlan()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_planning.go:1627 +0x27dd
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.restorePlanHook.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_planning.go:1297 +0x8bc
  github.com/cockroachdb/cockroach/pkg/sql.(*hookFnNode).startExec.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/planhook.go:146 +0x10d
==================
FAIL	github.com/cockroachdb/cockroach/pkg/ccl/backupccl	57.929s
More

Parameters:

  • GOFLAGS=-json
make stressrace TESTS=TestBackupRestoreDataDriven PKG=./pkg/ccl/backupccl TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(ccl/backupccl).TestBackupRestoreDataDriven failed on master@33e3abc57cccedd86dd1b2e1d58feb075fcb8dd4:

  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.(*restoreDataProcessor).Run()
      <autogenerated>:1 +0x57
  github.com/cockroachdb/cockroach/pkg/sql/flowinfra.(*FlowBase).Run()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/flowinfra/flow.go:392 +0x432
  github.com/cockroachdb/cockroach/pkg/sql/rowflow.(*rowBasedFlow).Run()
      <autogenerated>:1 +0x7f
  github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:422 +0x94e
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.distRestore()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_processor_planning.go:228 +0x19a0
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.restore()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_job.go:621 +0xede
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.(*restoreResumer).Resume()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_job.go:1187 +0xbcb
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1089 +0x144
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1090 +0xa8e
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).runJob()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:245 +0x46a
  github.com/cockroachdb/cockroach/pkg/jobs.(*StartableJob).Start.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/jobs.go:853 +0x2c9
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:347 +0x162

Goroutine 777 (running) created at:
  golang.org/x/sync/errgroup.(*Group).Go()
      /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x73
  github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:165 +0xda
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.runSplitAndScatter()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/split_and_scatter_processor.go:252 +0x2ef
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.(*splitAndScatterProcessor).Run.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/split_and_scatter_processor.go:182 +0x107

Goroutine 832 (running) created at:
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:342 +0x14a
  github.com/cockroachdb/cockroach/pkg/jobs.(*StartableJob).Start()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/jobs.go:852 +0x49e
  github.com/cockroachdb/cockroach/pkg/jobs.(*StartableJob).Run()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/jobs.go:871 +0x16d
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.doRestorePlan()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_planning.go:1627 +0x27dd
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.restorePlanHook.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_planning.go:1297 +0x8bc
  github.com/cockroachdb/cockroach/pkg/sql.(*hookFnNode).startExec.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/planhook.go:146 +0x10d
==================
FAIL	github.com/cockroachdb/cockroach/pkg/ccl/backupccl	35.128s
More

Parameters:

  • GOFLAGS=-json
make stressrace TESTS=TestBackupRestoreDataDriven PKG=./pkg/ccl/backupccl TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(ccl/backupccl).TestBackupRestoreDataDriven failed on master@33e3abc57cccedd86dd1b2e1d58feb075fcb8dd4:

  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.(*restoreDataProcessor).Run()
      <autogenerated>:1 +0x57
  github.com/cockroachdb/cockroach/pkg/sql/flowinfra.(*FlowBase).Run()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/flowinfra/flow.go:392 +0x432
  github.com/cockroachdb/cockroach/pkg/sql/rowflow.(*rowBasedFlow).Run()
      <autogenerated>:1 +0x7f
  github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:422 +0x94e
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.distRestore()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_processor_planning.go:228 +0x19a0
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.restore()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_job.go:621 +0xede
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.(*restoreResumer).Resume()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_job.go:1187 +0xbcb
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1089 +0x144
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1090 +0xa8e
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).runJob()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:245 +0x46a
  github.com/cockroachdb/cockroach/pkg/jobs.(*StartableJob).Start.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/jobs.go:853 +0x2c9
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:347 +0x162

Goroutine 455 (running) created at:
  golang.org/x/sync/errgroup.(*Group).Go()
      /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x73
  github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:165 +0xda
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.runSplitAndScatter()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/split_and_scatter_processor.go:252 +0x2ef
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.(*splitAndScatterProcessor).Run.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/split_and_scatter_processor.go:182 +0x107

Goroutine 121 (running) created at:
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:342 +0x14a
  github.com/cockroachdb/cockroach/pkg/jobs.(*StartableJob).Start()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/jobs.go:852 +0x49e
  github.com/cockroachdb/cockroach/pkg/jobs.(*StartableJob).Run()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/jobs.go:871 +0x16d
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.doRestorePlan()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_planning.go:1627 +0x27dd
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.restorePlanHook.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_planning.go:1297 +0x8bc
  github.com/cockroachdb/cockroach/pkg/sql.(*hookFnNode).startExec.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/planhook.go:146 +0x10d
==================
FAIL	github.com/cockroachdb/cockroach/pkg/ccl/backupccl	26.616s
More

Parameters:

  • GOFLAGS=-json
make stressrace TESTS=TestBackupRestoreDataDriven PKG=./pkg/ccl/backupccl TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(ccl/backupccl).TestBackupRestoreDataDriven failed on master@33e3abc57cccedd86dd1b2e1d58feb075fcb8dd4:

  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.(*restoreDataProcessor).Run()
      <autogenerated>:1 +0x57
  github.com/cockroachdb/cockroach/pkg/sql/flowinfra.(*FlowBase).Run()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/flowinfra/flow.go:392 +0x432
  github.com/cockroachdb/cockroach/pkg/sql/rowflow.(*rowBasedFlow).Run()
      <autogenerated>:1 +0x7f
  github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:422 +0x94e
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.distRestore()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_processor_planning.go:228 +0x19a0
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.restore()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_job.go:621 +0xede
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.(*restoreResumer).Resume()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_job.go:1187 +0xbcb
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1089 +0x144
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1090 +0xa8e
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).runJob()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:245 +0x46a
  github.com/cockroachdb/cockroach/pkg/jobs.(*StartableJob).Start.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/jobs.go:853 +0x2c9
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:347 +0x162

Goroutine 463 (running) created at:
  golang.org/x/sync/errgroup.(*Group).Go()
      /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x73
  github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:165 +0xda
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.runSplitAndScatter()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/split_and_scatter_processor.go:252 +0x2ef
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.(*splitAndScatterProcessor).Run.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/split_and_scatter_processor.go:182 +0x107

Goroutine 477 (running) created at:
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:342 +0x14a
  github.com/cockroachdb/cockroach/pkg/jobs.(*StartableJob).Start()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/jobs.go:852 +0x49e
  github.com/cockroachdb/cockroach/pkg/jobs.(*StartableJob).Run()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/jobs.go:871 +0x16d
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.doRestorePlan()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_planning.go:1627 +0x27dd
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.restorePlanHook.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_planning.go:1297 +0x8bc
  github.com/cockroachdb/cockroach/pkg/sql.(*hookFnNode).startExec.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/planhook.go:146 +0x10d
==================
FAIL	github.com/cockroachdb/cockroach/pkg/ccl/backupccl	26.761s
More

Parameters:

  • GOFLAGS=-json
make stressrace TESTS=TestBackupRestoreDataDriven PKG=./pkg/ccl/backupccl TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(ccl/backupccl).TestBackupRestoreDataDriven failed on master@75234294d22cc9a6e12724b043f49e78f0ac92d2:

  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).Send()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:772 +0xab3
  github.com/cockroachdb/cockroach/pkg/kv.(*CrossRangeTxnWrapperSender).Send()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:219 +0xfc
  github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:744 +0x1ad
  github.com/cockroachdb/cockroach/pkg/kv.(*DB).send-fm()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:726 +0xf6
  github.com/cockroachdb/cockroach/pkg/kv.sendAndFill()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:654 +0x182
  github.com/cockroachdb/cockroach/pkg/kv.(*DB).Run()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:677 +0xd1
  github.com/cockroachdb/cockroach/pkg/kv.(*DB).AdminSplit()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:532 +0xdf
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.dbSplitAndScatterer.splitAndScatterKey()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/split_and_scatter_processor.go:61 +0x228
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.(*dbSplitAndScatterer).splitAndScatterKey()
      <autogenerated>:1 +0xb5
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.runSplitAndScatter.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/split_and_scatter_processor.go:257 +0x378
  github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:166 +0x4e
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x71

Goroutine 432 (running) created at:
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:342 +0x14a
  github.com/cockroachdb/cockroach/pkg/jobs.(*StartableJob).Start()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/jobs.go:852 +0x49e
  github.com/cockroachdb/cockroach/pkg/jobs.(*StartableJob).Run()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/jobs.go:871 +0x16d
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.doRestorePlan()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_planning.go:1627 +0x27dd
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.restorePlanHook.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_planning.go:1297 +0x8bc
  github.com/cockroachdb/cockroach/pkg/sql.(*hookFnNode).startExec.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/planhook.go:146 +0x10d

Goroutine 196 (running) created at:
  golang.org/x/sync/errgroup.(*Group).Go()
      /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x73
  github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:165 +0xda
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.runSplitAndScatter()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/split_and_scatter_processor.go:252 +0x2ef
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.(*splitAndScatterProcessor).Run.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/split_and_scatter_processor.go:182 +0x107
==================
FAIL	github.com/cockroachdb/cockroach/pkg/ccl/backupccl	27.582s
More

Parameters:

  • GOFLAGS=-json
make stressrace TESTS=TestBackupRestoreDataDriven PKG=./pkg/ccl/backupccl TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(ccl/backupccl).TestBackupRestoreDataDriven failed on master@75234294d22cc9a6e12724b043f49e78f0ac92d2:

      /go/src/github.com/cockroachdb/cockroach/pkg/sql/flowinfra/flow.go:392 +0x432
  github.com/cockroachdb/cockroach/pkg/sql/rowflow.(*rowBasedFlow).Run()
      <autogenerated>:1 +0x7f
  github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:422 +0x94e
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.distRestore()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_processor_planning.go:228 +0x19a0
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.restore()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_job.go:621 +0xede
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.(*restoreResumer).Resume()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_job.go:1187 +0xbcb
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1089 +0x144
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1090 +0xa8e
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).runJob()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:245 +0x46a
  github.com/cockroachdb/cockroach/pkg/jobs.(*StartableJob).Start.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/jobs.go:853 +0x2c9
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:347 +0x162

Goroutine 184 (running) created at:
  golang.org/x/sync/errgroup.(*Group).Go()
      /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x73
  github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:165 +0xda
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.runSplitAndScatter()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/split_and_scatter_processor.go:252 +0x2ef
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.(*splitAndScatterProcessor).Run.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/split_and_scatter_processor.go:182 +0x107

Goroutine 493 (running) created at:
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:342 +0x14a
  github.com/cockroachdb/cockroach/pkg/jobs.(*StartableJob).Start()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/jobs.go:852 +0x49e
  github.com/cockroachdb/cockroach/pkg/jobs.(*StartableJob).Run()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/jobs.go:871 +0x16d
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.doRestorePlan()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_planning.go:1627 +0x27dd
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.restorePlanHook.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_planning.go:1297 +0x8bc
  github.com/cockroachdb/cockroach/pkg/sql.(*hookFnNode).startExec.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/planhook.go:146 +0x10d
==================
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestBackupRestoreDataDriven264680221
    --- FAIL: TestBackupRestoreDataDriven/user-defined-types (8.64s)
        testing.go:853: race detected during execution of test
    testing.go:853: race detected during execution of test
More

Parameters:

  • TAGS=
  • GOFLAGS=-race -parallel=2
make stressrace TESTS=TestBackupRestoreDataDriven PKG=./pkg/ccl/backupccl TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(ccl/backupccl).TestBackupRestoreDataDriven failed on master@75234294d22cc9a6e12724b043f49e78f0ac92d2:

  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.(*restoreDataProcessor).Run()
      <autogenerated>:1 +0x57
  github.com/cockroachdb/cockroach/pkg/sql/flowinfra.(*FlowBase).Run()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/flowinfra/flow.go:392 +0x432
  github.com/cockroachdb/cockroach/pkg/sql/rowflow.(*rowBasedFlow).Run()
      <autogenerated>:1 +0x7f
  github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:422 +0x94e
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.distRestore()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_processor_planning.go:228 +0x19a0
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.restore()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_job.go:621 +0xede
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.(*restoreResumer).Resume()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_job.go:1187 +0xbcb
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1089 +0x144
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1090 +0xa8e
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).runJob()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:245 +0x46a
  github.com/cockroachdb/cockroach/pkg/jobs.(*StartableJob).Start.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/jobs.go:853 +0x2c9
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:347 +0x162

Goroutine 355 (running) created at:
  golang.org/x/sync/errgroup.(*Group).Go()
      /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x73
  github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:165 +0xda
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.runSplitAndScatter()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/split_and_scatter_processor.go:252 +0x2ef
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.(*splitAndScatterProcessor).Run.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/split_and_scatter_processor.go:182 +0x107

Goroutine 117 (running) created at:
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:342 +0x14a
  github.com/cockroachdb/cockroach/pkg/jobs.(*StartableJob).Start()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/jobs.go:852 +0x49e
  github.com/cockroachdb/cockroach/pkg/jobs.(*StartableJob).Run()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/jobs.go:871 +0x16d
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.doRestorePlan()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_planning.go:1627 +0x27dd
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.restorePlanHook.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_planning.go:1297 +0x8bc
  github.com/cockroachdb/cockroach/pkg/sql.(*hookFnNode).startExec.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/planhook.go:146 +0x10d
==================
FAIL	github.com/cockroachdb/cockroach/pkg/ccl/backupccl	20.530s
More

Parameters:

  • GOFLAGS=-json
make stressrace TESTS=TestBackupRestoreDataDriven PKG=./pkg/ccl/backupccl TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(ccl/backupccl).TestBackupRestoreDataDriven failed on master@75234294d22cc9a6e12724b043f49e78f0ac92d2:

  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).Send()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:772 +0xab3
  github.com/cockroachdb/cockroach/pkg/kv.(*CrossRangeTxnWrapperSender).Send()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:219 +0xfc
  github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:744 +0x1ad
  github.com/cockroachdb/cockroach/pkg/kv.(*DB).send-fm()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:726 +0xf6
  github.com/cockroachdb/cockroach/pkg/kv.sendAndFill()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:654 +0x182
  github.com/cockroachdb/cockroach/pkg/kv.(*DB).Run()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:677 +0xd1
  github.com/cockroachdb/cockroach/pkg/kv.(*DB).AdminSplit()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:532 +0xdf
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.dbSplitAndScatterer.splitAndScatterKey()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/split_and_scatter_processor.go:61 +0x228
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.(*dbSplitAndScatterer).splitAndScatterKey()
      <autogenerated>:1 +0xb5
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.runSplitAndScatter.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/split_and_scatter_processor.go:257 +0x378
  github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:166 +0x4e
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x71

Goroutine 645 (running) created at:
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:342 +0x14a
  github.com/cockroachdb/cockroach/pkg/jobs.(*StartableJob).Start()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/jobs.go:852 +0x49e
  github.com/cockroachdb/cockroach/pkg/jobs.(*StartableJob).Run()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/jobs.go:871 +0x16d
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.doRestorePlan()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_planning.go:1627 +0x27dd
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.restorePlanHook.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_planning.go:1297 +0x8bc
  github.com/cockroachdb/cockroach/pkg/sql.(*hookFnNode).startExec.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/planhook.go:146 +0x10d

Goroutine 177 (running) created at:
  golang.org/x/sync/errgroup.(*Group).Go()
      /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x73
  github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:165 +0xda
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.runSplitAndScatter()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/split_and_scatter_processor.go:252 +0x2ef
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.(*splitAndScatterProcessor).Run.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/split_and_scatter_processor.go:182 +0x107
==================
FAIL	github.com/cockroachdb/cockroach/pkg/ccl/backupccl	32.612s
More

Parameters:

  • GOFLAGS=-json
make stressrace TESTS=TestBackupRestoreDataDriven PKG=./pkg/ccl/backupccl TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(ccl/backupccl).TestBackupRestoreDataDriven failed on master@75234294d22cc9a6e12724b043f49e78f0ac92d2:

  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.(*restoreDataProcessor).Run()
      <autogenerated>:1 +0x57
  github.com/cockroachdb/cockroach/pkg/sql/flowinfra.(*FlowBase).Run()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/flowinfra/flow.go:392 +0x432
  github.com/cockroachdb/cockroach/pkg/sql/rowflow.(*rowBasedFlow).Run()
      <autogenerated>:1 +0x7f
  github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:422 +0x94e
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.distRestore()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_processor_planning.go:228 +0x19a0
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.restore()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_job.go:621 +0xede
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.(*restoreResumer).Resume()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_job.go:1187 +0xbcb
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1089 +0x144
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1090 +0xa8e
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).runJob()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:245 +0x46a
  github.com/cockroachdb/cockroach/pkg/jobs.(*StartableJob).Start.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/jobs.go:853 +0x2c9
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:347 +0x162

Goroutine 464 (running) created at:
  golang.org/x/sync/errgroup.(*Group).Go()
      /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x73
  github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:165 +0xda
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.runSplitAndScatter()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/split_and_scatter_processor.go:252 +0x2ef
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.(*splitAndScatterProcessor).Run.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/split_and_scatter_processor.go:182 +0x107

Goroutine 817 (running) created at:
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:342 +0x14a
  github.com/cockroachdb/cockroach/pkg/jobs.(*StartableJob).Start()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/jobs.go:852 +0x49e
  github.com/cockroachdb/cockroach/pkg/jobs.(*StartableJob).Run()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/jobs.go:871 +0x16d
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.doRestorePlan()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_planning.go:1627 +0x27dd
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.restorePlanHook.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_planning.go:1297 +0x8bc
  github.com/cockroachdb/cockroach/pkg/sql.(*hookFnNode).startExec.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/planhook.go:146 +0x10d
==================
FAIL	github.com/cockroachdb/cockroach/pkg/ccl/backupccl	30.963s
More

Parameters:

  • GOFLAGS=-json
make stressrace TESTS=TestBackupRestoreDataDriven PKG=./pkg/ccl/backupccl TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(ccl/backupccl).TestBackupRestoreDataDriven failed on master@75234294d22cc9a6e12724b043f49e78f0ac92d2:

  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.(*restoreDataProcessor).Run()
      <autogenerated>:1 +0x57
  github.com/cockroachdb/cockroach/pkg/sql/flowinfra.(*FlowBase).Run()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/flowinfra/flow.go:392 +0x432
  github.com/cockroachdb/cockroach/pkg/sql/rowflow.(*rowBasedFlow).Run()
      <autogenerated>:1 +0x7f
  github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:422 +0x94e
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.distRestore()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_processor_planning.go:228 +0x19a0
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.restore()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_job.go:621 +0xede
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.(*restoreResumer).Resume()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_job.go:1187 +0xbcb
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1089 +0x144
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1090 +0xa8e
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).runJob()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:245 +0x46a
  github.com/cockroachdb/cockroach/pkg/jobs.(*StartableJob).Start.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/jobs.go:853 +0x2c9
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:347 +0x162

Goroutine 952 (running) created at:
  golang.org/x/sync/errgroup.(*Group).Go()
      /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x73
  github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:165 +0xda
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.runSplitAndScatter()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/split_and_scatter_processor.go:252 +0x2ef
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.(*splitAndScatterProcessor).Run.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/split_and_scatter_processor.go:182 +0x107

Goroutine 959 (running) created at:
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:342 +0x14a
  github.com/cockroachdb/cockroach/pkg/jobs.(*StartableJob).Start()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/jobs.go:852 +0x49e
  github.com/cockroachdb/cockroach/pkg/jobs.(*StartableJob).Run()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/jobs.go:871 +0x16d
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.doRestorePlan()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_planning.go:1627 +0x27dd
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.restorePlanHook.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_planning.go:1297 +0x8bc
  github.com/cockroachdb/cockroach/pkg/sql.(*hookFnNode).startExec.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/planhook.go:146 +0x10d
==================
FAIL	github.com/cockroachdb/cockroach/pkg/ccl/backupccl	27.696s
More

Parameters:

  • GOFLAGS=-json
make stressrace TESTS=TestBackupRestoreDataDriven PKG=./pkg/ccl/backupccl TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(ccl/backupccl).TestBackupRestoreDataDriven failed on master@75234294d22cc9a6e12724b043f49e78f0ac92d2:

  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.(*restoreDataProcessor).Run()
      <autogenerated>:1 +0x57
  github.com/cockroachdb/cockroach/pkg/sql/flowinfra.(*FlowBase).Run()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/flowinfra/flow.go:392 +0x432
  github.com/cockroachdb/cockroach/pkg/sql/rowflow.(*rowBasedFlow).Run()
      <autogenerated>:1 +0x7f
  github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:422 +0x94e
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.distRestore()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_processor_planning.go:228 +0x19a0
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.restore()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_job.go:621 +0xede
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.(*restoreResumer).Resume()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_job.go:1187 +0xbcb
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1089 +0x144
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1090 +0xa8e
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).runJob()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:245 +0x46a
  github.com/cockroachdb/cockroach/pkg/jobs.(*StartableJob).Start.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/jobs.go:853 +0x2c9
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:347 +0x162

Goroutine 222 (running) created at:
  golang.org/x/sync/errgroup.(*Group).Go()
      /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x73
  github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:165 +0xda
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.runSplitAndScatter()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/split_and_scatter_processor.go:252 +0x2ef
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.(*splitAndScatterProcessor).Run.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/split_and_scatter_processor.go:182 +0x107

Goroutine 804 (running) created at:
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:342 +0x14a
  github.com/cockroachdb/cockroach/pkg/jobs.(*StartableJob).Start()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/jobs.go:852 +0x49e
  github.com/cockroachdb/cockroach/pkg/jobs.(*StartableJob).Run()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/jobs.go:871 +0x16d
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.doRestorePlan()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_planning.go:1627 +0x27dd
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.restorePlanHook.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_planning.go:1297 +0x8bc
  github.com/cockroachdb/cockroach/pkg/sql.(*hookFnNode).startExec.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/planhook.go:146 +0x10d
==================
FAIL	github.com/cockroachdb/cockroach/pkg/ccl/backupccl	32.262s
More

Parameters:

  • GOFLAGS=-json
make stressrace TESTS=TestBackupRestoreDataDriven PKG=./pkg/ccl/backupccl TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
powered by pkg/cmd/internal/issues

@rytaft rytaft mentioned this issue Oct 24, 2020
Merged
@cockroach-teamcity
Copy link
Member Author

(ccl/backupccl).TestBackupRestoreDataDriven failed on master@75234294d22cc9a6e12724b043f49e78f0ac92d2:

  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).Send()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:772 +0xab3
  github.com/cockroachdb/cockroach/pkg/kv.(*CrossRangeTxnWrapperSender).Send()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:219 +0xfc
  github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:744 +0x1ad
  github.com/cockroachdb/cockroach/pkg/kv.(*DB).send-fm()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:726 +0xf6
  github.com/cockroachdb/cockroach/pkg/kv.sendAndFill()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:654 +0x182
  github.com/cockroachdb/cockroach/pkg/kv.(*DB).Run()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:677 +0xd1
  github.com/cockroachdb/cockroach/pkg/kv.(*DB).AdminSplit()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:532 +0xdf
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.dbSplitAndScatterer.splitAndScatterKey()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/split_and_scatter_processor.go:61 +0x228
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.(*dbSplitAndScatterer).splitAndScatterKey()
      <autogenerated>:1 +0xb5
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.runSplitAndScatter.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/split_and_scatter_processor.go:257 +0x378
  github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:166 +0x4e
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x71

Goroutine 679 (running) created at:
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:342 +0x14a
  github.com/cockroachdb/cockroach/pkg/jobs.(*StartableJob).Start()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/jobs.go:852 +0x49e
  github.com/cockroachdb/cockroach/pkg/jobs.(*StartableJob).Run()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/jobs.go:871 +0x16d
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.doRestorePlan()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_planning.go:1627 +0x27dd
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.restorePlanHook.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_planning.go:1297 +0x8bc
  github.com/cockroachdb/cockroach/pkg/sql.(*hookFnNode).startExec.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/planhook.go:146 +0x10d

Goroutine 132 (running) created at:
  golang.org/x/sync/errgroup.(*Group).Go()
      /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x73
  github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:165 +0xda
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.runSplitAndScatter()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/split_and_scatter_processor.go:252 +0x2ef
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.(*splitAndScatterProcessor).Run.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/split_and_scatter_processor.go:182 +0x107
==================
FAIL	github.com/cockroachdb/cockroach/pkg/ccl/backupccl	27.907s
More

Parameters:

  • GOFLAGS=-json
make stressrace TESTS=TestBackupRestoreDataDriven PKG=./pkg/ccl/backupccl TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
powered by pkg/cmd/internal/issues

@craig craig bot closed this as completed in #55937 Oct 24, 2020
@craig craig bot closed this as completed in 7fa301f Oct 24, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@solongordon solongordon

Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Milestone
20.2
Development

Successfully merging a pull request may close this issue.

tracing: avoid data races w.r.t noopSpan tbg/cockroach
3 participants
@solongordon @rytaft @cockroach-teamcity