ccl/backupccl: TestBackupRestoreAppend failed #54599

cockroach-teamcity · 2020-09-19T05:47:53Z

(ccl/backupccl).TestBackupRestoreAppend failed on master@d6f91638705d381eac18c0524264781e282e350e:

=== RUN   TestBackupRestoreAppend
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestBackupRestoreAppend782368301
--- FAIL: TestBackupRestoreAppend (146.16s)
    test_log_scope.go:154: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestBackupRestoreAppend782368301
    test_log_scope.go:63: use -show-logs to present logs inline
    backup_test.go:590: error executing 'RESTORE DATABASE data FROM $4 IN ($1, $2, $3) AS OF SYSTEM TIME 1600492874166087431.0000000000': pq: database "data" already exists

More

Parameters:

TAGS=
GOFLAGS=-race -parallel=2

make stressrace TESTS=TestBackupRestoreAppend PKG=./pkg/ccl/backupccl TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

Epic CRDB-7778

Jira issue: CRDB-3735

The text was updated successfully, but these errors were encountered:

cockroach-teamcity · 2020-09-21T16:40:16Z

(ccl/backupccl).TestBackupRestoreAppend failed on master@a6dc5e931154c2e7fff1f00ac4d6b272978be990:

=== RUN   TestBackupRestoreAppend
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestBackupRestoreAppend756252241
--- FAIL: TestBackupRestoreAppend (47.96s)
    test_log_scope.go:154: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestBackupRestoreAppend756252241
    test_log_scope.go:63: use -show-logs to present logs inline
    backup_test.go:578: error executing 'RESTORE DATABASE data FROM $4 IN ($1, $2, $3) AS OF SYSTEM TIME 1600699603203544523.0000000000': pq: database "data" already exists

More

Parameters:

GOFLAGS=-json

make stressrace TESTS=TestBackupRestoreAppend PKG=./pkg/ccl/backupccl TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

andreimatei · 2020-09-21T22:50:35Z

I just got a failure that reads

backup_test.go:589: error executing 'DROP DATABASE data CASCADE': pq: database "data" is offline: restoring

https://teamcity.cockroachdb.com/viewLog.html?buildId=2286556&buildTypeId=Cockroach_UnitTests

Refs: cockroachdb#50984, cockroachdb#54599 Reason: flaky test Generated by bin/skip-test. Release justification: non-production code changes Release note: None

55828: roachtest: disable range merge queue in tpchvec/tpcdsvec r=yuzefovich a=yuzefovich This commit disables the range merge queue on tpchvec and tpcdsvec roachtests in order to - hopefully - reduce some noise about the performance slowness. I'm guessing that some of the variation that we've been observing with tpchvec/perf tests might come due to range merge activity, so let's remove that variable. However, I don't have concrete data to back this guess up. Release note: None 55853: ccl/backupccl: skip TestBackupRestoreAppend under race r=rytaft a=rytaft Refs: #50984, #54599 Reason: flaky test Generated by bin/skip-test. Release justification: non-production code changes Release note: None 55857: cmd,skip: add flag to skip test under race r=rytaft a=rytaft This commit updates the `skip-test` command to support a flag `-under_race`, which only skips the test under race builds. This commit also adds a new function to the skip package, `UnderRaceWithIssue`, which makes it easier to skip a test under race and log the associated issue. Release note: None 55863: util: add skip.UnderMetamorphic; skip a test r=jordanlewis a=jordanlewis One test cannot be run under the metamorphic build configuration, since its output depends on the batch size: the test counts the number of spans scanned, which will go up with a small batch size due to the table reader resuming the scan until its finished. Release note: None Co-authored-by: Yahor Yuzefovich <yahor@cockroachlabs.com> Co-authored-by: Rebecca Taft <becca@cockroachlabs.com> Co-authored-by: Jordan Lewis <jordanthelewis@gmail.com>

cockroach-teamcity · 2020-12-17T18:12:36Z

(ccl/backupccl).TestBackupRestoreAppend failed on master@bdc14ea7c22e4fd78ffec35efdaa108f2c0bbc7a:

=== RUN   TestBackupRestoreAppend
    test_log_scope.go:72: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestBackupRestoreAppend920663505
    test_log_scope.go:73: use -show-logs to present logs inline
E201217 17:52:19.861405 96467 kv/kvserver/queue.go:1087  [n1,merge,s1,r36/1:‹/Table/53{-/1/100}›] while carrying out changes [{ADD_VOTER n1,s1}]: change replicas of r40 failed: descriptor changed: [expected] r40:/Table/53/1/{100-200} [(n2,s2):2, (n1,s1):4LEARNER, next=5, gen=16] != [actual] r40:/Table/53/1/{100-200} [(n2,s2):2, next=5, gen=17]
E201217 17:52:42.682069 123743 kv/kvclient/kvcoord/txn_interceptor_committer.go:449  making txn commit explicit failed for "unnamed" meta={id=76b8d00d pri=0.01066746 epo=0 ts=1608227562.666890646,0 min=1608227562.666890646,0 seq=6} lock=true stat=STAGING rts=1608227562.666890646,0 wto=false max=1608227562.666890646,0 int=1 ifw=5: result is ambiguous (server shutdown)
E201217 17:52:56.303072 148636 kv/kvclient/kvcoord/txn_interceptor_committer.go:449  making txn commit explicit failed for "unnamed" meta={id=d511ce4e pri=0.00126176 epo=0 ts=1608227576.282469634,0 min=1608227576.282469634,0 seq=6} lock=true stat=STAGING rts=1608227576.282469634,0 wto=false max=1608227576.282469634,0 int=1 ifw=5: result is ambiguous (server shutdown)
    backup_test.go:574: error scanning '&{<nil> 0xc00f9b8900}': pq: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=20d2be73 key=/Table/71/1 pri=0.00806116 epo=0 ts=1608227596.277805481,1 min=1608227595.679125504,0 seq=1000} lock=true stat=PENDING rts=1608227595.679125504,0 wto=false max=1608227596.179125504,0
    panic.go:636: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestBackupRestoreAppend920663505
--- FAIL: TestBackupRestoreAppend (69.60s)

More

Parameters:

GOFLAGS=-json

make stressrace TESTS=TestBackupRestoreAppend PKG=./pkg/ccl/backupccl TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

knz · 2020-12-22T18:53:55Z

New repro on master:

=== RUN   TestBackupRestoreAppend
test_log_scope.go:72: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestBackupRestoreAppend117608420
test_log_scope.go:73: use -show-logs to present logs inline
E201222 16:43:46.735532 123352 kv/kvclient/kvcoord/txn_interceptor_committer.go:449  making txn commit explicit failed for "unnamed" meta={id=f50e1f74 pri=0.03334652 epo=0 ts=1608655426.715293980,0 min=1608655426.715293980,0 seq=6} lock=true stat=STAGING rts=1608655426.715293980,0 wto=false max=1608655426.715293980,0 int=1 ifw=5: result is ambiguous (server shutdown)
E201222 16:44:01.029285 145719 kv/kvclient/kvcoord/txn_interceptor_committer.go:449  making txn commit explicit failed for "unnamed" meta={id=6daa7333 pri=0.01000813 epo=0 ts=1608655440.943437789,0 min=1608655440.943437789,0 seq=6} lock=true stat=STAGING rts=1608655440.943437789,0 wto=false max=1608655440.943437789,0 int=1 ifw=5: result is ambiguous (error=failed to connect to n3 at 127.0.0.1:46791: context canceled [exhausted])
backup_test.go:574: error scanning '&{<nil> 0xc00e550700}': pq: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=98f15d1f key=/Table/71/1 pri=0.02197954 epo=0 ts=1608655461.840806126,1 min=1608655461.061331760,0 seq=1000} lock=true stat=PENDING rts=1608655461.061331760,0 wto=false max=1608655461.561331760,0
E201222 16:44:21.973280 177257 kv/kvclient/kvcoord/txn_interceptor_committer.go:449  making txn commit explicit failed for "unnamed" meta={id=b3beff07 pri=0.00075609 epo=0 ts=1608655461.950481822,0 min=1608655461.950481822,0 seq=2} lock=true stat=STAGING rts=1608655461.950481822,0 wto=false max=1608655461.950481822,0 ifw=1: result is ambiguous (server shutdown)
E201222 16:44:21.977852 82833 kv/kvserver/reports/reporter.go:143  [n1,replication-reporter] failed to generate replication reports: failed to save locality report: timestamp-upsert-replication-critical-localities: failed to send RPC: sending to all replicas failed; last error: node unavailable; try another peer
panic.go:636: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestBackupRestoreAppend117608420
--- FAIL: TestBackupRestoreAppend (60.33s)

cockroach-teamcity · 2020-12-22T19:57:43Z

(ccl/backupccl).TestBackupRestoreAppend failed on master@a230e0e7ea375da92c6da1caf26508d883e90f6c:

=== RUN   TestBackupRestoreAppend
    test_log_scope.go:72: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestBackupRestoreAppend940487294
    test_log_scope.go:73: use -show-logs to present logs inline
E201222 19:36:40.550885 92021 kv/kvserver/queue.go:1087  [n2,merge,s2,r44/2:‹/Table/53/1/{100-200}›] while carrying out changes [{ADD_VOTER n2,s2}]: change replicas of r43 failed: descriptor changed: [expected] r43:/Table/53/1/{200-300} [(n3,s3):3, (n2,s2):4LEARNER, next=5, gen=15] != [actual] r43:/Table/53/1/{200-300} [(n3,s3):3, next=5, gen=16]
E201222 19:37:08.818811 146965 kv/kvclient/kvcoord/txn_interceptor_committer.go:449  making txn commit explicit failed for "unnamed" meta={id=8b86d4f1 pri=0.03054123 epo=0 ts=1608665828.807807029,0 min=1608665828.807807029,0 seq=6} lock=true stat=STAGING rts=1608665828.807807029,0 wto=false max=1608665828.807807029,0 int=1 ifw=5: result is ambiguous (server shutdown)
    backup_test.go:574: error scanning '&{<nil> 0xc0014a1380}': pq: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=55d24d79 key=/Table/71/1 pri=0.00024429 epo=0 ts=1608665847.729405425,1 min=1608665847.095551530,0 seq=1000} lock=true stat=PENDING rts=1608665847.095551530,0 wto=false max=1608665847.595551530,0
    panic.go:636: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestBackupRestoreAppend940487294
--- FAIL: TestBackupRestoreAppend (58.94s)

More

Parameters:

GOFLAGS=-json

make stressrace TESTS=TestBackupRestoreAppend PKG=./pkg/ccl/backupccl TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-12-24T02:06:45Z

(ccl/backupccl).TestBackupRestoreAppend failed on master@7e604ec4d47ed9817dc706956a0f73974ec10e74:

=== RUN   TestBackupRestoreAppend
    test_log_scope.go:72: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestBackupRestoreAppend808669847
    test_log_scope.go:73: use -show-logs to present logs inline
E201224 01:44:35.012063 96674 kv/kvserver/queue.go:1087  [n1,merge,s1,r36/1:‹/Table/53{-/1/400}›] while carrying out changes [{REMOVE_VOTER n2,s2}]: change replicas of r56 failed: descriptor changed: [expected] r56:/Table/53/1/{400-500} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=8] != [actual] r56:/Table/53/1/{400-700} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=11]
E201224 01:45:10.240738 146417 kv/kvclient/kvcoord/txn_interceptor_committer.go:449  making txn commit explicit failed for "unnamed" meta={id=712bcf3d pri=0.01409678 epo=0 ts=1608774310.231213396,0 min=1608774310.231213396,0 seq=6} lock=true stat=STAGING rts=1608774310.231213396,0 wto=false max=1608774310.231213396,0 int=1 ifw=5: result is ambiguous (server shutdown)
    backup_test.go:610: error executing 'RESTORE DATABASE data FROM $4 IN ($1, $2, $3) AS OF SYSTEM TIME 1608774310269232393.0000000000': pq: expected manifest 20201224/014529.20/BACKUP_PART_2_dc=dc2 not found in backup locations
    panic.go:636: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestBackupRestoreAppend808669847
--- FAIL: TestBackupRestoreAppend (264.72s)

More

Parameters:

GOFLAGS=-json

make stressrace TESTS=TestBackupRestoreAppend PKG=./pkg/ccl/backupccl TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-12-28T18:38:38Z

(ccl/backupccl).TestBackupRestoreAppend failed on master@4b2d11ba6831a4b5370fd90eb6faa2aafec59285:

=== RUN   TestBackupRestoreAppend
    test_log_scope.go:72: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestBackupRestoreAppend788847949
    test_log_scope.go:73: use -show-logs to present logs inline
E201228 18:17:31.413111 123761 kv/kvclient/kvcoord/txn_interceptor_committer.go:449  making txn commit explicit failed for "unnamed" meta={id=8ec34d1b pri=0.03179754 epo=0 ts=1609179451.371438780,0 min=1609179451.371438780,0 seq=6} lock=true stat=STAGING rts=1609179451.371438780,0 wto=false max=1609179451.371438780,0 int=1 ifw=5: result is ambiguous (server shutdown)
E201228 18:17:49.699234 147920 kv/kvclient/kvcoord/txn_interceptor_committer.go:449  making txn commit explicit failed for "unnamed" meta={id=d75d7220 pri=0.01363833 epo=0 ts=1609179469.680709931,0 min=1609179469.680709931,0 seq=6} lock=true stat=STAGING rts=1609179469.680709931,0 wto=false max=1609179469.680709931,0 int=1 ifw=5: result is ambiguous (server shutdown)
    backup_test.go:574: error scanning '&{<nil> 0xc00c36ef00}': pq: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=84280d40 key=/Table/71/1 pri=0.01230631 epo=0 ts=1609179486.353501511,1 min=1609179486.106810453,0 seq=1000} lock=true stat=PENDING rts=1609179486.106810453,0 wto=false max=1609179486.606810453,0
    panic.go:636: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestBackupRestoreAppend788847949
--- FAIL: TestBackupRestoreAppend (63.88s)

More

Parameters:

GOFLAGS=-json

make stressrace TESTS=TestBackupRestoreAppend PKG=./pkg/ccl/backupccl TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

yuzefovich · 2020-12-28T22:21:16Z

Hit this too.

cockroach-teamcity · 2021-01-07T00:10:55Z

(ccl/backupccl).TestBackupRestoreAppend failed on master@f62055c1239a8c101b26565828649c6d00e53043:

=== RUN   TestBackupRestoreAppend
    test_log_scope.go:72: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestBackupRestoreAppend765090177
    test_log_scope.go:73: use -show-logs to present logs inline
E210106 23:54:19.450659 94896 kv/kvserver/queue.go:1087  [n1,merge,s1,r36/1:/Table/53{-/1/100}] while carrying out changes [{ADD_VOTER n1,s1}]: change replicas of r43 failed: descriptor changed: [expected] r43:/Table/53/1/{100-200} [(n2,s2):2, (n1,s1):4LEARNER, next=5, gen=16] != [actual] r43:/Table/53/1/{100-200} [(n2,s2):2, next=5, gen=17]
E210106 23:54:52.541293 145816 kv/kvclient/kvcoord/txn_interceptor_committer.go:449  making txn commit explicit failed for "unnamed" meta={id=5753d66e pri=0.02696598 epo=0 ts=1609977292.525125880,0 min=1609977292.525125880,0 seq=6} lock=true stat=STAGING rts=1609977292.525125880,0 wto=false max=1609977292.525125880,0 int=1 ifw=5: result is ambiguous (error=failed to connect to n2 at 127.0.0.1:35663: context canceled [exhausted])
    backup_test.go:574: error scanning '&{<nil> 0xc0097b4200}': pq: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=4c24852f key=/Table/71/1 pri=0.01982031 epo=0 ts=1609977313.202792474,1 min=1609977313.196076964,0 seq=1000} lock=true stat=PENDING rts=1609977313.196076964,0 wto=false max=1609977313.696076964,0
    panic.go:636: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestBackupRestoreAppend765090177
--- FAIL: TestBackupRestoreAppend (65.04s)

More

Parameters:

GOFLAGS=-json

make stressrace TESTS=TestBackupRestoreAppend PKG=./pkg/ccl/backupccl TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-01-08T03:05:08Z

(ccl/backupccl).TestBackupRestoreAppend failed on master@339275585b7d30b9ee2d49b0c696b9ddb8d51ad4:

=== RUN   TestBackupRestoreAppend
    test_log_scope.go:72: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestBackupRestoreAppend757549060
    test_log_scope.go:73: use -show-logs to present logs inline
    backup_test.go:574: error scanning '&{<nil> 0xc00a4f8280}': pq: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=3961e045 key=/Table/71/1 pri=0.01623617 epo=0 ts=1610074020.805481381,1 min=1610074020.118737539,0 seq=1000} lock=true stat=PENDING rts=1610074020.118737539,0 wto=false max=1610074020.618737539,0
    panic.go:636: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestBackupRestoreAppend757549060
--- FAIL: TestBackupRestoreAppend (71.45s)

More

Parameters:

GOFLAGS=-json

make stressrace TESTS=TestBackupRestoreAppend PKG=./pkg/ccl/backupccl TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

Refs: cockroachdb#54599 Reason: flaky test Generated by bin/skip-test. Release justification: non-production code changes Release note: None

57805: changefeedccl: misc fixes for tenant compatibility r=dt a=dt see commits -- mostly just avoiding using NodeID, plus don't read meta2 directly. 58654: ccl/backupccl: skip TestBackupRestoreAppend r=adityamaru a=nkodali Refs: #54599 Reason: flaky test Generated by bin/skip-test. Release justification: non-production code changes Release note: None Co-authored-by: David Taylor <tinystatemachine@gmail.com> Co-authored-by: Namrata Kodali <namrata@cockroachlabs.com>

The test was rather hard to follow because of all the different flavours of backup we were trying to test. This change removes the deprecated `BACKUP TO` statements, and also removes the `BACKUP INTO subdir IN dir` testcase that is no longer a query we allow users to run. Fixes: #54599 Release note: None

cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Sep 19, 2020

cockroach-teamcity added this to the 20.2 milestone Sep 19, 2020

cockroach-teamcity assigned dt Sep 19, 2020

rytaft mentioned this issue Oct 22, 2020

ccl/backupccl: skip TestBackupRestoreAppend under race #55853

Merged

nkodali added a commit to nkodali/cockroach that referenced this issue Jan 8, 2021

ccl/backupccl: skip TestBackupRestoreAppend

2c450ee

Refs: cockroachdb#54599 Reason: flaky test Generated by bin/skip-test. Release justification: non-production code changes Release note: None

nkodali mentioned this issue Jan 8, 2021

ccl/backupccl: skip TestBackupRestoreAppend #58654

Merged

mwang1026 assigned adityamaru and unassigned dt Jan 12, 2021

exalate-issue-sync bot added the T-disaster-recovery label Sep 27, 2021

adityamaru added the skipped-test label Mar 14, 2022

adityamaru removed this from the 20.2 milestone Mar 14, 2022

craig bot closed this as completed in 3de2a52 Sep 9, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ccl/backupccl: TestBackupRestoreAppend failed #54599

ccl/backupccl: TestBackupRestoreAppend failed #54599

cockroach-teamcity commented Sep 19, 2020 •

edited by cockroach-jira-scripts

Loading

cockroach-teamcity commented Sep 21, 2020

andreimatei commented Sep 21, 2020

cockroach-teamcity commented Dec 17, 2020

knz commented Dec 22, 2020

cockroach-teamcity commented Dec 22, 2020

cockroach-teamcity commented Dec 24, 2020

cockroach-teamcity commented Dec 28, 2020

yuzefovich commented Dec 28, 2020

cockroach-teamcity commented Jan 7, 2021

cockroach-teamcity commented Jan 8, 2021

ccl/backupccl: TestBackupRestoreAppend failed #54599

ccl/backupccl: TestBackupRestoreAppend failed #54599

Comments

cockroach-teamcity commented Sep 19, 2020 • edited by cockroach-jira-scripts Loading

cockroach-teamcity commented Sep 21, 2020

andreimatei commented Sep 21, 2020

cockroach-teamcity commented Dec 17, 2020

knz commented Dec 22, 2020

cockroach-teamcity commented Dec 22, 2020

cockroach-teamcity commented Dec 24, 2020

cockroach-teamcity commented Dec 28, 2020

yuzefovich commented Dec 28, 2020

cockroach-teamcity commented Jan 7, 2021

cockroach-teamcity commented Jan 8, 2021

cockroach-teamcity commented Sep 19, 2020 •

edited by cockroach-jira-scripts

Loading