Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

github.com/cockroachdb/pebble/internal/metamorphic: TestMetaTwoInstance failed #3894

Closed
cockroach-teamcity opened this issue Aug 28, 2024 · 5 comments · Fixed by #3895
Closed

github.com/cockroachdb/pebble/internal/metamorphic: TestMetaTwoInstance failed #3894

cockroach-teamcity opened this issue Aug 28, 2024 · 5 comments · Fixed by #3895
Assignees
@RaduBerinde
Labels
A-storage branch-refs/heads/master C-test-failure metamorphic-failure O-robot release-blocker T-storage

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Aug 28, 2024 •
edited by cockroach-jira-scripts
Loading

github.com/cockroachdb/pebble/internal/metamorphic.TestMetaTwoInstance failed with artifacts on refs/heads/master @ 34e929e1fcac:

        iter37.Next("") // [false] <nil> #3850
        iter36.SeekPrefixGE("yvodk@7") // [false] pebble: SeekPrefixGE supplied with key outside of upper bound #3851
        iter38 = iter37.Clone(false, "hmwupnwtnwoj", "jlnvguh@11", 1 /* key types */, 0, 0, false /* use L6 filters */, "" /* masking suffix */) // <nil> #3852
        iter38.Next("") // [false] <nil> #3853
        db1.Get("ltaktczhi@9") // [""] pebble: not found #3854
        iter36.SeekPrefixGE("avncedojxmw@4") // [true,"avncedojxmw@4",<no point>,["avncedojxmw","avncedojxmw\x00")=>{"@4"="gmdsvaiuetjto"}*] <nil> #3855
        iter36.SeekGE("nggfnk", "") // [false] <nil> #3856
        iter37.Prev("") // [false] <nil> #3857
        iter37.Next("") // [false] <nil> #3858
        iter38.SeekPrefixGE("mrfd") // [false] pebble: SeekPrefixGE supplied with key outside of upper bound #3859
        db2.Set("nksptayrqmoi@11", "ehpuiwcrixfdq") // <nil> #3860
        iter38.First() // [false] <nil> #3861
        iter39 = iter36.Clone(false, "ahwuqfle@4", "dafszn", 2 /* key types */, 0, 0, false /* use L6 filters */, "@4" /* masking suffix */) // <nil> #3862
        iter39.SeekLT("lfyo", "") // [true,"cpql",<no point>,["cpql","dafszn")=>{"@4"="gmdsvaiuetjto"}*] <nil> #3863
        iter36.SeekPrefixGE("mdshz@2") // [false] pebble: SeekPrefixGE supplied with key outside of upper bound #3864
        iter39.SetBounds("ahwuqfle@4", "ahwuqfle@4") // <nil> #3865
        iter39.SeekLT("ahwuqfle@4", "") // [false] <nil> #3866
        iter39.Next("") // [false] <nil> #3867
        iter39.Prev("") // [false] <nil> #3868
        iter38.SeekGE("jzsyxhyha@9", "") // [false] <nil> #3869
        db2.Set("snkeitowwfpx", "ebbhek") // <nil> #3870
        iter36.SetBounds("ahwuqfle@4", "ahwuqfle@4") // <nil> #3871
        iter36.SeekLT("ahwuqfle@4", "") // [false] <nil> #3872
        iter36.Next("") // [false] <nil> #3873
        iter36.Prev("") // [false] <nil> #3874
        iter39.NextPrefix() // [false] NextPrefix not permitted with upper bound ahwuqfle@4 #3875
        iter37.Last() // [false] <nil> #3876
        db2.Get("rizbl@2") // [""] pebble: not found #3877
        iter40 = db1.NewIter("", "", 0 /* key types */, 0, 0, false /* use L6 filters */, "" /* masking suffix */) // <nil> #3878
        iter38.First() // [false] <nil> #3879
        db2.Get("coynhjqxt@4") // [""] pebble: not found #3880
        iter39.Last() // [false] <nil> #3881
        db1.DeleteRange("dyjwsimbo@11", "jktikyrig@11") // <nil> #3882
        iter37.Last() // [false] <nil> #3883
        iter40.Next("") // [false] <nil> #3884
        iter38.NextPrefix() // [false] NextPrefix not permitted with upper bound jlnvguh@11 #3885
        iter38.SeekPrefixGE("atajgsuke@10") // [false] pebble: SeekPrefixGE supplied with key outside of lower bound #3886
        iter39.Prev("") // [false] <nil> #3887
        iter39.SeekLT("rhalnitnuprh@8", "") // [false] <nil> #3888
        iter39.SeekPrefixGE("lkylmmg@9") // [false] pebble: SeekPrefixGE supplied with key outside of upper bound #3889
        iter39.Prev("") // [false] pebble: SeekPrefixGE supplied with key outside of upper bound #3890
        iter37.Next("") // [false] <nil> #3891
        iter39.SeekPrefixGE("uggkqhokp@3") // [false] pebble: SeekPrefixGE supplied with key outside of upper bound #3892
        // 04:36:22.208 INFO: [JOB 79] ingesting: sstable created 000071
        // 04:36:22.208 INFO: [JOB 80] WAL created 000072
        // 04:36:22.208 INFO: [JOB 81] flushing 1 memtable (4.3KB) to L0
        // 04:36:22.208 INFO: [JOB 81] flushing: sstable created 000073
        
        To reduce:  go test ./internal/metamorphic -tags invariants -run 'TestMetaTwoInstance$' --run-dir /artifacts/meta/240828-043620.1301987344096/random-003 --try-to-reduce -v
        --- FAIL: TestMetaTwoInstance/execution/random-003 (0.41s)
Help

To reproduce, try: 

go test -tags 'invariants' -exec 'stress -p 1' -timeout 0 -test.v -run 'TestMetaTwoInstance$' ./internal/metamorphic -seed 1724819780130833882 -ops "uniform:5000-10000"

This test on roachdash | Improve this report!

Jira issue: PEBBLE-252
@itsbilal
Copy link
Contributor 

[04:36:25] :	 [Step 1/1]         ===== SEED =====
[04:36:25] :	 [Step 1/1]         1724819780130833882
[04:36:25] :	 [Step 1/1]         ===== ERR =====
[04:36:25] :	 [Step 1/1]         exit status 2
[04:36:25] :	 [Step 1/1]         ===== OUT =====
[04:36:25] :	 [Step 1/1]         unexpected fault address 0x0
[04:36:25] :	 [Step 1/1]         fatal error: fault
[04:36:25] :	 [Step 1/1]         [signal SIGSEGV: segmentation violation code=0x80 addr=0x0 pc=0x417278]
[04:36:25] :	 [Step 1/1]         
[04:36:25] :	 [Step 1/1]         goroutine 447 gp=0xc0002116c0 m=10 mp=0xc000101008 [running]:
[04:36:25] :	 [Step 1/1]         runtime.throw({0xe88e43?, 0xe03320?})
[04:36:25] :	 [Step 1/1]         	GOROOT/src/runtime/panic.go:1023 +0x5c fp=0xc00022c8b0 sp=0xc00022c880 pc=0x44089c
[04:36:25] :	 [Step 1/1]         runtime.sigpanic()
[04:36:25] :	 [Step 1/1]         	GOROOT/src/runtime/signal_unix.go:895 +0x285 fp=0xc00022c910 sp=0xc00022c8b0 pc=0x459665
[04:36:25] :	 [Step 1/1]         runtime.moveToBmap(0xd94700, 0xc000729710, 0xc000044740?, 0x0?, 0x104b8b4830245c89)
[04:36:25] :	 [Step 1/1]         	GOROOT/src/runtime/map.go:1456 +0x38 fp=0xc00022c988 sp=0xc00022c910 pc=0x417278
[04:36:25] :	 [Step 1/1]         runtime.mapclone2(0xd94700, 0xc00024ddd0)
[04:36:25] :	 [Step 1/1]         	GOROOT/src/runtime/map.go:1538 +0x4af fp=0xc00022ca28 sp=0xc00022c988 pc=0x417a4f
[04:36:25] :	 [Step 1/1]         maps.clone({0xd94700, 0xc00024ddd0})
[04:36:25] :	 [Step 1/1]         	GOROOT/src/runtime/map.go:1448 +0x25 fp=0xc00022ca48 sp=0xc00022ca28 pc=0x474fe5
[04:36:25] :	 [Step 1/1]         maps.Clone[...](...)
[04:36:25] :	 [Step 1/1]         	GOROOT/src/maps/maps.go:46
[04:36:25] :	 [Step 1/1]         github.com/cockroachdb/pebble/vfs.cloneChildren(0xae90b2?)
[04:36:25] :	 [Step 1/1]         	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/mem_fs.go:641 +0x3b fp=0xc00022ca70 sp=0xc00022ca48 pc=0x7ed2db
[04:36:25] :	 [Step 1/1]         github.com/cockroachdb/pebble/vfs.(*memFile).Sync(0xc000278690)
[04:36:25] :	 [Step 1/1]         	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/mem_fs.go:853 +0x210 fp=0xc00022cb08 sp=0xc00022ca70 pc=0x7ef450
[04:36:25] :	 [Step 1/1]         github.com/cockroachdb/pebble/vfs/errorfs.(*errorFile).Sync(0xc0002786c0)
[04:36:25] :	 [Step 1/1]         	github.com/cockroachdb/pebble/vfs/errorfs/external/com_github_cockroachdb_pebble/vfs/errorfs/errorfs.go:533 +0x50 fp=0xc00022cb40 sp=0xc00022cb08 pc=0xae5750
[04:36:25] :	 [Step 1/1]         github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingDir).Sync.func1()
[04:36:25] :	 [Step 1/1]         	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:444 +0x28 fp=0xc00022cb60 sp=0xc00022cb40 pc=0x7e6bc8
[04:36:25] :	 [Step 1/1]         github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).timeFilesystemOp(0x0?, {0xc000044740?, 0x0?}, 0x0?, 0xc00022cbe8, 0x9fa865?)
[04:36:25] :	 [Step 1/1]         	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:653 +0x43 fp=0xc00022cba8 sp=0xc00022cb60 pc=0x7e71a3
[04:36:25] :	 [Step 1/1]         github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingDir).Sync(0xc0002786f0)
[04:36:25] :	 [Step 1/1]         	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:443 +0xb4 fp=0xc00022cc10 sp=0xc00022cba8 pc=0x7e6b74
[04:36:25] :	 [Step 1/1]         github.com/cockroachdb/pebble/objstorage/objstorageprovider.(*provider).vfsSync(0xc0002b4340)
[04:36:25] :	 [Step 1/1]         	github.com/cockroachdb/pebble/objstorage/objstorageprovider/external/com_github_cockroachdb_pebble/objstorage/objstorageprovider/vfs.go:96 +0x7e fp=0xc00022cc40 sp=0xc00022cc10 pc=0x9366de
[04:36:25] :	 [Step 1/1]         github.com/cockroachdb/pebble/objstorage/objstorageprovider.(*provider).Sync(0xc0002b4340)
[04:36:25] :	 [Step 1/1]         	github.com/cockroachdb/pebble/objstorage/objstorageprovider/external/com_github_cockroachdb_pebble/objstorage/objstorageprovider/provider.go:403 +0x18 fp=0xc00022cc58 sp=0xc00022cc40 pc=0x92d758
[04:36:25] :	 [Step 1/1]         github.com/cockroachdb/pebble.(*DB).compactAndWrite(0xc000150488, 0x51, 0xc0006cbb88, {0x0, 0x0, 0x0}, 0x6)
[04:36:25] :	 [Step 1/1]         	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/compaction.go:2550 +0xe62 fp=0xc00022d698 sp=0xc00022cc58 pc=0xa2c522
[04:36:25] :	 [Step 1/1]         github.com/cockroachdb/pebble.(*DB).runCompaction(0xc000150488, 0x51, 0xc0006cbb88)
[04:36:25] :	 [Step 1/1]         	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/compaction.go:2458 +0x34d fp=0xc00022d820 sp=0xc00022d698 pc=0xa2b22d
[04:36:25] :	 [Step 1/1]         github.com/cockroachdb/pebble.(*DB).flush1(0xc000150488)
[04:36:25] :	 [Step 1/1]         	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/compaction.go:1428 +0x3b4 fp=0xc00022de68 sp=0xc00022d820 pc=0xa23354
[04:36:25] :	 [Step 1/1]         github.com/cockroachdb/pebble.(*DB).flush.func1({0xfb8b40?, 0xc0007293e0?})
[04:36:25] :	 [Step 1/1]         	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/compaction.go:1163 +0xee fp=0xc00022df08 sp=0xc00022de68 pc=0xa21bae
[04:36:25] :	 [Step 1/1]         runtime/pprof.Do({0xfb8a28?, 0x15bc680?}, {{0xc00014cf20?, 0x0?, 0xc0008a6240?}}, 0xc000284fa8)
[04:36:25] :	 [Step 1/1]         	GOROOT/src/runtime/pprof/runtime.go:51 +0x9d fp=0xc00022df78 sp=0xc00022df08 pc=0x5c4d5d
[04:36:25] :	 [Step 1/1]         github.com/cockroachdb/pebble.(*DB).flush(0xc000284fb8?)
[04:36:25] :	 [Step 1/1]         	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/compaction.go:1156 +0x52 fp=0xc00022dfc8 sp=0xc00022df78 pc=0xa21a92
[04:36:25] :	 [Step 1/1]         github.com/cockroachdb/pebble.(*DB).maybeScheduleFlush.gowrap1()
[04:36:25] :	 [Step 1/1]         	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/compaction.go:1072 +0x25 fp=0xc00022dfe0 sp=0xc00022dfc8 pc=0xa21225
[04:36:25] :	 [Step 1/1]         runtime.goexit({})
[04:36:25] :	 [Step 1/1]         	src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00022dfe8 sp=0xc00022dfe0 pc=0x47b801
[04:36:25] :	 [Step 1/1]         created by github.com/cockroachdb/pebble.(*DB).maybeScheduleFlush in goroutine 36
[04:36:25] :	 [Step 1/1]         	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/compaction.go:1072 +0xdb
[04:36:25] :	 [Step 1/1]         
[04:36:25] :	 [Step 1/1]         goroutine 1 gp=0xc0000061c0 m=nil [chan receive]:
[04:36:25] :	 [Step 1/1]         runtime.gopark(0xc0002900d0?, 0xc000290158?, 0xd0?, 0x0?, 0xdb54e0?)
[04:36:25] :	 [Step 1/1]         	GOROOT/src/runtime/proc.go:402 +0xce fp=0xc0001ad890 sp=0xc0001ad870 pc=0x44384e
[04:36:25] :	 [Step 1/1]         runtime.chanrecv(0xc0002b2070, 0xc000055977, 0x1)
[04:36:25] :	 [Step 1/1]         	GOROOT/src/runtime/chan.go:583 +0x3bf fp=0xc0001ad908 sp=0xc0001ad890 pc=0x40c75f
[04:36:25] :	 [Step 1/1]         runtime.chanrecv1(0x1559c00?, 0xd59ec0?)
[04:36:25] :	 [Step 1/1]         	GOROOT/src/runtime/chan.go:442 +0x12 fp=0xc0001ad930 sp=0xc0001ad908 pc=0x40c372
[04:36:25] :	 [Step 1/1]         testing.(*T).Run(0xc0002b4000, {0xe916a6?, 0x0?}, 0xeda6d0)
[04:36:25] :	 [Step 1/1]         	GOROOT/src/testing/testing.go:1750 +0x3ab fp=0xc0001ad9f0 sp=0xc0001ad930 pc=0x5f33eb
[04:36:25] :	 [Step 1/1]         testing.runTests.func1(0xc0002b4000)
[04:36:25] :	 [Step 1/1]         	GOROOT/src/testing/testing.go:2161 +0x37 fp=0xc0001ada30 sp=0xc0001ad9f0 pc=0x5f5537
[04:36:25] :	 [Step 1/1]         testing.tRunner(0xc0002b4000, 0xc000055b40)
[04:36:25] :	 [Step 1/1]         	GOROOT/src/testing/testing.go:1689 +0xfb fp=0xc0001ada80 sp=0xc0001ada30 pc=0x5f251b
[04:36:25] :	 [Step 1/1]         testing.runTests(0xc000294000, {0x14fa7c0, 0x2, 0x2}, {0x7f086c249938?, 0x10?, 0x0?})
[04:36:25] :	 [Step 1/1]         	GOROOT/src/testing/testing.go:2159 +0x445 fp=0xc0001adb70 sp=0xc0001ada80 pc=0x5f5425
[04:36:25] :	 [Step 1/1]         testing.(*M).Run(0xc0002037c0)
[04:36:25] :	 [Step 1/1]         	GOROOT/src/testing/testing.go:2027 +0x68b fp=0xc0001adda0 sp=0xc0001adb70 pc=0x5f3e2b
[04:36:25] :	 [Step 1/1]         main.main()
[04:36:25] :	 [Step 1/1]         	main/bazel-out/k8-fastbuild/bin/external/com_github_cockroachdb_pebble/internal/metamorphic/metamorphic_test_/testmain.go:130 +0x4d7 fp=0xc0001adf50 sp=0xc0001adda0 pc=0xb4f937
[04:36:25] :	 [Step 1/1]         runtime.main()
[04:36:25] :	 [Step 1/1]         	GOROOT/src/runtime/proc.go:271 +0x29d fp=0xc0001adfe0 sp=0xc0001adf50 pc=0x4433fd
[04:36:25] :	 [Step 1/1]         runtime.goexit({})
[04:36:25] :	 [Step 1/1]         	src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0001adfe8 sp=0xc0001adfe0 pc=0x47b801
[04:36:25] :	 [Step 1/1]

@RaduBerinde probably a fallout of #3888 ?

@RaduBerinde RaduBerinde self-assigned this Aug 28, 2024
@RaduBerinde
Copy link
Member

Hm.. looks like it could be golang/go#62203. I'll investigate in which Go versions that was fixed.

@RaduBerinde
Copy link
Member

No, that fix is part of all 1.22 releases. I'll see if I can reproduce.

@RaduBerinde
Copy link
Member

I can't figure out how this is even possible:

			for srcBmap != nil {
				dstBmap, pos = moveToBmap(t, dst, dstBmap, pos, srcBmap)

func moveToBmap(t *maptype, h *hmap, dst *bmap, pos int, src *bmap) (*bmap, int) {
	for i := 0; i < bucketCnt; i++ {
		if isEmpty(src.tophash[i]) {  <- null pointer here

// A bucket for a Go map.
type bmap struct {
	// tophash generally contains the top byte of the hash value
	// for each key in this bucket. If tophash[0] < minTopHash,
	// tophash[0] is a bucket evacuation state instead.
	tophash [bucketCnt]uint8

We check srcBmap != nil right before calling..

RaduBerinde added a commit to RaduBerinde/pebble that referenced this issue Aug 28, 2024
@RaduBerinde
vfs: work around a maps.Clone bug
b2c26ff 
Stop using `maps.Clone` because of golang/go#69110

Fixes cockroachdb#3894
RaduBerinde added a commit to RaduBerinde/pebble that referenced this issue Aug 28, 2024
@RaduBerinde
vfs: workaround for a maps.Clone bug
b96e157 
Stop using `maps.Clone` because of golang/go#69110

Fixes cockroachdb#3894
@RaduBerinde RaduBerinde mentioned this issue Aug 28, 2024
Merged
@RaduBerinde
Copy link
Member

Managed to reproduce the Go issue: golang/go#69110

RaduBerinde added a commit that referenced this issue Aug 28, 2024
@RaduBerinde
vfs: workaround for a maps.Clone bug
95565b7 
Stop using `maps.Clone` because of golang/go#69110

Fixes #3894
@github-project-automation github-project-automation bot moved this from Incoming to Done in [Deprecated] Storage Aug 28, 2024
@RaduBerinde RaduBerinde mentioned this issue Aug 28, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@RaduBerinde RaduBerinde

Labels
A-storage branch-refs/heads/master C-test-failure metamorphic-failure O-robot release-blocker T-storage
Projects
No open projects
[Deprecated] Storage
Archived in project
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

vfs: workaround for a maps.Clone bug RaduBerinde/pebble
4 participants
@itsbilal @RaduBerinde @cockroach-teamcity and others