Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Release Blocker] valueLog.Read slice index crash #1389

Closed
jarifibrahim opened this issue Jun 30, 2020 · 1 comment
Closed

[Release Blocker] valueLog.Read slice index crash #1389

jarifibrahim opened this issue Jun 30, 2020 · 1 comment
Labels
area/crash This issue causes a panic or some other of exception that causes a crash. help wanted Please help us fix this! kind/bug Something is broken. priority/P1 Serious issue that requires eventual attention (can wait a bit) releaseblocker status/accepted We accept to investigate or work on it.

Comments

@jarifibrahim
Copy link
Contributor

The crash was first seen in cddf7c03451c . The line in question is

badger/value.go

Line 1518 in cddf7c0

return kv[h.klen : h.klen+h.vlen], cb, nil
.

Similar to the other index crash issues. This one is also not reproducible. The crash could be because the background compression/encryption corrupts the value pointer. The lower bound of [4294967295:7] is math.MaxUint32

panic: runtime error: slice bounds out of range [4294967295:7]

goroutine 90985 [running]:
github.com/dgraph-io/badger/v2.(*valueLog).Read(0xc0005549d0, 0xc7d00000000, 0x10000c0001b83b9, 0xc006e9c360, 0x10, 0xc006e959b0, 0x99e0de, 0xc08a0261a0, 0xc056f582a3, 0xc)
	/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-rc1.0.20200421062606-cddf7c03451c/value.go:1518 +0x604
github.com/dgraph-io/badger/v2.(*Item).yieldItemValue(0xc006ea0840, 0xd979c4, 0x1, 0xc05b94678d, 0xc006e95b00, 0x9598c8, 0x60)
	/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-rc1.0.20200421062606-cddf7c03451c/iterator.go:170 +0x11d
github.com/dgraph-io/badger/v2.(*Item).Value(0xc006ea0840, 0xc006e95bc0, 0x0, 0x0)
	/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-rc1.0.20200421062606-cddf7c03451c/iterator.go:105 +0xd6
github.com/dgraph-io/dgraph/posting.ReadPostingList(0xc056f582a0, 0x18, 0x18, 0xc006e98820, 0x0, 0x0, 0x0)
	/ext-go/1/src/github.com/dgraph-io/dgraph/posting/mvcc.go:301 +0x374
github.com/dgraph-io/dgraph/posting.getNew(0xc056f582a0, 0x18, 0x18, 0xc000554800, 0xffffffffffffffff, 0x0, 0x0, 0x0)
	/ext-go/1/src/github.com/dgraph-io/dgraph/posting/mvcc.go:347 +0x1dc
github.com/dgraph-io/dgraph/posting.(*LocalCache).getInternal(0x0, 0xc056f582a0, 0x18, 0x18, 0x1, 0x18, 0x0, 0x1)
	/ext-go/1/src/github.com/dgraph-io/dgraph/posting/lists.go:209 +0x342
github.com/dgraph-io/dgraph/posting.(*LocalCache).Get(...)
	/ext-go/1/src/github.com/dgraph-io/dgraph/posting/lists.go:242
github.com/dgraph-io/dgraph/worker.(*queryState).handleValuePostings.func1(0x0, 0x1d, 0xc042934680, 0x98b566)
	/ext-go/1/src/github.com/dgraph-io/dgraph/worker/task.go:385 +0x1fb
github.com/dgraph-io/dgraph/worker.(*queryState).handleValuePostings.func2(0xc006ea2000, 0xc006e9a100, 0x0, 0x1d)
	/ext-go/1/src/github.com/dgraph-io/dgraph/worker/task.go:494 +0x3a
created by github.com/dgraph-io/dgraph/worker.(*queryState).handleValuePostings
	/ext-go/1/src/github.com/dgraph-io/dgraph/worker/task.go:493 +0x3ee
@jarifibrahim jarifibrahim added kind/bug Something is broken. priority/P1 Serious issue that requires eventual attention (can wait a bit) area/crash This issue causes a panic or some other of exception that causes a crash. help wanted Please help us fix this! labels Jun 30, 2020
This was referenced Jun 30, 2020
Merged
Closed
@lgalatin lgalatin added the status/accepted We accept to investigate or work on it. label Jul 1, 2020
This was referenced Jul 10, 2020
Closed
Merged
Merged
jarifibrahim pushed a commit that referenced this issue Jul 10, 2020
Revert "add assert to check integer overflow for table size (#1402)" (#…
e0d058c 
…1406)

This reverts commit 7f4e4b5. 
This revert is necessary because b.sz won't exist after builder changes are
reverted and the assert is no longer valid.

The commit 7f4e4b5 is being reverted because we have seen some crashes
which could be caused by these changes. We haven't been able to reproduce the
crashes yet.

Related to #1389, #1388, #1387
Also see https://discuss.dgraph.io/t/current-state-of-badger-crashes/7602
jarifibrahim pushed a commit that referenced this issue Jul 10, 2020
Revert "fix: Fix race condition in block.incRef (#1337)" (#1407)
63d9309 
This reverts commit 21735af.

The commit is reverted because we have seen some crashes which
could be caused by this change. We haven't been able to reproduce
the crashes yet.

Related to #1389, #1388, #1387
Also, see https://discuss.dgraph.io/t/current-state-of-badger-crashes/7602
@jarifibrahim jarifibrahim mentioned this issue Jul 10, 2020
Merged
jarifibrahim pushed a commit that referenced this issue Jul 10, 2020
Revert "Buffer pool for decompression (#1308)" (#1408)
800305e 
This reverts commit aadda9a.

This commit is being reverted because we have seen some crashes
which could be caused by it. We haven't been able to reproduce the crashes yet.

Related to #1389, #1388, #1387
Also, see https://discuss.dgraph.io/t/current-state-of-badger-crashes/7602
@jarifibrahim jarifibrahim mentioned this issue Jul 10, 2020
Merged
jarifibrahim pushed a commit that referenced this issue Jul 10, 2020
Revert "Compress/Encrypt Blocks in the background (#1227)" (#1409)
6001230 
This reverts commit b13b927.
This commit is being reverted because we have seen some crashes which
could be caused by it. We haven't been able to reproduce the crashes yet.

Related to #1389, #1388, #1387
Also, see https://discuss.dgraph.io/t/current-state-of-badger-crashes/7602

This commit had some conflicts. See PR description for details.
@jarifibrahim
Copy link
Contributor Author

I wasn't able to reproduce this issue after multiple attempts. I'm closing this as non-reproducible.

jarifibrahim pushed a commit that referenced this issue Oct 2, 2020
Revert "add assert to check integer overflow for table size (#1402)" (#…
5eb5c41 
…1406)

This reverts commit 7f4e4b5. 
This revert is necessary because b.sz won't exist after builder changes are
reverted and the assert is no longer valid.

The commit 7f4e4b5 is being reverted because we have seen some crashes
which could be caused by these changes. We haven't been able to reproduce the
crashes yet.

Related to #1389, #1388, #1387
Also see https://discuss.dgraph.io/t/current-state-of-badger-crashes/7602
jarifibrahim pushed a commit that referenced this issue Oct 2, 2020
Revert "fix: Fix race condition in block.incRef (#1337)" (#1407)
268db59 
This reverts commit 21735af.

The commit is reverted because we have seen some crashes which
could be caused by this change. We haven't been able to reproduce
the crashes yet.

Related to #1389, #1388, #1387
Also, see https://discuss.dgraph.io/t/current-state-of-badger-crashes/7602
jarifibrahim pushed a commit that referenced this issue Oct 2, 2020
Revert "Buffer pool for decompression (#1308)" (#1408)
0cff38a 
This reverts commit aadda9a.

This commit is being reverted because we have seen some crashes
which could be caused by it. We haven't been able to reproduce the crashes yet.

Related to #1389, #1388, #1387
Also, see https://discuss.dgraph.io/t/current-state-of-badger-crashes/7602
jarifibrahim pushed a commit that referenced this issue Oct 2, 2020
Revert "Compress/Encrypt Blocks in the background (#1227)" (#1409)
f198dc7 
This reverts commit b13b927.
This commit is being reverted because we have seen some crashes which
could be caused by it. We haven't been able to reproduce the crashes yet.

Related to #1389, #1388, #1387
Also, see https://discuss.dgraph.io/t/current-state-of-badger-crashes/7602

This commit had some conflicts. See PR description for details.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/crash This issue causes a panic or some other of exception that causes a crash. help wanted Please help us fix this! kind/bug Something is broken. priority/P1 Serious issue that requires eventual attention (can wait a bit) releaseblocker status/accepted We accept to investigate or work on it.
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jarifibrahim @lgalatin