Skip to content
This repository has been archived by the owner on Aug 2, 2021. It is now read-only.

Data race in swarm/storage TestGetAllReferences #1206

Closed
frncmx opened this issue Feb 8, 2019 · 2 comments · Fixed by ethereum/go-ethereum#19028
Closed

Data race in swarm/storage TestGetAllReferences #1206

frncmx opened this issue Feb 8, 2019 · 2 comments · Fixed by ethereum/go-ethereum#19028
Assignees
@holisticode
Labels
bug test

Comments

@frncmx
Copy link
Contributor

frncmx commented Feb 8, 2019

How to reproduce?

go test -race -run TestGetAllReferences github.com/ethereum/go-ethereum/swarm/storage

introduced in: 3eff652

Acceptance criteria

The above go test shall pass.

Stack trace

=== RUN   TestFileStoreCapacity
--- PASS: TestFileStoreCapacity (0.11s)
=== RUN   TestGetAllReferences
==================
WARNING: DATA RACE
Read at 0x00c000bfea28 by goroutine 199:
  github.com/ethereum/go-ethereum/swarm/storage.(*HashExplorer).Put()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/swarm/storage/filestore.go:136 +0xf8
  github.com/ethereum/go-ethereum/swarm/storage.(*PyramidChunker).processChunk()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/swarm/storage/pyramid.go:283 +0xe1
  github.com/ethereum/go-ethereum/swarm/storage.(*PyramidChunker).processor()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/swarm/storage/pyramid.go:275 +0xab

Previous write at 0x00c000bfea28 by goroutine 324:
  github.com/ethereum/go-ethereum/swarm/storage.(*HashExplorer).Put()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/swarm/storage/filestore.go:136 +0x1b0
  github.com/ethereum/go-ethereum/swarm/storage.(*PyramidChunker).processChunk()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/swarm/storage/pyramid.go:283 +0xe1
  github.com/ethereum/go-ethereum/swarm/storage.(*PyramidChunker).processor()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/swarm/storage/pyramid.go:275 +0xab

Goroutine 199 (running) created at:
  github.com/ethereum/go-ethereum/swarm/storage.(*PyramidChunker).prepareChunks()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/swarm/storage/pyramid.go:521 +0x97c
  github.com/ethereum/go-ethereum/swarm/storage.(*PyramidChunker).Split()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/swarm/storage/pyramid.go:205 +0xd5
  github.com/ethereum/go-ethereum/swarm/storage.PyramidSplit()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/swarm/storage/pyramid.go:100 +0xe1
  github.com/ethereum/go-ethereum/swarm/storage.(*FileStore).GetAllReferences()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/swarm/storage/filestore.go:109 +0x1f6
  github.com/ethereum/go-ethereum/swarm/storage.TestGetAllReferences()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/swarm/storage/filestore_test.go:199 +0x421
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x279
  fmt.Sscanf()
      /home/travis/.gimme/versions/go1.11.5.linux.amd64/src/fmt/scan.go:114 +0xd0
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:643 +0x144
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x279
  fmt.Sscanf()
      /home/travis/.gimme/versions/go1.11.5.linux.amd64/src/fmt/scan.go:114 +0xd0
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:643 +0x144
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x279
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).checkAndCleanFiles()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_util.go:52 +0x2e4
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.openDB()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:130 +0xa00
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x279
  fmt.Sscanf()
      /home/travis/.gimme/versions/go1.11.5.linux.amd64/src/fmt/scan.go:114 +0xd0
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:657 +0x1ed
  fmt.Fscanf()
      /home/travis/.gimme/versions/go1.11.5.linux.amd64/src/fmt/scan.go:143 +0xe9
  fmt.Sscanf()
      /home/travis/.gimme/versions/go1.11.5.linux.amd64/src/fmt/scan.go:114 +0xd0
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:643 +0x144
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x279
  fmt.Sscanf()
      /home/travis/.gimme/versions/go1.11.5.linux.amd64/src/fmt/scan.go:114 +0xd0
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:643 +0x144
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x279
  fmt.Sscanf()
      /home/travis/.gimme/versions/go1.11.5.linux.amd64/src/fmt/scan.go:114 +0xd0
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:643 +0x144
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x279
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).recoverJournal()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:476 +0xe3
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.openDB()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:125 +0x9dc
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.Open()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:197 +0x1ed
  fmt.Sscanf()
      /home/travis/.gimme/versions/go1.11.5.linux.amd64/src/fmt/scan.go:114 +0xd0
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:643 +0x144
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x279
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*session).recover.func1()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/session.go:113 +0xbb
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*session).recover()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/session.go:121 +0x17b7
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.Open()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:183 +0xd7
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.OpenFile()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:219 +0xa5
  github.com/ethereum/go-ethereum/swarm/storage.NewLDBDatabase()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/swarm/storage/database.go:37 +0xc3
  github.com/ethereum/go-ethereum/swarm/storage.NewLDBStore()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/swarm/storage/ldbstore.go:152 +0x37c
  github.com/ethereum/go-ethereum/swarm/storage.newTestDbStore()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/swarm/storage/ldbstore_test.go:60 +0x6dc
  github.com/ethereum/go-ethereum/swarm/storage.TestGetAllReferences()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/swarm/storage/filestore_test.go:180 +0x4d
  testing.tRunner()
      /home/travis/.gimme/versions/go1.11.5.linux.amd64/src/testing/testing.go:827 +0x162

Goroutine 324 (running) created at:
  github.com/ethereum/go-ethereum/swarm/storage.(*PyramidChunker).prepareChunks()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/swarm/storage/pyramid.go:389 +0x14c
  github.com/ethereum/go-ethereum/swarm/storage.(*PyramidChunker).Split()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/swarm/storage/pyramid.go:205 +0xd5
  github.com/ethereum/go-ethereum/swarm/storage.PyramidSplit()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/swarm/storage/pyramid.go:100 +0xe1
  github.com/ethereum/go-ethereum/swarm/storage.(*FileStore).GetAllReferences()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/swarm/storage/filestore.go:109 +0x1f6
  github.com/ethereum/go-ethereum/swarm/storage.TestGetAllReferences()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/swarm/storage/filestore_test.go:199 +0x421
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x279
  fmt.Sscanf()
      /home/travis/.gimme/versions/go1.11.5.linux.amd64/src/fmt/scan.go:114 +0xd0
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:643 +0x144
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x279
  fmt.Sscanf()
      /home/travis/.gimme/versions/go1.11.5.linux.amd64/src/fmt/scan.go:114 +0xd0
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:643 +0x144
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x279
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).checkAndCleanFiles()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_util.go:52 +0x2e4
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.openDB()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:130 +0xa00
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x279
  fmt.Sscanf()
      /home/travis/.gimme/versions/go1.11.5.linux.amd64/src/fmt/scan.go:114 +0xd0
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:657 +0x1ed
  fmt.Fscanf()
      /home/travis/.gimme/versions/go1.11.5.linux.amd64/src/fmt/scan.go:143 +0xe9
  fmt.Sscanf()
      /home/travis/.gimme/versions/go1.11.5.linux.amd64/src/fmt/scan.go:114 +0xd0
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:643 +0x144
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x279
  fmt.Sscanf()
      /home/travis/.gimme/versions/go1.11.5.linux.amd64/src/fmt/scan.go:114 +0xd0
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:643 +0x144
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x279
  fmt.Sscanf()
      /home/travis/.gimme/versions/go1.11.5.linux.amd64/src/fmt/scan.go:114 +0xd0
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:643 +0x144
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x279
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).recoverJournal()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:476 +0xe3
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.openDB()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:125 +0x9dc
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.Open()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:197 +0x1ed
  fmt.Sscanf()
      /home/travis/.gimme/versions/go1.11.5.linux.amd64/src/fmt/scan.go:114 +0xd0
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:643 +0x144
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x279
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*session).recover.func1()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/session.go:113 +0xbb
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*session).recover()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/session.go:121 +0x17b7
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.Open()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:183 +0xd7
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.OpenFile()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:219 +0xa5
  github.com/ethereum/go-ethereum/swarm/storage.NewLDBDatabase()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/swarm/storage/database.go:37 +0xc3
  github.com/ethereum/go-ethereum/swarm/storage.NewLDBStore()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/swarm/storage/ldbstore.go:152 +0x37c
  github.com/ethereum/go-ethereum/swarm/storage.newTestDbStore()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/swarm/storage/ldbstore_test.go:60 +0x6dc
  github.com/ethereum/go-ethereum/swarm/storage.TestGetAllReferences()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/swarm/storage/filestore_test.go:180 +0x4d
  testing.tRunner()
      /home/travis/.gimme/versions/go1.11.5.linux.amd64/src/testing/testing.go:827 +0x162
==================
==================
WARNING: DATA RACE
Read at 0x00c00013e7e0 by goroutine 199:
  runtime.growslice()
      /home/travis/.gimme/versions/go1.11.5.linux.amd64/src/runtime/slice.go:98 +0x0
  github.com/ethereum/go-ethereum/swarm/storage.(*HashExplorer).Put()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/swarm/storage/filestore.go:136 +0x285
  github.com/ethereum/go-ethereum/swarm/storage.(*PyramidChunker).processChunk()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/swarm/storage/pyramid.go:283 +0xe1
  github.com/ethereum/go-ethereum/swarm/storage.(*PyramidChunker).processor()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/swarm/storage/pyramid.go:275 +0xab

Previous write at 0x00c00013e7e0 by goroutine 324:
  github.com/ethereum/go-ethereum/swarm/storage.(*HashExplorer).Put()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/swarm/storage/filestore.go:136 +0x165
  github.com/ethereum/go-ethereum/swarm/storage.(*PyramidChunker).processChunk()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/swarm/storage/pyramid.go:283 +0xe1
  github.com/ethereum/go-ethereum/swarm/storage.(*PyramidChunker).processor()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/swarm/storage/pyramid.go:275 +0xab

Goroutine 199 (running) created at:
  github.com/ethereum/go-ethereum/swarm/storage.(*PyramidChunker).prepareChunks()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/swarm/storage/pyramid.go:521 +0x97c
  github.com/ethereum/go-ethereum/swarm/storage.(*PyramidChunker).Split()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/swarm/storage/pyramid.go:205 +0xd5
  github.com/ethereum/go-ethereum/swarm/storage.PyramidSplit()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/swarm/storage/pyramid.go:100 +0xe1
  github.com/ethereum/go-ethereum/swarm/storage.(*FileStore).GetAllReferences()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/swarm/storage/filestore.go:109 +0x1f6
  github.com/ethereum/go-ethereum/swarm/storage.TestGetAllReferences()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/swarm/storage/filestore_test.go:199 +0x421
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x279
  fmt.Sscanf()
      /home/travis/.gimme/versions/go1.11.5.linux.amd64/src/fmt/scan.go:114 +0xd0
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:643 +0x144
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x279
  fmt.Sscanf()
      /home/travis/.gimme/versions/go1.11.5.linux.amd64/src/fmt/scan.go:114 +0xd0
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:643 +0x144
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x279
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).checkAndCleanFiles()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_util.go:52 +0x2e4
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.openDB()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:130 +0xa00
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x279
  fmt.Sscanf()
      /home/travis/.gimme/versions/go1.11.5.linux.amd64/src/fmt/scan.go:114 +0xd0
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:657 +0x1ed
  fmt.Fscanf()
      /home/travis/.gimme/versions/go1.11.5.linux.amd64/src/fmt/scan.go:143 +0xe9
  fmt.Sscanf()
      /home/travis/.gimme/versions/go1.11.5.linux.amd64/src/fmt/scan.go:114 +0xd0
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:643 +0x144
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x279
  fmt.Sscanf()
      /home/travis/.gimme/versions/go1.11.5.linux.amd64/src/fmt/scan.go:114 +0xd0
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:643 +0x144
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x279
  fmt.Sscanf()
      /home/travis/.gimme/versions/go1.11.5.linux.amd64/src/fmt/scan.go:114 +0xd0
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:643 +0x144
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x279
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).recoverJournal()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:476 +0xe3
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.openDB()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:125 +0x9dc
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.Open()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:197 +0x1ed
  fmt.Sscanf()
      /home/travis/.gimme/versions/go1.11.5.linux.amd64/src/fmt/scan.go:114 +0xd0
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:643 +0x144
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x279
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*session).recover.func1()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/session.go:113 +0xbb
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*session).recover()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/session.go:121 +0x17b7
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.Open()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:183 +0xd7
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.OpenFile()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:219 +0xa5
  github.com/ethereum/go-ethereum/swarm/storage.NewLDBDatabase()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/swarm/storage/database.go:37 +0xc3
  github.com/ethereum/go-ethereum/swarm/storage.NewLDBStore()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/swarm/storage/ldbstore.go:152 +0x37c
  github.com/ethereum/go-ethereum/swarm/storage.newTestDbStore()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/swarm/storage/ldbstore_test.go:60 +0x6dc
  github.com/ethereum/go-ethereum/swarm/storage.TestGetAllReferences()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/swarm/storage/filestore_test.go:180 +0x4d
  testing.tRunner()
      /home/travis/.gimme/versions/go1.11.5.linux.amd64/src/testing/testing.go:827 +0x162

Goroutine 324 (running) created at:
  github.com/ethereum/go-ethereum/swarm/storage.(*PyramidChunker).prepareChunks()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/swarm/storage/pyramid.go:389 +0x14c
  github.com/ethereum/go-ethereum/swarm/storage.(*PyramidChunker).Split()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/swarm/storage/pyramid.go:205 +0xd5
  github.com/ethereum/go-ethereum/swarm/storage.PyramidSplit()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/swarm/storage/pyramid.go:100 +0xe1
  github.com/ethereum/go-ethereum/swarm/storage.(*FileStore).GetAllReferences()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/swarm/storage/filestore.go:109 +0x1f6
  github.com/ethereum/go-ethereum/swarm/storage.TestGetAllReferences()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/swarm/storage/filestore_test.go:199 +0x421
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x279
  fmt.Sscanf()
      /home/travis/.gimme/versions/go1.11.5.linux.amd64/src/fmt/scan.go:114 +0xd0
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:643 +0x144
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x279
  fmt.Sscanf()
      /home/travis/.gimme/versions/go1.11.5.linux.amd64/src/fmt/scan.go:114 +0xd0
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:643 +0x144
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x279
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).checkAndCleanFiles()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_util.go:52 +0x2e4
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.openDB()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:130 +0xa00
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x279
  fmt.Sscanf()
      /home/travis/.gimme/versions/go1.11.5.linux.amd64/src/fmt/scan.go:114 +0xd0
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:657 +0x1ed
  fmt.Fscanf()
      /home/travis/.gimme/versions/go1.11.5.linux.amd64/src/fmt/scan.go:143 +0xe9
  fmt.Sscanf()
      /home/travis/.gimme/versions/go1.11.5.linux.amd64/src/fmt/scan.go:114 +0xd0
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:643 +0x144
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x279
  fmt.Sscanf()
      /home/travis/.gimme/versions/go1.11.5.linux.amd64/src/fmt/scan.go:114 +0xd0
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:643 +0x144
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x279
  fmt.Sscanf()
      /home/travis/.gimme/versions/go1.11.5.linux.amd64/src/fmt/scan.go:114 +0xd0
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:643 +0x144
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x279
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).recoverJournal()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:476 +0xe3
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.openDB()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:125 +0x9dc
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.Open()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:197 +0x1ed
  fmt.Sscanf()
      /home/travis/.gimme/versions/go1.11.5.linux.amd64/src/fmt/scan.go:114 +0xd0
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:643 +0x144
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x279
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*session).recover.func1()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/session.go:113 +0xbb
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*session).recover()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/session.go:121 +0x17b7
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.Open()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:183 +0xd7
  github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.OpenFile()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:219 +0xa5
  github.com/ethereum/go-ethereum/swarm/storage.NewLDBDatabase()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/swarm/storage/database.go:37 +0xc3
  github.com/ethereum/go-ethereum/swarm/storage.NewLDBStore()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/swarm/storage/ldbstore.go:152 +0x37c
  github.com/ethereum/go-ethereum/swarm/storage.newTestDbStore()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/swarm/storage/ldbstore_test.go:60 +0x6dc
  github.com/ethereum/go-ethereum/swarm/storage.TestGetAllReferences()
      /home/travis/gopath/src/github.com/ethereum/go-ethereum/swarm/storage/filestore_test.go:180 +0x4d
  testing.tRunner()
      /home/travis/.gimme/versions/go1.11.5.linux.amd64/src/testing/testing.go:827 +0x162
==================
--- FAIL: TestGetAllReferences (0.15s)
    filestore_test.go:204: Expected reference array length to be 9, but is 8
    testing.go:771: race detected during execution of test
[...]
FAIL
FAIL	github.com/ethereum/go-ethereum/swarm/storage	220.163s
@frncmx
Copy link
Contributor Author

frncmx commented Feb 8, 2019

For now I'll be a CI; sry @holisticode :)
The race was introduced in 3eff652

@holisticode holisticode mentioned this issue Feb 11, 2019
Merged
@holisticode
Copy link
Contributor

This seems to be fixed after the latest commit in ethereum/go-ethereum#19028


>$ go test -race -run TestGetAllReferences github.com/ethereum/go-ethereum/swarm/storage
ok  	github.com/ethereum/go-ethereum/swarm/storage	1.947s

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@holisticode holisticode

Labels
bug test
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

swarm/storage: fix HashExplore concurrency bug ethersphere#1211 ethersphere/swarm
2 participants
@holisticode @frncmx