Fix LevelDB.Open failures in TestSpawnEtcdRaft #869
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
TestSpawnEtcdRaft is structured as four sub-tests that start ordering
nodes. All sub-tests share a common temporary directory and, by
extension, a common block ledger directory. While the orderer(s) were
killed at the end of each sub-test, the code did not wait for the
termination to complete leaving a window where the block ledger file
lock for a terminating orderer could be held while a new orderer
instance was starting.
```
--- FAIL: TestSpawnEtcdRaft/Good (1.86s)
--- PASS: TestSpawnEtcdRaft/Good/TLS_disabled_dual_listener (1.63s)
--- FAIL: TestSpawnEtcdRaft/Good/TLS_enabled_single_listener (0.23s)
etcdraft_test.go:143:
No future change is possible. Bailing out early after 0.027s.
Got stuck at:
... <redacted> ...
Metrics.Statsd.WriteInterval = 30s
Metrics.Statsd.Prefix = ""
2020-03-16 20:51:07.566 EDT [orderer.common.server] initializeServerConfig -> INFO 003 Starting orderer with mutual TLS enabled
panic: Error opening leveldb: resource temporarily unavailable
goroutine 1 [running]:
github.com/hyperledger/fabric/common/ledger/util/leveldbhelper.(*DB).Open(0xc0002e5630)
/Users/sykesm/workspace/fabric/src/github.com/hyperledger/fabric/common/ledger/util/leveldbhelper/leveldb_helper.go:75 +0x285
github.com/hyperledger/fabric/common/ledger/util/leveldbhelper.openDBAndCheckFormat(0xc000408980, 0x0, 0x0, 0x0)
/Users/sykesm/workspace/fabric/src/github.com/hyperledger/fabric/common/ledger/util/leveldbhelper/leveldb_provider.go:63 +0x11b
github.com/hyperledger/fabric/common/ledger/util/leveldbhelper.NewProvider(0xc000408980, 0xc000408980, 0x4, 0xc00004cb00)
/Users/sykesm/workspace/fabric/src/github.com/hyperledger/fabric/common/ledger/util/leveldbhelper/leveldb_provider.go:51 +0x2f
github.com/hyperledger/fabric/common/ledger/blkstorage/fsblkstorage.NewProvider(0xc000408940, 0xc000408960, 0x4df07c0, 0x55e7060, 0xc00004cae0, 0x5a, 0x0, 0x0)
/Users/sykesm/workspace/fabric/src/github.com/hyperledger/fabric/common/ledger/blkstorage/fsblkstorage/fs_blockstore_provider.go:43 +0x158
github.com/hyperledger/fabric/common/ledger/blockledger/fileledger.New(0xc0002e6690, 0x4e, 0x4df07c0, 0x55e7060, 0x7100008, 0x0, 0xc0002ad630, 0x400ea28)
/Users/sykesm/workspace/fabric/src/github.com/hyperledger/fabric/common/ledger/blockledger/fileledger/factory.go:71 +0x107
github.com/hyperledger/fabric/orderer/common/server.createLedgerFactory(0xc000304480, 0x4df07c0, 0x55e7060, 0xc00002e000, 0x37b, 0x57b, 0xc00002c300, 0xf1, 0x2f1)
/Users/sykesm/workspace/fabric/src/github.com/hyperledger/fabric/orderer/common/server/util.go:29 +0xf8
github.com/hyperledger/fabric/orderer/common/server.Main()
/Users/sykesm/workspace/fabric/src/github.com/hyperledger/fabric/orderer/common/server/main.go:113 +0x671
main.main()
/Users/sykesm/workspace/fabric/src/github.com/hyperledger/fabric/cmd/orderer/main.go:15 +0x20
Waiting for:
EvictionSuspicion not set, defaulting to 10m
FAIL
FAIL github.com/hyperledger/fabric/orderer/common/server 23.793s
FAIL
```
This change has the sub-tests wait for orderer termination to complete
before proceeding.
Signed-off-by: Matthew Sykes <sykesmat@us.ibm.com>
yacovm
approved these changes
Mar 17, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
TestSpawnEtcdRaft is structured as four sub-tests that start ordering nodes. All sub-tests share a common temporary directory and, by extension, a common block ledger directory. While the orderer(s) were killed at the end of each sub-test, the code did not wait for the termination to complete leaving a window where the block ledger file lock for a terminating orderer could be held while a new orderer instance was starting.
This change has the sub-tests wait for orderer termination to complete before proceeding.