swarm sync fixes #3780

zelig · 2017-03-13T20:35:55Z

This PR provdes fixes to various incorrect behaviours and inefficiencies in the syncer and
it fixes the mysterious hash integrity bug in the dbstore

db store layout is changed, a reindex cmd is provided
data is now stored at 6|<proximity_order_as_byte>|<dataIdx> so can iterate by prox bin for syncing
correct setting of dataIdx after restart and the value is read from db fixes the invalid hash issues
syncer is much simplified and syncs only the relevant bins
sync state layout is changed, which breaks compatibility with older versions of the bzz protocol!

mention-bot · 2017-03-13T20:35:57Z

@zelig, thanks for your PR! By analyzing the history of the files in this pull request, we identified @karalabe, @mchusovlianov and @fjl to be potential reviewers.

GitCop · 2017-03-13T20:35:57Z

Thank you for your contribution! Your commits seem to not adhere to the repository coding standards

Commit: fd0650888bbf4a536e9c8d9cc7335586561cfbe0
Commits must be prefixed with the package(s) they modify

Please check the contribution guidelines for more details.

This message was auto-generated by https://gitcop.com

GitCop · 2017-03-15T11:01:55Z

Thank you for your contribution! Your commits seem to not adhere to the repository coding standards

Commit: fd0650888bbf4a536e9c8d9cc7335586561cfbe0
Commits must be prefixed with the package(s) they modify

Please check the contribution guidelines for more details.

This message was auto-generated by https://gitcop.com

lmars · 2017-03-22T13:06:59Z

cmd/swarm/import.go

+		utils.Fatalf("--datadir must be specificed")
+	}
+
+	sourcedatadirfull = fmt.Sprintf("%s/swarm/bzz-%s", path.Clean(importdatadir), importbzzaccount)


Doesn't this mean that <source-bzzaccount> is a required command line argument? Otherwise the source directory will be <importdir>/swarm/bzz- which won't exist?

lmars · 2017-03-22T13:08:54Z

swarm/network/messages.go

+	syncRequestMsg            // 0x04
+	deliveryRequestMsg        // 0x05
+	unsyncedKeysMsg           // 0x06
+	paymentMsg                // 0x07


Is this necessary? Will it not cause confusion between old and new nodes misinterpreting the meaning of the codes?

lmars · 2017-03-22T13:12:06Z

swarm/network/syncer.go

@@ -32,6 +33,7 @@ const (
 	requestDbBatchSize = 512  // size of batch before written to request db
 	keyBufferSize      = 1024 // size of buffer  for unsynced keys
 	syncBatchSize      = 128  // maximum batchsize for outgoing requests
+	historyBufferSize  = 128  // maximum batchsize for outgoing requests


This comment needs updating (it is copied from the line above)

lmars · 2017-03-22T13:21:21Z

@zelig @nolash I started reviewing this but realised there is a lot of commented code, is it your intention to merge it like this?

nolash · 2017-03-28T04:04:14Z

I don't think this is done yet. There is still a matter of the last batch of keys not being sent by syncer. You'll be putting it here I presume, @zelig ?

GitCop · 2017-03-29T14:53:52Z

Thank you for your contribution! Your commits seem to not adhere to the repository coding standards

Commit: fd0650888bbf4a536e9c8d9cc7335586561cfbe0
Commits must be prefixed with the package(s) they modify

Please check the contribution guidelines for more details.

This message was auto-generated by https://gitcop.com

GitCop · 2017-04-02T20:53:23Z

Thank you for your contribution! Your commits seem to not adhere to the repository coding standards

Commit: fd0650888bbf4a536e9c8d9cc7335586561cfbe0
Commits must be prefixed with the package(s) they modify

Please check the contribution guidelines for more details.

This message was auto-generated by https://gitcop.com

GitCop · 2017-04-02T21:02:02Z

Thank you for your contribution! Your commits seem to not adhere to the repository coding standards

Commit: fd0650888bbf4a536e9c8d9cc7335586561cfbe0
Commits must be prefixed with the package(s) they modify

Please check the contribution guidelines for more details.

This message was auto-generated by https://gitcop.com

nagydani

The 0-returning distance function in line 33 of cleandb.go is definitely dangerous and probably wrong. Pease do not approve before this is properly handled.

GitCop · 2017-04-03T13:25:54Z

Thank you for your contribution! Your commits seem to not adhere to the repository coding standards

Commit: eb458a7201e8da74e1f648c6347d06809b38ec9c
Commits must be prefixed with the package(s) they modify

Please check the contribution guidelines for more details.

This message was auto-generated by https://gitcop.com

GitCop · 2017-04-03T13:33:14Z

Thank you for your contribution! Your commits seem to not adhere to the repository coding standards

Commit: eb458a7201e8da74e1f648c6347d06809b38ec9c
Commits must be prefixed with the package(s) they modify
Commit: 3781eb92e7d7e13ac86b76ccb6e67e46293c451a
Commits must be prefixed with the package(s) they modify

Please check the contribution guidelines for more details.

This message was auto-generated by https://gitcop.com

GitCop · 2017-04-05T07:31:50Z

Thank you for your contribution! Your commits seem to not adhere to the repository coding standards

Commit: eb458a7201e8da74e1f648c6347d06809b38ec9c
Commits must be prefixed with the package(s) they modify
Commit: 3781eb92e7d7e13ac86b76ccb6e67e46293c451a
Commits must be prefixed with the package(s) they modify

Please check the contribution guidelines for more details.

This message was auto-generated by https://gitcop.com

GitCop · 2017-04-05T08:34:06Z

Thank you for your contribution! Your commits seem to not adhere to the repository coding standards

Commit: eb458a7201e8da74e1f648c6347d06809b38ec9c
Commits must be prefixed with the package(s) they modify
Commit: 3781eb92e7d7e13ac86b76ccb6e67e46293c451a
Commits must be prefixed with the package(s) they modify

Please check the contribution guidelines for more details.

This message was auto-generated by https://gitcop.com

GitCop · 2017-04-07T16:53:22Z

Thank you for your contribution! Your commits seem to not adhere to the repository coding standards

Commit: eb458a7201e8da74e1f648c6347d06809b38ec9c
Commits must be prefixed with the package(s) they modify
Commit: 3781eb92e7d7e13ac86b76ccb6e67e46293c451a
Commits must be prefixed with the package(s) they modify

Please check the contribution guidelines for more details.

This message was auto-generated by https://gitcop.com

karalabe · 2017-04-14T09:41:29Z

Postponed to 1.6.1 as it wasn't ready in time for the release.

lmars · 2017-04-19T21:23:45Z

The Invalid Chunk in Database error is still happening on this branch.

Using the scripts from the swarm-dev branch I span up a 9 node cluster, uploaded some data and restarted the cluster:

$ rm -rf cluster
$ scripts/boot-cluster.sh --size 9
$ scripts/random-uploads.sh --addr 192.168.33.101:8500 --size 40k --count 1000
$ scripts/boot-cluster.sh --size 9

I then observed the error in the logs for swarm02, swarm03 and swarm05:

cluster/logs/swarm02.log:TRACE[04-19|19:03:02] Apparent key/hash mismatch. Hash 0ed3a9a2e9b648019546e3aad47d1438190dc1b9a972f1b20577ea96efa80803, key 275f24aadb09413a08e53752606705469c76551a0eca27ebc1de307f38a5ddf6 
cluster/logs/swarm03.log:TRACE[04-19|19:03:00] Apparent key/hash mismatch. Hash 79b852c6cf1d49cdc56725e91197eb157fc14858de4cf32738d02c86a4190bd4, key 260a6eb7044ddbb85401e7ba4be4cad293c3b2a36ac266de9e853cc9f5256e73 
cluster/logs/swarm05.log:TRACE[04-19|19:03:49] Apparent key/hash mismatch. Hash d869f4139277a6b6484157958c42ffc58fd54320f9c09f22cf864065b887108b, key cbed2c20ddf29998e68e1c523c0e030244009c746edabfae0d118790818c6ae6

Looking at the invalid chunk on swarm02, it was stored at 19:02:12 and was only reported invalid at 19:03:02:

$ grep -B1 -F 'Apparent key/hash' cluster/logs/swarm02.log
TRACE[04-19|19:03:02] DBStore: Chunk 275f24aa indexkey 242 datakey 06010000000000000242 proximity 1 
TRACE[04-19|19:03:02] Apparent key/hash mismatch. Hash 0ed3a9a2e9b648019546e3aad47d1438190dc1b9a972f1b20577ea96efa80803, key 275f24aadb09413a08e53752606705469c76551a0eca27ebc1de307f38a5ddf6 

$ grep -F 275f24aa cluster/logs/swarm02.log 
TRACE[04-19|19:02:12] Depo.HandleUnsyncedKeysMsg: received [<Key: 27f40f67, Priority: 0> <Key: 25845ac1, Priority: 0> <Key: 2500aa4c, Priority: 0> <Key: 27c2e087, Priority: 0> <Key: 25a74f98, Priority: 0> <Key: 24335e10, Priority: 0> <Key: 25291854, Priority: 0> <Key: 24a84f71, Priority: 0> <Key: 25821462, Priority: 0> <Key: 24c327e7, Priority: 0> <Key: 25b012b3, Priority: 0> <Key: 2584511f, Priority: 0> <Key: 274aa93f, Priority: 0> <Key: 24378cd3, Priority: 0> <Key: 260f4769, Priority: 0> <Key: 2764e1b8, Priority: 0> <Key: 26e510e2, Priority: 0> <Key: 2672bd42, Priority: 0> <Key: 26e5c281, Priority: 0> <Key: 24cf42c3, Priority: 0> <Key: 27552c06, Priority: 0> <Key: 252e5216, Priority: 0> <Key: 2748f5ba, Priority: 0> <Key: 275f24aa, Priority: 0> <Key: 25cb0ecf, Priority: 0> <Key: 276d1654, Priority: 0> <Key: 275721f5, Priority: 0> <Key: 24b52e06, Priority: 0> <Key: 276ccc9e, Priority: 0> <Key: 263da458, Priority: 0> <Key: 24727674, Priority: 0> <Key: 27133081, Priority: 0> <Key: 24d48fcb, Priority: 0> <Key: 272af5ed, Priority: 0> <Key: 27756d60, Priority: 0> <Key: 252412e4, Priority: 0> <Key: 27c500a6, Priority: 0> <Key: 260a6eb7, Priority: 0> <Key: 2462cdfd, Priority: 0> <Key: 249205e9, Priority: 0> <Key: 24affac4, Priority: 0> <Key: 2599258e, Priority: 0> <Key: 24b5e49f, Priority: 0> <Key: 2641044e, Priority: 0> <Key: 26f30e40, Priority: 0> <Key: 27ceef65, Priority: 0>] 
TRACE[04-19|19:02:12] incoming store request: from: self, Key: 275f24aadb09413a08e53752606705469c76551a0eca27ebc1de307f38a5ddf6; ID: 2037830665595424798, requestTimeout: <nil>, storageTimeout: <nil>, SData 00100000000000005cb8 
TRACE[04-19|19:02:12] Depo.handleStoreRequest: 275f24aadb09413a08e53752606705469c76551a0eca27ebc1de307f38a5ddf6 not found locally. create new chunk/request 
TRACE[04-19|19:02:12] delivery of Key: 275f24aa TreeSize: 4096 Chunksize: 4104 from enode://eee23e1886f3e0580c495b1c38427d3905c19a6f30b5c271970348633abad005150e62dd193af6176e307ef99cf0e3d73ccca3a9dbb32ce0d86d07930f0f000b@192.168.33.104:30399?discport=0 
TRACE[04-19|19:02:12] NetStore.Put: localStore.Put 275f24aa stored locally 
TRACE[04-19|19:02:12] batch put: datai          dx 578 prox 1 chunkkey 275f24aadb09413a08e53752606705469c76551a0eca27ebc1de307f38a5ddf6 datakey [6 1 0 0 0 0 0 0 2 66] data 275f24aadb09413a08e53752606705469c76551a0eca27ebc1de307f38a5ddf600100000000000005cb8109dd96dd34e0ffb1ca0f347f46ba8eee2a1ade18bf9 
TRACE[04-19|19:02:12] DbStore.Put: 275f24aa. db storage counter: 579  
TRACE[04-19|19:02:12] serve 1 (<=0) nodes for target lookup 275f24aadb09413a08e53752606705469c76551a0eca27ebc1de307f38a5ddf6 (PO001) 
TRACE[04-19|19:02:12] forwarder.Store: enode://eee23e1886f3e0580c495b1c38427d3905c19a6f30b5c271970348633abad005150e62dd193af6176e307ef99cf0e3d73ccca3a9dbb32ce0d86d07930f0f000b@192.168.33.104:30399?discport=0 Key: 275f24aa TreeSize: 4096 Chunksize: 4104 
TRACE[04-19|19:02:12] forwarder.Store: sent to 0 peers (chunk = Key: 275f24aa TreeSize: 4096 Chunksize: 4104) 
TRACE[04-19|19:03:02] DBStore: Chunk 275f24aa indexkey 242 datakey 06010000000000000242 proximity 1 
TRACE[04-19|19:03:02] Apparent key/hash mismatch. Hash 0ed3a9a2e9b648019546e3aad47d1438190dc1b9a972f1b20577ea96efa80803, key 275f24aadb09413a08e53752606705469c76551a0eca27ebc1de307f38a5ddf6

fjl

Can you provide a concise explanation of the problem that this PR fixes?

fjl · 2017-04-27T10:34:06Z

swarm/storage/types_test.go

+// You should have received a copy of the GNU Lesser General Public License
+// along with the go-ethereum library. If not, see <http://www.gnu.org/licenses/>.
+
+package storage


This file is empty.

fjl · 2017-04-27T10:34:55Z

swarm/storage/database.go

@@ -29,6 +29,7 @@ import (
 )

 const openFileLimit = 128
+//const openFileLimit = -1


Please remove this one.

fjl · 2017-04-27T10:35:45Z

swarm/storage/chunker_test.go

+// func BenchmarkSplitPyramid_5(t *testing.B)  { benchmarkSplitPyramid(100000, t) }
+// func BenchmarkSplitPyramid_6(t *testing.B)  { benchmarkSplitPyramid(1000000, t) }
+// func BenchmarkSplitPyramid_7(t *testing.B)  { benchmarkSplitPyramid(10000000, t) }
+// func BenchmarkSplitPyramid_8(t *testing.B)  { benchmarkSplitPyramid(100000000, t) }


Please remove or uncomment.

zelig · 2017-04-27T10:47:27Z

thanks @fjl pushing the changes

* new index and structure in db_store * syncronisation: iterator more efficient * syncer history sync is simplified * swarm/api: fix tests NOTE: this represents an interim fix until network layer rewrite is complete Update cleandb.go Just a comment, not to forget fixing this. Update dbstore.go Added a TODO comment to refactor the database constructor. swarm/storage: Adding bucketed chunk counters to chunk database cmd/swarm: cleandb now takes proper PO function based on bzzaccount swarm/storage: fix bucketcount stats in dbstore swarm/storage, cmd/swarm: add db dump as command

* increment dataindex read from db; fixes invalid hash error * increment last index; fixes repeated keys in history sync * only use MaxPO (=8) proxbins in dbstore; simplifies iterations * reduce logging output and make it useful * fix tests for fuse and http API * simplify chunkstore tests in common_test, streamline tests for dbstore, memstore and add benchmarks

GitCop · 2017-07-11T12:28:56Z

Thank you for your contribution! Your commits seem to not adhere to the repository coding standards

Commit: 163cef7
Commits must be prefixed with the package(s) they modify

Please check the contribution guidelines for more details.

This message was auto-generated by https://gitcop.com

zelig added type:bug type:feature in progress labels Mar 13, 2017

zelig added this to the 1.6.0 milestone Mar 13, 2017

zelig self-assigned this Mar 13, 2017

lmars reviewed Mar 22, 2017

View reviewed changes

zelig force-pushed the swarm-db-sync-fix-tests branch from f935bd1 to 0ee8038 Compare April 2, 2017 20:53

zelig force-pushed the swarm-db-sync-fix-tests branch from 0ee8038 to 5564bb0 Compare April 2, 2017 21:02

zelig force-pushed the swarm-db-sync-fix-tests branch from 5564bb0 to 442bf23 Compare April 3, 2017 12:33

nagydani reviewed Apr 3, 2017

View reviewed changes

karalabe modified the milestones: 1.6.1, 1.6.0 Apr 14, 2017

zelig force-pushed the swarm-db-sync-fix-tests branch from 28c6a55 to 44bcce0 Compare April 27, 2017 10:27

fjl requested changes Apr 27, 2017

View reviewed changes

zelig force-pushed the swarm-db-sync-fix-tests branch 2 times, most recently from 186e42e to 54a7faf Compare April 27, 2017 11:29

fjl modified the milestones: 1.6.1, 1.6.3 May 31, 2017

karalabe modified the milestones: 1.6.4, 1.6.3 Jun 1, 2017

trappist mentioned this pull request Jun 8, 2017

Synchronisation failed, dropping peer peer=b1710c8648aa839a err="retrieved hash chain is invalid" #14381

Closed

zelig mentioned this pull request Jun 26, 2017

swarm related PRs - Q2 merge plan #14706

Closed

13 tasks

zelig force-pushed the swarm-db-sync-fix-tests branch from 54a7faf to b7a08cb Compare July 3, 2017 14:11

zelig added 2 commits July 3, 2017 19:49

jmozah force-pushed the swarm-db-sync-fix-tests branch from b7a08cb to 73cc315 Compare July 3, 2017 14:58

swarm/network;swarm/dev:fix new syncer bugs

163cef7

fjl closed this Nov 30, 2017

gbalint deleted the swarm-db-sync-fix-tests branch May 25, 2018 14:44

adamschmideg removed in progress labels Dec 14, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

swarm sync fixes #3780

swarm sync fixes #3780

zelig commented Mar 13, 2017 •

edited

Loading

mention-bot commented Mar 13, 2017

GitCop commented Mar 13, 2017

GitCop commented Mar 15, 2017

lmars Mar 22, 2017

lmars Mar 22, 2017

lmars Mar 22, 2017

lmars commented Mar 22, 2017

nolash commented Mar 28, 2017

GitCop commented Mar 29, 2017

GitCop commented Apr 2, 2017

GitCop commented Apr 2, 2017

nagydani left a comment

GitCop commented Apr 3, 2017

GitCop commented Apr 3, 2017

GitCop commented Apr 5, 2017

GitCop commented Apr 5, 2017

GitCop commented Apr 7, 2017

karalabe commented Apr 14, 2017

lmars commented Apr 19, 2017

fjl left a comment

fjl Apr 27, 2017

fjl Apr 27, 2017

fjl Apr 27, 2017

zelig commented Apr 27, 2017

GitCop commented Jul 11, 2017

swarm sync fixes #3780

swarm sync fixes #3780

Conversation

zelig commented Mar 13, 2017 • edited Loading

mention-bot commented Mar 13, 2017

GitCop commented Mar 13, 2017

GitCop commented Mar 15, 2017

lmars Mar 22, 2017

Choose a reason for hiding this comment

lmars Mar 22, 2017

Choose a reason for hiding this comment

lmars Mar 22, 2017

Choose a reason for hiding this comment

lmars commented Mar 22, 2017

nolash commented Mar 28, 2017

GitCop commented Mar 29, 2017

GitCop commented Apr 2, 2017

GitCop commented Apr 2, 2017

nagydani left a comment

Choose a reason for hiding this comment

GitCop commented Apr 3, 2017

GitCop commented Apr 3, 2017

GitCop commented Apr 5, 2017

GitCop commented Apr 5, 2017

GitCop commented Apr 7, 2017

karalabe commented Apr 14, 2017

lmars commented Apr 19, 2017

fjl left a comment

Choose a reason for hiding this comment

fjl Apr 27, 2017

Choose a reason for hiding this comment

fjl Apr 27, 2017

Choose a reason for hiding this comment

fjl Apr 27, 2017

Choose a reason for hiding this comment

zelig commented Apr 27, 2017

GitCop commented Jul 11, 2017

zelig commented Mar 13, 2017 •

edited

Loading