rpc: implement iterator sessions #2555

AnnaShaleva · 2022-06-15T18:27:55Z

Port neo-project/neo-modules#721 and neo-project/neo-modules#715.

Depends on #2556 and #2558.

TODO:

RPC server/client documentation update.
Deal with Allow traverse iterators neo-project/neo-modules#715 (comment) on the client side.
Port Rpc Client iterators neo-project/neo-modules#721 (comment) after resolution.
~~Port Allow traverse iterators neo-project/neo-modules#715 (comment) if changed.~~ Not needed.
Tests.

codecov · 2022-06-16T13:09:20Z

Codecov Report

Merging #2555 (8d549a3) into master (251c9bd) will decrease coverage by 0.03%.
The diff coverage is 72.47%.

❗ Current head 8d549a3 differs from pull request most recent head 0117581. Consider uploading reports for the commit 0117581 to get more accurate results

@@            Coverage Diff             @@
##           master    #2555      +/-   ##
==========================================
- Coverage   84.63%   84.59%   -0.04%     
==========================================
  Files         297      297              
  Lines       37491    37901     +410     
==========================================
+ Hits        31730    32062     +332     
- Misses       4379     4429      +50     
- Partials     1382     1410      +28

Impacted Files	Coverage Δ
pkg/rpc/client/nep11.go	`52.19% <34.00%> (-8.68%)`	⬇️
pkg/rpc/client/rpc.go	`79.65% <65.38%> (-0.48%)`	⬇️
pkg/rpc/response/result/invoke.go	`75.56% <68.00%> (-6.88%)`	⬇️
pkg/rpc/client/helper.go	`59.11% <77.90%> (+12.23%)`	⬆️
pkg/rpc/server/server.go	`76.38% <78.17%> (+0.09%)`	⬆️
pkg/rpc/client/native.go	`70.73% <83.33%> (+1.57%)`	⬆️
cli/wallet/nep11.go	`76.42% <100.00%> (ø)`
pkg/config/config.go	`82.35% <100.00%> (ø)`
pkg/core/interop/iterator/interop.go	`100.00% <100.00%> (ø)`
pkg/rpc/request/param.go	`87.72% <100.00%> (+0.37%)`	⬆️
... and 8 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 34ddc99...0117581. Read the comment docs.

AnnaShaleva · 2022-06-16T16:48:02Z

Need to write some more tests, but the implementation can be reviewed.

pkg/config/config.go

pkg/core/statesync/neotest_test.go

fyrchik · 2022-06-17T07:50:08Z

pkg/rpc/client/helper.go

+		bytes, err := json.Marshal(params)
+		if err != nil {
+			return nil, fmt.Errorf("failed to convert operation parameters into System.Contract.Call arguments: marshalling error %w", err)
+		}
+		var slice request.Params
+		err = json.Unmarshal(bytes, &slice)


What's the point with marshal/unmarshal? Can we convert/emit directly?

Actually, this marshal/unmarshal code performs exactly the same thing that client/server perform during invokefunction call processing. The case is that ExpandArrayIntoScript works with request.Param only and has such artifacts like emitting Booleans like Ints:

case smartcontract.BoolType: val, err := fp.Value.GetBoolean() // not GetBooleanStrict(), because that's the way C# code works if err != nil { return errors.New("not a bool") } if val { emit.Int(script, 1) } else { emit.Int(script, 0) }

To remove marshal/unmarshal we have to create a copy of ExpandArrayIntoScript for smartcontract.Parameter, which will be almost exact copy of ExpandArrayIntoScript. I'm not sure we really need this.

So do you think it's better to create an analogue of ExpandArrayIntoScript for smartcontrct.Parameter?

Fixed by creating a separate function for smartcontract.Parameter.

pkg/rpc/client/helper.go

pkg/rpc/client/rpc.go

pkg/rpc/server/server.go

fyrchik · 2022-06-17T08:37:00Z

pkg/rpc/server/server.go

+	iteratorValues := iterator.Values(it, count)
+	result := make([]json.RawMessage, len(iteratorValues))
+	for j := range iteratorValues {
+		result[j], err = stackitem.ToJSONWithTypes(iteratorValues[j])
+		if err != nil {
+			return nil, response.NewInternalServerError(fmt.Sprintf("failed to marshal iterator value: %s", err))
+		}
+	}
+	return result, nil


Can we execute some of this code without s.sessionsLock mutex?

Not sure, we still need a way to lock iterator during traversal process. We either need to perform it under s.sessionsLock or to create a mutex for each iterator. What do you think?

Fixed by adding a separate mutex for sessions.

JSONisation is moved outside of the mutex.

pkg/rpc/server/client_test.go

pkg/rpc/server/server.go

pkg/rpc/response/result/invoke.go

pkg/rpc/server/server.go

docs/node-configuration.md

pkg/rpc/server/server.go

Add ability to switch between current blockchain storage and MPT-backed storage for iterator traversing process. It may be useful because iterator implementation traverses underlying backed storage (BoltDB, LevelDB) inside DB's Seek which is blocking operation for BoltDB: ``` Opening a read transaction and a write transaction in the same goroutine can cause the writer to deadlock because the database periodically needs to re-mmap itself as it grows and it cannot do that while a read transaction is open. If a long running read transaction (for example, a snapshot transaction) is needed, you might want to set DB.InitialMmapSize to a large enough value to avoid potential blocking of write transaction. ``` So during bbolt re-mmaping, standard blockchain DB operations (i.e. persist) can be blocked until iterator resourses release. The described behaviour is tested and confirmed on four-nodes privnet with BoltDB and `SessionExpirationTime` set to be 180 seconds. After new iterator session is added to the server, the subsequent persist took ~5m21s, see the log record `2022-06-17T18:58:21.563+0300`: ``` anna@kiwi:~/Documents/GitProjects/nspcc-dev/neo-go$ ./bin/neo-go node -p 2022-06-17T18:52:21.535+0300 INFO initial gas supply is not set or wrong, setting default value {"InitialGASSupply": "52000000"} 2022-06-17T18:52:21.535+0300 INFO MaxBlockSize is not set or wrong, setting default value {"MaxBlockSize": 262144} 2022-06-17T18:52:21.535+0300 INFO MaxBlockSystemFee is not set or wrong, setting default value {"MaxBlockSystemFee": 900000000000} 2022-06-17T18:52:21.535+0300 INFO MaxTransactionsPerBlock is not set or wrong, using default value {"MaxTransactionsPerBlock": 512} 2022-06-17T18:52:21.535+0300 INFO MaxValidUntilBlockIncrement is not set or wrong, using default value {"MaxValidUntilBlockIncrement": 5760} 2022-06-17T18:52:21.535+0300 INFO Hardforks are not set, using default value 2022-06-17T18:52:21.543+0300 INFO no storage version found! creating genesis block 2022-06-17T18:52:21.546+0300 INFO ExtensiblePoolSize is not set or wrong, using default value {"ExtensiblePoolSize": 20} 2022-06-17T18:52:21.546+0300 INFO service is running {"service": "Prometheus", "endpoint": ":2112"} 2022-06-17T18:52:21.547+0300 INFO starting rpc-server {"endpoint": ":20331"} 2022-06-17T18:52:21.547+0300 INFO rpc-server iterator sessions are enabled 2022-06-17T18:52:21.547+0300 INFO service hasn't started since it's disabled {"service": "Pprof"} 2022-06-17T18:52:21.547+0300 INFO node started {"blockHeight": 0, "headerHeight": 0} _ ____________ __________ / | / / ____/ __ \ / ____/ __ \ / |/ / __/ / / / /_____/ / __/ / / / / /| / /___/ /_/ /_____/ /_/ / /_/ / /_/ |_/_____/\____/ \____/\____/ /NEO-GO:0.99.1-pre-53-g7ccb646e/ 2022-06-17T18:52:21.548+0300 INFO new peer connected {"addr": "127.0.0.1:20336", "peerCount": 1} 2022-06-17T18:52:21.550+0300 INFO started protocol {"addr": "127.0.0.1:20336", "userAgent": "/NEO-GO:0.99.1-pre-53-g7ccb646e/", "startHeight": 65, "id": 1475228436} 2022-06-17T18:52:22.575+0300 INFO persisted to disk {"blocks": 65, "keys": 1410, "headerHeight": 65, "blockHeight": 65, "took": "28.193409ms"} 2022-06-17T18:52:24.548+0300 INFO new peer connected {"addr": "127.0.0.1:20333", "peerCount": 2} 2022-06-17T18:52:24.548+0300 INFO new peer connected {"addr": "127.0.0.1:20336", "peerCount": 3} 2022-06-17T18:52:24.548+0300 INFO new peer connected {"addr": "127.0.0.1:20334", "peerCount": 4} 2022-06-17T18:52:24.549+0300 INFO new peer connected {"addr": "127.0.0.1:20335", "peerCount": 5} 2022-06-17T18:52:24.549+0300 INFO new peer connected {"addr": "127.0.0.1:20335", "peerCount": 6} 2022-06-17T18:52:24.549+0300 INFO started protocol {"addr": "127.0.0.1:20333", "userAgent": "/NEO-GO:0.99.1-pre-53-g7ccb646e/", "startHeight": 65, "id": 3444438498} 2022-06-17T18:52:24.549+0300 INFO new peer connected {"addr": "127.0.0.1:20334", "peerCount": 7} 2022-06-17T18:52:24.549+0300 INFO new peer connected {"addr": "127.0.0.1:20333", "peerCount": 8} 2022-06-17T18:52:24.550+0300 INFO node reached synchronized state, starting services 2022-06-17T18:52:24.550+0300 INFO started protocol {"addr": "127.0.0.1:20334", "userAgent": "/NEO-GO:0.99.1-pre-53-g7ccb646e/", "startHeight": 65, "id": 2435677826} 2022-06-17T18:52:24.550+0300 INFO starting state validation service 2022-06-17T18:52:24.550+0300 INFO RPC server already started 2022-06-17T18:52:24.550+0300 INFO new peer connected {"addr": "127.0.0.1:20335", "peerCount": 9} 2022-06-17T18:52:24.550+0300 INFO new peer connected {"addr": "127.0.0.1:20335", "peerCount": 10} 2022-06-17T18:52:24.550+0300 WARN peer disconnected {"addr": "127.0.0.1:20334", "error": "already connected", "peerCount": 9} 2022-06-17T18:52:24.550+0300 WARN peer disconnected {"addr": "127.0.0.1:20336", "error": "already connected", "peerCount": 8} 2022-06-17T18:52:24.550+0300 WARN peer disconnected {"addr": "127.0.0.1:20333", "error": "already connected", "peerCount": 7} 2022-06-17T18:52:24.550+0300 WARN peer disconnected {"addr": "127.0.0.1:20335", "error": "unexpected empty payload: CMDVersion", "peerCount": 6} 2022-06-17T18:52:24.550+0300 INFO started protocol {"addr": "127.0.0.1:20335", "userAgent": "/NEO-GO:0.99.1-pre-53-g7ccb646e/", "startHeight": 65, "id": 970555896} 2022-06-17T18:52:24.551+0300 INFO new peer connected {"addr": "127.0.0.1:20334", "peerCount": 7} 2022-06-17T18:52:24.551+0300 WARN peer disconnected {"addr": "127.0.0.1:20335", "error": "unexpected empty payload: CMDVersion", "peerCount": 6} 2022-06-17T18:52:24.551+0300 WARN peer disconnected {"addr": "127.0.0.1:20335", "error": "unexpected empty payload: CMDVersion", "peerCount": 5} 2022-06-17T18:52:24.551+0300 WARN peer disconnected {"addr": "127.0.0.1:20334", "error": "already connected", "peerCount": 4} 2022-06-17T18:52:29.564+0300 INFO persisted to disk {"blocks": 1, "keys": 19, "headerHeight": 66, "blockHeight": 66, "took": "12.51808ms"} 2022-06-17T18:52:44.558+0300 INFO persisted to disk {"blocks": 1, "keys": 19, "headerHeight": 67, "blockHeight": 67, "took": "1.563137ms"} 2022-06-17T18:55:21.549+0300 WARN peer disconnected {"addr": "127.0.0.1:20335", "error": "ping/pong timeout", "peerCount": 3} 2022-06-17T18:55:21.550+0300 WARN peer disconnected {"addr": "127.0.0.1:20333", "error": "ping/pong timeout", "peerCount": 2} 2022-06-17T18:55:21.550+0300 WARN peer disconnected {"addr": "127.0.0.1:20334", "error": "ping/pong timeout", "peerCount": 1} 2022-06-17T18:55:21.550+0300 WARN peer disconnected {"addr": "127.0.0.1:20336", "error": "ping/pong timeout", "peerCount": 0} 2022-06-17T18:55:21.553+0300 INFO new peer connected {"addr": "127.0.0.1:20335", "peerCount": 1} 2022-06-17T18:55:21.554+0300 INFO started protocol {"addr": "127.0.0.1:20335", "userAgent": "/NEO-GO:0.99.1-pre-53-g7ccb646e/", "startHeight": 77, "id": 970555896} 2022-06-17T18:55:24.554+0300 INFO new peer connected {"addr": "172.200.0.4:20333", "peerCount": 2} 2022-06-17T18:55:24.555+0300 INFO new peer connected {"addr": "172.200.0.3:20334", "peerCount": 3} 2022-06-17T18:55:24.555+0300 INFO new peer connected {"addr": "10.78.13.84:59876", "peerCount": 4} 2022-06-17T18:55:24.555+0300 INFO new peer connected {"addr": "127.0.0.1:20335", "peerCount": 5} 2022-06-17T18:55:24.556+0300 INFO new peer connected {"addr": "172.200.0.254:20332", "peerCount": 6} 2022-06-17T18:55:24.556+0300 INFO new peer connected {"addr": "127.0.0.1:20336", "peerCount": 7} 2022-06-17T18:55:24.556+0300 INFO started protocol {"addr": "172.200.0.4:20333", "userAgent": "/NEO-GO:0.99.1-pre-53-g7ccb646e/", "startHeight": 76, "id": 3444438498} 2022-06-17T18:55:24.556+0300 INFO new peer connected {"addr": "172.200.0.1:20335", "peerCount": 8} 2022-06-17T18:55:24.558+0300 INFO started protocol {"addr": "127.0.0.1:20336", "userAgent": "/NEO-GO:0.99.1-pre-53-g7ccb646e/", "startHeight": 77, "id": 1475228436} 2022-06-17T18:55:24.559+0300 INFO new peer connected {"addr": "127.0.0.1:20334", "peerCount": 9} 2022-06-17T18:55:24.558+0300 INFO started protocol {"addr": "172.200.0.3:20334", "userAgent": "/NEO-GO:0.99.1-pre-53-g7ccb646e/", "startHeight": 77, "id": 2435677826} 2022-06-17T18:55:24.559+0300 INFO new peer connected {"addr": "127.0.0.1:20336", "peerCount": 10} 2022-06-17T18:55:24.559+0300 WARN peer disconnected {"addr": "172.200.0.1:20335", "error": "unexpected empty payload: CMDVersion", "peerCount": 9} 2022-06-17T18:55:24.559+0300 INFO new peer connected {"addr": "127.0.0.1:20333", "peerCount": 10} 2022-06-17T18:55:24.560+0300 INFO new peer connected {"addr": "172.200.0.2:20336", "peerCount": 11} 2022-06-17T18:55:24.560+0300 WARN peer disconnected {"addr": "172.200.0.254:20332", "error": "identical node id", "peerCount": 10} 2022-06-17T18:55:24.561+0300 WARN peer disconnected {"addr": "127.0.0.1:20335", "error": "already connected", "peerCount": 9} 2022-06-17T18:55:24.561+0300 INFO new peer connected {"addr": "127.0.0.1:20334", "peerCount": 10} 2022-06-17T18:55:24.561+0300 WARN peer disconnected {"addr": "10.78.13.84:59876", "error": "unexpected empty payload: CMDVersion", "peerCount": 9} 2022-06-17T18:55:24.561+0300 WARN peer disconnected {"addr": "127.0.0.1:20336", "error": "already connected", "peerCount": 8} 2022-06-17T18:55:24.561+0300 INFO new peer connected {"addr": "127.0.0.1:20335", "peerCount": 9} 2022-06-17T18:55:24.561+0300 WARN peer disconnected {"addr": "127.0.0.1:20333", "error": "unexpected empty payload: CMDVersion", "peerCount": 8} 2022-06-17T18:55:24.561+0300 INFO new peer connected {"addr": "127.0.0.1:20333", "peerCount": 9} 2022-06-17T18:55:24.561+0300 WARN peer disconnected {"addr": "127.0.0.1:20334", "error": "unexpected empty payload: CMDVersion", "peerCount": 8} 2022-06-17T18:55:24.561+0300 WARN peer disconnected {"addr": "172.200.0.2:20336", "error": "unexpected empty payload: CMDVersion", "peerCount": 7} 2022-06-17T18:55:24.561+0300 INFO new peer connected {"addr": "127.0.0.1:20336", "peerCount": 8} 2022-06-17T18:55:24.561+0300 INFO new peer connected {"addr": "127.0.0.1:20333", "peerCount": 9} 2022-06-17T18:55:24.561+0300 WARN peer disconnected {"addr": "127.0.0.1:20336", "error": "already connected", "peerCount": 8} 2022-06-17T18:55:24.561+0300 INFO new peer connected {"addr": "127.0.0.1:20336", "peerCount": 9} 2022-06-17T18:55:24.561+0300 INFO new peer connected {"addr": "127.0.0.1:20334", "peerCount": 10} 2022-06-17T18:55:24.561+0300 INFO new peer connected {"addr": "127.0.0.1:20334", "peerCount": 11} 2022-06-17T18:55:24.561+0300 INFO new peer connected {"addr": "127.0.0.1:20333", "peerCount": 12} 2022-06-17T18:55:24.562+0300 WARN peer disconnected {"addr": "127.0.0.1:20335", "error": "already connected", "peerCount": 11} 2022-06-17T18:55:24.562+0300 INFO new peer connected {"addr": "127.0.0.1:20333", "peerCount": 12} 2022-06-17T18:55:24.562+0300 INFO new peer connected {"addr": "127.0.0.1:20334", "peerCount": 13} 2022-06-17T18:55:24.562+0300 WARN peer disconnected {"addr": "127.0.0.1:20333", "error": "already connected", "peerCount": 12} 2022-06-17T18:55:24.562+0300 WARN peer disconnected {"addr": "127.0.0.1:20336", "error": "already connected", "peerCount": 11} 2022-06-17T18:55:24.562+0300 WARN peer disconnected {"addr": "127.0.0.1:20334", "error": "already connected", "peerCount": 10} 2022-06-17T18:55:24.562+0300 WARN peer disconnected {"addr": "127.0.0.1:20333", "error": "unexpected empty payload: CMDVersion", "peerCount": 9} 2022-06-17T18:55:24.563+0300 INFO new peer connected {"addr": "127.0.0.1:20336", "peerCount": 10} 2022-06-17T18:55:24.563+0300 WARN peer disconnected {"addr": "127.0.0.1:20334", "error": "already connected", "peerCount": 9} 2022-06-17T18:55:24.563+0300 WARN peer disconnected {"addr": "127.0.0.1:20334", "error": "unexpected empty payload: CMDVersion", "peerCount": 8} 2022-06-17T18:55:24.563+0300 WARN peer disconnected {"addr": "127.0.0.1:20333", "error": "already connected", "peerCount": 7} 2022-06-17T18:55:24.563+0300 WARN peer disconnected {"addr": "127.0.0.1:20335", "error": "max peers reached", "peerCount": 6} 2022-06-17T18:55:24.563+0300 WARN peer disconnected {"addr": "127.0.0.1:20333", "error": "already connected", "peerCount": 5} 2022-06-17T18:55:24.563+0300 WARN peer disconnected {"addr": "127.0.0.1:20334", "error": "max peers reached", "peerCount": 4} 2022-06-17T18:55:24.563+0300 WARN peer disconnected {"addr": "127.0.0.1:20336", "error": "already connected", "peerCount": 3} 2022-06-17T18:57:21.551+0300 WARN peer disconnected {"addr": "172.200.0.4:20333", "error": "ping/pong timeout", "peerCount": 2} 2022-06-17T18:57:21.552+0300 WARN peer disconnected {"addr": "172.200.0.3:20334", "error": "ping/pong timeout", "peerCount": 1} 2022-06-17T18:57:21.552+0300 WARN peer disconnected {"addr": "127.0.0.1:20336", "error": "ping/pong timeout", "peerCount": 0} 2022-06-17T18:57:21.553+0300 INFO new peer connected {"addr": "172.200.0.4:20333", "peerCount": 1} 2022-06-17T18:57:21.554+0300 INFO new peer connected {"addr": "10.78.13.84:20332", "peerCount": 2} 2022-06-17T18:57:21.555+0300 INFO started protocol {"addr": "172.200.0.4:20333", "userAgent": "/NEO-GO:0.99.1-pre-53-g7ccb646e/", "startHeight": 82, "id": 3444438498} 2022-06-17T18:57:21.556+0300 INFO new peer connected {"addr": "127.0.0.1:20334", "peerCount": 3} 2022-06-17T18:57:21.556+0300 INFO new peer connected {"addr": "10.78.13.84:46076", "peerCount": 4} 2022-06-17T18:57:21.556+0300 INFO new peer connected {"addr": "172.200.0.1:20335", "peerCount": 5} 2022-06-17T18:57:21.556+0300 INFO new peer connected {"addr": "172.200.0.254:20332", "peerCount": 6} 2022-06-17T18:57:21.556+0300 INFO new peer connected {"addr": "10.78.13.84:59972", "peerCount": 7} 2022-06-17T18:57:21.557+0300 INFO new peer connected {"addr": "127.0.0.1:20333", "peerCount": 8} 2022-06-17T18:57:21.557+0300 INFO new peer connected {"addr": "127.0.0.1:20335", "peerCount": 9} 2022-06-17T18:57:21.557+0300 INFO new peer connected {"addr": "172.200.0.2:20336", "peerCount": 10} 2022-06-17T18:57:21.557+0300 INFO new peer connected {"addr": "127.0.0.1:20333", "peerCount": 11} 2022-06-17T18:57:21.557+0300 INFO new peer connected {"addr": "127.0.0.1:20334", "peerCount": 12} 2022-06-17T18:57:21.557+0300 INFO new peer connected {"addr": "172.200.0.3:20334", "peerCount": 13} 2022-06-17T18:57:21.557+0300 INFO new peer connected {"addr": "127.0.0.1:20336", "peerCount": 14} 2022-06-17T18:57:21.557+0300 INFO started protocol {"addr": "127.0.0.1:20334", "userAgent": "/NEO-GO:0.99.1-pre-53-g7ccb646e/", "startHeight": 82, "id": 2435677826} 2022-06-17T18:57:21.557+0300 WARN peer disconnected {"addr": "172.200.0.2:20336", "error": "max peers reached", "peerCount": 13} 2022-06-17T18:57:21.557+0300 INFO new peer connected {"addr": "127.0.0.1:20335", "peerCount": 14} 2022-06-17T18:57:21.558+0300 INFO started protocol {"addr": "172.200.0.1:20335", "userAgent": "/NEO-GO:0.99.1-pre-53-g7ccb646e/", "startHeight": 82, "id": 970555896} 2022-06-17T18:57:21.558+0300 WARN peer disconnected {"addr": "172.200.0.254:20332", "error": "identical node id", "peerCount": 13} 2022-06-17T18:57:21.558+0300 INFO new peer connected {"addr": "127.0.0.1:20334", "peerCount": 14} 2022-06-17T18:57:21.558+0300 WARN peer disconnected {"addr": "127.0.0.1:20335", "error": "max peers reached", "peerCount": 13} 2022-06-17T18:57:21.558+0300 WARN peer disconnected {"addr": "10.78.13.84:46076", "error": "identical node id", "peerCount": 12} 2022-06-17T18:57:21.558+0300 INFO new peer connected {"addr": "127.0.0.1:20333", "peerCount": 13} 2022-06-17T18:57:21.558+0300 INFO new peer connected {"addr": "127.0.0.1:20335", "peerCount": 14} 2022-06-17T18:57:21.558+0300 INFO new peer connected {"addr": "127.0.0.1:20336", "peerCount": 15} 2022-06-17T18:57:21.558+0300 WARN peer disconnected {"addr": "10.78.13.84:59972", "error": "identical node id", "peerCount": 14} 2022-06-17T18:57:21.558+0300 WARN peer disconnected {"addr": "127.0.0.1:20334", "error": "already connected", "peerCount": 13} 2022-06-17T18:57:21.559+0300 WARN peer disconnected {"addr": "10.78.13.84:20332", "error": "unexpected empty payload: CMDVersion", "peerCount": 12} 2022-06-17T18:57:21.559+0300 WARN peer disconnected {"addr": "127.0.0.1:20333", "error": "already connected", "peerCount": 11} 2022-06-17T18:57:21.559+0300 WARN peer disconnected {"addr": "172.200.0.3:20334", "error": "unexpected empty payload: CMDVersion", "peerCount": 10} 2022-06-17T18:57:21.559+0300 WARN peer disconnected {"addr": "127.0.0.1:20335", "error": "unexpected empty payload: CMDVersion", "peerCount": 9} 2022-06-17T18:57:21.559+0300 WARN peer disconnected {"addr": "127.0.0.1:20334", "error": "already connected", "peerCount": 8} 2022-06-17T18:57:21.559+0300 WARN peer disconnected {"addr": "127.0.0.1:20333", "error": "unexpected empty payload: CMDVersion", "peerCount": 7} 2022-06-17T18:57:21.559+0300 INFO started protocol {"addr": "127.0.0.1:20336", "userAgent": "/NEO-GO:0.99.1-pre-53-g7ccb646e/", "startHeight": 82, "id": 1475228436} 2022-06-17T18:57:21.559+0300 WARN peer disconnected {"addr": "127.0.0.1:20333", "error": "already connected", "peerCount": 6} 2022-06-17T18:57:21.559+0300 WARN peer disconnected {"addr": "127.0.0.1:20335", "error": "already connected", "peerCount": 5} 2022-06-17T18:57:21.559+0300 WARN peer disconnected {"addr": "127.0.0.1:20336", "error": "already connected", "peerCount": 4} 2022-06-17T18:58:21.561+0300 INFO persisted to disk {"blocks": 1, "keys": 20, "headerHeight": 68, "blockHeight": 68, "took": "5m21.993873018s"} 2022-06-17T18:58:21.563+0300 INFO persisted to disk {"blocks": 8, "keys": 111, "headerHeight": 76, "blockHeight": 76, "took": "2.243347ms"} 2022-06-17T18:58:22.567+0300 INFO persisted to disk {"blocks": 10, "keys": 135, "headerHeight": 86, "blockHeight": 86, "took": "5.637669ms"} 2022-06-17T18:58:25.565+0300 INFO persisted to disk {"blocks": 1, "keys": 19, "headerHeight": 87, "blockHeight": 87, "took": "1.879912ms"} 2022-06-17T18:58:40.572+0300 INFO persisted to disk {"blocks": 1, "keys": 20, "headerHeight": 88, "blockHeight": 88, "took": "1.560317ms"} 2022-06-17T18:58:55.579+0300 INFO persisted to disk {"blocks": 1, "keys": 19, "headerHeight": 89, "blockHeight": 89, "took": "1.925225ms"} 2022-06-17T18:59:10.587+0300 INFO persisted to disk {"blocks": 1, "keys": 19, "headerHeight": 90, "blockHeight": 90, "took": "3.118073ms"} 2022-06-17T18:59:25.592+0300 INFO persisted to disk {"blocks": 1, "keys": 19, "headerHeight": 91, "blockHeight": 91, "took": "1.607248ms"} 2022-06-17T18:59:40.600+0300 INFO persisted to disk {"blocks": 1, "keys": 20, "headerHeight": 92, "blockHeight": 92, "took": "931.806µs"} 2022-06-17T18:59:55.610+0300 INFO persisted to disk {"blocks": 1, "keys": 19, "headerHeight": 93, "blockHeight": 93, "took": "2.019041ms"} ```

Create a set of functions that are able to work with both session-based iterators, default unpacked iterators and client-side unpacked iterators.

Do not unwrap the whole set of iterator values even on demand.

…rScript

It's server who should be responsible for iterator ID creation and iterator registration.

AnnaShaleva added this to the v0.99.1 milestone Jun 15, 2022

AnnaShaleva requested review from fyrchik and roman-khimov June 15, 2022 18:27

AnnaShaleva force-pushed the rpc/sessions branch 4 times, most recently from b27e4c7 to 5a05d87 Compare June 16, 2022 13:03

AnnaShaleva force-pushed the rpc/sessions branch 2 times, most recently from b062be8 to 8036c9e Compare June 16, 2022 16:46

fyrchik reviewed Jun 17, 2022

View reviewed changes

AnnaShaleva force-pushed the rpc/sessions branch 2 times, most recently from 70634c2 to 2b6f82f Compare June 17, 2022 09:27

AnnaShaleva requested a review from fyrchik June 17, 2022 09:57

AnnaShaleva commented Jun 17, 2022

View reviewed changes

pkg/rpc/server/server.go Outdated Show resolved Hide resolved

AnnaShaleva force-pushed the rpc/sessions branch 9 times, most recently from 861347d to 293f425 Compare June 20, 2022 16:08

github-advanced-security bot found potential problems Jun 20, 2022

View reviewed changes

pkg/rpc/server/server.go Fixed Show fixed Hide fixed

AnnaShaleva force-pushed the rpc/sessions branch from 293f425 to 62d48df Compare June 21, 2022 15:24

AnnaShaleva changed the title ~~WIP rpc: implement iterator sessions~~ rpc: implement iterator sessions Jun 21, 2022

AnnaShaleva marked this pull request as ready for review June 21, 2022 15:26

AnnaShaleva force-pushed the rpc/sessions branch from 62d48df to 0e2b8cd Compare June 22, 2022 14:14

AnnaShaleva force-pushed the rpc/sessions branch from 845a537 to e7a82c8 Compare July 8, 2022 09:56

AnnaShaleva marked this pull request as draft July 8, 2022 10:01

AnnaShaleva force-pushed the rpc/sessions branch from e7a82c8 to 2702e26 Compare July 8, 2022 10:25

AnnaShaleva marked this pull request as ready for review July 8, 2022 10:28

AnnaShaleva force-pushed the rpc/sessions branch 2 times, most recently from e5859cb to 457442c Compare July 8, 2022 12:54

roman-khimov reviewed Jul 8, 2022

View reviewed changes

pkg/rpc/response/result/invoke.go Outdated Show resolved Hide resolved

pkg/rpc/server/server.go Outdated Show resolved Hide resolved

docs/node-configuration.md Show resolved Hide resolved

pkg/rpc/server/server.go Outdated Show resolved Hide resolved

AnnaShaleva force-pushed the rpc/sessions branch 5 times, most recently from 189643d to b6f9404 Compare July 8, 2022 14:00

AnnaShaleva added 9 commits July 8, 2022 17:05

rpc: implement iterator sessions

cbd20eb

rpc: restrict default SessionExpirationTime

47ffc1f

rpc: extend iterator-related client functionality

fad061f

Create a set of functions that are able to work with both session-based iterators, default unpacked iterators and client-side unpacked iterators.

rpc: restrict (*Client).TraverseIterator with single RPC call

9bdd815

Do not unwrap the whole set of iterator values even on demand.

rpc: restrict max number of iterator items for createIteratorUnwrappe…

4581cc3

…rScript

rpc: move session maintenance related code out of the result.Invoke

8f73ce0

It's server who should be responsible for iterator ID creation and iterator registration.

rpc: restrict the amount of concurrently running iterator sessions

445cca1

rpc: fail invoke.Result unmarshalling on stack unmarshalling error

ef114d6

roman-khimov added the rpc RPC server and client label Jul 8, 2022

roman-khimov approved these changes Jul 8, 2022

View reviewed changes

AnnaShaleva force-pushed the rpc/sessions branch from b6f9404 to 8d549a3 Compare July 8, 2022 14:15

rpc: add implementation note about iterator sessions

0117581

AnnaShaleva force-pushed the rpc/sessions branch from 8d549a3 to 0117581 Compare July 8, 2022 14:16

roman-khimov approved these changes Jul 8, 2022

View reviewed changes

roman-khimov merged commit bbeef6e into master Jul 8, 2022

roman-khimov deleted the rpc/sessions branch July 8, 2022 14:29

roman-khimov mentioned this pull request Jul 10, 2022

Move response-related RPC code #2586

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rpc: implement iterator sessions #2555

rpc: implement iterator sessions #2555

AnnaShaleva commented Jun 15, 2022 •

edited

Loading

codecov bot commented Jun 16, 2022 •

edited

Loading

AnnaShaleva commented Jun 16, 2022

fyrchik Jun 17, 2022

AnnaShaleva Jun 17, 2022

AnnaShaleva Jun 17, 2022

AnnaShaleva Jun 21, 2022

fyrchik Jun 17, 2022

AnnaShaleva Jun 17, 2022

AnnaShaleva Jun 17, 2022

AnnaShaleva Jun 21, 2022

rpc: implement iterator sessions #2555

rpc: implement iterator sessions #2555

Conversation

AnnaShaleva commented Jun 15, 2022 • edited Loading

codecov bot commented Jun 16, 2022 • edited Loading

Codecov Report

AnnaShaleva commented Jun 16, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AnnaShaleva commented Jun 15, 2022 •

edited

Loading

codecov bot commented Jun 16, 2022 •

edited

Loading