forked from near/nearcore
-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
NEP-364 #1
Draft
blasrodri
wants to merge
334
commits into
master
Choose a base branch
from
blas-implement-signature-verif-host-functions
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
NEP-364 #1
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
For tracking here are the needed functions |
I think we've got everything now. I just need to hook this with the near-sdk to verify that we can construct our own:
|
Progress here: near#7165 |
* fix validator info * fix * add next epoch validators Co-authored-by: near-bulldozer[bot] <73298989+near-bulldozer[bot]@users.noreply.github.com>
In shardnet still can't reach all validators: ``` Validators found: 122. Visited nodes: 917. ```
* Export more information about peers (network stats etc) * fmt * fix field name Co-authored-by: near-bulldozer[bot] <73298989+near-bulldozer[bot]@users.noreply.github.com>
…7282) We haven’t been sending those messages for a while now so support for borsh serialising and deserialising them is unnecessary. Meanwhile, the serialisation forces bunch of other types to be borsh-serialisable as well making it harder to change those types. And changing them is something we might want to do. The message used to be sent when view client couldn’t handle a Query due to lack of state. In that case it would route the message to other nodes. This feature has been removed in commit f68140e: ‘Add structured errors to query method in ViewClient and RPC.’
…e conveniently (near#7256) With cops, we now need to somehow configure `setup_mock_all_validators` to also specify cops. It *could* be done by adding just another argument, but I'd rather avoid that. Instead, what I suggest is bunching up all related configs into a single builder object, `ValidatorSchedule`, and use that as a universal tool to configure validators for KeyValueRuntime. Note that this PR doesn't change `setup_mock_all_validators` -- that's going to be a bigger diff, so I want to land that separately. I might also merge this with `EpochValidatorSet`, the structs handle similar data, but, at this stage, `EpochValidatorSet` is implementation detail, while `ValidatorSchedule` is the API for authors of tests. Can be reviewed per commit.
…ar#7286) * info about current chunk producers (and how many are reachable) * list of unreachable chunk/block producers * added colors to height * added information about the type of connection (IN vs OUT) * added columns about first connection time and last ping (they will start working once the changes to the neard are submitted). <img width="1398" alt="image" src="https://user-images.githubusercontent.com/91919554/181454724-62b3bb7b-682b-4f1a-93a0-9e1d35f57bd6.png">
Specifically this PR implements broadcasting data by validator accounts for the current epoch. Currently it just includes the list of IPs under which the validator is reachable. It will be used to establish TIER1 network. This is just implementation of the broadcast. Seeding the data at the validators will be implemented in the next PR. jira issue: https://nearinc.atlassian.net/browse/CPR-94
Introduce Trie::retrieve_raw_node method and use it in three places which had practically identical code.
and adjusted the accounts_data::Cache accordingly.
This'll allow configuring chunk-only producers for tests using this infra follow up to near#7256
While at it, convert the fuzz target to use encoders from near primitives rather than using base64 and bs58 crates directly.
Because 1.x.y tags we’re using are not annotated, git describe ignores them when describing a commit. Add --tags flag to make it take a look at such tags. While at it, include --match=[0-9]* so that the command excludes non-nearcore-release tags such as crates-0.14.0.
near#7301) enter the commit message for your changes. Lines starting
The get_validator_info function doesn't work with the current epoch id, see comments in ValidatorIdentifier.
…7308) Rather than declaring a field String and having all the users do base64 conversion manually, declare it Vec<u8> with base64_format serialisation scheme. This way, only the serialisation code needs to worry how the data looks on the wire. This also means that users reading the data don’t need to worry about decoding errors. Fixes: near#7232
* doc: prefer for loops to for_each and try_for_each methods * Update chain/chunks/src/lib.rs Co-authored-by: Akhilesh Singhania <akhi@near.org> * Update runtime/runtime-params-estimator/estimator-warehouse/src/check.rs Co-authored-by: Akhilesh Singhania <akhi@near.org> * fix * a few more * some better wording Co-authored-by: Akhilesh Singhania <akhi@near.org>
Uptime is represented as the timestamp of starting the client.
* Configurable telemetry reporting interval * Implement Default for TelemetryConfig. * Set default telemetry reporting interval Include endpoint in the error message
Adding option to state dump to kick non-whitelisted validators.
Turns out `rustflags` here are not additive!
Co-authored-by: Aleksey Kladov <aleksey.kladov@gmail.com>
See near#7287 for similar work. This should be the final PR that is based splitting out tests based on `ProtocolFeature`. I suspect that we could split some more tests from `process_blocks.rs` as well. But we will have to figure out how to organise them.
This is part of a preparation to support TIER1 peers. I've moved as much logic as possible inside the constructor, so that the number of arguments is minimal. I've also added a PeerActor::Config which is passed to the constructor, because PeerActor::new cannot return an error (due to Actix API constraints). Also moved accepting new connections from PeerManagerActor to NetworkState.
Allow to select only one metric instead of always running both in `estimator-warehouse estimate`. This is necessary because `icount` estimations have become very slow.
When profiling memory usage, I've noticed non-trivial amount of space is occupied by net IO buffers. So, let's add metrics to track their size! We use SockAddr, rather than `PeerId`, as a label here, as we don't know peer ids for outgoing connections. I don't worry much about perf hit here -- we more or less do a syscall here, that should dominate metric bump. The implementation is a bit hacky, but the "proper" fix would probably require unforking the FramedRead/Write infrastructure.
Debug page now has as section for catchup information <img width="785" alt="Screen Shot 2022-09-02 at 11 58 39 AM" src="https://user-images.githubusercontent.com/34969888/188193706-291ae020-888d-4215-ac75-bd43056d280b.png"> Catchup info will also be printed as part of stats in log. ``` 2022-09-02T15:58:30.730076Z INFO stats: Catchups Sync block `CZL3BCD15QpZ4FLqWu7vAJ97JsVHJ1ngtZQupfhiVaL6`@61 Shard sync status: Shard 0 done, Shard 1 split applying total parts 3 done 1, Shard 2 split applying not started, Shard 3 split applying not started Next blocks to catch up: `CZL3BCD15QpZ4FLqWu7vAJ97JsVHJ1ngtZQupfhiVaL6`@61 ```
…ease (near#7538) * add feature RejectBlocksWithOutdatedProtocolVersions to the next shardnet release * address comments
Deprecate near_peer_message_received_total Prometheus metric in favour of aggregating near_peer_message_received_by_type_total metric instead.
Taking slice as argument is more flexible than taking reference to a vector since the caller does not need to have to own the data. In addition, slices also work if caller owns a boxed slice which would need to be converted into a vector if function required reference to a vector. Slices also allow to take sub-slices easily. The only remaining place is Store::push_component in near-network crate. This is needed because near-network’s schema’s StoreUpdate::set doesn’t work with borrowed types. More generic magic is needed to make it work.
Separate crate for test-only logging utils doesn't really pull its weight, let's just add a `testonly` module to the general o11y crate. Especially given that there already were some prod dependencies on this crate.
…ear#7544) With cold storage approaching, it’s gonig to be necessary to be able to open RocksDB instance with only a subset of defined columns. Change how RocksDB deals with column family handles. Specifically, get_cf_handles now accepts an iterator over DBCol values as argument and cf_handle will return an error if trying to get a handle for a column that wasn’t set up.
check_free_space_interval and free_space_threshold fields of RocksDB struct have always the same value. Turn them into constants.
Extend test_iterator_seek by adding verification of known existing keys.
Metrics *are* part of o11y package, it's much better organizationally to not split this stuff into two crates.
The database version definitions make more sense inside of near-store than near-primitives. Especially since majority of usage of them is in near-store and the definitions aren’t used for anything specific to the chain or other operations not involving the storage. While at it, change RocksDB::get_version such that it checks whether the database exists (and returns None if it doesn’t) and opens the database with only DbVersion column configured (so that’ll properly read version from old databases without modifying them).
Co-authored-by: Aleksey Kladov <aleksey.kladov@gmail.com>
…ure-verif-host-functions
also adjusted a bit the internal API for sending routed messages.
The serde::Serialize and serde::Deserialize implementations for the VMContext type are never used. Get rid of them. This makes base58_format unused so get rid of it as well.
Instead of introducing a new dependency, we can generate request id using generate_random_string function. The id doesn’t really matter anyway (apart from JSON RPC specification requiring it to be present) so any string really will work there.
Just a tiny bit of refactoring separating configuration part of the storage and code responsible for opening the storage into separate modules.
The default Debug derive for types is quite verbose with a lot of vertical spacing. For example, TrieNode branches are printed as a list with each empty branch as a None on its own line. This isn’t very ergonomic when trying to debug trie nodes and printing a lot of the nodes to output. Replace the default Debug implementation for TrieNode, NodeHandle and ValueHandle by custom ones which are more concise and don’t waste space on empty nodes.
This will be needed in future commit where we will need access to the raw bytes of the node. See near#7509.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
WIP