Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

backport 19361 v16 #19379

Closed
wants to merge 470 commits into from
Closed

Conversation

brooksprumo
Copy link
Contributor

Problem

Summary of Changes

Fixes #

mergify bot and others added 30 commits April 23, 2021 07:20
…-labs#16724)

(cherry picked from commit 4c94f89)

Co-authored-by: carllin <carl@solana.com>
) (solana-labs#16770)

first_coding_index is the same as the set_index and is so redundant:
https://github.com/solana-labs/solana/blob/37b8587d4/ledger/src/blockstore_meta.rs#L49-L60

(cherry picked from commit 0319414)

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
…solana-labs#16771)

Local timestamps are updated for records associated with a pubkey if the
origin is still active:
https://github.com/solana-labs/solana/blob/c8ed14c64/core/src/crds.rs#L301-L311

However this is done inconsistently on some gossip paths (pull requests
and pull responses) but not all (e.g. push messages). Additionally
update_record_timestamp is inefficient since there can be ~800 values
associated with each pubkey.

This commit updates records timestamps only on contact-infos; and,
instead utilizes origin's timestamp when purging old values.

(cherry picked from commit 2c82f21)

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
(cherry picked from commit c217ee3)

Co-authored-by: strykerin <dacosta.pereirafabio@gmail.com>
…a-labs#16775)

(cherry picked from commit e9a616c)

Co-authored-by: strykerin <dacosta.pereirafabio@gmail.com>
(cherry picked from commit da58f20)

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
(cherry picked from commit be29568)

# Conflicts:
#	runtime/src/bank.rs

Co-authored-by: Trent Nelson <trent@solana.com>
(cherry picked from commit 63436cc)

Co-authored-by: Michael Vines <mvines@gmail.com>
…ase (solana-labs#16802) (solana-labs#16805)

(cherry picked from commit ec37a84)

Co-authored-by: Josh <josh.hundley@gmail.com>
…16633) (solana-labs#16828)

(cherry picked from commit 9706512)

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
…olana-labs#16845)

* Update bpf loader info on native-programs docs

* Link to program deployment docs

(cherry picked from commit 5eb5d9b)

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
(cherry picked from commit 3af8cb0)

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
…solana-labs#16858)

(cherry picked from commit b66a689)

Co-authored-by: Michael Vines <mvines@gmail.com>
…gle vote account (solana-labs#16859)

(cherry picked from commit 59fc336)

Co-authored-by: Michael Vines <mvines@gmail.com>
…16800) (solana-labs#16871)

process_pull_responses should record hash of values purged by expired
responses (as well as unexpired ones):
https://github.com/solana-labs/solana/blob/c1829dd00/core/src/crds_gossip_pull.rs#L385-L387

otherwise, these values are not excluded from following pull-requests
(from likely different nodes):
https://github.com/solana-labs/solana/blob/c1829dd00/core/src/crds_gossip_pull.rs#L447-L452

and would waste bandwidth should they be included in subsequent
pull-responses.

(cherry picked from commit 3b8d6b5)

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
…olana-labs#16708) (solana-labs#16870)

* limits to data_header.size when combining shreds' payloads (solana-labs#16708)

Shredder::deshred is ignoring data_header.size when combining shreds' payloads:
https://github.com/solana-labs/solana/blob/37b8587d4/ledger/src/shred.rs#L940-L961

Also adding more sanity checks on the alignment of data shreds indices.

(cherry picked from commit 0f3ac51)

# Conflicts:
#	ledger/src/shred.rs

* removes backport merge conflicts

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
…olana-labs#16892)

* Enable multiple payer keypairs

* Suppress tx creation if batch size == 0

* Suppress logs when waiting to create txs

* Double airdrop threshold to prevent stall when closing accounts

(cherry picked from commit 283f587)

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
…olana-labs#16890)

* Retain alloc'd and updated data in cpi (solana-labs#16850)

(cherry picked from commit 9b3a59f)

# Conflicts:
#	programs/bpf_loader/src/syscalls.rs
#	sdk/src/feature_set.rs

* resolve conflicts

Co-authored-by: Jack May <jack@solana.com>
…ana-labs#16894)

* SDK: More conversions for `Pubkey`

(cherry picked from commit 9b7120b)

* SDK: More conversion for `DerivationPath`

(cherry picked from commit 722de94)

* remote-wallet: Add helpers for locating remote wallets

(cherry picked from commit 64fcb79)

* remote-wallet: Plumb `Locator` into `RemoteWalletInfo`

(cherry picked from commit 3d12be2)

* remote-wallet: `derivation-path` crate doesn't like empty trailing child indexes

(cherry picked from commit 4ce4f04)

* remote-wallet: Move `Locator` to its own module

(cherry picked from commit cac666d)

Co-authored-by: Trent Nelson <trent@solana.com>
…-labs#16808) (solana-labs#16896)

local_message_pending_push_queue is recording timestamps at the time the
value is created, and uses that when the pending values are flushed:
https://github.com/solana-labs/solana/blob/ec37a843a/core/src/cluster_info.rs#L321
https://github.com/solana-labs/solana/blob/ec37a843a/core/src/crds_gossip.rs#L96-L102

which is then used as the insert_timestamp when inserting values in the
crds table:
https://github.com/solana-labs/solana/blob/ec37a843a/core/src/crds_gossip_push.rs#L183

The flushing may happen 100ms after the values are created (or even
later if there is a lock contention). This will cause non-monotone
insert_timestamps in the crds table (where time goes backward),
hindering the usability of insert_timestamps for other computations.

For example both ClusterInfo::get_votes and get_epoch_slots_since rely
on monotone insert_timestamps when values are inserted into the table:
https://github.com/solana-labs/solana/blob/ec37a843a/core/src/cluster_info.rs#L1197-L1215
https://github.com/solana-labs/solana/blob/ec37a843a/core/src/cluster_info.rs#L1274-L1298

This commit removes timestamps from local_message_pending_push_queue and
uses current timestamp when flushing the queue.

(cherry picked from commit b468ead)

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
(cherry picked from commit 36574c3)

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
…olana-labs#16898)

(cherry picked from commit b17d5ee)

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
…#16806) (solana-labs#16905)

It is crucial that VersionedCrdsValue::insert_timestamp does not go
backward in time:
https://github.com/solana-labs/solana/blob/ec37a843a/core/src/crds.rs#L67-L79

Otherwise methods such as get_votes and get_epoch_slots_since will
break, which will break their downstream flow, including vote-listener
and optimistic confirmation:
https://github.com/solana-labs/solana/blob/ec37a843a/core/src/cluster_info.rs#L1197-L1215
https://github.com/solana-labs/solana/blob/ec37a843a/core/src/cluster_info.rs#L1274-L1298

For that, Crds::new_versioned is intended to be called "atomically" with
Crds::insert_verioned (as the comment already says so):
https://github.com/solana-labs/solana/blob/ec37a843a/core/src/crds.rs#L126-L129

However, currently this is violated in the code. For example,
filter_pull_responses creates VersionedCrdsValues (with the current
timestamp), then acquires an exclusive lock on gossip, then
process_pull_responses writes those values to the crds table:
https://github.com/solana-labs/solana/blob/ec37a843a/core/src/cluster_info.rs#L2375-L2392

Depending on the workload and lock contention, the insert_timestamps may
well be in the past when these values finally are inserted into gossip.

To avoid such scenarios, this commit:
  * removes Crds::new_versioned and Crd::insert_versioned.
  * makes VersionedCrdsValue constructor private, only invoked in
    Crds::insert, so that insert_timestamp is populated right before
    insert.

This will improve insert_timestamp monotonicity as long as Crds::insert
is not called with a stalled timestamp. Following commits may further
improve this by calling timestamp() inside Crds::insert, and/or
switching to std::time::Instant which guarantees monotonicity.

(cherry picked from commit 1ac2a8c)

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
) (solana-labs#16907)

ClusterInfo::new_pull_requests has to lookup contact-infos:
https://github.com/solana-labs/solana/blob/a1ef2bd74/core/src/cluster_info.rs#L1663-L1673

when it was already available when making pull requests:
https://github.com/solana-labs/solana/blob/a1ef2bd74/core/src/crds_gossip_pull.rs#L232

(cherry picked from commit 25054bf)

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
(cherry picked from commit 783bd79)

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
mergify bot and others added 28 commits July 13, 2021 07:18
…18626)

* Add Fees struct to client

* Add complete RpcClient::get_fees methods

* Switch cli to last_valid_block_height

(cherry picked from commit 8ad4ffd)

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
)

* hijack secp256k1 enablement feature plumbing for libsecp256k1 upgrade

* Bump libsecp256k1 to v0.5.0

* gate libsecp256k1 upgrade to v0.5.0

Co-authored-by: Trent Nelson <trent@solana.com>
…#18665)

* rpc: more params for `GetVoteAccountsConfig`

(cherry picked from commit bf90ea2)

# Conflicts:
#	docs/src/developing/clients/jsonrpc-api.md

* cli: allow returning more `solana validators`

(cherry picked from commit a4a24b6)

# Conflicts:
#	Cargo.lock
#	cli/Cargo.toml
#	cli/src/cluster_query.rs

Co-authored-by: Trent Nelson <trent@solana.com>
(cherry picked from commit 2ec81f6)

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
(cherry picked from commit e316586)

# Conflicts:
#	core/src/broadcast_stage.rs
#	core/src/cluster_info.rs

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
* ci: fix typo

(cherry picked from commit 96a7ced)

* ci: suppress cargo tree output

(cherry picked from commit 59cd055)

Co-authored-by: Trent Nelson <trent@solana.com>
* backporting bank_hash and program_costs column families from master to 1.6 for rocksdb backward compatibility

* missed a line to allow dead code

* include code for purge
…#18797)

(cherry picked from commit a4c3db5)

Co-authored-by: Trent Nelson <trent@solana.com>
…18780) (solana-labs#18804)

* feature: add new token program feature

* Fixup test

* Cargo fmt

* Add back whitespace for cargo fmt

* Revert file totally
…ana-labs#18823) (solana-labs#18836)

(cherry picked from commit d6f5945)

Co-authored-by: Jon Cinque <jon.cinque@gmail.com>
(cherry picked from commit 578f2aa)

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
…lana-labs#18969)

(cherry picked from commit 611af87)

Co-authored-by: Ryo Onodera <ryoqun@gmail.com>
@brooksprumo brooksprumo deleted the backport-19361-v16 branch August 23, 2021 23:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.