Mixnet integration #1346

zdave-parity · 2023-08-31T23:34:14Z

This adds all the necessary mixnet components, and puts them together in the "kitchen-sink" node/runtime. The components added are:

A pallet (frame/mixnet). This is responsible for determining the current mixnet session and phase, and the mixnodes to use in each session. It provides a function that validators can call to register a mixnode for the next session. The logic of this pallet is very similar to that of the im-online pallet.
A service (client/mixnet). This implements the core mixnet logic, building on the mixnet crate. The service communicates with other nodes using notifications sent over the "mixnet" protocol.
An RPC interface. This currently only supports sending transactions over the mixnet.

zdave-parity · 2023-08-31T23:39:28Z

Moved PR from the old Substrate repo (old PR here). I think I've addressed all the comments except two...

Not sure where the CLI param stuff should go. On the Substrate PR I made this comment:

Hmm I'm actually thinking the kitchen-sink node would be the right place for this? It makes assumptions that are not made by sc-mixnet. For example, sc-mixnet does not assume that non-authorities cannot be mixnodes (sc-mixnet doesn't care whether nodes are authorities or not). Similarly, sc-mixnet doesn't really know anything about RPC (so the comments about RPC would be out of place).

Not sure how log levels should be picked. On the Substrate PR I made this comment:

Is there a reference somewhere for which log levels should be used when? For this mixnet stuff I have so far been using error for things which should only happen if some node misbehaves, warn for things which might happen even if all nodes behave (but isn't likely to), and debug for everything else.

arkpar · 2023-09-01T08:47:40Z

Not sure how log levels should be picked.

info for informational messages that are useful to the user. Progress, detected network addresses, etc.

error should be used for fatal errors that typically lead to process termination.

warn is for

Things that the user can take action for. E.g. misconfiguration.
Invalid/suspicious state caused by logic bugs.

debug and trace should be used for everything else with debug being higer priority/lower verbosity.

So normally, I'd say network misbehaviour should be a debug, since there's nothing that can be done by the user about it. Unless we can detect that something is terribly wrong and show a general warning. E.g. when majority is misbehaving.

arkpar · 2023-09-01T08:56:42Z

Not sure where the CLI param stuff should go. On the Substrate PR I made this comment:

Hmm I'm actually thinking the kitchen-sink node would be the right place for this? It makes assumptions that are not made by sc-mixnet. For example, sc-mixnet does not assume that non-authorities cannot be mixnodes (sc-mixnet doesn't care whether nodes are authorities or not). Similarly, sc-mixnet doesn't really know anything about RPC (so the comments about RPC would be out of place).

I'm fine with keeping it in sc-cli

zdave-parity · 2023-09-01T14:31:01Z

Thanks. Fixed up all the log levels.

bkchr · 2023-09-13T08:20:32Z

substrate/primitives/mixnet/src/runtime_api.rs

+		///
+		/// `session_index` should match `session_status().current_index`; if it does not, `false`
+		/// is returned immediately.


I still don't get this requirement. If you know the current_index why do I need to pass it then?

It's just to avoid needing two calls into the runtime every block (one to check the session index and one to register). I can drop this argument if you'd prefer the two calls.

substrate/frame/mixnet/src/lib.rs

bkchr · 2023-09-13T08:33:34Z

substrate/frame/mixnet/src/lib.rs

+	///
+	/// `session_index` should match `session_status().current_index`; if it does not, `false` is
+	/// returned immediately.


Same comment as for the runtime api. I don't get why we not just fetch the session index?

substrate/frame/mixnet/src/lib.rs

zdave-parity · 2023-10-05T13:19:04Z

Does anyone have any idea what's going on with the Zombienet block-building test (https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/3884822)? The CI output is not particularly enlightening.

pepoviola · 2023-10-06T10:02:27Z

Does anyone have any idea what's going on with the Zombienet block-building test (https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/3884822)? The CI output is not particularly enlightening.

We should update the version of zombienet used in ci to the latest v1.3.69 that includes your pr in zombienet code 👍 .

zdave-parity · 2023-10-06T11:44:42Z

This PR doesn't actually depend on that change as it doesn't include the Zombienet test; I was going to add that in a follow-up commit.

It looks like there is some issue starting the nodes up in the block building test but I'm not sure how to start debugging this. The test works fine locally using the native backend, but this isn't exactly what CI is running. If I could run what is being run under CI locally that would be a good start but I'm not sure how to do that.

pepoviola · 2023-10-06T11:57:19Z

This is the error, the node crash.

2023-10-05 14:19:35.897	
Error: Service(Other("Error parsing spec file: missing field `mixnet` at line 129 column 13"))

https://grafana.teleport.parity.io/goto/PZf_nnGIg?orgId=1

zdave-parity · 2023-10-06T12:06:29Z

Ah ok, I guess that change is required then, my mistake. Thanks!

Responsible for determining the mixnode set for each session.

ordian · 2023-10-07T15:11:20Z

substrate/frame/mixnet/src/lib.rs

+	impl<T: Config> Pallet<T> {
+		/// Register a mixnode for the following session.
+		#[pallet::call_index(0)]
+		#[pallet::weight(1)] // TODO


Does this need a benchmark?

Yes, but this pallet doesn't go to production directly?

Sure, can be done as a follow-up, but our policy is to use tracking issues instead of TODOs :P

Yeah, fair point.

* master: (33 commits) ci: set CI_IMAGE back to (now updated) .ci-unified (#1854) ci: bump ci image to rust 1.73.0 (#1830) Refactor Identity to benchmark v2 (#1838) PVF worker: bump landlock, update ABI docs (#1850) Xcm emulator nits (#1649) Fixes path issue in derive-impl (#1823) upgrade to macro_magic 0.4.3 (#1832) Use safe math when pruning statuses (#1835) remote-ext: fix state download stall on slow connections and reduce memory usage (#1295) Update testnet bootnode dns name (#1712) [FRAME] Warn on unchecked weight witness (#1818) [xcm] Use `Weight::MAX` for `reserve_asset_deposited`, `receive_teleported_asset` benchmarks (#1726) Update bridges subtree (#1803) Check for parent of first ready block being on chain (#1812) Make CheckNonce refuse transactions signed by accounts with no providers (#1578) Fix Asset Hub collator crashing when starting from genesis (#1788) Mixnet integration (#1346) [xcm-emulator] Decouple the `AccountId` type from `AccountId32` (#1458) Treasury spends various asset kinds (#1333) chore: bump zombienter version (#1806) ...

See paritytech#1345, <paritytech/substrate#14207>. This adds all the necessary mixnet components, and puts them together in the "kitchen-sink" node/runtime. The components added are: - A pallet (`frame/mixnet`). This is responsible for determining the current mixnet session and phase, and the mixnodes to use in each session. It provides a function that validators can call to register a mixnode for the next session. The logic of this pallet is very similar to that of the `im-online` pallet. - A service (`client/mixnet`). This implements the core mixnet logic, building on the `mixnet` crate. The service communicates with other nodes using notifications sent over the "mixnet" protocol. - An RPC interface. This currently only supports sending transactions over the mixnet. --------- Co-authored-by: David Emett <dave@sp4m.net> Co-authored-by: Javier Viola <javier@parity.io>

zdave-parity requested review from arkpar and bkchr and removed request for arkpar August 31, 2023 23:34

paritytech-ci requested a review from a team August 31, 2023 23:35

zdave-parity force-pushed the mixnet branch from c1aac56 to 5c10b61 Compare September 1, 2023 14:29

arkpar approved these changes Sep 1, 2023

View reviewed changes

bkchr approved these changes Sep 13, 2023

View reviewed changes

zdave-parity requested a review from a team September 14, 2023 11:42

zdave-parity force-pushed the mixnet branch from cfb405a to f028ab1 Compare October 4, 2023 12:02

zdave-parity added the T0-node This PR/Issue is related to the topic “node”. label Oct 5, 2023

zdave-parity force-pushed the mixnet branch from f028ab1 to 6d9046d Compare October 5, 2023 11:28

zdave-parity mentioned this pull request Oct 5, 2023

Mixnet support paritytech/zombienet#1394

Merged

zdave and others added 7 commits October 6, 2023 13:56

Disconnect peer on unreserving if no free slot for

92e5f79

Mixnet pallet

f512dd9

Responsible for determining the mixnode set for each session.

Mixnet service

9ea2a1d

Mixnet RPC interface

6c49a33

Basic mixnet CLI options

5e356f6

Add mixnet to kitchen sink node/runtime

00b2536

Update Cargo.lock

601d32c

zdave-parity force-pushed the mixnet branch from 6d9046d to 601d32c Compare October 6, 2023 13:06

ordian reviewed Oct 7, 2023

View reviewed changes

Merge branch 'master' into mixnet

c2039cd

zdave-parity merged commit a808a3a into paritytech:master Oct 9, 2023
8 checks passed

ordian mentioned this pull request Nov 3, 2023

Approve multiple candidates with a single signature #1191

Merged

7 tasks

kiltbot mentioned this pull request Nov 7, 2023

[AUTOMATIC] Update Polkadot dependencies from 1.2.0 to 1.3.0 KILTprotocol/kilt-node#575

Closed

ahmadkaouk mentioned this pull request Nov 16, 2023

Update polkadot-sdk from v.1.1.0 to v1.3.0 moonbeam-foundation/moonbeam#2565

Closed

zdave-parity deleted the mixnet branch December 22, 2023 13:48

ordian mentioned this pull request Feb 5, 2024

Add availability-recovery from systematic chunks #1644

Merged

10 tasks

serban300 pushed a commit to serban300/polkadot-sdk that referenced this pull request Apr 8, 2024

edition = "2021" (paritytech#1346)

5d86ec3

serban300 pushed a commit to serban300/polkadot-sdk that referenced this pull request Apr 8, 2024

edition = "2021" (paritytech#1346)

57ba6de

serban300 pushed a commit to serban300/polkadot-sdk that referenced this pull request Apr 8, 2024

edition = "2021" (paritytech#1346)

ef20d9b

serban300 pushed a commit to serban300/polkadot-sdk that referenced this pull request Apr 8, 2024

edition = "2021" (paritytech#1346)

2ee8652

serban300 pushed a commit to serban300/polkadot-sdk that referenced this pull request Apr 9, 2024

edition = "2021" (paritytech#1346)

f9471ee

serban300 pushed a commit to serban300/polkadot-sdk that referenced this pull request Apr 9, 2024

edition = "2021" (paritytech#1346)

fae3b1a

serban300 pushed a commit to serban300/polkadot-sdk that referenced this pull request Apr 9, 2024

edition = "2021" (paritytech#1346)

4360739

serban300 pushed a commit to serban300/polkadot-sdk that referenced this pull request Apr 9, 2024

edition = "2021" (paritytech#1346)

1bec2f7

serban300 pushed a commit to serban300/polkadot-sdk that referenced this pull request Apr 9, 2024

edition = "2021" (paritytech#1346)

5fd7c5a

serban300 pushed a commit to serban300/polkadot-sdk that referenced this pull request Apr 9, 2024

edition = "2021" (paritytech#1346)

800b268

serban300 pushed a commit to serban300/polkadot-sdk that referenced this pull request Apr 10, 2024

edition = "2021" (paritytech#1346)

beead9e

serban300 pushed a commit to serban300/polkadot-sdk that referenced this pull request Apr 10, 2024

edition = "2021" (paritytech#1346)

e5fad20

bkchr pushed a commit that referenced this pull request Apr 10, 2024

edition = "2021" (#1346)

2bf5b23

This was referenced Jun 5, 2024

Update polkadot-sdk from v1.7.0 to v1.11.0 moondance-labs/tanssi#573

Closed

Update polkadot-sdk from v1.10.0 to v1.11.0 moondance-labs/tanssi#577

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mixnet integration #1346

Mixnet integration #1346

zdave-parity commented Aug 31, 2023

zdave-parity commented Aug 31, 2023

arkpar commented Sep 1, 2023 •

edited

Loading

arkpar commented Sep 1, 2023

zdave-parity commented Sep 1, 2023

bkchr Sep 13, 2023

zdave-parity Sep 14, 2023

bkchr Sep 13, 2023

zdave-parity commented Oct 5, 2023

pepoviola commented Oct 6, 2023

zdave-parity commented Oct 6, 2023

pepoviola commented Oct 6, 2023

zdave-parity commented Oct 6, 2023

ordian Oct 7, 2023

bkchr Oct 7, 2023

ordian Oct 7, 2023

bkchr Oct 7, 2023

Mixnet integration #1346

Mixnet integration #1346

Conversation

zdave-parity commented Aug 31, 2023

zdave-parity commented Aug 31, 2023

arkpar commented Sep 1, 2023 • edited Loading

arkpar commented Sep 1, 2023

zdave-parity commented Sep 1, 2023

bkchr Sep 13, 2023

Choose a reason for hiding this comment

zdave-parity Sep 14, 2023

Choose a reason for hiding this comment

bkchr Sep 13, 2023

Choose a reason for hiding this comment

zdave-parity commented Oct 5, 2023

pepoviola commented Oct 6, 2023

zdave-parity commented Oct 6, 2023

pepoviola commented Oct 6, 2023

zdave-parity commented Oct 6, 2023

ordian Oct 7, 2023

Choose a reason for hiding this comment

bkchr Oct 7, 2023

Choose a reason for hiding this comment

ordian Oct 7, 2023

Choose a reason for hiding this comment

bkchr Oct 7, 2023

Choose a reason for hiding this comment

arkpar commented Sep 1, 2023 •

edited

Loading