Update to Substrate master and Parachains V1 (#102)

Overview

This PR is primarily an update of four key Moonbeam dependencies, Substrate, Cumulus, Polkadot, and Frontier. For a while Moonbeam has been using old versions of Substrate, Cumulus, and Polkadot, and backporting Frontier for compatability. This PR updates to the master branch of all of those.1
New Features

Although no new features are added directly to moonbeam with this upgrade, we gain several new features or fixes from upstream.

    Parachains v1 (Polkadot)
    Full multi collator support (Cumulus)
    Weight gas mapping (Frontier)
    More rpc endpoints (Frontier)
    Rename trait Trait to trait Config (Substrate)
    Estimate gas from EVM (Frontier)

Hotfix Status

Two hotfixes were removed. We no longer use a patched Frontier runtime API. Because of the Substrate update, we no longer use a non-standard ref_count type.

Two hotfixes remain, but were moved to Moonbeam as drop-in replacements for their upstream counterparts. This allows us to depend on the stock Frontier, and keep better track of hotfixes because they live in the Moonbeam repo. Specifically the EthApi was moved.This includes two orthogonal patches:

    The backend optimization
    The headers patch

docker/polkadot-relay.Dockerfile

@@ -0,0 +1,57 @@
+# Inspired by Polkadot Dockerfile
+
+FROM phusion/baseimage:0.11 as builder
+LABEL maintainer "alan@purestake.com"
+LABEL description="This is the build stage for Polkadot. Here we create the binary."
+
+ARG PROFILE=release
+ARG POLKADOT_COMMIT=master
+RUN echo "Using polkadot ${POLKADOT_COMMIT}"
+WORKDIR /
+
+# Install OS dependencies
+RUN apt-get update && \
+	apt-get upgrade -y && \
+	apt-get install -y cmake pkg-config libssl-dev git clang
+
+# Grab the Polkadot Code
+# TODO how to grab the correct commit from the lock file?
+RUN git clone https://github.com/paritytech/polkadot
+WORKDIR /polkadot
+RUN git checkout ${POLKADOT_COMMIT}
+
+# Download rust dependencies and build the rust binary
+RUN curl https://sh.rustup.rs -sSf | sh -s -- -y && \
+	export PATH=$PATH:$HOME/.cargo/bin && \
+	scripts/init.sh && \
+	cargo build --$PROFILE --features=real-overseer
+
+# ===== SECOND STAGE ======
+
+FROM phusion/baseimage:0.11
+LABEL maintainer "alan@purestake.com"
+LABEL description="Polkadot for Moonbeam Alphanet Relay Chain"
+ARG PROFILE=release
+COPY --from=builder /polkadot/target/$PROFILE/polkadot /usr/local/bin
+
+RUN mv /usr/share/ca* /tmp && \
+	rm -rf /usr/share/*  && \
+	mv /tmp/ca-certificates /usr/share/ && \
+	rm -rf /usr/lib/python* && \
+	useradd -m -u 1000 -U -s /bin/sh -d /moonbeam moonbeam && \
+	mkdir -p /moonbeam/.local/share/moonbeam && \
+	chown -R moonbeam:moonbeam /moonbeam/.local && \
+	ln -s /moonbeam/.local/share/moonbeam /data && \
+	rm -rf /usr/bin /usr/sbin
+
+USER moonbeam
+
+# 30333 for p2p traffic
+# 9933 for RPC call
+# 9944 for Websocket
+# 9615 for Prometheus (metrics)
+EXPOSE 30333 9933 9944 9615
+
+VOLUME ["/data"]
+
+CMD ["/usr/local/bin/polkadot"]

launch-notes.md

@@ -0,0 +1,208 @@
+# Build
+
+## Compile Polkadot with the real overseer feature
+```bash
+git checkout d7257026
+cargo build --release --features=real-overseer
+./target/release/polkadot --version
+```
+
+## Compile Moonbeam
+```bash
+git checkout a4d257b0
+cargo build --release
+./target/release/moonbase-alphanet --version
+```
+
+# Launch Relay
+
+## Validator Keys
+
+Determining what session keys are needed requires looking at the code in a few places.
+* Rococo Session Key Definition - https://github.com/paritytech/polkadot/blob/master/runtime/rococo/src/lib.rs#L148-L152
+* Substrate Session key codes - https://github.com/paritytech/substrate/blob/master/primitives/core/src/crypto.rs#L1085
+* Polkadot Session key codes - https://github.com/paritytech/polkadot/blob/master/primitives/src/v0.rs#L71
+* Upcoming (but not currently used) Polkadot session keys - https://github.com/paritytech/polkadot/blob/master/primitives/src/v1.rs#L62-L83
+* Observation: We don't need session keys for "acco" and "stak", but they were in the old validator script.
+
+```bash
+# Generate a key and note the mnemonic
+./polkadot-d7257026-real-overseer key generate
+```
+
+For this write up I'll be using these example keys.
+
+### Alfie
+```bash
+# SR25519 - most keys
+$ ./polkadot-d7257026-real-overseer key inspect-key --scheme sr25519 "rail order express dynamic sketch tip mask double cave medal guitar between"
+Secret phrase `rail order express dynamic sketch tip mask double cave medal guitar between` is account:
+  Secret seed:      0xb7c888bf9f01da9a2cb5805d6d6ada744ed22d8a24aa41c509a26268ec701461
+  Public key (hex): 0xc0671bd602df3430ea648f52baef5068f1082ce03e6563677255d163220ddd42
+  Account ID:       0xc0671bd602df3430ea648f52baef5068f1082ce03e6563677255d163220ddd42
+  SS58 Address:     5GQygSBSa7BjEeYiE41Q7uFNMUVfMEuU3bjtPmjZQBUEvgjn
+
+# ED25519 - grandpa
+$ ./polkadot-d7257026-real-overseer key inspect-key --scheme ed25519 "rail order express dynamic sketch tip mask double cave medal guitar between"
+Secret phrase `rail order express dynamic sketch tip mask double cave medal guitar between` is account:
+  Secret seed:      0xb7c888bf9f01da9a2cb5805d6d6ada744ed22d8a24aa41c509a26268ec701461
+  Public key (hex): 0x9a35999189aeac73680dbccc89b18335a545f62045a54c9225105a428976cc16
+  Account ID:       0x9a35999189aeac73680dbccc89b18335a545f62045a54c9225105a428976cc16
+  SS58 Address:     5FYu9sxGFZ15SapDCUJiJy8JXFCDBLv56iLrWrEDeT6BfybQ
+
+```
+
+### Bet
+```bash
+# SR25519 - most keys
+$ ./polkadot-d7257026-real-overseer key inspect-key --scheme sr25519 "planet ill puzzle mirror fog system admit genre subject dance aim limit"
+Secret phrase `planet ill puzzle mirror fog system admit genre subject dance aim limit` is account:
+  Secret seed:      0xa7854968eaa257ed5e87b3dc765feca8e7dadf5322b2d04e6ba60e080164da15
+  Public key (hex): 0x80d42ee5b73818f4cf65d9a4cbe87c990c83af82c9baceb565d85d0f2a3e4807
+  Account ID:       0x80d42ee5b73818f4cf65d9a4cbe87c990c83af82c9baceb565d85d0f2a3e4807
+  SS58 Address:     5Eyd1zj8BSFaAt66y152neBFhZTaQ1TJNoxZYdYkXqqXRf1X
+
+# ED25519 - grandpa
+$ ./polkadot-d7257026-real-overseer key inspect-key --scheme ed25519 "planet ill puzzle mirror fog system admit genre subject dance aim limit"
+Secret phrase `planet ill puzzle mirror fog system admit genre subject dance aim limit` is account:
+  Secret seed:      0xa7854968eaa257ed5e87b3dc765feca8e7dadf5322b2d04e6ba60e080164da15
+  Public key (hex): 0xbd8d705b2742bb237d662b0a60414c9d7cef8f9407f816dac0912cac6b933f13
+  Account ID:       0xbd8d705b2742bb237d662b0a60414c9d7cef8f9407f816dac0912cac6b933f13
+  SS58 Address:     5GMEvWtt8CyxRwZeRAuxDy2rmuzsZAuwCochbHCkQVv1Mfev
+```
+
+Insert Alfie's session keys
+```bash
+./polkadot-d7257026-real-overseer key insert --keystore-path ./alfie/chains/rococo_local_testnet/keystore --base-path alfie --suri "rail order express dynamic sketch tip mask double cave medal guitar between"  --key-type gran --scheme ed25519 && \
+./polkadot-d7257026-real-overseer key insert --keystore-path ./alfie/chains/rococo_local_testnet/keystore --base-path alfie --suri "rail order express dynamic sketch tip mask double cave medal guitar between"  --key-type babe && \
+./polkadot-d7257026-real-overseer key insert --keystore-path ./alfie/chains/rococo_local_testnet/keystore --base-path alfie --suri "rail order express dynamic sketch tip mask double cave medal guitar between"  --key-type imon && \
+./polkadot-d7257026-real-overseer key insert --keystore-path ./alfie/chains/rococo_local_testnet/keystore --base-path alfie --suri "rail order express dynamic sketch tip mask double cave medal guitar between"  --key-type para && \
+./polkadot-d7257026-real-overseer key insert --keystore-path ./alfie/chains/rococo_local_testnet/keystore --base-path alfie --suri "rail order express dynamic sketch tip mask double cave medal guitar between"  --key-type audi
+
+#TODO Should we use different derivation paths for each keypair like Telmo recommended? Probably. But NOT for stash or controller. Session keys are HOT keys.
+```
+
+Repeat similarly for Bet
+```bash
+./polkadot-d7257026-real-overseer key insert --keystore-path ./bet/chains/rococo_local_testnet/keystore --base-path bet --suri "planet ill puzzle mirror fog system admit genre subject dance aim limit"  --key-type gran --scheme ed25519 && \
+./polkadot-d7257026-real-overseer key insert --keystore-path ./bet/chains/rococo_local_testnet/keystore --base-path bet --suri "planet ill puzzle mirror fog system admit genre subject dance aim limit"  --key-type babe && \
+./polkadot-d7257026-real-overseer key insert --keystore-path ./bet/chains/rococo_local_testnet/keystore --base-path bet --suri "planet ill puzzle mirror fog system admit genre subject dance aim limit"  --key-type imon && \
+./polkadot-d7257026-real-overseer key insert --keystore-path ./bet/chains/rococo_local_testnet/keystore --base-path bet --suri "planet ill puzzle mirror fog system admit genre subject dance aim limit"  --key-type para && \
+./polkadot-d7257026-real-overseer key insert --keystore-path ./bet/chains/rococo_local_testnet/keystore --base-path bet --suri "planet ill puzzle mirror fog system admit genre subject dance aim limit"  --key-type audi
+```
+
+`polkadot key insert` was unintuitive to me for several reasons. I've reported these in https://github.com/paritytech/polkadot/issues/2072
+* Specifying `--chain` doesn't seem to have any effect.
+* Specifying `--base-path` alone "works" but puts the keys in the wrong place.
+* Sepcifying `--keystore-path` alone complains that `--base-path` is not specified.
+
+## Chain Spec
+
+Rococo local is known to throw the warning `💸 Chain does not have enough staking candidates to operate. Era Some(0)`. This is normal and harmless.
+
+```bash
+./polkadot-d7257026-real-overseer build-spec --chain rococo-local --disable-default-bootnode > rococo-local-d7257026-real-overseer.json
+```
+
+Insert the custom session keys like so in the chain spec. (TODO as I mentioned above, we should eventually use different offline keys for stash and controller.)
+
+```json
+"palletSession": {
+  "keys": [
+    [
+      "5GQygSBSa7BjEeYiE41Q7uFNMUVfMEuU3bjtPmjZQBUEvgjn",
+      "5GQygSBSa7BjEeYiE41Q7uFNMUVfMEuU3bjtPmjZQBUEvgjn",
+      {
+        "grandpa": "5FYu9sxGFZ15SapDCUJiJy8JXFCDBLv56iLrWrEDeT6BfybQ",
+        "babe": "5GQygSBSa7BjEeYiE41Q7uFNMUVfMEuU3bjtPmjZQBUEvgjn",
+        "im_online": "5GQygSBSa7BjEeYiE41Q7uFNMUVfMEuU3bjtPmjZQBUEvgjn",
+        "parachain_validator": "5GQygSBSa7BjEeYiE41Q7uFNMUVfMEuU3bjtPmjZQBUEvgjn",
+        "authority_discovery": "5GQygSBSa7BjEeYiE41Q7uFNMUVfMEuU3bjtPmjZQBUEvgjn"
+      }
+    ],
+    [
+      "5Eyd1zj8BSFaAt66y152neBFhZTaQ1TJNoxZYdYkXqqXRf1X",
+      "5Eyd1zj8BSFaAt66y152neBFhZTaQ1TJNoxZYdYkXqqXRf1X",
+      {
+        "grandpa": "5GMEvWtt8CyxRwZeRAuxDy2rmuzsZAuwCochbHCkQVv1Mfev",
+        "babe": "5Eyd1zj8BSFaAt66y152neBFhZTaQ1TJNoxZYdYkXqqXRf1X",
+        "im_online": "5Eyd1zj8BSFaAt66y152neBFhZTaQ1TJNoxZYdYkXqqXRf1X",
+        "parachain_validator": "5Eyd1zj8BSFaAt66y152neBFhZTaQ1TJNoxZYdYkXqqXRf1X",
+        "authority_discovery": "5Eyd1zj8BSFaAt66y152neBFhZTaQ1TJNoxZYdYkXqqXRf1X"
+      }
+    ]
+  ]
+},
+```
+
+Finally, bake a raw spec
+
+```bash
+./polkadot-d7257026-real-overseer build-spec --chain rococo-local-d7257026-real-overseer.json --disable-default-bootnode > rococo-local-d7257026-real-overseer-raw.json
+```
+
+## Validator Commands
+
+This version of Polkadot is known to throw the warning `Ran out of free WASM instances`. This harmless, and is [issue #2070](https://github.com/paritytech/polkadot/issues/2070) and addressed by [PR #2069](https://github.com/paritytech/polkadot/pull/2069).
+
+```bash
+# Alfie
+./polkadot-d7257026-real-overseer --chain rococo-local-d7257026-real-overseer-raw.json --validator --base-path ./alfie/
+
+# Bet
+./polkadot-d7257026-real-overseer --chain rococo-local-d7257026-real-overseer-raw.json --validator  --base-path ./bet/ --port 30334
+```
+
+# Launch Parachain
+
+## Export genesis state and wasm
+
+```bash
+./target/release/moonbase-alphanet export-genesis-state --parachain-id 200 > genesis-state
+./target/release/moonbase-alphanet export-genesis-wasm > genesis-wasm
+```
+
+## Launch Collators
+
+Collators don't need session keys yet (They will once we have aura on the parachain). They only differ from each other in port numbers.
+
+```bash
+./target/release/moonbase-alphanet --collator --tmp --parachain-id 200 --port 40335 --ws-port 9946 -- --execution wasm --chain ../polkadot/rococo-local-d7257026-real-overseer-raw.json --port 30335
+./target/release/moonbase-alphanet --collator --tmp --parachain-id 200 --port 40336 --ws-port 9947 -- --execution wasm --chain ../polkadot/rococo-local-d7257026-real-overseer-raw.json --port 30336
+```
+
+## Launch Parachain Full Nodes
+
+Same as the Collators but no `--collator` flag (and different ports)
+
+```bash
+./target/release/moonbase-alphanet --tmp --parachain-id 200 --port 40337 --ws-port 9948 -- --execution wasm --chain ../polkadot/rococo-local-d7257026-real-overseer-raw.json --port 30337
+./target/release/moonbase-alphanet --tmp --parachain-id 200 --port 40338 --ws-port 9949 -- --execution wasm --chain ../polkadot/rococo-local-d7257026-real-overseer-raw.json --port 30338
+```
+
+# Registration Transaction
+
+Here we use the polkadot js tools docker image. The only tag available is `latest` which includes what we need as of 4 Dec 2020. For reference I'm using this image:
+```
+REPOSITORY                                          TAG                 IMAGE ID            CREATED             SIZE
+jacogr/polkadot-js-tools                            latest              755149046430        29 hours ago        664MB
+```
+
+The runtime wasm is too large (~1MB) to be passed directly on the CLI. Instead we use the `--parmas` flag introduced in https://github.com/polkadot-js/tools/pull/91. It requires all the params to be in a single space-separated file, so we make that first.
+
+```bash
+# Create the file that holds all the parameters
+echo -n "200 {\"genesis_head\":\"$(cat genesis-state)\",\"validation_code\":\"" > parachain-config && \
+cat genesis-wasm  >> parachain-config && \
+echo -n "\",\"parachain\":true}" >> parachain-config
+
+# Submit the transaction
+docker run --rm --network=host \
+  -v $(pwd)/parachain-config:/config \
+  jacogr/polkadot-js-tools:latest api \
+    --ws "ws://localhost:9944" \
+    --sudo \
+    --seed "//Alice" \
+    --params /config \
+    tx.parasSudoWrapper.sudoScheduleParaInitialize
+```