diff --git a/docs/dev/agents/agent-development.md b/docs/dev/agents/agent-development.md deleted file mode 100644 index b15342e..0000000 --- a/docs/dev/agents/agent-development.md +++ /dev/null @@ -1,83 +0,0 @@ ---- -title: Agent Development -lang: en-US ---- - -# Developing the Agents - -## Configuration - -Agents read settings from the config files and/or env. - -Config files are loaded from `rust/config/default` unless specified otherwise. Currently deployment config directories are labeled by the timestamp at which they were deployed - -Configuration key/value pairs are loaded in the following order, with later sources taking precedence: - -1. The config file specified by the `RUN_ENV` and `BASE_CONFIG` env vars. `$RUN_ENV/$BASE_CONFIG` -2. The config file specified by the `RUN_ENV` env var and the agent's name. `$RUN_ENV/{agent}-partial.json`. - E.g. `$RUN_ENV/updater-partial.json` -3. Configuration env vars with the prefix `OPT_BASE` intended to be shared by multiple agents in the same environment - E.g. `export OPT_BASE_REPLICAS_KOVAN_DOMAIN=3000` -4. Configuration env vars with the prefix `OPT_{agent name}` intended to be used by a specific agent. - E.g. `export OPT_KATHY_CHAT_TYPE="static message"` - -## Building an Agent for Development - -For contributing to the Rust codebase, it is advantageous and preferable to build agents using your host dev environment. As mentioned in the previous section, configuration precedence is your friend here. You can specify the base config json to use, and then override variables via the environment. - -Below is a sample env file with appropriate variables to run an agent instance against the development environment. - -Note: You will need to fetch dev keys (or generate your own via a contract deployment) for this to work properly. - -``` -RUN_ENV=1625169020727 -OPT_BASE_TRACING_LEVEL=info - -OPT_UPDATER_UPDATER_KEY= - -OPT_UPDATER_DB=updaterdb -OPT_RELAYER_DB=relayerdb -OPT_KATHY_DB=kathydb - -OPT_KATHY_SIGNERS_ALFAJORES_KEY= -OPT_UPDATER_SIGNERS_ALFAJORES_KEY= -OPT_RELAYER_SIGNERS_ALFAJORES_KEY= -OPT_PROCESSOR_SIGNERS_ALFAJORES_KEY= - -OPT_KATHY_SIGNERS_KOVAN_KEY= -OPT_UPDATER_SIGNERS_KOVAN_KEY= -OPT_RELAYER_SIGNERS_KOVAN_KEY= -OPT_PROCESSOR_SIGNERS_KOVAN_KEY= -``` - -Lets walk through the variables here: - -`RUN_ENV` - Specifies the config folder to load configuration from, defaults to `default`. -`OPT_BASE_TRACING_LEVEL` - Specifies the log level the agents should boot up with. -`OPT_UPDATER_UPDATER_KEY` - The Updater attestation key. - - - -For a full list of potentially useful common environment variables, check out the Agent Helm Chart's [ConfigMap](https://github.com/nomad-xyz/nomad-monorepo/blob/main/rust/helm/nomad-agent/templates/configmap.yaml#L8-L34) - -Agents also have role-specific environment variables in their StatefulSet definitions: -- [Updater](https://github.com/nomad-xyz/nomad-monorepo/blob/main/rust/helm/nomad-agent/templates/updater-statefulset.yaml#L54-L89) -- [Relayer](https://github.com/nomad-xyz/nomad-monorepo/blob/main/rust/helm/nomad-agent/templates/relayer-statefulset.yaml#L54-L74) -- [Processor](https://github.com/nomad-xyz/nomad-monorepo/blob/main/rust/helm/nomad-agent/templates/processor-statefulset.yaml#L54-L74) -- [Kathy](https://github.com/nomad-xyz/nomad-monorepo/blob/main/rust/helm/nomad-agent/templates/kathy-statefulset.yaml#L54-L74) - -To run an agent, you can use the following command: -`BASE_CONFIG=kovan_config.json env $(cat ../tmp.env | xargs) cargo run --bin ` - -This will build the codebase and run the specified `` binary using the provided environment variables. - -## Production Builds - -It is important when making changes to the Rust codebase, to ensure the Docker build used in production environments still works. You can check this automatically in CI as it is built on every PR ([see docker workflow here](https://github.com/nomad-xyz/nomad-monorepo/blob/main/.github/workflows/docker.yml)), however you can check it much faster usually by attempting to build it locally. - -You can build the docker image by running the following script in the `rust` directory: - -`./build.sh latest` - -If that goes smoothly, you can rest assured it will most likely also work in CI. diff --git a/docs/dev/agents/agent-operations.md b/docs/dev/agents/agent-operations.md index a68b135..60da54a 100644 --- a/docs/dev/agents/agent-operations.md +++ b/docs/dev/agents/agent-operations.md @@ -5,6 +5,39 @@ lang: en-US # Agent Operations +## Agent Overview + +Agents are the off-chain component of the Nomad system, their purpose is to ferry data between the constituent `Domains` that make up the Nomad network. Agents are written in Rust and globally share the following properties: + +- Agents are modeled after a [12-factor app](https://12factor.net/) + - Deployable via a Docker container + - Configurable via Environment Variables (or a configuration file if necessary) + - Suitable for use in Infrastructure automation systems +- Agents funded Transaction Signers to pay for gas when sending transactions +- Agents are (optionally) deployable via the [nomad-bridge Helm Chart](https://github.com/nomad-xyz/helm-charts/tree/main/charts/nomad-bridge) + +There are 4 production agent roles, with an optional additional agent that is only suitable for use in Development environments: + +- **Updater** + - The `Updater` is the most important Agent in regards to _Liveness_. + - The `Updater` issues attestations of merkle root transitions and is bonded to disincentivize fraud. + - [Link to Updater Docs](./agent-updater.md) +- **Relayer** + - Relays root transitions from home to replica(s) + - [Link to Relayer Docs](./agent-relayer.md) +- **Processor** + - The `Processor` is entirely optional, however it implements the property of _subsidized channels_. The `Processor` polls for messages that have reached the expiration of their fraud proof window, and claims them on behalf of the user, obviating the need for the user to do so on their own or pay the required gas. + - The `Processor` can be tuned to look for a subset of messages to process, such that it can be deployed by a particuar xApp operator who would like to subsidize transactions on behalf of the users of their xApp _exclusively_. + - The Nomad Core Team operates processors for most Domains with execution environments that are affordable, with the main notable unsubsidized Domain being Ethereum. + - [Link to Processor Docs](./agent-processor.md) +- **Watcher** + - The `Watcher` is the most importent Agent in regards to _Safety_. + - The `Watcher` _watches_ a designated Home contract and its corresponding replicas on remote domains. It keeps an internal state of root transitions, and if an improper update is detected, it proves it to the protocol -- slashing the Updater and preventing further fraudulant updates from being passed. + - [Link to Watcher Docs](./agent-watcher.md) +- **Kathy** + - `Kathy` (aka Chatty Kathy) is a development agent which sends Nomad messages on an interval in order to test bridge channels. + - It is not advised that you deploy `Kathy` to a production environment as it will spend your hard-earned money with no regards. + ## Deployment Environments There will exist several logical deployments of Nomad to enable us to test new code/logic before releasing it to Mainnet. Each environment encompasses the various Home/Replica contracts deployed to many blockchains, as well as the agent deployments and their associated account secrets. @@ -13,60 +46,60 @@ The environments have various purposes and can be described as follows: ### Development -Purpose: Allows us to test changes to contracts and agents. *Bugs should be found here.* +Purpose: Allows us to test changes to contracts and agents. _Bugs should be found here._ - Deployed against testnets - Agent Accounts: HexKeys stored in a secret manager for ease of rotation/access - Agent Infrastructure: Nomad core team will operate agent infrastructure for this. - Node Infrastructure: Forno/Infura - Agent Deployments: Automatic, continuous deployment -- Contract Deployments: Automatic, with human intervention required for updating the **upgradeBeacon**. +- Contract Deployments: Automatic, with human intervention required for updating the [UpgradeBeacon](../upgrade-setup.md). **Current Dev Contract Deployment:** -[development](https://github.com/nomad-xyz/nomad-monorepo/rust/config/development/) +[Development](https://github.com/nomad-xyz/rust/tree/main/configuration/configs/development.json) ### Staging -Purpose: Allows us to test the full-stack integration, specifically surrounding the KMS access control and federated secret management. *Issues with process should be found here.* +Purpose: Allows us to test the full-stack integration, specifically surrounding the KMS access control and federated secret management. _Issues with process should be found here._ - Deployed against testnets, mirrors Mainnet deployment. - Agent Accounts: KMS-provisioned keys - Agent Infrastructure: Agent operations will be decentralized - Node Infrastructure: Node infrastructure will be decentralized - Agent Deployments: Determined by whoever is running the agents -- Contract Deployments: Automatic, with human intervention required for updating the **upgradeBeacon**. +- Contract Deployments: Automatic, with human intervention required for updating the [UpgradeBeacon](../upgrade-setup.md). **Current Staging Contract Deployment:** -[staging](https://github.com/nomad-xyz/nomad-monorepo/rust/config/staging/) +[Staging](https://github.com/nomad-xyz/rust/tree/main/configuration/configs/staging.json) ### Production -Purpose: Where the magic happens, **things should not break here.** +Purpose: Where the magic happens, **things should not break here.** - Deployed against Mainnets - Agent Accounts: KMS-provisioned keys - Agent Infrastructure: Agent operations will be decentralized - Node Infrastructure: Node infrastructure will be decentralized - Agent Deployments: Determined by whoever is running the agents -- Contract Deployments: ***Manual*** - Existing tooling can be used, but deploys will be gated and require approval as contract deployments are expensive on Mainnet. +- Contract Deployments: **_Manual_** - Existing tooling can be used, but deploys will be gated and require approval as contract deployments are expensive on Mainnet. **Current Production Contract Deployment:** -[mainnet](https://github.com/nomad-xyz/nomad-monorepo/rust/config/mainnet/) +[Production](https://github.com/nomad-xyz/rust/tree/main/configuration/configs/production.json) ## Key Material -Keys for Staging and Production environments will be stored in AWS KMS, which is a highly flexible solution in terms of granting access. It guarantees nobody will ever have access to the key material itself, while still allowing granular permissions over access to remote signing. +Keys for Staging and Production environments will be stored in AWS KMS, which is a highly flexible solution in terms of granting access. It guarantees nobody will ever have access to the key material itself, while still allowing granular permissions over access to remote signing. At the outset, the Nomad team will have full control over agent keys, and any contracted party will simply be granted access through existing IAM tooling/roles. ### Provision KMS Keys -There exists a script in the monorepo [(`rust/provision_kms_keys.py`)](https://github.com/nomad-xyz/nomad-monorepo/blob/main/rust/provision_kms_keys.py) that facilitates KMS key provisioning for agent roles. +There exists a script in the Rust repo [`provision_kms_keys.py`](https://github.com/nomad-xyz/rust/blob/main/provision_kms_keys.py) that facilitates KMS key provisioning for agent roles. -The script will produce a single set of keys per "environment." Where an __environment__ is a logical set of smart contract deployments, as documented [here](#deployment-environments). By default there are two environments configured, `staging` and `production`. +The script will produce a single set of keys per "environment." Where an **environment** is a logical set of smart contract deployments, as documented [here](#deployment-environments). By default there are two environments configured, `staging` and `production`. #### Keys Explained @@ -76,11 +109,12 @@ One signer key should be provisioned for each agent per-network. These keys are **Attestation Signers** -One additional key is provisioned for both the Watcher and Updater Agents. The Updater uses its key to sign updates to its assigned Home contract, while the Watcher uses its key to sign fraud proofs when it observes the Updater commiting fraud. +One additional key is provisioned for both the Watcher and Updater Agents. The Updater uses its key to sign updates to its assigned Home contract, while the Watcher uses its key to sign fraud proofs when it observes the Updater commiting fraud. + +Note: Attestation signer addresses are used as input to the contract deployment process. They can be configured in the `nomad-deploy` package [like so](https://github.com/nomad-xyz/nomad-monorepo/blob/main/typescript/nomad-deploy/config/testnets/kovan.ts#L28-L30). -Note: Attestation signer addresses are used as input to the contract deployment process. They can be configured in the `nomad-deploy` package [like so](https://github.com/nomad-xyz/nomad-monorepo/blob/main/typescript/nomad-deploy/config/testnets/kovan.ts#L28-L30). +You may configure the script to generate arbitrary signer keys on a per-environment basis. -You may configure the script to generate arbitrary signer keys on a per-environment basis. ```python # Agent Keys agent_keys = { @@ -104,7 +138,8 @@ agent_keys = { } ``` -Additionally, the supported networks for each environment are configured below. +Additionally, the supported networks for each environment are configured below. + ```python networks = { "production": [ @@ -114,21 +149,20 @@ networks = { ], "staging": [ "moonbasealpha", - "kovan" + "kovan" ] } ``` - #### Run the Key Provisioning Script ```bash -AWS_ACCESS_KEY_ID=accesskey AWS_SECRET_ACCESS_KEY=secretkey python3 provision_kms_keys.py +AWS_ACCESS_KEY_ID=accesskey AWS_SECRET_ACCESS_KEY=secretkey python3 provision_kms_keys.py ``` -If the required keys are not present, the script will generate them. If they keys _are_ present, their information will be fetched and displayed non-destructively. +If the required keys are not present, the script will generate them. If they keys _are_ present, their information will be fetched and displayed non-destructively. -Upon successful operation, the script will output a table of the required keys, their ARNs, ETH addresses (for funding the accounts), and their regions. +Upon successful operation, the script will output a table of the required keys, their ARNs, ETH addresses (for funding the accounts), and their regions. #### Provision IAM Policies and Users @@ -148,6 +182,7 @@ The following sequence describes how to set up IAM policies staging and producti - kms-admin - Save IAM credential CSV - Create staging signer policies + - staging-processor-signer - staging-relayer-signer - staging-updater-signer @@ -155,25 +190,22 @@ The following sequence describes how to set up IAM policies staging and producti - With the following policy, modified appropriately: ```json - { - "Version": "2012-10-17", - "Statement": [ - { - "Sid": "NomadStagingPolicy", - "Effect": "Allow", - "Action": [ - "kms:GetPublicKey", - "kms:Sign" - ], - "Resource": "arn:aws:kms:*:11111111111:key/*", - "Condition": { - "ForAnyValue:StringLike": { - "kms:ResourceAliases": "alias/staging-processor*" - } + { + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "NomadStagingPolicy", + "Effect": "Allow", + "Action": ["kms:GetPublicKey", "kms:Sign"], + "Resource": "arn:aws:kms:*:11111111111:key/*", + "Condition": { + "ForAnyValue:StringLike": { + "kms:ResourceAliases": "alias/staging-processor*" } } - ] - } + } + ] + } ``` - production-processor-signer @@ -183,61 +215,58 @@ The following sequence describes how to set up IAM policies staging and producti - With the following policy, modified appropriately: ```json - { - "Version": "2012-10-17", - "Statement": [ - { - "Sid": "NomadProductionPolicy", - "Effect": "Allow", - "Action": [ - "kms:GetPublicKey", - "kms:Sign" - ], - "Resource": "arn:aws:kms:*:11111111111:key/*", - "Condition": { - "ForAnyValue:StringLike": { - "kms:ResourceAliases": "alias/production-processor*" - } + { + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "NomadProductionPolicy", + "Effect": "Allow", + "Action": ["kms:GetPublicKey", "kms:Sign"], + "Resource": "arn:aws:kms:*:11111111111:key/*", + "Condition": { + "ForAnyValue:StringLike": { + "kms:ResourceAliases": "alias/production-processor*" } } - ] - } - ``` + } + ] + } + ``` - Create kms-admin policy ```json - { - "Version": "2012-10-17", - "Statement": [ - { - "Sid": "KMSAdminPolicy", - "Effect": "Allow", - "Action": [ - "kms:DescribeCustomKeyStores", - "kms:ListKeys", - "kms:DeleteCustomKeyStore", - "kms:GenerateRandom", - "kms:UpdateCustomKeyStore", - "kms:ListAliases", - "kms:DisconnectCustomKeyStore", - "kms:CreateKey", - "kms:ConnectCustomKeyStore", - "kms:CreateCustomKeyStore" - ], - "Resource": "*" - }, - { - "Sid": "VisualEditor1", - "Effect": "Allow", - "Action": "kms:*", - "Resource": [ - "arn:aws:kms:*:756467427867:alias/*", - "arn:aws:kms:*:756467427867:key/*" - ] - } - ] - } + { + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "KMSAdminPolicy", + "Effect": "Allow", + "Action": [ + "kms:DescribeCustomKeyStores", + "kms:ListKeys", + "kms:DeleteCustomKeyStore", + "kms:GenerateRandom", + "kms:UpdateCustomKeyStore", + "kms:ListAliases", + "kms:DisconnectCustomKeyStore", + "kms:CreateKey", + "kms:ConnectCustomKeyStore", + "kms:CreateCustomKeyStore" + ], + "Resource": "*" + }, + { + "Sid": "VisualEditor1", + "Effect": "Allow", + "Action": "kms:*", + "Resource": [ + "arn:aws:kms:*:756467427867:alias/*", + "arn:aws:kms:*:756467427867:key/*" + ] + } + ] + } ``` - Create IAM groups @@ -251,23 +280,24 @@ The following sequence describes how to set up IAM policies staging and producti ## Funding Addresses -Each agent should be configured with a unique wallet to be used to signing transactions and paying gas. In order to automate the process of monitoring and topping up agent wallets, the Nomad core team built a CLI tool called [The Keymaster](the-keymaster.md). +Each agent should be configured with a unique wallet to be used to signing transactions and paying gas. In order to automate the process of monitoring and topping up agent wallets, the Nomad core team built a CLI tool called [The Keymaster](the-keymaster.md). -The Keymaster, upon configuration, enables the manual one-off topping up of agent wallets on an arbitrary number of netorks. Additionally, it is capable of running this functionality as a service, topping up accounts on an interval and exposing prometheus metrics about the addresses it is monitoring for use in dashboards. +The Keymaster, upon configuration, enables the manual one-off topping up of agent wallets on an arbitrary number of netorks. Additionally, it is capable of running this functionality as a service, topping up accounts on an interval and exposing prometheus metrics about the addresses it is monitoring for use in dashboards. ## Self-Service Proofs in S3 -In order to facilitate users processing their own proofs in the GUI, agents (specifically the Processor), have the functionality to upload raw proofs to an AWS S3 bucket. In the default configuration, the bucket is publicly accessible and allows end-users to fetch them via the GUI and submit them in a transaction to the apropriate blockchain. +In order to facilitate users processing their own proofs in the GUI, agents (specifically the Processor), have the functionality to upload raw proofs to an AWS S3 bucket. In the default configuration, the bucket is publicly accessible and allows end-users to fetch them via the GUI and submit them in a transaction to the apropriate blockchain. -### Pre-Requisites -- AWS Account +### Pre-Requisites + +- AWS Account - Agent Infrastructure -### Bucket Setup +### Bucket Setup Setup is simple, create a bucket in your desired region via the AWS UI, ensuring to uncheck "Block Public Access" as the desired outcome is for the contents of this bucket to be publicly accessible on the internet. -Use the following bucket policy to enable public access to `s3:getObject` in your newly created bucket: +Use the following bucket policy to enable public access to `s3:getObject` in your newly created bucket: ``` { @@ -287,9 +317,9 @@ Use the following bucket policy to enable public access to `s3:getObject` in you ### AWS IAM Permissions -NOTE: Currently, Agents only support a single AWS key for both KMS Signing and S3 Upload. This enforces a non-functional requirement that the S3 bucket proofs are uploaded to and the KMS keys used to sign transactions are in the same logical AWS account. +NOTE: Currently, Agents only support a single AWS key for both KMS Signing and S3 Upload. This enforces a non-functional requirement that the S3 bucket proofs are uploaded to and the KMS keys used to sign transactions are in the same logical AWS account. -Create an IAM policy that allows a user to write to the S3 bucket you created in the previous step: +Create an IAM policy that allows a user to write to the S3 bucket you created in the previous step: ``` { @@ -321,19 +351,19 @@ Attach this policy to an IAM user and provision/download AWS keys. ### Configuring the Agent -The Processor agent has special config for S3 proof indexing, located in the code [here](https://github.com/nomad-xyz/nomad-monorepo-archive/blob/main/rust/agents/processor/src/settings.rs#L9-L24). +The Processor agent has special config for S3 proof indexing, located in the code [here](https://github.com/nomad-xyz/nomad-monorepo-archive/blob/main/rust/agents/processor/src/settings.rs#L9-L24). -Buckets can be configured at agent runtime via the following environment variables: +Buckets can be configured at agent runtime via the following environment variables: `OPT_PROCESSOR_S3_BUCKET` -> Name of the bucket. Ex. `nomadxyz-development-proofs` `OPT_PROCESSOR_S3_REGION` -> AWS region the bucket lives in. Ex. `us-west-1` -If you are using the helm chart, these values can be passed via [`values.yaml`](https://github.com/nomad-xyz/nomad-monorepo/blob/main/rust/helm/nomad-agent/values.yaml#L147-L149) like so: +If you are using the helm chart, these values can be passed via [`values.yaml`](https://github.com/nomad-xyz/nomad-monorepo/blob/main/rust/helm/nomad-agent/values.yaml#L147-L149) like so: ``` -processor: +processor: ... - s3Proofs: + s3Proofs: bucket: nomadxyz-development-proofs region: us-west-1 -``` \ No newline at end of file +``` diff --git a/docs/dev/agents/agent-processor.md b/docs/dev/agents/agent-processor.md new file mode 100644 index 0000000..e69de29 diff --git a/docs/dev/agents/agent-relayer.md b/docs/dev/agents/agent-relayer.md new file mode 100644 index 0000000..e69de29 diff --git a/docs/dev/agents/agent-updater.md b/docs/dev/agents/agent-updater.md new file mode 100644 index 0000000..e69de29 diff --git a/docs/dev/agents/agent-watcher.md b/docs/dev/agents/agent-watcher.md new file mode 100644 index 0000000..e69de29 diff --git a/docs/dev/agents/index.md b/docs/dev/agents/index.md index 748f718..e61705c 100644 --- a/docs/dev/agents/index.md +++ b/docs/dev/agents/index.md @@ -5,7 +5,7 @@ lang: en-US # Agent Resources -- [Agent Development](./agent-development.md) - How to Contribute to the Agent Codebase -- [Agent Operations](./agent-operations.md) - How to Run Agents in Production +- [Running Agents](./running-agents.md) - How to Configure/Run Agents +- [Agent Operations](./agent-operations.md) - How to Operate Agents in Production - [Agent Debugging](./debug-agents.md) - Runbook for Debugging Agent Regressions - [The Keymaster](./the-keymaster.md) - Tooling for Agent Wallets diff --git a/docs/dev/agents/running-agents.md b/docs/dev/agents/running-agents.md new file mode 100644 index 0000000..51a271e --- /dev/null +++ b/docs/dev/agents/running-agents.md @@ -0,0 +1,140 @@ +--- +title: Running Agents +lang: en-US +--- + +# Running Agents + +## Overview + +Agents read settings from a mix of our public [JSON configuration files](https://github.com/nomad-xyz/rust/tree/main/configuration/configs) and private environment variables. + +Our hosted JSON configs provide public network info such as contract addresses and chain finality settings. Environment variables specify what networks you want to run the agent against as well as secrets such as signer keys or private RPC endpoints. + +## Configuring an Agent + +To configure an agent, you must populate the proper environment variables. The key fields one must specify are: + +- **Run Environment** + - `RUN_ENV`: Development, staging, or production +- **Agent Home** + - `AGENT_HOME_NAME`: What home the agent is running against +- **Agent Replicas** + + - Specify networks: + - `AGENT_REPLICA_0_NAME`, `AGENT_REPLICA_1_NAME`, `AGENT_REPLICA_2_NAME`, etc... + - What replica(s) the agent will run against + - Default to all connected networks: + - `AGENT_REPLICAS_ALL` + - Expects all connected replicas if `true` + - Expects specified networks if `false` or not set + +- **RPC Info** + - Network-specific: + - `{network}_RPCSTYLE`: What RPC style `network` is; "ethereum" for all EVM chains + - `{network}_CONNECTION_URL`: RPC endpoint url + - Default: + - `DEFAULT_RPCSTYLE`: Default rpc style for any network not explicitly configured +- **Transaction Submission Info** + - Network-specific: + - Transaction Submission Type: + - `{network}_SUBMITTER_TYPE` + - `local` for local signing/submitting + - `gelato` if you are integrated with Gelato Relay + - Local Submission: + - Transaction signer key: + - Hex key: + - `{network}_TXSIGNER_KEY` + - Raw 0x-prefixed hex key + - AWS Key: + - `{network}_TXSIGNER_ID` + - AWS key id + - Gelato Submission (ignore if you do not plan on using Gelato Relay): + - Sponsor signer: + - Hex key: + - `{network}_GELATO_SPONSOR_KEY` + - Raw 0x-prefixed hex key + - AWS Key: + - `{network}_GELATO_SPONSOR_ID` + - AWS key id + - Fee token + - `{network}_GELATO_SPONSOR_FEETOKEN` + - 0x-prefixed token contract address + - Default: + - Default for any network not explicitly configured + - Same as network-specific (above) but replacing specific `{network}` with `DEFAULT` + - Example: + - `DEFAULT_SUBMITTER_TYPE=local` + - `DEFAULT_TXSIGNER_ID=some_aws_id` + - All networks use `local` transaction submission with the default txsigner key +- **Attestation Signer (optional)** + - Required _only_ for updater and watcher + - Hex key: + - `ATTESTATION_SIGNER_KEY` + - Raw 0x-prefixed hex key + - AWS Key: + - `ATTESTATION_SIGNER_ID` + - AWS key id + +
+ +Note that default values are only used if a network-specific value is not provided. In other words, network-specific values override the default if both are provided. + +For more info on our different run environments and key configuration/provisioning, please refer to our [agents operations page](./agent-operations.md). + +You can see an example .env file below: + +``` +# Only runs agent for Ethereum <> Moonbeam channel (production) +RUN_ENV=production +AGENT_HOME_NAME=ethereum +AGENT_REPLICA_0_NAME=moonbeam + +# can provide default rpc style for all networks, or specify network specific +# network-specific values always override the default +DEFAULT_RPCSTYLE=ethereum +ETHEREUM_RPCSTYLE=ethereum +MOONBEAM_RPCSTYLE=ethereum + +# provide network-specific RPC endpoints +ETHEREUM_CONNECTION_URL=https://main-light.eth.linkpool.io/ +MOONBEAM_CONNECTION_URL=https://rpc.api.moonbeam.network + +# we will default to local transaction signing/submission +DEFAULT_SUBMITTER_TYPE=local + +# can provide tx signer as hex key (for ethereum) or aws key (for moonbeam) +# again, default tx signer is overriden by network-specifics +DEFAULT_TXSIGNER_KEY=0x1111111111111111111111111111111111111111111111111111111111111111 +ETHEREUM_TXSIGNER_KEY=0x1111111111111111111111111111111111111111111111111111111111111111 +MOONBEAM_TXSIGNER_ID=dummy_id + +# can provide attestation signer as aws or hex key +ATTESTATION_SIGNER_ID=dummy_id +``` + +If you would like to configure an agent to run against all connected networks (against all replicas the home is connected to), see [this example](https://github.com/nomad-xyz/rust/blob/main/fixtures/env.test). For more examples of .env files, see our [test fixtures folder](https://github.com/nomad-xyz/rust/tree/main/fixtures). + +## Running Agent + +Once you have populated a .env file, running an agent is as simple as running the following command: + +`env $(cat .env | xargs) cargo run --bin ` + +This will build the codebase and run the specified `` binary (updater, relayer, processor, or watcher) using the provided environment variables. + +## Agents Release Process + +Our release process follows a monthly cadence. We follow [Semantic Versioning](https://semver.org/), where breaking changes constitute changes that break agent configuration compatibility. + +We manage releases through GitHub. You can find new per-agent releases [here](https://github.com/nomad-xyz/rust/releases). + +## Production Builds + +When making changes to the Rust codebase, it is important to ensure the Docker build used in production environments still works. You can check this automatically in CI as it is built on every PR ([see docker workflow here](https://github.com/nomad-xyz/rust/blob/main/.github/workflows/docker.yml)), however you can check it much faster usually by attempting to build it locally. + +You can build the docker image by running the following script in the `rust` directory: + +`./build.sh latest` + +If that goes smoothly, you can rest assured it will most likely also work in CI. diff --git a/package-lock.json b/package-lock.json index 3c7db48..92a66b5 100644 --- a/package-lock.json +++ b/package-lock.json @@ -360,6 +360,24 @@ "url": "https://github.com/sponsors/ljharb" } }, + "node_modules/js-tokens": { + "version": "4.0.0", + "resolved": "https://registry.npmjs.org/js-tokens/-/js-tokens-4.0.0.tgz", + "integrity": "sha512-RdJUflcE3cUzKiMqQgsCu06FPu9UdIJO0beYbPhHN4k6apgJtifcoCtT9bcxOpYBtpD2kCM6Sbzg4CausW/PKQ==", + "peer": true + }, + "node_modules/loose-envify": { + "version": "1.4.0", + "resolved": "https://registry.npmjs.org/loose-envify/-/loose-envify-1.4.0.tgz", + "integrity": "sha512-lyuxPGr/Wfhrlem2CL/UcnUc1zcqKAImBDzukY7Y5F/yQiNdko6+fRLevlw1HgMySw7f611UIY408EtxRSoK3Q==", + "peer": true, + "dependencies": { + "js-tokens": "^3.0.0 || ^4.0.0" + }, + "bin": { + "loose-envify": "cli.js" + } + }, "node_modules/magic-string": { "version": "0.25.7", "license": "MIT", @@ -377,6 +395,15 @@ "node": "^10 || ^12 || ^13.7 || ^14 || >=15.0.1" } }, + "node_modules/object-assign": { + "version": "4.1.1", + "resolved": "https://registry.npmjs.org/object-assign/-/object-assign-4.1.1.tgz", + "integrity": "sha512-rJgTQnkUnH1sFw8yT6VSU3zD3sWmu6sZhIseY8VX+GRu3P6F7Fu+JNDoXfklElbLJSnc3FUQHVe4cU5hj+BcUg==", + "peer": true, + "engines": { + "node": ">=0.10.0" + } + }, "node_modules/path-parse": { "version": "1.0.7", "license": "MIT" @@ -413,6 +440,52 @@ "version": "1.25.0", "license": "MIT" }, + "node_modules/prop-types": { + "version": "15.8.1", + "resolved": "https://registry.npmjs.org/prop-types/-/prop-types-15.8.1.tgz", + "integrity": "sha512-oj87CgZICdulUohogVAR7AjlC0327U4el4L6eAvOqCeudMDVU0NThNaV+b9Df4dXgSP1gXMTnPdhfe/2qDH5cg==", + "peer": true, + "dependencies": { + "loose-envify": "^1.4.0", + "object-assign": "^4.1.1", + "react-is": "^16.13.1" + } + }, + "node_modules/react": { + "version": "16.14.0", + "resolved": "https://registry.npmjs.org/react/-/react-16.14.0.tgz", + "integrity": "sha512-0X2CImDkJGApiAlcf0ODKIneSwBPhqJawOa5wCtKbu7ZECrmS26NvtSILynQ66cgkT/RJ4LidJOc3bUESwmU8g==", + "peer": true, + "dependencies": { + "loose-envify": "^1.1.0", + "object-assign": "^4.1.1", + "prop-types": "^15.6.2" + }, + "engines": { + "node": ">=0.10.0" + } + }, + "node_modules/react-dom": { + "version": "16.14.0", + "resolved": "https://registry.npmjs.org/react-dom/-/react-dom-16.14.0.tgz", + "integrity": "sha512-1gCeQXDLoIqMgqD3IO2Ah9bnf0w9kzhwN5q4FGnHZ67hBm9yePzB5JJAIQCc8x3pFnNlwFq4RidZggNAAkzWWw==", + "peer": true, + "dependencies": { + "loose-envify": "^1.1.0", + "object-assign": "^4.1.1", + "prop-types": "^15.6.2", + "scheduler": "^0.19.1" + }, + "peerDependencies": { + "react": "^16.14.0" + } + }, + "node_modules/react-is": { + "version": "16.13.1", + "resolved": "https://registry.npmjs.org/react-is/-/react-is-16.13.1.tgz", + "integrity": "sha512-24e6ynE2H+OKt4kqsOvNd8kBpV65zoxbA4BVsEOB3ARVWQki/DHzaUoC5KuON/BiccDaCCTZBuOcfZs70kR8bQ==", + "peer": true + }, "node_modules/resolve": { "version": "1.20.0", "license": "MIT", @@ -437,6 +510,16 @@ "fsevents": "~2.3.2" } }, + "node_modules/scheduler": { + "version": "0.19.1", + "resolved": "https://registry.npmjs.org/scheduler/-/scheduler-0.19.1.tgz", + "integrity": "sha512-n/zwRWRYSUj0/3g/otKDRPMh6qv2SYMWNq85IEa8iZyAv8od9zDYpGSnpBEjNgcMNq6Scbu5KfIPxNF72R/2EA==", + "peer": true, + "dependencies": { + "loose-envify": "^1.1.0", + "object-assign": "^4.1.1" + } + }, "node_modules/source-map": { "version": "0.6.1", "license": "BSD-3-Clause", @@ -789,6 +872,21 @@ "has": "^1.0.3" } }, + "js-tokens": { + "version": "4.0.0", + "resolved": "https://registry.npmjs.org/js-tokens/-/js-tokens-4.0.0.tgz", + "integrity": "sha512-RdJUflcE3cUzKiMqQgsCu06FPu9UdIJO0beYbPhHN4k6apgJtifcoCtT9bcxOpYBtpD2kCM6Sbzg4CausW/PKQ==", + "peer": true + }, + "loose-envify": { + "version": "1.4.0", + "resolved": "https://registry.npmjs.org/loose-envify/-/loose-envify-1.4.0.tgz", + "integrity": "sha512-lyuxPGr/Wfhrlem2CL/UcnUc1zcqKAImBDzukY7Y5F/yQiNdko6+fRLevlw1HgMySw7f611UIY408EtxRSoK3Q==", + "peer": true, + "requires": { + "js-tokens": "^3.0.0 || ^4.0.0" + } + }, "magic-string": { "version": "0.25.7", "requires": { @@ -798,6 +896,12 @@ "nanoid": { "version": "3.1.30" }, + "object-assign": { + "version": "4.1.1", + "resolved": "https://registry.npmjs.org/object-assign/-/object-assign-4.1.1.tgz", + "integrity": "sha512-rJgTQnkUnH1sFw8yT6VSU3zD3sWmu6sZhIseY8VX+GRu3P6F7Fu+JNDoXfklElbLJSnc3FUQHVe4cU5hj+BcUg==", + "peer": true + }, "path-parse": { "version": "1.0.7" }, @@ -818,6 +922,46 @@ "prismjs": { "version": "1.25.0" }, + "prop-types": { + "version": "15.8.1", + "resolved": "https://registry.npmjs.org/prop-types/-/prop-types-15.8.1.tgz", + "integrity": "sha512-oj87CgZICdulUohogVAR7AjlC0327U4el4L6eAvOqCeudMDVU0NThNaV+b9Df4dXgSP1gXMTnPdhfe/2qDH5cg==", + "peer": true, + "requires": { + "loose-envify": "^1.4.0", + "object-assign": "^4.1.1", + "react-is": "^16.13.1" + } + }, + "react": { + "version": "16.14.0", + "resolved": "https://registry.npmjs.org/react/-/react-16.14.0.tgz", + "integrity": "sha512-0X2CImDkJGApiAlcf0ODKIneSwBPhqJawOa5wCtKbu7ZECrmS26NvtSILynQ66cgkT/RJ4LidJOc3bUESwmU8g==", + "peer": true, + "requires": { + "loose-envify": "^1.1.0", + "object-assign": "^4.1.1", + "prop-types": "^15.6.2" + } + }, + "react-dom": { + "version": "16.14.0", + "resolved": "https://registry.npmjs.org/react-dom/-/react-dom-16.14.0.tgz", + "integrity": "sha512-1gCeQXDLoIqMgqD3IO2Ah9bnf0w9kzhwN5q4FGnHZ67hBm9yePzB5JJAIQCc8x3pFnNlwFq4RidZggNAAkzWWw==", + "peer": true, + "requires": { + "loose-envify": "^1.1.0", + "object-assign": "^4.1.1", + "prop-types": "^15.6.2", + "scheduler": "^0.19.1" + } + }, + "react-is": { + "version": "16.13.1", + "resolved": "https://registry.npmjs.org/react-is/-/react-is-16.13.1.tgz", + "integrity": "sha512-24e6ynE2H+OKt4kqsOvNd8kBpV65zoxbA4BVsEOB3ARVWQki/DHzaUoC5KuON/BiccDaCCTZBuOcfZs70kR8bQ==", + "peer": true + }, "resolve": { "version": "1.20.0", "requires": { @@ -831,6 +975,16 @@ "fsevents": "~2.3.2" } }, + "scheduler": { + "version": "0.19.1", + "resolved": "https://registry.npmjs.org/scheduler/-/scheduler-0.19.1.tgz", + "integrity": "sha512-n/zwRWRYSUj0/3g/otKDRPMh6qv2SYMWNq85IEa8iZyAv8od9zDYpGSnpBEjNgcMNq6Scbu5KfIPxNF72R/2EA==", + "peer": true, + "requires": { + "loose-envify": "^1.1.0", + "object-assign": "^4.1.1" + } + }, "source-map": { "version": "0.6.1" },