From 3087f31747b19da02a5fbd0e2075376b40721dd2 Mon Sep 17 00:00:00 2001 From: Olga Telezhnaya Date: Thu, 26 Sep 2024 10:39:04 +0100 Subject: [PATCH] Increase HW requirements for RPC nodes (#105) * Add more detailed HW requirements * Fix md links * Fix SSD/HDD requirements and pricings * Add comment about gc_num_epochs_to_keep --- docs/archival/hardware-archival.md | 77 ++++++---- .../run-archival-node-without-nearup.md | 2 +- docs/rpc/hardware-rpc.md | 79 +++++++---- docs/troubleshooting/common-errors.md | 9 ++ docs/troubleshooting/resharding.md | 4 +- docs/validator/hardware-validator.md | 133 ++++++++++++++++++ docs/validator/hardware.md | 73 ---------- docs/validator/running-a-node-macos-linux.md | 3 +- docs/validator/validator-bootcamp.md | 2 +- 9 files changed, 255 insertions(+), 127 deletions(-) create mode 100644 docs/validator/hardware-validator.md delete mode 100644 docs/validator/hardware.md diff --git a/docs/archival/hardware-archival.md b/docs/archival/hardware-archival.md index b8c815f..01db08c 100644 --- a/docs/archival/hardware-archival.md +++ b/docs/archival/hardware-archival.md @@ -8,48 +8,77 @@ description: NEAR Archival Node Hardware Requirements This page covers the minimum and recommended hardware requirements for engaging with the NEAR platform as an Archival node. +# Mainnet ## Recommended Hardware Specifications {#recommended-hardware-specifications} -| Hardware | Recommended Specifications | -| -------------- | ----------------------------------------------------------------------- | -| CPU | 8-Core (16-Thread) Intel i7/Xeon or equivalent with AVX support | -| CPU Features | CMPXCHG16B, POPCNT, SSE4.1, SSE4.2, AVX, SHA-NI | -| RAM | 24GB DDR4 | -| Hot Storage | 3 Terabyte SSD | -| Cold Storage | 40 Terabyte NON-SSD Persistent Disks | +| Hardware | Recommended Specifications | +| -------------- |-----------------------------------------------------------------| +| CPU | 8-Core (16-Thread) Intel i7/Xeon or equivalent with AVX support | +| CPU Features | CMPXCHG16B, POPCNT, SSE4.1, SSE4.2, AVX, SHA-NI | +| RAM | 32GB DDR4 | +| Hot Storage | 3 Terabyte SSD | +| Cold Storage | 60 Terabyte NON-SSD Persistent Disks | -_Verify AVX support on Linux by issuing the command ```$ lscpu | grep -oh avx```. If the output is empty, your CPU is not supported._ +## Minimal Hardware Specifications {#minimal-hardware-specifications} +| Hardware | Minimal Specifications | +| -------------- |-----------------------------------------------------------------| +| CPU | 8-Core (16-Thread) Intel i7/Xeon or equivalent with AVX support | +| RAM | 24GB DDR4 | +| Hot Storage | 3 Terabyte SSD | +| Cold Storage | 60 Terabyte NON-SSD Persistent Disks | -## Minimal Hardware Specifications {#minimal-hardware-specifications} +## Cost Estimation {#cost-estimation} -| Hardware | Minimal Specifications | -| -------------- | -------------------------------------------------------------------------- | -| CPU | 8-Core (16-Thread) Intel i7/Xeon or equivalent with AVX support | -| RAM | 16GB DDR4 | -| Hot Storage | 3 Terabyte SSD | -| Cold Storage | 40 Terabyte NON-SSD Persistent Disks | +Estimated monthly costs depending on cloud provider: -_Verify AVX support on Linux by issuing the command ```$ lscpu | grep -oh avx```. If the output is empty, your CPU is not supported._ +| Cloud Provider | Machine Size | Linux | +| -------------- | --------------- |----------------------------| +| AWS | m5a.2xlarge | $160 CPU + $1200 storage † | +| GCP | n2-standard-8 | $280 CPU + $2100 storage † | +| Azure | Standard_D8s_v5 | $180 CPU + $400 storage † | -## Cost Estimation {#cost-estimation} +_( † ) The storage cost will grow overtime as an archival node stores more data from the growing NEAR blockchain._ + +# Testnet -Estimated monthly costs depending on operating system: +## Recommended Hardware Specifications {#recommended-hardware-specifications-testnet} + +| Hardware | Recommended Specifications | +| -------------- |-----------------------------------------------------------------| +| CPU | 8-Core (16-Thread) Intel i7/Xeon or equivalent with AVX support | +| CPU Features | CMPXCHG16B, POPCNT, SSE4.1, SSE4.2, AVX, SHA-NI | +| RAM | 24GB DDR4 | +| Hot Storage | 1 Terabyte SSD | +| Cold Storage | 15 Terabyte NON-SSD Persistent Disks | + +## Minimal Hardware Specifications {#minimal-hardware-specifications-testnet} + +| Hardware | Minimal Specifications | +| -------------- |-----------------------------------------------------------------| +| CPU | 8-Core (16-Thread) Intel i7/Xeon or equivalent with AVX support | +| RAM | 16GB DDR4 | +| Hot Storage | 1 Terabyte SSD | +| Cold Storage | 15 Terabyte NON-SSD Persistent Disks | + +## Cost Estimation {#cost-estimation-testnet} + +Estimated monthly costs depending on cloud provider: | Cloud Provider | Machine Size | Linux | -| -------------- | --------------- | ------------------------ | -| AWS | m5.2xlarge | $330 CPU + $800 storage † | -| GCP | n2-standard-8 | $280 CPU + $800 storage † | -| Azure | Standard_F8s_v2 | $180 CPU + $800 storage † | +| -------------- | --------------- |---------------------------| +| AWS | m5a.2xlarge | $160 CPU + $350 storage † | +| GCP | n2-standard-8 | $280 CPU + $600 storage † | +| Azure | Standard_D8s_v5 | $180 CPU + $150 storage † | _( † ) The storage cost will grow overtime as an archival node stores more data from the growing NEAR blockchain._ -
Resources for Cost Estimation

-All prices reflect *reserved instances* which offer deep discounts on all platforms with a 1 year commitment +All prices reflect *reserved instances* which offer deep discounts on all platforms with a 1-year commitment. +We cannot guarantee the prices mentioned above as they may change at any moment. - AWS - cpu: https://aws.amazon.com/ec2/pricing/reserved-instances/pricing diff --git a/docs/archival/run-archival-node-without-nearup.md b/docs/archival/run-archival-node-without-nearup.md index 9e9bbd6..9a09c72 100644 --- a/docs/archival/run-archival-node-without-nearup.md +++ b/docs/archival/run-archival-node-without-nearup.md @@ -25,7 +25,7 @@ Running an archival node is very similar to running a [validator node](/validato ## Prerequisites {#prerequisites} -- [Rust](https://www.rust-lang.org/). If not already installed, please [follow these instructions](https://docs.near.org/sdk/rust/introduction#install-rust-and-wasm-toolchain). +- [Rust](https://www.rust-lang.org/) - [Git](https://git-scm.com/) - Installed developer tools: - MacOS diff --git a/docs/rpc/hardware-rpc.md b/docs/rpc/hardware-rpc.md index 6bd3a75..d9fed01 100644 --- a/docs/rpc/hardware-rpc.md +++ b/docs/rpc/hardware-rpc.md @@ -6,47 +6,78 @@ sidebar_position: 1 description: NEAR RPC Node Hardware Requirements --- -This page covers the minimum and recommended hardware requirements for engaging with the NEAR platform as a RPC node. +This page covers the minimum and recommended hardware requirements for engaging with the NEAR platform as an RPC node. For testing your RPC once the node is fully sync'd, see [this example RPC request.](https://docs.near.org/api/rpc/network#node-status) -## Recommended Hardware Specifications {#recommended-hardware-specifications} +The `gc_num_epochs_to_keep` config parameter controls how many epochs are stored on your RPC node. +The default is 5, with a minimum of 3. Increasing it requires more SSD storage and keeps data for a longer period. -| Hardware | Recommended Specifications | -| -------------- |--------------------------------------------------------------------------| -| CPU | 8-Core (16-Thread) Intel i7/Xeon or equivalent | -| CPU Features | CMPXCHG16B, POPCNT, SSE4.1, SSE4.2, AVX, SHA-NI | -| RAM | 20GB DDR4 | -| Storage | 1TB SSD (NVMe SSD is recommended. HDD will be enough for localnet only ) | +# Mainnet -_Verify AVX support on Linux by issuing the command ```$ lscpu | grep -oh avx```. If the output is empty, your CPU is not supported._ +## Recommended Hardware Specifications {#recommended-hardware-specifications} +| Hardware | Recommended Specifications | +| -------------- |-------------------------------------------------| +| CPU | 8-Core (16-Thread) Intel i7/Xeon or equivalent | +| CPU Features | CMPXCHG16B, POPCNT, SSE4.1, SSE4.2, AVX, SHA-NI | +| RAM | 32GB DDR4 | +| Storage | 3TB NVMe SSD | ## Minimal Hardware Specifications {#minimal-hardware-specifications} -| Hardware | Minimal Specifications | -| -------------- |---------------------------------------------------------------------------| -| CPU | 8-Core (16-Thread) Intel i7/Xeon or equivalent | -| CPU Features | CMPXCHG16B, POPCNT, SSE4.1, SSE4.2, AVX, SHA-NI | -| RAM | 12GB DDR4 | -| Storage | 500GB SSD (NVMe SSD is recommended. HDD will be enough for localnet only) | - -_Verify AVX support on Linux by issuing the command ```$ lscpu | grep -oh avx```. If the output is empty, your CPU is not supported._ +| Hardware | Minimal Specifications | +| -------------- |-------------------------------------------------| +| CPU | 8-Core (16-Thread) Intel i7/Xeon or equivalent | +| CPU Features | CMPXCHG16B, POPCNT, SSE4.1, SSE4.2, AVX, SHA-NI | +| RAM | 24GB DDR4 | +| Storage | 2.5TB NVMe SSD | ## Cost Estimation {#cost-estimation} -Estimated monthly costs depending on operating system: +Estimated monthly costs depending on cloud provider: + +| Cloud Provider | Machine Size | Linux | +| -------------- |------------------|-------------------------| +| AWS | m5a.2xlarge | $160 CPU + $300 storage | +| GCP | n2-standard-8 | $280 CPU + $400 storage | +| Azure | Standard_D8s_v5 | $180 CPU + $300 storage | + +# Testnet + +## Recommended Hardware Specifications {#recommended-hardware-specifications-testnet} + +| Hardware | Recommended Specifications | +| -------------- |-------------------------------------------------| +| CPU | 8-Core (16-Thread) Intel i7/Xeon or equivalent | +| CPU Features | CMPXCHG16B, POPCNT, SSE4.1, SSE4.2, AVX, SHA-NI | +| RAM | 32GB DDR4 | +| Storage | 1TB NVMe SSD | + +## Minimal Hardware Specifications {#minimal-hardware-specifications-testnet} + +| Hardware | Minimal Specifications | +| -------------- |-------------------------------------------------| +| CPU | 8-Core (16-Thread) Intel i7/Xeon or equivalent | +| CPU Features | CMPXCHG16B, POPCNT, SSE4.1, SSE4.2, AVX, SHA-NI | +| RAM | 24GB DDR4 | +| Storage | 600GB NVMe SSD | + +## Cost Estimation {#cost-estimation-testnet} + +Estimated monthly costs depending on cloud provider: -| Cloud Provider | Machine Size | Linux | -| -------------- | --------------- | ---------------------- | -| AWS | m5.2xlarge | $330 CPU + $80 storage | -| GCP | n2-standard-8 | $280 CPU + $80 storage | -| Azure | Standard_F8s_v2 | $180 CPU + $40 storage | +| Cloud Provider | Machine Size | Linux | +| -------------- |------------------|-------------------------| +| AWS | m5a.2xlarge | $160 CPU + $80 storage | +| GCP | n2-standard-8 | $280 CPU + $120 storage | +| Azure | Standard_D8s_v5 | $180 CPU + $100 storage |
Resources for Cost Estimation

-All prices reflect *reserved instances* which offer deep discounts on all platforms with a 1 year commitment +All prices reflect *reserved instances* which offer deep discounts on all platforms with a 1-year commitment. +We cannot guarantee the prices mentioned above as they may change at any moment. - AWS - cpu: https://aws.amazon.com/ec2/pricing/reserved-instances/pricing diff --git a/docs/troubleshooting/common-errors.md b/docs/troubleshooting/common-errors.md index 926eb64..bb935c0 100644 --- a/docs/troubleshooting/common-errors.md +++ b/docs/troubleshooting/common-errors.md @@ -5,6 +5,15 @@ sidebar_label: Common Node Errors and Solutions description: Common Node Errors and Solutions --- +***How can I check if my CPU supports all the needed features?*** + +Verify CPU feature support by running the following command on Linux: + +``` +lscpu | grep -P '(?=.*avx )(?=.*sse4.2 )(?=.*cx16 )(?=.*popcnt )(?=.*sha_ni )' > /dev/null \ + && echo "Supported" \ + || echo "Not supported" +``` ***My node is looking for peers on guildnet, testnet, and mainnet. Why?*** diff --git a/docs/troubleshooting/resharding.md b/docs/troubleshooting/resharding.md index 7a5001a..51e57f9 100644 --- a/docs/troubleshooting/resharding.md +++ b/docs/troubleshooting/resharding.md @@ -54,11 +54,11 @@ You can find instructions on how to migrate to split storage on [Split Storage p #### Monitoring {#monitoring 1.37} To monitor resharding you can use metrics `near_resharding_status`, `near_resharding_batch_size`, and `near_resharding_batch_prepare_time_bucket`. -You can read more [on github](https://github.com/near/nearcore/blob/master/docs/architecture/how/resharding.md#monitoring). +You can read more [on github](https://github.com/near/nearcore/blob/master/docs/architecture/how/resharding_v2.md#monitoring). If you observe problems with block production or resharding performance, you can adjust resharding throttling configuration. This does not require a node restart, you can send a signal to the neard process to load the new config. -Read more [on github](https://github.com/near/nearcore/blob/master/docs/architecture/how/resharding.md#monitoring). +Read more [on github](https://github.com/near/nearcore/blob/master/docs/architecture/how/resharding_v2.md#monitoring). ### After resharding {#after 1.37} If your node failed to reshard or is not able to sync with the network after the protocol upgrade, you will need to download the latest DB snapshot provided by Pagoda from s3 diff --git a/docs/validator/hardware-validator.md b/docs/validator/hardware-validator.md new file mode 100644 index 0000000..6662975 --- /dev/null +++ b/docs/validator/hardware-validator.md @@ -0,0 +1,133 @@ +--- +id: hardware-validator +title: Hardware Requirements for Validator Node +sidebar_label: Hardware Requirements +sidebar_position: 1 +description: NEAR Validator Node Hardware Requirements +--- + +This page covers the minimum and recommended hardware requirements for engaging with the NEAR platform as a validator node. + +# Mainnet + +## Recommended Hardware Specifications {#recommended-hardware-specifications} + +### Chunk/Block Producers + +| Hardware | Recommended Specifications | +| -------------- |--------------------------------------------------------------| +| CPU | x86_64 (Intel, AMD) processor with at least 8 physical cores | +| CPU Features | CMPXCHG16B, POPCNT, SSE4.1, SSE4.2, AVX, SHA-NI | +| RAM | 32GB DDR4 | +| Storage | 2TB NVMe SSD | + +### Chunk validators + +| Hardware | Recommended Specifications | +| -------------- |--------------------------------------------------------------| +| CPU | x86_64 (Intel, AMD) processor with at least 8 physical cores | +| CPU Features | CMPXCHG16B, POPCNT, SSE4.1, SSE4.2, AVX, SHA-NI | +| RAM | 32GB DDR4 | +| Storage | 2TB NVMe SSD | + +## Minimal Hardware Specifications {#minimal-hardware-specifications} + +### Chunk/Block Producers + +| Hardware | Minimal Specifications | +| -------------- |--------------------------------------------------------------| +| CPU | x86_64 (Intel, AMD) processor with at least 8 physical cores | +| CPU Features | CMPXCHG16B, POPCNT, SSE4.1, SSE4.2, AVX, SHA-NI | +| RAM | 24GB DDR4 | +| Storage | 1TB NVMe SSD | + +### Chunk validators + +| Hardware | Minimal Specifications | +| -------------- |--------------------------------------------------------------| +| CPU | x86_64 (Intel, AMD) processor with at least 8 physical cores | +| CPU Features | CMPXCHG16B, POPCNT, SSE4.1, SSE4.2, AVX, SHA-NI | +| RAM | 24GB DDR4 | +| Storage | 1TB NVMe SSD | + +## Cost Estimation {#cost-estimation} + +Estimated monthly costs depending on cloud provider: + +| Cloud Provider | Machine Size | Linux | +| -------------- |------------------|-------------------------| +| AWS | m5a.2xlarge | $160 CPU + $160 storage | +| GCP | n2-standard-8 | $280 CPU + $240 storage | +| Azure | Standard_D8s_v5 | $180 CPU + $200 storage | + +# Testnet + +## Recommended Hardware Specifications {#recommended-hardware-specifications} + +### Chunk/Block Producers + +| Hardware | Recommended Specifications | +| -------------- |--------------------------------------------------------------| +| CPU | x86_64 (Intel, AMD) processor with at least 8 physical cores | +| CPU Features | CMPXCHG16B, POPCNT, SSE4.1, SSE4.2, AVX, SHA-NI | +| RAM | 32GB DDR4 | +| Storage | 1TB NVMe SSD | + +### Chunk validators + +| Hardware | Recommended Specifications | +| -------------- |--------------------------------------------------------------| +| CPU | x86_64 (Intel, AMD) processor with at least 8 physical cores | +| CPU Features | CMPXCHG16B, POPCNT, SSE4.1, SSE4.2, AVX, SHA-NI | +| RAM | 32GB DDR4 | +| Storage | 1TB NVMe SSD | + +## Minimal Hardware Specifications {#minimal-hardware-specifications} + +### Chunk/Block Producers + +| Hardware | Minimal Specifications | +| -------------- |--------------------------------------------------------------| +| CPU | x86_64 (Intel, AMD) processor with at least 8 physical cores | +| CPU Features | CMPXCHG16B, POPCNT, SSE4.1, SSE4.2, AVX, SHA-NI | +| RAM | 24GB DDR4 | +| Storage | 600GB NVMe SSD | + +### Chunk validators + +| Hardware | Minimal Specifications | +| -------------- |--------------------------------------------------------------| +| CPU | x86_64 (Intel, AMD) processor with at least 8 physical cores | +| CPU Features | CMPXCHG16B, POPCNT, SSE4.1, SSE4.2, AVX, SHA-NI | +| RAM | 24GB DDR4 | +| Storage | 600GB NVMe SSD | + +## Cost Estimation {#cost-estimation} + +Estimated monthly costs depending on cloud provider: + +| Cloud Provider | Machine Size | Linux | +| -------------- |------------------|-------------------------| +| AWS | m5a.2xlarge | $160 CPU + $80 storage | +| GCP | n2-standard-8 | $280 CPU + $120 storage | +| Azure | Standard_D8s_v5 | $180 CPU + $100 storage | + +
+Resources for Cost Estimation

+ +All prices reflect *reserved instances* which offer deep discounts on all platforms with a 1-year commitment. +We cannot guarantee the prices mentioned above as they may change at any moment. + +- AWS + - cpu: https://aws.amazon.com/ec2/pricing/reserved-instances/pricing + - storage: https://aws.amazon.com/ebs/pricing +- GCP + - cpu: https://cloud.google.com/compute/vm-instance-pricing + - storage: https://cloud.google.com/compute/disks-image-pricing +- Azure — https://azure.microsoft.com/en-us/pricing/calculator + +
+ +>Got a question? + + Ask it on StackOverflow! diff --git a/docs/validator/hardware.md b/docs/validator/hardware.md deleted file mode 100644 index d8af77b..0000000 --- a/docs/validator/hardware.md +++ /dev/null @@ -1,73 +0,0 @@ ---- -id: hardware -title: Hardware Requirements for Validator Node -sidebar_label: Hardware Requirements -sidebar_position: 1 -description: NEAR Validator Node Hardware Requirements ---- - -This page covers the minimum and recommended hardware requirements for engaging with the NEAR platform as a validator node. - - -## Recommended Hardware Specifications {#recommended-hardware-specifications} - -| Hardware | Recommended Specifications | -| -------------- | --------------------------------------------------------------- | -| CPU | x86_64 (Intel, AMD) processor with at least 8 physical cores | -| CPU Features | CMPXCHG16B, POPCNT, SSE4.1, SSE4.2, AVX, SHA-NI | -| RAM | 24GB DDR4 | -| Storage | 2TB SSD (NVMe SSD is recommended. HDD will be enough for localnet only) | - -Verify CPU feature support by running the following command on Linux: - -``` -lscpu | grep -P '(?=.*avx )(?=.*sse4.2 )(?=.*cx16 )(?=.*popcnt )(?=.*sha_ni )' > /dev/null \ - && echo "Supported" \ - || echo "Not supported" -``` - -## Minimal Hardware Specifications {#minimal-hardware-specifications} - -| Hardware | Minimal Specifications | -| -------------- | --------------------------------------------------------------- | -| CPU | x86_64 (Intel, AMD) processor with at least 8 physical cores | -| CPU Features | CMPXCHG16B, POPCNT, SSE4.1, SSE4.2, AVX, SHA-NI | -| RAM | 16GB DDR4 | -| Storage | 1TB SSD (NVMe SSD is recommended. HDD will be enough for localnet only) | - -Verify CPU feature support by running the following command on Linux: - -``` -lscpu | grep -P '(?=.*avx )(?=.*sse4.2 )(?=.*cx16 )(?=.*popcnt )(?=.*sha_ni )' > /dev/null \ - && echo "Supported" \ - || echo "Not supported" -``` - -## Cost Estimation {#cost-estimation} - -Estimated monthly costs depending on operating system: - -| Cloud Provider | Machine Size | Linux | -| -------------- | --------------- | ---------------------- | -| AWS | m5.2xlarge | $330 CPU + $80 storage | -| GCP | n2-standard-8 | $280 CPU + $80 storage | -| Azure | Standard_F8s_v2 | $180 CPU + $40 storage | - -
-Resources for Cost Estimation

- -All prices reflect *reserved instances* which offer deep discounts on all platforms with a 1 year commitment - -- AWS - - cpu: https://aws.amazon.com/ec2/pricing/reserved-instances/pricing - - storage: https://aws.amazon.com/ebs/pricing -- GCP - - cpu: https://cloud.google.com/compute/vm-instance-pricing - - storage: https://cloud.google.com/compute/disks-image-pricing -- Azure — https://azure.microsoft.com/en-us/pricing/calculator - -
- ->Got a question? - - Ask it on StackOverflow! diff --git a/docs/validator/running-a-node-macos-linux.md b/docs/validator/running-a-node-macos-linux.md index ba85f4d..6347247 100644 --- a/docs/validator/running-a-node-macos-linux.md +++ b/docs/validator/running-a-node-macos-linux.md @@ -179,8 +179,7 @@ Enter your account ID (leave empty if not going to be a validator): ## Running a Node on the Cloud {#running-a-node-on-the-cloud} -Create a new instance following the [Hardware -requirements](hardware.md). +Create a new instance following the [Hardware requirements](hardware-validator.md). Add firewall rules to allow traffic to port 24567 from all IPs (0.0.0.0/0). diff --git a/docs/validator/validator-bootcamp.md b/docs/validator/validator-bootcamp.md index f037a90..0dc97d5 100644 --- a/docs/validator/validator-bootcamp.md +++ b/docs/validator/validator-bootcamp.md @@ -869,7 +869,7 @@ near call {pool_id}.{staking_pool_factory} update_staking_key '{"stake_public_ke In order to get a validator seat you must first submit a proposal with an appropriate amount of stake. Proposals are sent for epoch +2. Meaning if you send a proposal now, if approved, you would get the seat in 3 epochs. You should submit a proposal every epoch to ensure your seat. To send a proposal we use the ping command. A proposal is also sent if a stake or unstake command is sent to the staking pool contract. -To note, a ping also updates the staking balances for your delegators. A ping should be issued each epoch to keep reported rewards current on the pool contract. You could set up a ping using a cron job or use [Cron Cat](https://cron.cat/). +To note, a ping also updates the staking balances for your delegators. A ping should be issued each epoch to keep reported rewards current on the pool contract. You could set up a ping using a cron job. Staking Pools Factories for each network: