Increase HW requirements for RPC nodes (#105)

* Add more detailed HW requirements * Fix md links * Fix SSD/HDD requirements and pricings * Add comment about gc_num_epochs_to_keep
near · Sep 26, 2024 · 3087f31 · 3087f31
1 parent 917b9ce
commit 3087f31
Show file tree

Hide file tree

Showing 9 changed files with 255 additions and 127 deletions.
diff --git a/docs/archival/hardware-archival.md b/docs/archival/hardware-archival.md
@@ -8,48 +8,77 @@ description: NEAR Archival Node Hardware Requirements
 
 This page covers the minimum and recommended hardware requirements for engaging with the NEAR platform as an Archival node.
 
+# Mainnet
 
 ## Recommended Hardware Specifications {#recommended-hardware-specifications}
 
-| Hardware       |  Recommended Specifications                                               |
-| -------------- | -----------------------------------------------------------------------   |
-| CPU            | 8-Core (16-Thread) Intel i7/Xeon or equivalent with AVX support           |
-| CPU Features   | CMPXCHG16B, POPCNT, SSE4.1, SSE4.2, AVX, SHA-NI                           |
-| RAM            | 24GB DDR4                                                                 |
-| Hot Storage    | 3 Terabyte SSD                                                            |
-| Cold Storage   | 40 Terabyte  NON-SSD Persistent Disks                                     |
+| Hardware       | Recommended Specifications                                      |
+| -------------- |-----------------------------------------------------------------|
+| CPU            | 8-Core (16-Thread) Intel i7/Xeon or equivalent with AVX support |
+| CPU Features   | CMPXCHG16B, POPCNT, SSE4.1, SSE4.2, AVX, SHA-NI                 |
+| RAM            | 32GB DDR4                                                       |
+| Hot Storage    | 3 Terabyte SSD                                                  |
+| Cold Storage   | 60 Terabyte  NON-SSD Persistent Disks                           |
 
-_Verify AVX support on Linux by issuing the command ```$ lscpu | grep -oh  avx```. If the output is empty, your CPU is not supported._
+## Minimal Hardware Specifications {#minimal-hardware-specifications}
 
+| Hardware       | Minimal Specifications                                          |
+| -------------- |-----------------------------------------------------------------|
+| CPU            | 8-Core (16-Thread) Intel i7/Xeon or equivalent with AVX support |
+| RAM            | 24GB DDR4                                                       |
+| Hot Storage    | 3 Terabyte SSD                                                  |
+| Cold Storage   | 60 Terabyte NON-SSD Persistent Disks                            |
 
-## Minimal Hardware Specifications {#minimal-hardware-specifications}
+## Cost Estimation {#cost-estimation}
 
-| Hardware       |  Minimal Specifications                                                    |
-| -------------- | -------------------------------------------------------------------------- |
-| CPU            | 8-Core (16-Thread) Intel i7/Xeon or equivalent with AVX support            |
-| RAM            | 16GB DDR4                                                                  |
-| Hot Storage    | 3 Terabyte SSD                                                             |
-| Cold Storage   | 40 Terabyte NON-SSD Persistent Disks                                       |
+Estimated monthly costs depending on cloud provider:
 
-_Verify AVX support on Linux by issuing the command ```$ lscpu | grep -oh  avx```. If the output is empty, your CPU is not supported._
+| Cloud Provider | Machine Size    | Linux                      |
+| -------------- | --------------- |----------------------------|
+| AWS            | m5a.2xlarge     | $160 CPU + $1200 storage † |
+| GCP            | n2-standard-8   | $280 CPU + $2100 storage † |
+| Azure          | Standard_D8s_v5 | $180 CPU + $400 storage †  |
 
-## Cost Estimation {#cost-estimation}
+_( † ) The storage cost will grow overtime as an archival node stores more data from the growing NEAR blockchain._
+
+# Testnet
 
-Estimated monthly costs depending on operating system:
+## Recommended Hardware Specifications {#recommended-hardware-specifications-testnet}
+
+| Hardware       | Recommended Specifications                                      |
+| -------------- |-----------------------------------------------------------------|
+| CPU            | 8-Core (16-Thread) Intel i7/Xeon or equivalent with AVX support |
+| CPU Features   | CMPXCHG16B, POPCNT, SSE4.1, SSE4.2, AVX, SHA-NI                 |
+| RAM            | 24GB DDR4                                                       |
+| Hot Storage    | 1 Terabyte SSD                                                  |
+| Cold Storage   | 15 Terabyte NON-SSD Persistent Disks                            |
+
+## Minimal Hardware Specifications {#minimal-hardware-specifications-testnet}
+
+| Hardware       | Minimal Specifications                                          |
+| -------------- |-----------------------------------------------------------------|
+| CPU            | 8-Core (16-Thread) Intel i7/Xeon or equivalent with AVX support |
+| RAM            | 16GB DDR4                                                       |
+| Hot Storage    | 1 Terabyte SSD                                                  |
+| Cold Storage   | 15 Terabyte NON-SSD Persistent Disks                            |
+
+## Cost Estimation {#cost-estimation-testnet}
+
+Estimated monthly costs depending on cloud provider:
 
 | Cloud Provider | Machine Size    | Linux                     |
-| -------------- | --------------- | ------------------------  |
-| AWS            | m5.2xlarge      | $330 CPU + $800 storage † |
-| GCP            | n2-standard-8   | $280 CPU + $800 storage † |
-| Azure          | Standard_F8s_v2 | $180 CPU + $800 storage † |
+| -------------- | --------------- |---------------------------|
+| AWS            | m5a.2xlarge     | $160 CPU + $350 storage † |
+| GCP            | n2-standard-8   | $280 CPU + $600 storage † |
+| Azure          | Standard_D8s_v5 | $180 CPU + $150 storage † |
 
 _( † ) The storage cost will grow overtime as an archival node stores more data from the growing NEAR blockchain._
 
-
 <blockquote class="info">
 <strong>Resources for Cost Estimation</strong><br /><br />
 
-All prices reflect *reserved instances* which offer deep discounts on all platforms with a 1 year commitment
+All prices reflect *reserved instances* which offer deep discounts on all platforms with a 1-year commitment.
+We cannot guarantee the prices mentioned above as they may change at any moment.
 
 - AWS
   - cpu: https://aws.amazon.com/ec2/pricing/reserved-instances/pricing

diff --git a/docs/archival/run-archival-node-without-nearup.md b/docs/archival/run-archival-node-without-nearup.md
@@ -25,7 +25,7 @@ Running an archival node is very similar to running a [validator node](/validato
 
 ## Prerequisites {#prerequisites}
 
-- [Rust](https://www.rust-lang.org/). If not already installed, please [follow these instructions](https://docs.near.org/sdk/rust/introduction#install-rust-and-wasm-toolchain).
+- [Rust](https://www.rust-lang.org/)
 - [Git](https://git-scm.com/)
 - Installed developer tools:
   - MacOS

diff --git a/docs/rpc/hardware-rpc.md b/docs/rpc/hardware-rpc.md
@@ -6,47 +6,78 @@ sidebar_position: 1
 description: NEAR RPC Node Hardware Requirements
 ---
 
-This page covers the minimum and recommended hardware requirements for engaging with the NEAR platform as a RPC node.
+This page covers the minimum and recommended hardware requirements for engaging with the NEAR platform as an RPC node.
 
 For testing your RPC once the node is fully sync'd, see [this example RPC request.](https://docs.near.org/api/rpc/network#node-status)
 
-## Recommended Hardware Specifications {#recommended-hardware-specifications}
+The `gc_num_epochs_to_keep` config parameter controls how many epochs are stored on your RPC node.
+The default is 5, with a minimum of 3. Increasing it requires more SSD storage and keeps data for a longer period.
 
-| Hardware       | Recommended Specifications                                               |
-| -------------- |--------------------------------------------------------------------------|
-| CPU            | 8-Core (16-Thread) Intel i7/Xeon or equivalent                           |
-| CPU Features   | CMPXCHG16B, POPCNT, SSE4.1, SSE4.2, AVX, SHA-NI                          |
-| RAM            | 20GB DDR4                                                                 |
-| Storage        | 1TB SSD (NVMe SSD is recommended. HDD will be enough for localnet only ) |
+# Mainnet
 
-_Verify AVX support on Linux by issuing the command ```$ lscpu | grep -oh  avx```. If the output is empty, your CPU is not supported._
+## Recommended Hardware Specifications {#recommended-hardware-specifications}
 
+| Hardware       | Recommended Specifications                      |
+| -------------- |-------------------------------------------------|
+| CPU            | 8-Core (16-Thread) Intel i7/Xeon or equivalent  |
+| CPU Features   | CMPXCHG16B, POPCNT, SSE4.1, SSE4.2, AVX, SHA-NI |
+| RAM            | 32GB DDR4                                       |
+| Storage        | 3TB NVMe SSD                                    |
 
 ## Minimal Hardware Specifications {#minimal-hardware-specifications}
 
-| Hardware       | Minimal Specifications                                                    |
-| -------------- |---------------------------------------------------------------------------|
-| CPU            | 8-Core (16-Thread) Intel i7/Xeon or equivalent                            |
-| CPU Features   | CMPXCHG16B, POPCNT, SSE4.1, SSE4.2, AVX, SHA-NI                           |
-| RAM            | 12GB DDR4                                                                 |
-| Storage        | 500GB SSD (NVMe SSD is recommended. HDD will be enough for localnet only) |
-
-_Verify AVX support on Linux by issuing the command ```$ lscpu | grep -oh  avx```. If the output is empty, your CPU is not supported._
+| Hardware       | Minimal Specifications                          |
+| -------------- |-------------------------------------------------|
+| CPU            | 8-Core (16-Thread) Intel i7/Xeon or equivalent  |
+| CPU Features   | CMPXCHG16B, POPCNT, SSE4.1, SSE4.2, AVX, SHA-NI |
+| RAM            | 24GB DDR4                                       |
+| Storage        | 2.5TB NVMe SSD                                  |
 
 ## Cost Estimation {#cost-estimation}
 
-Estimated monthly costs depending on operating system:
+Estimated monthly costs depending on cloud provider:
+
+| Cloud Provider | Machine Size     | Linux                   |
+| -------------- |------------------|-------------------------|
+| AWS            | m5a.2xlarge      | $160 CPU + $300 storage |
+| GCP            | n2-standard-8    | $280 CPU + $400 storage |
+| Azure          | Standard_D8s_v5  | $180 CPU + $300 storage |
+
+# Testnet
+
+## Recommended Hardware Specifications {#recommended-hardware-specifications-testnet}
+
+| Hardware       | Recommended Specifications                      |
+| -------------- |-------------------------------------------------|
+| CPU            | 8-Core (16-Thread) Intel i7/Xeon or equivalent  |
+| CPU Features   | CMPXCHG16B, POPCNT, SSE4.1, SSE4.2, AVX, SHA-NI |
+| RAM            | 32GB DDR4                                       |
+| Storage        | 1TB NVMe SSD                                    |
+
+## Minimal Hardware Specifications {#minimal-hardware-specifications-testnet}
+
+| Hardware       | Minimal Specifications                          |
+| -------------- |-------------------------------------------------|
+| CPU            | 8-Core (16-Thread) Intel i7/Xeon or equivalent  |
+| CPU Features   | CMPXCHG16B, POPCNT, SSE4.1, SSE4.2, AVX, SHA-NI |
+| RAM            | 24GB DDR4                                       |
+| Storage        | 600GB NVMe SSD                                  |
+
+## Cost Estimation {#cost-estimation-testnet}
+
+Estimated monthly costs depending on cloud provider:
 
-| Cloud Provider | Machine Size    | Linux                  |
-| -------------- | --------------- | ---------------------- |
-| AWS            | m5.2xlarge      | $330 CPU + $80 storage |
-| GCP            | n2-standard-8   | $280 CPU + $80 storage |
-| Azure          | Standard_F8s_v2 | $180 CPU + $40 storage |
+| Cloud Provider | Machine Size     | Linux                   |
+| -------------- |------------------|-------------------------|
+| AWS            | m5a.2xlarge      | $160 CPU + $80 storage  |
+| GCP            | n2-standard-8    | $280 CPU + $120 storage |
+| Azure          | Standard_D8s_v5  | $180 CPU + $100 storage |
 
 <blockquote class="info">
 <strong>Resources for Cost Estimation</strong><br /><br />
 
-All prices reflect *reserved instances* which offer deep discounts on all platforms with a 1 year commitment
+All prices reflect *reserved instances* which offer deep discounts on all platforms with a 1-year commitment.
+We cannot guarantee the prices mentioned above as they may change at any moment.
 
 - AWS
   - cpu: https://aws.amazon.com/ec2/pricing/reserved-instances/pricing

diff --git a/docs/troubleshooting/common-errors.md b/docs/troubleshooting/common-errors.md
@@ -5,6 +5,15 @@ sidebar_label: Common Node Errors and Solutions
 description: Common Node Errors and Solutions
 ---
 
+***How can I check if my CPU supports all the needed features?***
+
+Verify CPU feature support by running the following command on Linux:
+
+```
+lscpu | grep -P '(?=.*avx )(?=.*sse4.2 )(?=.*cx16 )(?=.*popcnt )(?=.*sha_ni )' > /dev/null \
+  && echo "Supported" \
+  || echo "Not supported"
+```
 
 ***My node is looking for peers on guildnet, testnet, and mainnet. Why?***
 

diff --git a/docs/troubleshooting/resharding.md b/docs/troubleshooting/resharding.md
@@ -54,11 +54,11 @@ You can find instructions on how to migrate to split storage on [Split Storage p
 
 #### Monitoring {#monitoring 1.37}
 To monitor resharding you can use metrics `near_resharding_status`, `near_resharding_batch_size`, and `near_resharding_batch_prepare_time_bucket`.
-You can read more [on github](https://github.com/near/nearcore/blob/master/docs/architecture/how/resharding.md#monitoring).
+You can read more [on github](https://github.com/near/nearcore/blob/master/docs/architecture/how/resharding_v2.md#monitoring).
 
 If you observe problems with block production or resharding performance, you can adjust resharding throttling configuration.
 This does not require a node restart, you can send a signal to the neard process to load the new config.
-Read more [on github](https://github.com/near/nearcore/blob/master/docs/architecture/how/resharding.md#monitoring).
+Read more [on github](https://github.com/near/nearcore/blob/master/docs/architecture/how/resharding_v2.md#monitoring).
 
 ### After resharding {#after 1.37}
 If your node failed to reshard or is not able to sync with the network after the protocol upgrade, you will need to download the latest DB snapshot provided by Pagoda from s3