Skip to content

Commit

Permalink
update archival and rpc docs with latest how to download info
Browse files Browse the repository at this point in the history
  • Loading branch information
@rtsainear
rtsainear committed Jun 14, 2024
1 parent 9a1cf98 commit 444bce8
Show file tree
Hide file tree
Showing 7 changed files with 165 additions and 94 deletions.
10 changes: 6 additions & 4 deletions docs/archival/hardware-archival.md
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
id: hardware-archival
title: Hardware Requirements for Archival Node
title: Hardware Requirements for Archival Split-Storage Node
sidebar_label: Hardware Requirements
sidebar_position: 1
description: NEAR Archival Node Hardware Requirements
Expand All @@ -16,7 +16,8 @@ This page covers the minimum and recommended hardware requirements for engaging
| CPU | 8-Core (16-Thread) Intel i7/Xeon or equivalent with AVX support |
| CPU Features | CMPXCHG16B, POPCNT, SSE4.1, SSE4.2, AVX |
| RAM | 24GB DDR4 |
| Storage | 9 Terabyte SSD |
| Hot Storage | 3 Terabyte SSD |
| Cold Storage | 40 Terabyte SSD |

_Verify AVX support on Linux by issuing the command ```$ lscpu | grep -oh avx```. If the output is empty, your CPU is not supported._

Expand All @@ -26,8 +27,9 @@ _Verify AVX support on Linux by issuing the command ```$ lscpu | grep -oh avx``
| Hardware | Minimal Specifications |
| -------------- | -------------------------------------------------------------------------- |
| CPU | 8-Core (16-Thread) Intel i7/Xeon or equivalent with AVX support |
| RAM | 16GB DDR4 |
| Storage | 9 Terabyte SSD |
| RAM | 16GB DDR4 |
| Hot Storage | 3 Terabyte SSD |
| Cold Storage | 40 Terabyte SSD |

_Verify AVX support on Linux by issuing the command ```$ lscpu | grep -oh avx```. If the output is empty, your CPU is not supported._

Expand Down
2 changes: 2 additions & 0 deletions docs/archival/run-archival-node-with-nearup.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@ sidebar_position: 3
description: How to run an Archival Node with nearup
---

*This page is DEPRECATED Please Use [Run a Node](/archival/split-storage-archival)

*We encourage you to set up your node with Neard instead of Nearup as Nearup is not used on Mainnet. Please head to [Run a Node](/archival/run-archival-node-without-nearup) for instructions on how to setup an archival node with Neard.*

<blockquote class="info">
Expand Down
1 change: 1 addition & 0 deletions docs/archival/run-archival-node-without-nearup.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ sidebar_label: Run a Node 🚀
sidebar_position: 2
description: How to run an Archival Node without nearup
---
*This page is DEPRECATED in favor of split storage archival. Please Use [Run a Node](/archival/split-storage-archival)

The following instructions are applicable across localnet, testnet, and mainnet.

Expand Down
43 changes: 37 additions & 6 deletions docs/archival/split-storage-archival.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -62,17 +62,50 @@ Example:
}
```

## Using Pagoda-provided S3 split-storage snapshots {#S3 migration}
## Using Pagoda-provided S3/CloudFrontsplit-storage snapshots {#S3 migration}

Prerequisite:

Recommended download client [`rclone`](https://rclone.org).
This tool is present in many Linux distributions. There is also a version for Windows.
And its main merit is multithread.
You can [read about it on](https://rclone.org)
** rclone version needs to be v1.66.0 or higher

First, install rclone:
```
$ sudo -v ; curl https://rclone.org/install.sh | sudo bash
```
Next, prepare config, so you don't need to specify all the parameters interactively:
```
mkdir -p ~/.config/rclone
touch ~/.config/rclone/rclone.conf
```

, and paste exactly the following config into `rclone.conf`:
```
[near_cf]
type = s3
provider = AWS
download_url = https://dcf58hz8pnro2.cloudfront.net/
acl = public-read
server_side_encryption = AES256
region = ca-central-1
```

1. Find latest snapshot
```bash
chain=testnet/mainnet
aws s3 --no-sign-request cp s3://near-protocol-public/backups/$chain/archive/latest_split_storage .
rclone copy --no-check-certificate near_cf://near-protocol-public/backups/${chain:?}/archive/latest_split_storage ./
latest=$(cat latest_split_storage)
latest=$(cat latest_split_storage)
```
2. Download cold and hot databases
```bash
NEAR_HOME=/home/ubuntu/.near
aws s3 --no-sign-request cp --recursive s3://near-protocol-public/backups/$chain/archive/$latest/ $NEAR_HOME
rclone copy --no-check-certificate --progress --transfers=6 --checkers=6 \
near_cf://near-protocol-public/backups/${chain:?}/archive/${latest:?} $NEAR_HOME
```
This will download cold database to `$NEAR_HOME/cold-data`, and hot database to `$NEAR_HOME/hot-data`
3. Make changes to your config.json
Expand Down Expand Up @@ -266,6 +299,4 @@ These two metrics should be very close to each other.
`near_cold_store_copy_result` – status of the continuous block-by-block migration that is performed in background to block production and can be interrupted.
Size of the hot db – should be close to the size of an RPC node’s storage.
Size of the cold db – should be close to the size of a legacy archival node’s storage.
Size of the cold db – should be close to the size of a legacy archival node’s storage.
96 changes: 19 additions & 77 deletions docs/intro/snapshots.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,19 +6,18 @@ sidebar_position: 3
description: Node Data Snapshots
---

**Notice: The AWS S3 endpoints will be inaccessible starting January 1st, 2024. Please use the updated Cloudfront endpoints for snapshot downloads to ensure uninterrupted service.**

Before you start running a node, you must first sync with the network. This means your node needs to download all the headers and blocks that other nodes in the network already have. You can speed up this process by downloading the latest data snapshots from a public cloudfront endpoint.

Here are the available snapshots directories based on node type and network. Please note that the snapshots are updated every 12 hours.


| Node Type and Network| S3 Path |
| -------------------- | ------------------------------------------------------------------------ |
| RPC testnet | `https://dcf58hz8pnro2.cloudfront.net/backups/testnet/rpc/latest` |
| RPC mainnet | `https://dcf58hz8pnro2.cloudfront.net/backups/mainnet/rpc/latest` |
| Archival testnet | `https://dcf58hz8pnro2.cloudfront.net/backups/testnet/archive/latest` |
| Archival mainnet | `https://dcf58hz8pnro2.cloudfront.net/backups/mainnet/archive/latest` |
| Node Type and Network| CloudFront Path |
| -------------------- | ---------------------------------------------------------------------------------------|
| RPC testnet | `https://dcf58hz8pnro2.cloudfront.net/backups/testnet/rpc/latest` |
| RPC mainnet | `https://dcf58hz8pnro2.cloudfront.net/backups/mainnet/rpc/latest` |
| Archival testnet | `https://dcf58hz8pnro2.cloudfront.net/backups/testnet/archive/latest_split_storage` |
| Archival mainnet | `https://dcf58hz8pnro2.cloudfront.net/backups/mainnet/archive/latest_split_storage` |

----
Prerequisite:
Expand All @@ -27,18 +26,19 @@ Recommended download client [`rclone`](https://rclone.org).
This tool is present in many Linux distributions. There is also a version for Windows.
And its main merit is multithread.
You can [read about it on](https://rclone.org)
** rclone version needs to be v1.66.0 or higher

First, install rclone:
```
$ sudo apt install rclone
$ sudo -v ; curl https://rclone.org/install.sh | sudo bash
```
Next, prepare config, so you don't need to specify all the parameters interactively:
```
mkdir -p ~/.config/rclone
touch ~/.config/rclone/rclone.conf
```

, and paste the following config into `rclone.conf`:
, and paste exactly the following config into `rclone.conf`:
```
[near_cf]
type = s3
Expand All @@ -49,85 +49,27 @@ server_side_encryption = AES256
region = ca-central-1
```
Commands to run:
Commands to run for RPC|Validator Data Download:
```
chain="mainnet" # or "testnet"
kind="rpc" # or "archive"
kind="rpc"
rclone copy --no-check-certificate near_cf://near-protocol-public/backups/${chain:?}/${kind:?}/latest ./
latest=$(cat latest)
rclone copy --no-check-certificate --progress --transfers=6 \
rclone copy --no-check-certificate --progress --transfers=6 --checkers=6 \
near_cf://near-protocol-public/backups/${chain:?}/${kind:?}/${latest:?} ~/.near/data
```

----


**Notice: The following section are the older instructions to download snapshots from s3. Direct s3 access will be deprecated starting January 1, 2024.**


| Node Type and Network| S3 Path |
| -------------------- | ------------------------------------------------------------- |
| RPC testnet | s3://near-protocol-public/backups/testnet/rpc/latest |
| RPC mainnet | s3://near-protocol-public/backups/mainnet/rpc/latest |
| Archival testnet | s3://near-protocol-public/backups/testnet/archive/latest |
| Archival mainnet | s3://near-protocol-public/backups/mainnet/archive/latest |


----


If you've [initialized the working directory for your node](/validator/compile-and-run-a-node#3-initialize-working-directory-1) without passing in a preferred location, the default working directory for your node is `~/.near`. It is recommended that you wget and untar into a `data` folder under `~/.near/`. The new `~/.near/data` is where your node will store historical states and write its state. To use the default location, run the following commands.

First, please install AWS CLI:
```bash
$ sudo apt-get install awscli -y
Commands to run for Archival Split Storage Data Download:
```

Then, download the snapshot using the AWS CLI:
```bash
$ chain="mainnet" # or "testnet"
$ kind="rpc" # or "archive"
$ aws s3 --no-sign-request cp "s3://near-protocol-public/backups/${chain:?}/${kind:?}/latest" .
$ latest=$(cat latest)
$ aws s3 sync --delete --no-sign-request "s3://near-protocol-public/backups/${chain:?}/${kind:?}/${latest:?}" ~/.near/data
chain="mainnet" # or "testnet"
kind="archive"
rclone copy --no-check-certificate near_cf://near-protocol-public/backups/${chain:?}/${kind:?}/latest_split_storage ./
latest=$(cat latest_split_storage)
rclone copy --no-check-certificate --progress --transfers=6 --checkers=6 \
near_cf://near-protocol-public/backups/${chain:?}/${kind:?}/${latest:?} ~/.near/data
```

For a faster snapshot download speed, use s5cmd, the download accelerator for S3 written in Go. For download instruction, please see https://github.com/peak/s5cmd.

Another alternative is [`rclone`](https://rclone.org).
This tool is present in many Linux distributions. There is also a version for Windows.
And its main merit is multithread.
You can [read about it on](https://rclone.org)

First, install rclone:
```
$ sudo apt install rclone
```
Next, prepare config, so you don't need to specify all the parameters interactively:
```
mkdir -p ~/.config/rclone
touch ~/.config/rclone/rclone.conf
```

, and paste the following config into `rclone.conf`:
```
[near_s3]
type = s3
provider = AWS
location_constraint = EU
acl = public-read
server_side_encryption = AES256
region = ca-central-1
```
Next step very similar with aws-cli.
```
chain="mainnet" # or "testnet"
kind="rpc" # or "archive"
rclone copy --no-check-certificate near_s3://near-protocol-public/backups/${chain:?}/${kind:?}/latest ./
latest=$(cat latest)
rclone copy --no-check-certificate --progress --transfers=6 \
near_s3://near-protocol-public/backups/${chain:?}/${kind:?}/${latest:?} ~/.near/data
```

>Got a question?
<a href="https://stackoverflow.com/questions/tagged/nearprotocol">
Expand Down
37 changes: 34 additions & 3 deletions docs/rpc/run-rpc-node-with-nearup.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -77,11 +77,42 @@ Then run:
$ nearup stop
```

Retrieve a copy of the latest RPC snapshot from S3:
Retrieve a copy of the latest RPC snapshot from S3 using rclone:
Prerequisite:

Recommended download client [`rclone`](https://rclone.org).
This tool is present in many Linux distributions. There is also a version for Windows.
And its main merit is multithread.
You can [read about it on](https://rclone.org)
** rclone version needs to be v1.66.0 or higher

First, install rclone:
```
$ sudo -v ; curl https://rclone.org/install.sh | sudo bash
```
Next, prepare config, so you don't need to specify all the parameters interactively:
```
mkdir -p ~/.config/rclone
touch ~/.config/rclone/rclone.conf
```

, and paste exactly the following config into `rclone.conf`:
```
[near_cf]
type = s3
provider = AWS
download_url = https://dcf58hz8pnro2.cloudfront.net/
acl = public-read
server_side_encryption = AES256
region = ca-central-1
```

```bash
$ aws s3 --no-sign-request cp s3://near-protocol-public/backups/testnet/rpc/latest .
$ rclone copy --no-check-certificate near_cf://near-protocol-public/backups/testnet/rpc/latest ./
$ LATEST=$(cat latest)
$ aws s3 --no-sign-request cp --no-sign-request --recursive s3://near-protocol-public/backups/testnet/rpc/$LATEST ~/.near/data
$ rclone copy --no-check-certificate --progress --transfers=6 --checkers=6 \
near_cf://near-protocol-public/backups/testnet/rpc/${latest:?} ~/.near/data
```


Expand Down
70 changes: 66 additions & 4 deletions docs/rpc/run-rpc-node-without-nearup.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -129,10 +129,41 @@ $ wget https://s3-us-west-1.amazonaws.com/build.nearprotocol.com/nearcore-deploy
The node is ready to be started. However, you must first sync up with the network. This means your node needs to download all the headers and blocks that other nodes in the network already have.
Prerequisite:
Recommended download client [`rclone`](https://rclone.org).
This tool is present in many Linux distributions. There is also a version for Windows.
And its main merit is multithread.
You can [read about it on](https://rclone.org)
** rclone version needs to be v1.66.0 or higher
First, install rclone:
```
$ sudo -v ; curl https://rclone.org/install.sh | sudo bash
```
Next, prepare config, so you don't need to specify all the parameters interactively:
```
mkdir -p ~/.config/rclone
touch ~/.config/rclone/rclone.conf
```

, and paste exactly the following config into `rclone.conf`:
```
[near_cf]
type = s3
provider = AWS
download_url = https://dcf58hz8pnro2.cloudfront.net/
acl = public-read
server_side_encryption = AES256
region = ca-central-1
```

```bash
$ aws s3 --no-sign-request cp s3://near-protocol-public/backups/testnet/rpc/latest .
$ rclone copy --no-check-certificate near_cf://near-protocol-public/backups/testnet/rpc/latest ./
$ LATEST=$(cat latest)
$ aws s3 --no-sign-request cp --no-sign-request --recursive s3://near-protocol-public/backups/testnet/rpc/$LATEST ~/.near/data
$ rclone copy --no-check-certificate --progress --transfers=6 --checkers=6 \
near_cf://near-protocol-public/backups/testnet/rpc/${latest:?} ~/.near/data
```

### 6. Run the node
Expand Down Expand Up @@ -222,10 +253,41 @@ $ wget https://s3-us-west-1.amazonaws.com/build.nearprotocol.com/nearcore-deploy
The node is ready to be started. However, you must first sync up with the network. This means your node needs to download all the headers and blocks that other nodes in the network already have.
Prerequisite:
Recommended download client [`rclone`](https://rclone.org).
This tool is present in many Linux distributions. There is also a version for Windows.
And its main merit is multithread.
You can [read about it on](https://rclone.org)
** rclone version needs to be v1.66.0 or higher
First, install rclone:
```
$ sudo -v ; curl https://rclone.org/install.sh | sudo bash
```
Next, prepare config, so you don't need to specify all the parameters interactively:
```
mkdir -p ~/.config/rclone
touch ~/.config/rclone/rclone.conf
```

, and paste exactly the following config into `rclone.conf`:
```
[near_cf]
type = s3
provider = AWS
download_url = https://dcf58hz8pnro2.cloudfront.net/
acl = public-read
server_side_encryption = AES256
region = ca-central-1
```

```bash
$ aws s3 --no-sign-request cp s3://near-protocol-public/backups/mainnet/rpc/latest .
$ rclone copy --no-check-certificate near_cf://near-protocol-public/backups/mainnet/rpc/latest ./
$ LATEST=$(cat latest)
$ aws s3 --no-sign-request cp --no-sign-request --recursive s3://near-protocol-public/backups/mainnet/rpc/$LATEST ~/.near/data
$ rclone copy --no-check-certificate --progress --transfers=6 --checkers=6 \
near_cf://near-protocol-public/backups/mainnet/rpc/${latest:?} ~/.near/data
```

### 6. Run the node
Expand Down

0 comments on commit 444bce8

Please sign in to comment.