Unify and improve on sandbox documentation

docs/.vuepress/sidebar.ts

@@ -476,6 +476,7 @@ export const getSidebar = (locale: string) =>
               `${locale}/indexer/build/mapping/stellar.md`,
               `${locale}/indexer/build/mapping/cache.md`,
               `${locale}/indexer/build/mapping/store.md`,
+              `${locale}/indexer/build/mapping/sandbox.md`,
             ],
           },
           `${locale}/indexer/build/testing.md`,

docs/indexer/build/introduction.md

@@ -5,9 +5,9 @@ In the [quick start](../quickstart/quickstart.md) guide, we very quickly ran thr
 Some of the following examples will assume you have successfully initialized the starter package in the [Quick start](../quickstart/quickstart.md) section. From that starter package, we'll walk through the standard process to customise and implement your SubQuery project.
 
 1. Initialise your project using `subql init PROJECT_NAME`.
-2. Update the Manifest file (`project.ts`) to include information about your blockchain, and the entities that you will map - see [Manifest File](./manifest/polkadot.md).
+2. Update the Manifest file (`project.ts`) to include information about your blockchain, and the entities that you will map - see [Manifest File](./manifest/ethereum.md).
 3. Create GraphQL entities in your schema (`schema.graphql`) that defines the shape of the data that you will extract and persist for querying - see [GraphQL Schema](./graphql.md).
-4. Add all the mapping functions (eg `mappingHandlers.ts`) you wish to invoke to transform chain data to the GraphQL entities that you have defined - see [Mapping](./mapping/polkadot.md).
+4. Add all the mapping functions (eg `mappingHandlers.ts`) you wish to invoke to transform chain data to the GraphQL entities that you have defined - see [Mapping](./mapping/ethereum.md).
 5. Generate, build, and publish your code to SubQuery Projects (or run in your own local node) - see how to [Run](../run_publish/run.md) and [Publish](../run_publish/introduction.md) your Starter Project in our quick start guide.
 
 ## Directory Structure

docs/indexer/build/mapping/algorand.md

@@ -5,6 +5,7 @@ Mapping functions define how chain data is transformed into the optimised GraphQ
 - Mappings are defined in the `src/mappings` directory and are exported as a function.
 - These mappings are also exported in `src/index.ts`.
 - The mappings files are reference in `project.ts` under the mapping handlers.
+- The mappings are run from within a [Sandbox](./sandbox.md)
 
 There are different classes of mappings functions for Algorand: [Block handlers](#block-handler), and [Transaction Handlers](#transaction-handler).
 
@@ -59,57 +60,3 @@ const txGroup: AlgorandTransaction[] = tx.block.getTransactionsByGroup(
 );
 ```
 
-## Third-party Library Support - the Sandbox
-
-SubQuery is deterministic by design, that means that each SubQuery project is guaranteed to index the same data set. This is a critical factor that is makes it possible to verify data in the decentralised SubQuery Network. This limitation means that in default configuration, the indexer is by default run in a strict virtual machine, with access to a strict number of third party libraries.
-
-**You can easily bypass this limitation however, allowing you to retrieve data from external API endpoints, non historical RPC calls, and import your own external libraries into your projects.** In order to do to, you must run your project in `unsafe` mode, you can read more about this in the [references](../../run_publish/references.md#unsafe-node-service). An easy way to do this while developing (and running in Docker) is to add the following line to your `docker-compose.yml`:
-
-```yml
-subquery-node:
-  image: onfinality/subql-node-algorand:latest
-  ...
-  command:
-    - -f=/app
-    - --db-schema=app
-    - --unsafe
-  ...
-```
-
-When run in `unsafe` mode, you can import any custom libraries into your project and make external API calls using tools like node-fetch. A simple example is given below:
-
-```ts
-import { AlgorandTransaction } from "@subql/types-algorand";
-import fetch from "node-fetch";
-
-export async function handleTransaction(
-  tx: AlgorandTransaction,
-): Promise<void> {
-  const httpData = await fetch("https://api.github.com/users/github");
-  logger.info(`httpData: ${JSON.stringify(httpData.body)}`);
-  // Do something with this data
-}
-```
-
-By default (when in safe mode), the [VM2](https://www.npmjs.com/package/vm2) sandbox only allows the following:
-
-- only some certain built-in modules, e.g. `assert`, `buffer`, `crypto`,`util` and `path`
-- third-party libraries written by _CommonJS_.
-- external `HTTP` and `WebSocket` connections are forbidden
-
-## Modules and Libraries
-
-To improve SubQuery's data processing capabilities, we have allowed some of the NodeJS's built-in modules for running mapping functions in the [sandbox](#third-party-library-support---the-sandbox), and have allowed users to call third-party libraries.
-
-Please note this is an **experimental feature** and you may encounter bugs or issues that may negatively impact your mapping functions. Please report any bugs you find by creating an issue in [GitHub](https://github.com/subquery/subql).
-
-### Built-in modules
-
-Currently, we allow the following NodeJS modules: `assert`, `buffer`, `crypto`, `util`, and `path`.
-
-Rather than importing the whole module, we recommend only importing the required method(s) that you need. Some methods in these modules may have dependencies that are unsupported and will fail on import.
-
-```ts
-import { hashMessage } from "ethers/lib/utils"; // Good way
-import { utils } from "ethers"; // Bad way
-```

docs/indexer/build/mapping/arbitrum.md

@@ -9,6 +9,7 @@ Mapping functions define how chain data is transformed into the optimised GraphQ
 - Mappings are defined in the `src/mappings` directory and are exported as a function.
 - These mappings are also exported in `src/index.ts`.
 - The mappings files are reference in `project.ts` under the mapping handlers.
+- The mappings are run from within a [Sandbox](./sandbox.md)
 
 There are different classes of mappings functions for Arbitrum; [Block handlers](#block-handler), [Transaction Handlers](#transaction-handler), and [Log Handlers](#log-handler).
 
@@ -90,57 +91,3 @@ const balance = await erc20.balanceOf(address);
 
 The above example assumes that the user has an ABI file named `erc20.json`, so that TypeChain generates `ERC20__factory` class for them. Check out [this example](https://github.com/dethcrypto/TypeChain/tree/master/examples/ethers-v5) to see how to generate factory code around your contract ABI using TypeChain.
 
-## Third-party Library Support - the Sandbox
-
-SubQuery is deterministic by design, that means that each SubQuery project is guaranteed to index the same data set. This is a critical factor that is makes it possible to verify data in the decentralised SubQuery Network. This limitation means that in default configuration, the indexer is by default run in a strict virtual machine, with access to a strict number of third party libraries.
-
-**You can easily bypass this limitation however, allowing you to retrieve data from external API endpoints, non historical RPC calls, and import your own external libraries into your projects.** In order to do to, you must run your project in `unsafe` mode, you can read more about this in the [references](../../run_publish/references.md#unsafe-node-service). An easy way to do this while developing (and running in Docker) is to add the following line to your `docker-compose.yml`:
-
-```yml
-subquery-node:
-  image: onfinality/subql-node-ethereum:latest
-  ...
-  command:
-    - -f=/app
-    - --db-schema=app
-    - --unsafe
-  ...
-```
-
-When run in `unsafe` mode, you can import any custom libraries into your project and make external API calls using tools like node-fetch. A simple example is given below:
-
-```ts
-import { EthereumTransaction } from "@subql/types-ethereum"; // We use ethereum types since Arbitrum is a layer-2 that is compatible
-import fetch from "node-fetch";
-
-export async function handleTransaction(
-  tx: EthereumTransaction,
-): Promise<void> {
-  const httpData = await fetch("https://api.github.com/users/github");
-  logger.info(`httpData: ${JSON.stringify(httpData.body)}`);
-  // Do something with this data
-}
-```
-
-By default (when in safe mode), the [VM2](https://www.npmjs.com/package/vm2) sandbox only allows the following:
-
-- only some certain built-in modules, e.g. `assert`, `buffer`, `crypto`,`util` and `path`
-- third-party libraries written by _CommonJS_.
-- external `HTTP` and `WebSocket` connections are forbidden
-
-## Modules and Libraries
-
-To improve SubQuery's data processing capabilities, we have allowed some of the NodeJS's built-in modules for running mapping functions in the [sandbox](#third-party-library-support---the-sandbox), and have allowed users to call third-party libraries.
-
-Please note this is an **experimental feature** and you may encounter bugs or issues that may negatively impact your mapping functions. Please report any bugs you find by creating an issue in [GitHub](https://github.com/subquery/subql).
-
-### Built-in modules
-
-Currently, we allow the following NodeJS modules: `assert`, `buffer`, `crypto`, `util`, and `path`.
-
-Rather than importing the whole module, we recommend only importing the required method(s) that you need. Some methods in these modules may have dependencies that are unsupported and will fail on import.
-
-```ts
-import { hashMessage } from "ethers/lib/utils"; // Good way
-import { utils } from "ethers"; // Bad way
-```

docs/indexer/build/mapping/avalanche.md

@@ -17,6 +17,7 @@ Mapping functions define how chain data is transformed into the optimised GraphQ
 - Mappings are defined in the `src/mappings` directory and are exported as a function.
 - These mappings are also exported in `src/index.ts`.
 - The mappings files are reference in `project.ts` under the mapping handlers.
+- The mappings are run from within a [Sandbox](./sandbox.md)
 
 There are different classes of mappings functions for Avalanche; [Block handlers](#block-handler), [Transaction Handlers](#transaction-handler), and [Log Handlers](#log-handler).
 
@@ -99,58 +100,3 @@ const balance = await erc20.balanceOf(address);
 ```
 
 The above example assumes that the user has an ABI file named `erc20.json`, so that TypeChain generates `ERC20__factory` class for them. Check out [this example](https://github.com/dethcrypto/TypeChain/tree/master/examples/ethers-v5) to see how to generate factory code around your contract ABI using TypeChain.
-
-## Third-party Library Support - the Sandbox
-
-SubQuery is deterministic by design, that means that each SubQuery project is guaranteed to index the same data set. This is a critical factor that is makes it possible to verify data in the decentralised SubQuery Network. This limitation means that in default configuration, the indexer is by default run in a strict virtual machine, with access to a strict number of third party libraries.
-
-**You can easily bypass this limitation however, allowing you to retrieve data from external API endpoints, non historical RPC calls, and import your own external libraries into your projects.** In order to do to, you must run your project in `unsafe` mode, you can read more about this in the [references](../../run_publish/references.md#unsafe-node-service). An easy way to do this while developing (and running in Docker) is to add the following line to your `docker-compose.yml`:
-
-```yml
-subquery-node:
-  image: onfinality/subql-node-ethereum:latest
-  ...
-  command:
-    - -f=/app
-    - --db-schema=app
-    - --unsafe
-  ...
-```
-
-When run in `unsafe` mode, you can import any custom libraries into your project and make external API calls using tools like node-fetch. A simple example is given below:
-
-```ts
-import { EthereumTransaction } from "@subql/types-ethereum"; // We use ethereum types since Avalanche is compatible
-import fetch from "node-fetch";
-
-export async function handleTransaction(
-  tx: EthereumTransaction,
-): Promise<void> {
-  const httpData = await fetch("https://api.github.com/users/github");
-  logger.info(`httpData: ${JSON.stringify(httpData.body)}`);
-  // Do something with this data
-}
-```
-
-By default (when in safe mode), the [VM2](https://www.npmjs.com/package/vm2) sandbox only allows the following:
-
-- only some certain built-in modules, e.g. `assert`, `buffer`, `crypto`,`util` and `path`
-- third-party libraries written by _CommonJS_.
-- external `HTTP` and `WebSocket` connections are forbidden
-
-## Modules and Libraries
-
-To improve SubQuery's data processing capabilities, we have allowed some of the NodeJS's built-in modules for running mapping functions in the [sandbox](#third-party-library-support---the-sandbox), and have allowed users to call third-party libraries.
-
-Please note this is an **experimental feature** and you may encounter bugs or issues that may negatively impact your mapping functions. Please report any bugs you find by creating an issue in [GitHub](https://github.com/subquery/subql).
-
-### Built-in modules
-
-Currently, we allow the following NodeJS modules: `assert`, `buffer`, `crypto`, `util`, and `path`.
-
-Rather than importing the whole module, we recommend only importing the required method(s) that you need. Some methods in these modules may have dependencies that are unsupported and will fail on import.
-
-```ts
-import { hashMessage } from "ethers/lib/utils"; // Good way
-import { utils } from "ethers"; // Bad way
-```

docs/indexer/build/mapping/bsc.md

@@ -9,6 +9,7 @@ Mapping functions define how chain data is transformed into the optimised GraphQ
 - Mappings are defined in the `src/mappings` directory and are exported as a function.
 - These mappings are also exported in `src/index.ts`.
 - The mappings files are reference in `project.ts` under the mapping handlers.
+- The mappings are run from within a [Sandbox](./sandbox.md)
 
 There are different classes of mappings functions for BSC; [Block handlers](#block-handler), [Transaction Handlers](#transaction-handler), and [Log Handlers](#log-handler).
 
@@ -89,58 +90,3 @@ const balance = await erc20.balanceOf(address);
 ```
 
 The above example assumes that the user has an ABI file named `erc20.json`, so that TypeChain generates `ERC20__factory` class for them. Check out [this example](https://github.com/dethcrypto/TypeChain/tree/master/examples/ethers-v5) to see how to generate factory code around your contract ABI using TypeChain.
-
-## Third-party Library Support - the Sandbox
-
-SubQuery is deterministic by design, that means that each SubQuery project is guaranteed to index the same data set. This is a critical factor that is makes it possible to verify data in the decentralised SubQuery Network. This limitation means that in default configuration, the indexer is by default run in a strict virtual machine, with access to a strict number of third party libraries.
-
-**You can easily bypass this limitation however, allowing you to retrieve data from external API endpoints, non historical RPC calls, and import your own external libraries into your projects.** In order to do to, you must run your project in `unsafe` mode, you can read more about this in the [references](../../run_publish/references.md#unsafe-node-service). An easy way to do this while developing (and running in Docker) is to add the following line to your `docker-compose.yml`:
-
-```yml
-subquery-node:
-  image: onfinality/subql-node-ethereum:latest
-  ...
-  command:
-    - -f=/app
-    - --db-schema=app
-    - --unsafe
-  ...
-```
-
-When run in `unsafe` mode, you can import any custom libraries into your project and make external API calls using tools like node-fetch. A simple example is given below:
-
-```ts
-import { EthereumTransaction } from "@subql/types-ethereum"; // We use ethereum types since BSC is a layer-2 that is compatible
-import fetch from "node-fetch";
-
-export async function handleTransaction(
-  tx: EthereumTransaction,
-): Promise<void> {
-  const httpData = await fetch("https://api.github.com/users/github");
-  logger.info(`httpData: ${JSON.stringify(httpData.body)}`);
-  // Do something with this data
-}
-```
-
-By default (when in safe mode), the [VM2](https://www.npmjs.com/package/vm2) sandbox only allows the following:
-
-- only some certain built-in modules, e.g. `assert`, `buffer`, `crypto`,`util` and `path`
-- third-party libraries written by _CommonJS_.
-- external `HTTP` and `WebSocket` connections are forbidden
-
-## Modules and Libraries
-
-To improve SubQuery's data processing capabilities, we have allowed some of the NodeJS's built-in modules for running mapping functions in the [sandbox](#third-party-library-support---the-sandbox), and have allowed users to call third-party libraries.
-
-Please note this is an **experimental feature** and you may encounter bugs or issues that may negatively impact your mapping functions. Please report any bugs you find by creating an issue in [GitHub](https://github.com/subquery/subql).
-
-### Built-in modules
-
-Currently, we allow the following NodeJS modules: `assert`, `buffer`, `crypto`, `util`, and `path`.
-
-Rather than importing the whole module, we recommend only importing the required method(s) that you need. Some methods in these modules may have dependencies that are unsupported and will fail on import.
-
-```ts
-import { hashMessage } from "ethers/lib/utils"; // Good way
-import { utils } from "ethers"; // Bad way
-```