diff --git a/docs/.vuepress/sidebar.ts b/docs/.vuepress/sidebar.ts index fbf07e08e06..8375d0298ca 100644 --- a/docs/.vuepress/sidebar.ts +++ b/docs/.vuepress/sidebar.ts @@ -476,6 +476,7 @@ export const getSidebar = (locale: string) => `${locale}/indexer/build/mapping/stellar.md`, `${locale}/indexer/build/mapping/cache.md`, `${locale}/indexer/build/mapping/store.md`, + `${locale}/indexer/build/mapping/sandbox.md`, ], }, `${locale}/indexer/build/testing.md`, diff --git a/docs/indexer/build/introduction.md b/docs/indexer/build/introduction.md index f18ccdbfb6b..57591ba7247 100644 --- a/docs/indexer/build/introduction.md +++ b/docs/indexer/build/introduction.md @@ -5,9 +5,9 @@ In the [quick start](../quickstart/quickstart.md) guide, we very quickly ran thr Some of the following examples will assume you have successfully initialized the starter package in the [Quick start](../quickstart/quickstart.md) section. From that starter package, we'll walk through the standard process to customise and implement your SubQuery project. 1. Initialise your project using `subql init PROJECT_NAME`. -2. Update the Manifest file (`project.ts`) to include information about your blockchain, and the entities that you will map - see [Manifest File](./manifest/polkadot.md). +2. Update the Manifest file (`project.ts`) to include information about your blockchain, and the entities that you will map - see [Manifest File](./manifest/ethereum.md). 3. Create GraphQL entities in your schema (`schema.graphql`) that defines the shape of the data that you will extract and persist for querying - see [GraphQL Schema](./graphql.md). -4. Add all the mapping functions (eg `mappingHandlers.ts`) you wish to invoke to transform chain data to the GraphQL entities that you have defined - see [Mapping](./mapping/polkadot.md). +4. Add all the mapping functions (eg `mappingHandlers.ts`) you wish to invoke to transform chain data to the GraphQL entities that you have defined - see [Mapping](./mapping/ethereum.md). 5. Generate, build, and publish your code to SubQuery Projects (or run in your own local node) - see how to [Run](../run_publish/run.md) and [Publish](../run_publish/introduction.md) your Starter Project in our quick start guide. ## Directory Structure diff --git a/docs/indexer/build/mapping/algorand.md b/docs/indexer/build/mapping/algorand.md index 897e9af680e..e531003d328 100644 --- a/docs/indexer/build/mapping/algorand.md +++ b/docs/indexer/build/mapping/algorand.md @@ -5,6 +5,7 @@ Mapping functions define how chain data is transformed into the optimised GraphQ - Mappings are defined in the `src/mappings` directory and are exported as a function. - These mappings are also exported in `src/index.ts`. - The mappings files are reference in `project.ts` under the mapping handlers. +- The mappings are run from within a [Sandbox](./sandbox.md) There are different classes of mappings functions for Algorand: [Block handlers](#block-handler), and [Transaction Handlers](#transaction-handler). @@ -59,57 +60,3 @@ const txGroup: AlgorandTransaction[] = tx.block.getTransactionsByGroup( ); ``` -## Third-party Library Support - the Sandbox - -SubQuery is deterministic by design, that means that each SubQuery project is guaranteed to index the same data set. This is a critical factor that is makes it possible to verify data in the decentralised SubQuery Network. This limitation means that in default configuration, the indexer is by default run in a strict virtual machine, with access to a strict number of third party libraries. - -**You can easily bypass this limitation however, allowing you to retrieve data from external API endpoints, non historical RPC calls, and import your own external libraries into your projects.** In order to do to, you must run your project in `unsafe` mode, you can read more about this in the [references](../../run_publish/references.md#unsafe-node-service). An easy way to do this while developing (and running in Docker) is to add the following line to your `docker-compose.yml`: - -```yml -subquery-node: - image: onfinality/subql-node-algorand:latest - ... - command: - - -f=/app - - --db-schema=app - - --unsafe - ... -``` - -When run in `unsafe` mode, you can import any custom libraries into your project and make external API calls using tools like node-fetch. A simple example is given below: - -```ts -import { AlgorandTransaction } from "@subql/types-algorand"; -import fetch from "node-fetch"; - -export async function handleTransaction( - tx: AlgorandTransaction, -): Promise { - const httpData = await fetch("https://api.github.com/users/github"); - logger.info(`httpData: ${JSON.stringify(httpData.body)}`); - // Do something with this data -} -``` - -By default (when in safe mode), the [VM2](https://www.npmjs.com/package/vm2) sandbox only allows the following: - -- only some certain built-in modules, e.g. `assert`, `buffer`, `crypto`,`util` and `path` -- third-party libraries written by _CommonJS_. -- external `HTTP` and `WebSocket` connections are forbidden - -## Modules and Libraries - -To improve SubQuery's data processing capabilities, we have allowed some of the NodeJS's built-in modules for running mapping functions in the [sandbox](#third-party-library-support---the-sandbox), and have allowed users to call third-party libraries. - -Please note this is an **experimental feature** and you may encounter bugs or issues that may negatively impact your mapping functions. Please report any bugs you find by creating an issue in [GitHub](https://github.com/subquery/subql). - -### Built-in modules - -Currently, we allow the following NodeJS modules: `assert`, `buffer`, `crypto`, `util`, and `path`. - -Rather than importing the whole module, we recommend only importing the required method(s) that you need. Some methods in these modules may have dependencies that are unsupported and will fail on import. - -```ts -import { hashMessage } from "ethers/lib/utils"; // Good way -import { utils } from "ethers"; // Bad way -``` diff --git a/docs/indexer/build/mapping/arbitrum.md b/docs/indexer/build/mapping/arbitrum.md index 839f1416459..b15f0f57b6b 100644 --- a/docs/indexer/build/mapping/arbitrum.md +++ b/docs/indexer/build/mapping/arbitrum.md @@ -9,6 +9,7 @@ Mapping functions define how chain data is transformed into the optimised GraphQ - Mappings are defined in the `src/mappings` directory and are exported as a function. - These mappings are also exported in `src/index.ts`. - The mappings files are reference in `project.ts` under the mapping handlers. +- The mappings are run from within a [Sandbox](./sandbox.md) There are different classes of mappings functions for Arbitrum; [Block handlers](#block-handler), [Transaction Handlers](#transaction-handler), and [Log Handlers](#log-handler). @@ -90,57 +91,3 @@ const balance = await erc20.balanceOf(address); The above example assumes that the user has an ABI file named `erc20.json`, so that TypeChain generates `ERC20__factory` class for them. Check out [this example](https://github.com/dethcrypto/TypeChain/tree/master/examples/ethers-v5) to see how to generate factory code around your contract ABI using TypeChain. -## Third-party Library Support - the Sandbox - -SubQuery is deterministic by design, that means that each SubQuery project is guaranteed to index the same data set. This is a critical factor that is makes it possible to verify data in the decentralised SubQuery Network. This limitation means that in default configuration, the indexer is by default run in a strict virtual machine, with access to a strict number of third party libraries. - -**You can easily bypass this limitation however, allowing you to retrieve data from external API endpoints, non historical RPC calls, and import your own external libraries into your projects.** In order to do to, you must run your project in `unsafe` mode, you can read more about this in the [references](../../run_publish/references.md#unsafe-node-service). An easy way to do this while developing (and running in Docker) is to add the following line to your `docker-compose.yml`: - -```yml -subquery-node: - image: onfinality/subql-node-ethereum:latest - ... - command: - - -f=/app - - --db-schema=app - - --unsafe - ... -``` - -When run in `unsafe` mode, you can import any custom libraries into your project and make external API calls using tools like node-fetch. A simple example is given below: - -```ts -import { EthereumTransaction } from "@subql/types-ethereum"; // We use ethereum types since Arbitrum is a layer-2 that is compatible -import fetch from "node-fetch"; - -export async function handleTransaction( - tx: EthereumTransaction, -): Promise { - const httpData = await fetch("https://api.github.com/users/github"); - logger.info(`httpData: ${JSON.stringify(httpData.body)}`); - // Do something with this data -} -``` - -By default (when in safe mode), the [VM2](https://www.npmjs.com/package/vm2) sandbox only allows the following: - -- only some certain built-in modules, e.g. `assert`, `buffer`, `crypto`,`util` and `path` -- third-party libraries written by _CommonJS_. -- external `HTTP` and `WebSocket` connections are forbidden - -## Modules and Libraries - -To improve SubQuery's data processing capabilities, we have allowed some of the NodeJS's built-in modules for running mapping functions in the [sandbox](#third-party-library-support---the-sandbox), and have allowed users to call third-party libraries. - -Please note this is an **experimental feature** and you may encounter bugs or issues that may negatively impact your mapping functions. Please report any bugs you find by creating an issue in [GitHub](https://github.com/subquery/subql). - -### Built-in modules - -Currently, we allow the following NodeJS modules: `assert`, `buffer`, `crypto`, `util`, and `path`. - -Rather than importing the whole module, we recommend only importing the required method(s) that you need. Some methods in these modules may have dependencies that are unsupported and will fail on import. - -```ts -import { hashMessage } from "ethers/lib/utils"; // Good way -import { utils } from "ethers"; // Bad way -``` diff --git a/docs/indexer/build/mapping/avalanche.md b/docs/indexer/build/mapping/avalanche.md index 5343bf90c8c..b4f937a0714 100644 --- a/docs/indexer/build/mapping/avalanche.md +++ b/docs/indexer/build/mapping/avalanche.md @@ -17,6 +17,7 @@ Mapping functions define how chain data is transformed into the optimised GraphQ - Mappings are defined in the `src/mappings` directory and are exported as a function. - These mappings are also exported in `src/index.ts`. - The mappings files are reference in `project.ts` under the mapping handlers. +- The mappings are run from within a [Sandbox](./sandbox.md) There are different classes of mappings functions for Avalanche; [Block handlers](#block-handler), [Transaction Handlers](#transaction-handler), and [Log Handlers](#log-handler). @@ -99,58 +100,3 @@ const balance = await erc20.balanceOf(address); ``` The above example assumes that the user has an ABI file named `erc20.json`, so that TypeChain generates `ERC20__factory` class for them. Check out [this example](https://github.com/dethcrypto/TypeChain/tree/master/examples/ethers-v5) to see how to generate factory code around your contract ABI using TypeChain. - -## Third-party Library Support - the Sandbox - -SubQuery is deterministic by design, that means that each SubQuery project is guaranteed to index the same data set. This is a critical factor that is makes it possible to verify data in the decentralised SubQuery Network. This limitation means that in default configuration, the indexer is by default run in a strict virtual machine, with access to a strict number of third party libraries. - -**You can easily bypass this limitation however, allowing you to retrieve data from external API endpoints, non historical RPC calls, and import your own external libraries into your projects.** In order to do to, you must run your project in `unsafe` mode, you can read more about this in the [references](../../run_publish/references.md#unsafe-node-service). An easy way to do this while developing (and running in Docker) is to add the following line to your `docker-compose.yml`: - -```yml -subquery-node: - image: onfinality/subql-node-ethereum:latest - ... - command: - - -f=/app - - --db-schema=app - - --unsafe - ... -``` - -When run in `unsafe` mode, you can import any custom libraries into your project and make external API calls using tools like node-fetch. A simple example is given below: - -```ts -import { EthereumTransaction } from "@subql/types-ethereum"; // We use ethereum types since Avalanche is compatible -import fetch from "node-fetch"; - -export async function handleTransaction( - tx: EthereumTransaction, -): Promise { - const httpData = await fetch("https://api.github.com/users/github"); - logger.info(`httpData: ${JSON.stringify(httpData.body)}`); - // Do something with this data -} -``` - -By default (when in safe mode), the [VM2](https://www.npmjs.com/package/vm2) sandbox only allows the following: - -- only some certain built-in modules, e.g. `assert`, `buffer`, `crypto`,`util` and `path` -- third-party libraries written by _CommonJS_. -- external `HTTP` and `WebSocket` connections are forbidden - -## Modules and Libraries - -To improve SubQuery's data processing capabilities, we have allowed some of the NodeJS's built-in modules for running mapping functions in the [sandbox](#third-party-library-support---the-sandbox), and have allowed users to call third-party libraries. - -Please note this is an **experimental feature** and you may encounter bugs or issues that may negatively impact your mapping functions. Please report any bugs you find by creating an issue in [GitHub](https://github.com/subquery/subql). - -### Built-in modules - -Currently, we allow the following NodeJS modules: `assert`, `buffer`, `crypto`, `util`, and `path`. - -Rather than importing the whole module, we recommend only importing the required method(s) that you need. Some methods in these modules may have dependencies that are unsupported and will fail on import. - -```ts -import { hashMessage } from "ethers/lib/utils"; // Good way -import { utils } from "ethers"; // Bad way -``` diff --git a/docs/indexer/build/mapping/bsc.md b/docs/indexer/build/mapping/bsc.md index c21d4a0e505..3927ab613fb 100644 --- a/docs/indexer/build/mapping/bsc.md +++ b/docs/indexer/build/mapping/bsc.md @@ -9,6 +9,7 @@ Mapping functions define how chain data is transformed into the optimised GraphQ - Mappings are defined in the `src/mappings` directory and are exported as a function. - These mappings are also exported in `src/index.ts`. - The mappings files are reference in `project.ts` under the mapping handlers. +- The mappings are run from within a [Sandbox](./sandbox.md) There are different classes of mappings functions for BSC; [Block handlers](#block-handler), [Transaction Handlers](#transaction-handler), and [Log Handlers](#log-handler). @@ -89,58 +90,3 @@ const balance = await erc20.balanceOf(address); ``` The above example assumes that the user has an ABI file named `erc20.json`, so that TypeChain generates `ERC20__factory` class for them. Check out [this example](https://github.com/dethcrypto/TypeChain/tree/master/examples/ethers-v5) to see how to generate factory code around your contract ABI using TypeChain. - -## Third-party Library Support - the Sandbox - -SubQuery is deterministic by design, that means that each SubQuery project is guaranteed to index the same data set. This is a critical factor that is makes it possible to verify data in the decentralised SubQuery Network. This limitation means that in default configuration, the indexer is by default run in a strict virtual machine, with access to a strict number of third party libraries. - -**You can easily bypass this limitation however, allowing you to retrieve data from external API endpoints, non historical RPC calls, and import your own external libraries into your projects.** In order to do to, you must run your project in `unsafe` mode, you can read more about this in the [references](../../run_publish/references.md#unsafe-node-service). An easy way to do this while developing (and running in Docker) is to add the following line to your `docker-compose.yml`: - -```yml -subquery-node: - image: onfinality/subql-node-ethereum:latest - ... - command: - - -f=/app - - --db-schema=app - - --unsafe - ... -``` - -When run in `unsafe` mode, you can import any custom libraries into your project and make external API calls using tools like node-fetch. A simple example is given below: - -```ts -import { EthereumTransaction } from "@subql/types-ethereum"; // We use ethereum types since BSC is a layer-2 that is compatible -import fetch from "node-fetch"; - -export async function handleTransaction( - tx: EthereumTransaction, -): Promise { - const httpData = await fetch("https://api.github.com/users/github"); - logger.info(`httpData: ${JSON.stringify(httpData.body)}`); - // Do something with this data -} -``` - -By default (when in safe mode), the [VM2](https://www.npmjs.com/package/vm2) sandbox only allows the following: - -- only some certain built-in modules, e.g. `assert`, `buffer`, `crypto`,`util` and `path` -- third-party libraries written by _CommonJS_. -- external `HTTP` and `WebSocket` connections are forbidden - -## Modules and Libraries - -To improve SubQuery's data processing capabilities, we have allowed some of the NodeJS's built-in modules for running mapping functions in the [sandbox](#third-party-library-support---the-sandbox), and have allowed users to call third-party libraries. - -Please note this is an **experimental feature** and you may encounter bugs or issues that may negatively impact your mapping functions. Please report any bugs you find by creating an issue in [GitHub](https://github.com/subquery/subql). - -### Built-in modules - -Currently, we allow the following NodeJS modules: `assert`, `buffer`, `crypto`, `util`, and `path`. - -Rather than importing the whole module, we recommend only importing the required method(s) that you need. Some methods in these modules may have dependencies that are unsupported and will fail on import. - -```ts -import { hashMessage } from "ethers/lib/utils"; // Good way -import { utils } from "ethers"; // Bad way -``` diff --git a/docs/indexer/build/mapping/concordium.md b/docs/indexer/build/mapping/concordium.md index 6d5d9488a7d..e34b50ffb12 100644 --- a/docs/indexer/build/mapping/concordium.md +++ b/docs/indexer/build/mapping/concordium.md @@ -5,6 +5,7 @@ Mapping functions define how chain data is transformed into the optimised GraphQ - Mappings are defined in the `src/mappings` directory and are exported as a function. - These mappings are also exported in `src/index.ts`. - The mappings files are reference in `project.ts` under the mapping handlers. +- The mappings are run from within a [Sandbox](./sandbox.md) There are different classes of mappings functions for Concordium: [Block handlers](#block-handler), [Transaction Handlers](#transaction-handler), [Transaction Event Handlers](#transaction-event-handler) and [Special Event Handlers](#special-event-handler). @@ -92,58 +93,3 @@ export async function handleSpecialEvent( await record.save(); } ``` - -## Third-party Library Support - the Sandbox - -SubQuery is deterministic by design, that means that each SubQuery project is guaranteed to index the same data set. This is a critical factor that is makes it possible to verify data in the decentralised SubQuery Network. This limitation means that in default configuration, the indexer is by default run in a strict virtual machine, with access to a strict number of third party libraries. - -**You can easily bypass this limitation however, allowing you to retrieve data from external API endpoints, non historical RPC calls, and import your own external libraries into your projects.** In order to do to, you must run your project in `unsafe` mode, you can read more about this in the [references](../../run_publish/references.md#unsafe-node-service). An easy way to do this while developing (and running in Docker) is to add the following line to your `docker-compose.yml`: - -```yml -subquery-node: - image: onfinality/subql-node-concordium:latest - ... - command: - - -f=/app - - --db-schema=app - - --unsafe - ... -``` - -When run in `unsafe` mode, you can import any custom libraries into your project and make external API calls using tools like node-fetch. A simple example is given below: - -```ts -import { ConcordiumTransaction } from "@subql/types-concordium"; -import fetch from "node-fetch"; - -export async function handleTransaction( - tx: ConcordiumTransaction, -): Promise { - const httpData = await fetch("https://api.github.com/users/github"); - logger.info(`httpData: ${JSON.stringify(httpData.body)}`); - // Do something with this data -} -``` - -By default (when in safe mode), the [VM2](https://www.npmjs.com/package/vm2) sandbox only allows the following: - -- only some certain built-in modules, e.g. `assert`, `buffer`, `crypto`,`util` and `path` -- third-party libraries written by _CommonJS_. -- external `HTTP` and `WebSocket` connections are forbidden - -## Modules and Libraries - -To improve SubQuery's data processing capabilities, we have allowed some of the NodeJS's built-in modules for running mapping functions in the [sandbox](#third-party-library-support---the-sandbox), and have allowed users to call third-party libraries. - -Please note this is an **experimental feature** and you may encounter bugs or issues that may negatively impact your mapping functions. Please report any bugs you find by creating an issue in [GitHub](https://github.com/subquery/subql). - -### Built-in modules - -Currently, we allow the following NodeJS modules: `assert`, `buffer`, `crypto`, `util`, and `path`. - -Rather than importing the whole module, we recommend only importing the required method(s) that you need. Some methods in these modules may have dependencies that are unsupported and will fail on import. - -```ts -import { hashMessage } from "ethers/lib/utils"; // Good way -import { utils } from "ethers"; // Bad way -``` diff --git a/docs/indexer/build/mapping/ethereum.md b/docs/indexer/build/mapping/ethereum.md index a8986cf8ecc..1e8f9866ef5 100644 --- a/docs/indexer/build/mapping/ethereum.md +++ b/docs/indexer/build/mapping/ethereum.md @@ -5,6 +5,7 @@ Mapping functions define how chain data is transformed into the optimised GraphQ - Mappings are defined in the `src/mappings` directory and are exported as a function. - These mappings are also exported in `src/index.ts`. - The mappings files are reference in `project.ts` under the mapping handlers. +- The mappings are run from within a [Sandbox](./sandbox.md) There are different classes of mappings functions for Ethereum; [Block handlers](#block-handler), [Transaction Handlers](#transaction-handler), and [Log Handlers](#log-handler). @@ -85,58 +86,3 @@ const balance = await erc20.balanceOf(address); ``` The above example assumes that the user has an ABI file named `erc20.json`, so that TypeChain generates `ERC20__factory` class for them. Check out [this example](https://github.com/dethcrypto/TypeChain/tree/master/examples/ethers-v5) to see how to generate factory code around your contract ABI using TypeChain. - -## Third-party Library Support - the Sandbox - -SubQuery is deterministic by design, that means that each SubQuery project is guaranteed to index the same data set. This is a critical factor that is makes it possible to verify data in the decentralised SubQuery Network. This limitation means that in default configuration, the indexer is by default run in a strict virtual machine, with access to a strict number of third party libraries. - -**You can easily bypass this limitation however, allowing you to retrieve data from external API endpoints, non historical RPC calls, and import your own external libraries into your projects.** In order to do to, you must run your project in `unsafe` mode, you can read more about this in the [references](../../run_publish/references.md#unsafe-node-service). An easy way to do this while developing (and running in Docker) is to add the following line to your `docker-compose.yml`: - -```yml -subquery-node: - image: onfinality/subql-node-ethereum:latest - ... - command: - - -f=/app - - --db-schema=app - - --unsafe - ... -``` - -When run in `unsafe` mode, you can import any custom libraries into your project and make external API calls using tools like node-fetch. A simple example is given below: - -```ts -import { EthereumTransaction } from "@subql/types-ethereum"; -import fetch from "node-fetch"; - -export async function handleTransaction( - tx: EthereumTransaction, -): Promise { - const httpData = await fetch("https://api.github.com/users/github"); - logger.info(`httpData: ${JSON.stringify(httpData.body)}`); - // Do something with this data -} -``` - -By default (when in safe mode), the [VM2](https://www.npmjs.com/package/vm2) sandbox only allows the following: - -- only some certain built-in modules, e.g. `assert`, `buffer`, `crypto`,`util` and `path` -- third-party libraries written by _CommonJS_. -- external `HTTP` and `WebSocket` connections are forbidden - -## Modules and Libraries - -To improve SubQuery's data processing capabilities, we have allowed some of the NodeJS's built-in modules for running mapping functions in the [sandbox](#third-party-library-support---the-sandbox), and have allowed users to call third-party libraries. - -Please note this is an **experimental feature** and you may encounter bugs or issues that may negatively impact your mapping functions. Please report any bugs you find by creating an issue in [GitHub](https://github.com/subquery/subql). - -### Built-in modules - -Currently, we allow the following NodeJS modules: `assert`, `buffer`, `crypto`, `util`, and `path`. - -Rather than importing the whole module, we recommend only importing the required method(s) that you need. Some methods in these modules may have dependencies that are unsupported and will fail on import. - -```ts -import { hashMessage } from "ethers/lib/utils"; // Good way -import { utils } from "ethers"; // Bad way -``` diff --git a/docs/indexer/build/mapping/flare.md b/docs/indexer/build/mapping/flare.md index 493303b9fd3..d16d196b6e8 100644 --- a/docs/indexer/build/mapping/flare.md +++ b/docs/indexer/build/mapping/flare.md @@ -5,6 +5,7 @@ Mapping functions define how chain data is transformed into the optimised GraphQ - Mappings are defined in the `src/mappings` directory and are exported as a function. - These mappings are also exported in `src/index.ts`. - The mappings files are reference in `project.ts` under the mapping handlers. +- The mappings are run from within a [Sandbox](./sandbox.md) There are different classes of mappings functions for Flare; [Block handlers](#block-handler), [Transaction Handlers](#transaction-handler), and [Log Handlers](#log-handler). @@ -92,57 +93,3 @@ const balance = await erc20.balanceOf(address); The above example assumes that the user has an ABI file named `erc20.json`, so that TypeChain generates `ERC20__factory` class for them. Check out [this example](https://github.com/dethcrypto/TypeChain/tree/master/examples/ethers-v5) to see how to generate factory code around your contract ABI using TypeChain. -## Third-party Library Support - the Sandbox - -SubQuery is deterministic by design, that means that each SubQuery project is guaranteed to index the same data set. This is a critical factor that is makes it possible to verify data in the decentralised SubQuery Network. This limitation means that in default configuration, the indexer is by default run in a strict virtual machine, with access to a strict number of third party libraries. - -**You can easily bypass this limitation however, allowing you to retrieve data from external API endpoints, non historical RPC calls, and import your own external libraries into your projects.** In order to do to, you must run your project in `unsafe` mode, you can read more about this in the [references](../../run_publish/references.md#unsafe-node-service). An easy way to do this while developing (and running in Docker) is to add the following line to your `docker-compose.yml`: - -```yml -subquery-node: - image: onfinality/subql-node-ethereum:latest - ... - command: - - -f=/app - - --db-schema=app - - --unsafe - ... -``` - -When run in `unsafe` mode, you can import any custom libraries into your project and make external API calls using tools like node-fetch. A simple example is given below: - -```ts -import { EthereumTransaction } from "@subql/types-ethereum"; -import fetch from "node-fetch"; - -export async function handleTransaction( - tx: EthereumTransaction, -): Promise { - const httpData = await fetch("https://api.github.com/users/github"); - logger.info(`httpData: ${JSON.stringify(httpData.body)}`); - // Do something with this data -} -``` - -By default (when in safe mode), the [VM2](https://www.npmjs.com/package/vm2) sandbox only allows the following: - -- only some certain built-in modules, e.g. `assert`, `buffer`, `crypto`,`util` and `path` -- third-party libraries written by _CommonJS_. -- external `HTTP` and `WebSocket` connections are forbidden - -## Modules and Libraries - -To improve SubQuery's data processing capabilities, we have allowed some of the NodeJS's built-in modules for running mapping functions in the [sandbox](#third-party-library-support---the-sandbox), and have allowed users to call third-party libraries. - -Please note this is an **experimental feature** and you may encounter bugs or issues that may negatively impact your mapping functions. Please report any bugs you find by creating an issue in [GitHub](https://github.com/subquery/subql). - -### Built-in modules - -Currently, we allow the following NodeJS modules: `assert`, `buffer`, `crypto`, `util`, and `path`. - -Rather than importing the whole module, we recommend only importing the required method(s) that you need. Some methods in these modules may have dependencies that are unsupported and will fail on import. - -```ts -import { hashMessage } from "ethers/lib/utils"; // Good way -import { utils } from "ethers"; // Bad way -``` diff --git a/docs/indexer/build/mapping/gnosis.md b/docs/indexer/build/mapping/gnosis.md index 7049281390a..77e37e607b5 100644 --- a/docs/indexer/build/mapping/gnosis.md +++ b/docs/indexer/build/mapping/gnosis.md @@ -9,6 +9,7 @@ Mapping functions define how chain data is transformed into the optimised GraphQ - Mappings are defined in the `src/mappings` directory and are exported as a function. - These mappings are also exported in `src/index.ts`. - The mappings files are reference in `project.ts` under the mapping handlers. +- The mappings are run from within a [Sandbox](./sandbox.md) There are different classes of mappings functions for Gnosis; [Block handlers](#block-handler), [Transaction Handlers](#transaction-handler), and [Log Handlers](#log-handler). @@ -89,58 +90,3 @@ const balance = await erc20.balanceOf(address); ``` The above example assumes that the user has an ABI file named `erc20.json`, so that TypeChain generates `ERC20__factory` class for them. Check out [this example](https://github.com/dethcrypto/TypeChain/tree/master/examples/ethers-v5) to see how to generate factory code around your contract ABI using TypeChain. - -## Third-party Library Support - the Sandbox - -SubQuery is deterministic by design, that means that each SubQuery project is guaranteed to index the same data set. This is a critical factor that is makes it possible to verify data in the decentralised SubQuery Network. This limitation means that in default configuration, the indexer is by default run in a strict virtual machine, with access to a strict number of third party libraries. - -**You can easily bypass this limitation however, allowing you to retrieve data from external API endpoints, non historical RPC calls, and import your own external libraries into your projects.** In order to do to, you must run your project in `unsafe` mode, you can read more about this in the [references](../../run_publish/references.md#unsafe-node-service). An easy way to do this while developing (and running in Docker) is to add the following line to your `docker-compose.yml`: - -```yml -subquery-node: - image: onfinality/subql-node-ethereum:latest - ... - command: - - -f=/app - - --db-schema=app - - --unsafe - ... -``` - -When run in `unsafe` mode, you can import any custom libraries into your project and make external API calls using tools like node-fetch. A simple example is given below: - -```ts -import { EthereumTransaction } from "@subql/types-ethereum"; // We use ethereum types since Gnosis is a layer-2 that is compatible -import fetch from "node-fetch"; - -export async function handleTransaction( - tx: EthereumTransaction, -): Promise { - const httpData = await fetch("https://api.github.com/users/github"); - logger.info(`httpData: ${JSON.stringify(httpData.body)}`); - // Do something with this data -} -``` - -By default (when in safe mode), the [VM2](https://www.npmjs.com/package/vm2) sandbox only allows the following: - -- only some certain built-in modules, e.g. `assert`, `buffer`, `crypto`,`util` and `path` -- third-party libraries written by _CommonJS_. -- external `HTTP` and `WebSocket` connections are forbidden - -## Modules and Libraries - -To improve SubQuery's data processing capabilities, we have allowed some of the NodeJS's built-in modules for running mapping functions in the [sandbox](#third-party-library-support---the-sandbox), and have allowed users to call third-party libraries. - -Please note this is an **experimental feature** and you may encounter bugs or issues that may negatively impact your mapping functions. Please report any bugs you find by creating an issue in [GitHub](https://github.com/subquery/subql). - -### Built-in modules - -Currently, we allow the following NodeJS modules: `assert`, `buffer`, `crypto`, `util`, and `path`. - -Rather than importing the whole module, we recommend only importing the required method(s) that you need. Some methods in these modules may have dependencies that are unsupported and will fail on import. - -```ts -import { hashMessage } from "ethers/lib/utils"; // Good way -import { utils } from "ethers"; // Bad way -``` diff --git a/docs/indexer/build/mapping/near.md b/docs/indexer/build/mapping/near.md index b3bc4e99dfd..62aca664761 100644 --- a/docs/indexer/build/mapping/near.md +++ b/docs/indexer/build/mapping/near.md @@ -5,6 +5,7 @@ Mapping functions define how chain data is transformed into the optimised GraphQ - Mappings are defined in the `src/mappings` directory and are exported as a function. - These mappings are also exported in `src/index.ts`. - The mappings files are reference in `project.ts` under the mapping handlers. +- The mappings are run from within a [Sandbox](./sandbox.md) There are different classes of mappings functions for NEAR; [Block handlers](#block-handler), [Transaction Handlers](#transaction-handler), and [Action Handlers](#action-handler). @@ -106,58 +107,3 @@ We also support some [API RPC methods here](https://github.com/subquery/subql-ne Documents in [NEAR `JsonRpcProvider`](https://docs.near.org/tools/near-api-js/reference/classes/providers_json_rpc_provider.JsonRpcProvider.html) provide some methods to interact with the NEAR RPC API. -## Third-party Library Support - the Sandbox - -SubQuery is deterministic by design, that means that each SubQuery project is guaranteed to index the same data set. This is a critical factor that is makes it possible to verify data in the decentralised SubQuery Network. This limitation means that in default configuration, the indexer is by default run in a strict virtual machine, with access to a strict number of third party libraries. - -**You can easily bypass this limitation however, allowing you to retrieve data from external API endpoints, non historical RPC calls, and import your own external libraries into your projects.** In order to do to, you must run your project in `unsafe` mode, you can read more about this in the [references](../../run_publish/references.md#unsafe-node-service). An easy way to do this while developing (and running in Docker) is to add the following line to your `docker-compose.yml`: - -```yml -subquery-node: - image: onfinality/subql-node-near:latest - ... - command: - - -f=/app - - --db-schema=app - - --unsafe - ... -``` - -When run in `unsafe` mode, you can import any custom libraries into your project and make external API calls using tools like node-fetch. A simple example is given below: - -```ts -import { NearAction, Transfer } from "@subql/types-near"; -import fetch from "node-fetch"; - -export async function handleAction( - action: NearAction, -): Promise { - const httpData = await fetch("https://api.github.com/users/github"); - logger.info(`httpData: ${JSON.stringify(httpData.body)}`); - // Do something with this data -} -``` - -By default (when in safe mode), the [VM2](https://www.npmjs.com/package/vm2) sandbox only allows the following: - -- only some certain built-in modules, e.g. `assert`, `buffer`, `crypto`,`util` and `path` -- third-party libraries written by _CommonJS_. -- Historical/safe queries, see [RPC Calls](#rpc-calls). -- external `HTTP` and `WebSocket` connections are forbidden - -## Modules and Libraries - -To improve SubQuery's data processing capabilities, we have allowed some of the NodeJS's built-in modules for running mapping functions in the [sandbox](#third-party-library-support---the-sandbox), and have allowed users to call third-party libraries. - -Please note this is an **experimental feature** and you may encounter bugs or issues that may negatively impact your mapping functions. Please report any bugs you find by creating an issue in [GitHub](https://github.com/subquery/subql). - -### Built-in modules - -Currently, we allow the following NodeJS modules: `assert`, `buffer`, `crypto`, `util`, and `path`. - -Rather than importing the whole module, we recommend only importing the required method(s) that you need. Some methods in these modules may have dependencies that are unsupported and will fail on import. - -```ts -import { hashMessage } from "ethers/lib/utils"; // Good way -import { utils } from "ethers"; // Bad way -``` diff --git a/docs/indexer/build/mapping/optimism.md b/docs/indexer/build/mapping/optimism.md index c2420fc9091..4c3d77d80b7 100644 --- a/docs/indexer/build/mapping/optimism.md +++ b/docs/indexer/build/mapping/optimism.md @@ -9,6 +9,7 @@ Mapping functions define how chain data is transformed into the optimised GraphQ - Mappings are defined in the `src/mappings` directory and are exported as a function. - These mappings are also exported in `src/index.ts`. - The mappings files are reference in `project.ts` under the mapping handlers. +- The mappings are run from within a [Sandbox](./sandbox.md) There are different classes of mappings functions for Optimism; [Block handlers](#block-handler), [Transaction Handlers](#transaction-handler), and [Log Handlers](#log-handler). @@ -89,58 +90,3 @@ const balance = await erc20.balanceOf(address); ``` The above example assumes that the user has an ABI file named `erc20.json`, so that TypeChain generates `ERC20__factory` class for them. Check out [this example](https://github.com/dethcrypto/TypeChain/tree/master/examples/ethers-v5) to see how to generate factory code around your contract ABI using TypeChain. - -## Third-party Library Support - the Sandbox - -SubQuery is deterministic by design, that means that each SubQuery project is guaranteed to index the same data set. This is a critical factor that is makes it possible to verify data in the decentralised SubQuery Network. This limitation means that in default configuration, the indexer is by default run in a strict virtual machine, with access to a strict number of third party libraries. - -**You can easily bypass this limitation however, allowing you to retrieve data from external API endpoints, non historical RPC calls, and import your own external libraries into your projects.** In order to do to, you must run your project in `unsafe` mode, you can read more about this in the [references](../../run_publish/references.md#unsafe-node-service). An easy way to do this while developing (and running in Docker) is to add the following line to your `docker-compose.yml`: - -```yml -subquery-node: - image: onfinality/subql-node-ethereum:latest - ... - command: - - -f=/app - - --db-schema=app - - --unsafe - ... -``` - -When run in `unsafe` mode, you can import any custom libraries into your project and make external API calls using tools like node-fetch. A simple example is given below: - -```ts -import { EthereumTransaction } from "@subql/types-ethereum"; // We use ethereum types since Optimism is a layer-2 that is compatible -import fetch from "node-fetch"; - -export async function handleTransaction( - tx: EthereumTransaction, -): Promise { - const httpData = await fetch("https://api.github.com/users/github"); - logger.info(`httpData: ${JSON.stringify(httpData.body)}`); - // Do something with this data -} -``` - -By default (when in safe mode), the [VM2](https://www.npmjs.com/package/vm2) sandbox only allows the following: - -- only some certain built-in modules, e.g. `assert`, `buffer`, `crypto`,`util` and `path` -- third-party libraries written by _CommonJS_. -- external `HTTP` and `WebSocket` connections are forbidden - -## Modules and Libraries - -To improve SubQuery's data processing capabilities, we have allowed some of the NodeJS's built-in modules for running mapping functions in the [sandbox](#third-party-library-support---the-sandbox), and have allowed users to call third-party libraries. - -Please note this is an **experimental feature** and you may encounter bugs or issues that may negatively impact your mapping functions. Please report any bugs you find by creating an issue in [GitHub](https://github.com/subquery/subql). - -### Built-in modules - -Currently, we allow the following NodeJS modules: `assert`, `buffer`, `crypto`, `util`, and `path`. - -Rather than importing the whole module, we recommend only importing the required method(s) that you need. Some methods in these modules may have dependencies that are unsupported and will fail on import. - -```ts -import { hashMessage } from "ethers/lib/utils"; // Good way -import { utils } from "ethers"; // Bad way -``` diff --git a/docs/indexer/build/mapping/polkadot.md b/docs/indexer/build/mapping/polkadot.md index 410925fd022..1371b422424 100644 --- a/docs/indexer/build/mapping/polkadot.md +++ b/docs/indexer/build/mapping/polkadot.md @@ -5,6 +5,7 @@ Mapping functions define how chain data is transformed into the optimised GraphQ - Mappings are defined in the `src/mappings` directory and are exported as a function. - These mappings are also exported in `src/index.ts`. - The mappings files are referenced in `project.ts` under the mapping handlers. +- The mappings are run from within a [Sandbox](./sandbox.md) There are different classes of mappings functions for Polkadot/Substrate; [Block handlers](#block-handler), [Event Handlers](#event-handler), and [Call Handlers](#call-handler). @@ -93,60 +94,6 @@ async function handleEvmCall( ::: -## Third-party Library Support - the Sandbox - -SubQuery is deterministic by design, that means that each SubQuery project is guaranteed to index the same data set. This is a critical factor that is makes it possible to verify data in the decentralised SubQuery Network. This limitation means that in default configuration, the indexer is by default run in a strict virtual machine, with access to a strict number of third party libraries. - -**You can easily bypass this limitation however, allowing you to retrieve data from external API endpoints, non historical RPC calls, and import your own external libraries into your projects.** In order to do so, you must run your project in `unsafe` mode, you can read more about this in the [references](../../run_publish/references.md#unsafe-node-service). An easy way to do this while developing (and running in Docker) is to add the following line to your `docker-compose.yml`: - -```yml -subquery-node: - image: onfinality/subql-node:latest - ... - command: - - -f=/app - - --db-schema=app - - --unsafe - ... -``` - -When run in `unsafe` mode, you can import any custom libraries into your project and make external API calls using tools like node-fetch. A simple example is given below: - -```ts -import { SubstrateEvent } from "@subql/types"; -import fetch from "node-fetch"; - -export async function handleEvent(event: SubstrateEvent): Promise { - const httpData = await fetch("https://api.github.com/users/github"); - logger.info(`httpData: ${JSON.stringify(httpData.body)}`); - // Do something with this data -} -``` - -By default (when in safe mode), the [VM2](https://www.npmjs.com/package/vm2) sandbox only allows the following: - -- only some certain built-in modules, e.g. `assert`, `buffer`, `crypto`,`util` and `path` -- third-party libraries written by _CommonJS_. -- hybrid libraries like `@polkadot/*` that uses ESM as default. However, if any other libraries depend on any modules in _ESM_ format, the virtual machine will _NOT_ compile and return an error. -- Historical/safe queries, see [RPC Calls](#rpc-calls). -- external `HTTP` and `WebSocket` connections are forbidden - -## Modules and Libraries - -To improve SubQuery's data processing capabilities, we have allowed some of the NodeJS's built-in modules for running mapping functions in the [sandbox](=#third-party-library-support---the-sandbox), and have allowed users to call third-party libraries. - -Please note this is an **experimental feature** and you may encounter bugs or issues that may negatively impact your mapping functions. Please report any bugs you find by creating an issue in [GitHub](https://github.com/subquery/subql). - -### Built-in modules - -Currently, we allow the following NodeJS modules: `assert`, `buffer`, `crypto`, `util`, and `path`. - -Rather than importing the whole module, we recommend only importing the required method(s) that you need. Some methods in these modules may have dependencies that are unsupported and will fail on import. - -```ts -import { hashMessage } from "ethers/lib/utils"; // Good way -import { utils } from "ethers"; // Bad way -``` ### Query States diff --git a/docs/indexer/build/mapping/polygon.md b/docs/indexer/build/mapping/polygon.md index ef3f1b0396b..c2d0c97c8d5 100644 --- a/docs/indexer/build/mapping/polygon.md +++ b/docs/indexer/build/mapping/polygon.md @@ -9,6 +9,7 @@ Mapping functions define how chain data is transformed into the optimised GraphQ - Mappings are defined in the `src/mappings` directory and are exported as a function. - These mappings are also exported in `src/index.ts`. - The mappings files are reference in `project.ts` under the mapping handlers. +- The mappings are run from within a [Sandbox](./sandbox.md) There are different classes of mappings functions for Polygon; [Block handlers](#block-handler), [Transaction Handlers](#transaction-handler), and [Log Handlers](#log-handler). @@ -89,58 +90,3 @@ const balance = await erc20.balanceOf(address); ``` The above example assumes that the user has an ABI file named `erc20.json`, so that TypeChain generates `ERC20__factory` class for them. Check out [this example](https://github.com/dethcrypto/TypeChain/tree/master/examples/ethers-v5) to see how to generate factory code around your contract ABI using TypeChain. - -## Third-party Library Support - the Sandbox - -SubQuery is deterministic by design, that means that each SubQuery project is guaranteed to index the same data set. This is a critical factor that is makes it possible to verify data in the decentralised SubQuery Network. This limitation means that in default configuration, the indexer is by default run in a strict virtual machine, with access to a strict number of third party libraries. - -**You can easily bypass this limitation however, allowing you to retrieve data from external API endpoints, non historical RPC calls, and import your own external libraries into your projects.** In order to do to, you must run your project in `unsafe` mode, you can read more about this in the [references](../../run_publish/references.md#unsafe-node-service). An easy way to do this while developing (and running in Docker) is to add the following line to your `docker-compose.yml`: - -```yml -subquery-node: - image: onfinality/subql-node-ethereum:latest - ... - command: - - -f=/app - - --db-schema=app - - --unsafe - ... -``` - -When run in `unsafe` mode, you can import any custom libraries into your project and make external API calls using tools like node-fetch. A simple example is given below: - -```ts -import { EthereumTransaction } from "@subql/types-ethereum"; // We use ethereum types since Polygon is a layer-2 that is compatible -import fetch from "node-fetch"; - -export async function handleTransaction( - tx: EthereumTransaction, -): Promise { - const httpData = await fetch("https://api.github.com/users/github"); - logger.info(`httpData: ${JSON.stringify(httpData.body)}`); - // Do something with this data -} -``` - -By default (when in safe mode), the [VM2](https://www.npmjs.com/package/vm2) sandbox only allows the following: - -- only some certain built-in modules, e.g. `assert`, `buffer`, `crypto`,`util` and `path` -- third-party libraries written by _CommonJS_. -- external `HTTP` and `WebSocket` connections are forbidden - -## Modules and Libraries - -To improve SubQuery's data processing capabilities, we have allowed some of the NodeJS's built-in modules for running mapping functions in the [sandbox](#third-party-library-support---the-sandbox), and have allowed users to call third-party libraries. - -Please note this is an **experimental feature** and you may encounter bugs or issues that may negatively impact your mapping functions. Please report any bugs you find by creating an issue in [GitHub](https://github.com/subquery/subql). - -### Built-in modules - -Currently, we allow the following NodeJS modules: `assert`, `buffer`, `crypto`, `util`, and `path`. - -Rather than importing the whole module, we recommend only importing the required method(s) that you need. Some methods in these modules may have dependencies that are unsupported and will fail on import. - -```ts -import { hashMessage } from "ethers/lib/utils"; // Good way -import { utils } from "ethers"; // Bad way -``` diff --git a/docs/indexer/build/mapping/sandbox.md b/docs/indexer/build/mapping/sandbox.md new file mode 100644 index 00000000000..1e5b57dc9f9 --- /dev/null +++ b/docs/indexer/build/mapping/sandbox.md @@ -0,0 +1,102 @@ +# Sandbox + +SubQuery is deterministic by design, that means that each SubQuery project is guaranteed to index the same data set. This is a critical factor that is makes it possible to verify data in the decentralised SubQuery Network. This limitation means that in default configuration, the indexer is by default run in a strict virtual machine. + +By default (when in safe mode), the [VM2](https://www.npmjs.com/package/vm2) sandbox only allows the following: + +- only some certain nodejs built-in modules, e.g. `assert`, `buffer`, `crypto`, `util`, `path`, `url` and `stream` +- third-party libraries written by _CommonJS_. +- external `HTTP` and `WebSocket` connections are forbidden + +## Global Variables + +In order to make indexing easier SubQuery provides some global variables for convenience. With typescript your editor should be aware of these variables and building should result in errors if the types are incorrect. + +### `chainId` + +If your project is deployed on multiple networks and you have a need for chain specific code or behaviour then you can access this to get the chainId that the handler is running for/ + +```ts +logger.info(`The chain id is ${chainId}`); +```` + +### `api` + +This is an instance of the networks API that is connected to the provided network endpoints. +This is only available if the network supports an API that is scoped to the current indexed block height. i.e querying state will return the state at the current block height. + + +### `unsafeApi` + +This is the same as the `api` described above but not scoped to the current block being processed. This is available with all chains. + +### `store` + +The data store, for more details please read the [store](./store.md) documentation. For most use cases this will not be needed and you should use the generated entities instead. + +### `cache` + +If you wish to store in memory variables outside of your handler the cache should be used instead. This is to ensure that data is shared between instances of the sandbox when workers are used. For more details read the [In-Memory Cache](./cache.md) documentation. + +### `logger` + +From within the sandbox `console` doesn't work and you will not see any output from this. Instead we inject a logger as an alternative. This logger is based on Pino and each of the logger methods take a single argument. + +```ts +logger.info('Hello world'); +``` + +## Third Party Libraries + +SubQuery allows importing third party libraries but there are some caveats and things to be aware of. + +* Only _CommonJS_ is supported, not _ESM_ modules. For some libraries this might mean using older versions of the package. + - For example, `node-fetch` is only available as a CommonJS module up to version 2.6.7. + +* Rather than importing the whole module, we recommend only importing the required method(s) that you need and narrowing your imports as much as possible. Some methods in these modules may have dependencies that are unsupported and will fail on import. + +```ts +import { hashMessage } from "ethers/lib/utils"; // Good way +import { utils } from "ethers"; // Bad way +``` + +## Unsafe + +**You can easily bypass this limitation however, allowing you to retrieve data from external API endpoints, non historical RPC calls, and import your own external libraries into your projects.** In order to do to, you must run your project in `unsafe` mode, you can read more about this in the [references](../../run_publish/references.md#unsafe-node-service). An easy way to do this while developing (and running in Docker) is to add the following line to your `docker-compose.yml`: + +```yml +subquery-node: + image: onfinality/subql-node-ethereum:latest + ... + command: + - -f=/app + - --db-schema=app + - --unsafe + ... +``` + +When run in `unsafe` mode, you can import any custom libraries into your project and make external API calls using tools like node-fetch. A simple example is given below: + +```ts +import { EthereumTransaction } from "@subql/types-ethereum"; +import fetch from "node-fetch"; + +export async function handleTransaction( + tx: EthereumTransaction, +): Promise { + const httpData = await fetch("https://api.github.com/users/github"); + logger.info(`httpData: ${JSON.stringify(httpData.body)}`); + // Do something with this data +} +``` + + +## Other differences and incompatibilities + +* `TextEncoder` and `TextDecoder` are not available by default. If your handlers or a dependency need these they can be polyfilled like so: +```ts +// Add `text-encoding` as a dependency and include this code in index.ts before any other imports +import { TextDecoder, TextEncoder } from 'text-encoding'; +global.TextEncoder = TextEncoder; +global.TextDecoder = TextDecoder; +``` diff --git a/docs/indexer/build/mapping/starknet.md b/docs/indexer/build/mapping/starknet.md index 49687eea5aa..e076b2f3ccb 100644 --- a/docs/indexer/build/mapping/starknet.md +++ b/docs/indexer/build/mapping/starknet.md @@ -5,6 +5,7 @@ Mapping functions define how chain data is transformed into the optimised GraphQ - Mappings are defined in the `src/mappings` directory and are exported as a function. - These mappings are also exported in `src/index.ts`. - The mappings files are reference in `project.ts` under the mapping handlers. +- The mappings are run from within a [Sandbox](./sandbox.md) There are different classes of mappings functions for Starknet; [Block handlers](#block-handler), [Transaction Handlers](#transaction-handler), and [Log Handlers](#log-handler). @@ -125,53 +126,3 @@ const myTestContract = new Contract(compiledTest.abi, testAddress, provider); const bal1 = await myTestContract.get_balance(); logger.info(`Initial balance: ${bal1.toString()}`); // Cairo 1 contract ``` - -## Third-party Library Support - the Sandbox - -SubQuery is deterministic by design, that means that each SubQuery project is guaranteed to index the same data set. This is a critical factor that is makes it possible to verify data in the decentralised SubQuery Network. This limitation means that in default configuration, the indexer is by default run in a strict virtual machine, with access to a strict number of third party libraries. - -**You can easily bypass this limitation however, allowing you to retrieve data from external API endpoints, non historical RPC calls, and import your own external libraries into your projects.** In order to do to, you must run your project in `unsafe` mode, you can read more about this in the [references](../../run_publish/references.md#unsafe-node-service). An easy way to do this while developing (and running in Docker) is to add the following line to your `docker-compose.yml`: - -```yml -subquery-node: - image: onfinality/subql-node-starknet:latest - ... - command: - - -f=/app - - --db-schema=app - - --unsafe - ... -``` - -When run in `unsafe` mode, you can import any custom libraries into your project and make external API calls using tools like node-fetch. A simple example is given below: - -```ts -import { StarknetTransaction } from "@subql/types-starknet"; -import fetch from "node-fetch"; - -export async function handleTransaction( - tx: StarknetTransaction, -): Promise { - const httpData = await fetch("https://api.github.com/users/github"); - logger.info(`httpData: ${JSON.stringify(httpData.body)}`); - // Do something with this data -} -``` - -By default (when in safe mode), the [VM2](https://www.npmjs.com/package/vm2) sandbox only allows the following: - -- only some certain built-in modules, e.g. `assert`, `buffer`, `crypto`,`util` and `path` -- third-party libraries written by _CommonJS_. -- external `HTTP` and `WebSocket` connections are forbidden - -## Modules and Libraries - -To improve SubQuery's data processing capabilities, we have allowed some of the NodeJS's built-in modules for running mapping functions in the [sandbox](#third-party-library-support---the-sandbox), and have allowed users to call third-party libraries. - -Please note this is an **experimental feature** and you may encounter bugs or issues that may negatively impact your mapping functions. Please report any bugs you find by creating an issue in [GitHub](https://github.com/subquery/subql). - -### Built-in modules - -Currently, we allow the following NodeJS modules: `assert`, `buffer`, `crypto`, `util`, and `path`. - -Rather than importing the whole module, we recommend only importing the required method(s) that you need. Some methods in these modules may have dependencies that are unsupported and will fail on import. diff --git a/docs/indexer/build/mapping/stellar.md b/docs/indexer/build/mapping/stellar.md index c6e4e9d2070..8d7f039cb78 100644 --- a/docs/indexer/build/mapping/stellar.md +++ b/docs/indexer/build/mapping/stellar.md @@ -5,6 +5,7 @@ Mapping functions define how chain data is transformed into the optimised GraphQ - Mappings are defined in the `src/mappings` directory and are exported as a function. - These mappings are also exported in `src/index.ts`. - The mappings files are reference in `project.ts` under the mapping handlers. +- The mappings are run from within a [Sandbox](./sandbox.md) There are different classes of mapping functions for Stellar; [Block handlers](#block-handler), [Operation Handlers](#operation-handler), and [Effect Handlers](#effect-handler). @@ -127,45 +128,3 @@ export async function handleEvent(event: SorobanEvent): Promise { await transfer.save(); } ``` - -## Third-party Library Support - the Sandbox - -SubQuery is deterministic by design, that means that each SubQuery project is guaranteed to index the same data set. This is a critical factor that is makes it possible to verify data in the decentralised SubQuery Network. This limitation means that in default configuration, the indexer is by default run in a strict virtual machine, with access to a strict number of third party libraries. - -**You can easily bypass this limitation however, allowing you to retrieve data from external API endpoints, non historical RPC calls, and import your own external libraries into your projects.** In order to do to, you must run your project in `unsafe` mode, you can read more about this in the [references](../../run_publish/references.md#unsafe-node-service). An easy way to do this while developing (and running in Docker) is to add the following line to your `docker-compose.yml`: - -```yml -subquery-node: - image: subquerynetwork/subql-node-stellar:latest - ... - command: - - -f=/app - - --db-schema=app - - --unsafe - ... -``` - -When run in `unsafe` mode, you can import any custom libraries into your project and make external API calls using tools like node-fetch. A simple example is given below: - -```ts -import { StellarEvent } from "@subql/types-stellar"; -import fetch from "node-fetch"; - -export async function handleEvent(event: StellarEvent): Promise { - const httpData = await fetch("https://api.github.com/users/github"); - logger.info(`httpData: ${JSON.stringify(httpData.body)}`); - // Do something with this data -} -``` - -By default (when in safe mode), the [VM2](https://www.npmjs.com/package/vm2) sandbox only allows the following: - -- only some certain built-in modules, e.g. `assert`, `buffer`, `crypto`,`util` and `path` -- third-party libraries written by _CommonJS_. -- external `HTTP` and `WebSocket` connections are forbidden. - -## Modules and Libraries - -To improve SubQuery's data processing capabilities, we have allowed some of the NodeJS's built-in modules for running mapping functions in the [sandbox](#third-party-library-support---the-sandbox), and have allowed users to call third-party libraries. - -Please note this is an **experimental feature** and you may encounter bugs or issues that may negatively impact your mapping functions. Please report any bugs you find by creating an issue in [GitHub](https://github.com/subquery/subql).