Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(cactus-plugin-persistence-ethereum): add new persistence plugin #2259

Merged
merged 1 commit into from
Mar 30, 2023
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion .cspell.json
Original file line number Diff line number Diff line change
@@ -139,7 +139,11 @@
"uuidv",
"vscc",
"wasm",
"Xdai"
"Xdai",
"goquorum",
"hada",
"undici",
"ossp"
],
"dictionaries": [
"typescript,node,npm,go,rust"
27 changes: 27 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -933,6 +933,33 @@ jobs:
restore-keys: |
${{ runner.os }}-yarn-
- run: ./tools/ci.sh
cactus-plugin-persistence-ethereum:
continue-on-error: false
env:
FULL_BUILD_DISABLED: true
JEST_TEST_PATTERN: packages/cactus-plugin-persistence-ethereum/src/test/typescript/(unit|integration|benchmark)/.*/*.test.ts
JEST_TEST_RUNNER_DISABLED: false
TAPE_TEST_RUNNER_DISABLED: true
needs: build-dev
runs-on: ubuntu-20.04
steps:
- name: Use Node.js v16.14.2
uses: actions/setup-node@v2.1.2
with:
node-version: v16.14.2
- uses: actions/checkout@v2.3.4
- id: yarn-cache-dir-path
name: Get yarn cache directory path
run: echo "::set-output name=dir::$(yarn cache dir)"
- id: yarn-cache
name: Restore Yarn Cache
uses: actions/cache@v3.0.4
with:
key: ${{ runner.os }}-yarn-${{ hashFiles('**/yarn.lock') }}
path: ${{ steps.yarn-cache-dir-path.outputs.dir }}
restore-keys: |
${{ runner.os }}-yarn-
- run: ./tools/ci.sh
cactus-plugin-object-store-ipfs:
continue-on-error: false
env:
14 changes: 14 additions & 0 deletions packages/cactus-plugin-persistence-ethereum/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
FROM node:16

ENV PACKAGE_PATH=/opt/cactus-plugin-persistence-ethereum

WORKDIR ${PACKAGE_PATH}

# CMake is required by one of npm dependencies (install other packages in this step as well in the future)
RUN apt-get update && apt-get install -y cmake && rm -rf /var/lib/apt/lists/*

COPY ./dist/yarn.lock ./package.json ./
RUN yarn install --production --ignore-engines --non-interactive --cache-folder ./.yarnCache && \
rm -rf ./.yarnCache

COPY ./dist ./dist
237 changes: 237 additions & 0 deletions packages/cactus-plugin-persistence-ethereum/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,237 @@
# `@hyperledger/cactus-plugin-persistence-ethereum`

This plugin allows `Cactus` to persist Ethereum data into some storage (currently to a `PostgreSQL` database, but this concept can be extended further).
Data in the database can later be analyzed and viewed in a GUI tool.
petermetz marked this conversation as resolved.
Show resolved Hide resolved

## Summary

- [Remarks](#remarks)
- [Getting Started](#getting-started)
- [Endpoints](#endpoints)
- [Running the tests](#running-the-tests)
- [Contributing](#contributing)
- [License](#license)
- [Acknowledgments](#acknowledgments)

## Remarks

- This plugin was only tested with small, permissioned Ethereum ledgers. Running it to archive and monitor large ledgers (like main net) is not recommended yet.
- For now, the database schema is not considered public and can change over time (i.e., writing own application that reads data directly from the database is discouraged).
- Only `status` endpoint is available, all the methods must be called directly on the plugin instance for now.
- Monitored ERC20 tokens should be added before synchronizing the database (previous transfers will not be parsed correctly if you later add the token).

## Getting Started

Clone the git repository on your local machine. Follow these instructions that will get you a copy of the project up and running on your local machine for development and testing purposes.

### Prerequisites

In the root of the project, execute the command to install and build the dependencies. It will also build this persistence plugin:

```sh
yarn run configure
```

### Usage

Instantiate a new `PluginPersistenceEthereum` instance:

```typescript
import { PluginPersistenceEthereum } from "@hyperledger/cactus-plugin-persistence-ethereum";
import { v4 as uuidv4 } from "uuid";

const persistencePlugin = new PluginPersistenceEthereum({
instanceId: uuidv4(),
apiClient: new SocketIOApiClient(apiConfigOptions),
logLevel: "info",
connectionString:
"postgresql://postgres:your-super-secret-and-long-postgres-password@localhost:5432/postgres",
});

// Initialize the connection to the DB
await persistencePlugin.onPluginInit();
```

Alternatively, import `PluginFactoryLedgerPersistence` from the plugin package and use it to create a plugin.

```typescript
import { PluginFactoryLedgerPersistence } from "@hyperledger/cactus-plugin-persistence-ethereum";
import { PluginImportType } from "@hyperledger/cactus-core-api";
import { v4 as uuidv4 } from "uuid";

const factory = new PluginFactoryLedgerPersistence({
pluginImportType: PluginImportType.Local,
});

const persistencePlugin = await factory.create({
instanceId: uuidv4(),
apiClient: new SocketIOApiClient(apiConfigOptions),
logLevel: "info",
connectionString:
"postgresql://postgres:your-super-secret-and-long-postgres-password@localhost:5432/postgres",
});

// Initialize the connection to the DB
await persistencePlugin.onPluginInit();
```

You can use the persistent plugin to start monitoring token balance changes and synchronize ledger state with the database.
Here is a sample script that adds two tokens to monitor, synchronizes all currently issued ERC721 tokens and starts monitoring for new blocks:

```typescript
// Add ERC20 token under address erc20ContractAddress, monitor all transfers.
await persistencePlugin.addTokenERC20(erc20ContractAddress);

// Add ERC721 token as well to monitor transfers
await persistencePlugin.addTokenERC721(erc721ContractAddress);

// Synchronize all issued ERC721 token balances that we currently monitor
await persistencePlugin.syncERC721Tokens();

// Start monitoring new blocks.
// Transactions in each block are parsed, token transfers update current token balances.
// Entire ledger is synchronized first with the DB (`syncAll` is called) so this operation can take a while on large ledgers!
persistencePlugin.startMonitor((err) => {
reject(err);
});

// Show current status of the plugin
persistencePlugin.getStatus();
```

> See [plugin integration tests](./src/test/typescript/integration) for complete usage examples.

### Building/running the container image locally

In the Cactus project root say:

```sh
DOCKER_BUILDKIT=1 docker build ./packages/cactus-plugin-persistence-ethereum/ -f ./packages/cactus-plugin-persistence-ethereum/Dockerfile -t cactus-plugin-persistence-ethereum
```

## Endpoints

### StatusV1 (`/api/v1/plugins/@hyperledger/cactus-plugin-persistence-ethereum/status`)

- Returns status of the plugin (latest block read, failed blocks, is monitor running, etc...)

### Plugin Methods

- Most of the plugin functionalities are currently not available through OpenAPI interface, please use direct method calls instead.

#### `onPluginInit`

- Should be called before using the plugin.

#### `shutdown`

- Close the connection to the DB, cleanup any allocated resources.

#### `getStatus`

- Get status report of this instance of persistence plugin.

#### `refreshMonitoredTokens`

- Fetch the metadata of all tokens to be monitored by this persistence plugin.

#### `syncERC721Tokens`

- Synchronize issued tokens for all ERC721 token contract monitored by this persistence plugin.

#### `startMonitor`

- Start the block monitoring process. New blocks from the ledger will be parsed and pushed to the database.

#### `stopMonitor`

- Stop the block monitoring process.

#### `addTokenERC20`

- Add new ERC20 token to be monitored by this plugin.

#### `addTokenERC721`

- Add new ERC721 token to be monitored by this plugin.

#### `syncFailedBlocks`

- Walk through all the blocks that could not be synchronized with the DB for some reasons and try pushing them again.

#### `syncAll`

- Synchronize entire ledger state with the database.

## Running the tests

To run all the tests for this persistence plugin to ensure it's working correctly execute the following from the root of the `cactus` project:

```sh
npx jest cactus-plugin-persistence-ethereum
```

## Contributing

We welcome contributions to Hyperledger Cactus in many forms, and there’s always plenty to do!

Please review [CONTIRBUTING.md](../../CONTRIBUTING.md) to get started.

### Quick plugin project walkthrough

#### ./src/main/json/contract_abi

- Contains reference token ABIs used to call and identify token transfers.

#### `./src/main/json/openapi.json`

- Contains OpenAPI definition.

#### `./src/main/sql/schema.sql`

- Database schema for Ethereum data.

#### `./src/main/typescript/token-client`

- Client used to execute methods on token contracts.

#### `./src/main/typescript/web-services`

- Folder that contains web service endpoint definitions.

#### `./plugin-persistence-ethereum`

- Main persistent plugin logic file

#### `./src/test/typescript/integration/`

- Integration test of various plugin functionalities.

### Generating types from the database schema

- Current setup assume use of Supabase that has utility for generating types from the database schema.
- We use this tool to generate type definitions and store them in `./src/main/typescript/db-client/database.types.ts`
- Upstream instructions: https://supabase.com/docs/guides/api/generating-types
- Step by step manual on updating the types (must be done after changing the database schema):
- Install `supabase` package
- Init and start development supabase server:
- `npx supabase init`
- `npx supabase start`
- Fill in current schema:
- `psql -h localhost -p 54322 -U postgres -d postgres -a -f src/main/sql/schema.sql` (password: `postgres`)
- Generate the file with type definitions:
- `npx supabase gen types typescript --schema public --local > src/main/typescript/db-client/database.types.ts`
- Cleanup:
- `npx supabase stop`
- `rm -rf ./supabase`

#### Insert sample data

- Can be used to test GUI applications without running entire ledger / persistence setup.
- `psql -h localhost -p 54322 -U postgres -d postgres -a -f src/test/sql/insert-test-data.sql` (password: `postgres`)

## License

This distribution is published under the Apache License Version 2.0 found in the [LICENSE](../../LICENSE) file.

## Acknowledgments
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
{
"$schema": "node_modules/@openapitools/openapi-generator-cli/config.schema.json",
"spaces": 2,
"generator-cli": {
"version": "5.2.1"
}
}
91 changes: 91 additions & 0 deletions packages/cactus-plugin-persistence-ethereum/package.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
{
"name": "@hyperledger/cactus-plugin-persistence-ethereum",
"version": "1.2.0",
"description": "Persistence plugin for Ethereum ledgers to store data into a database.",
"keywords": [
"Hyperledger",
"Cactus",
"Integration",
"Blockchain",
"Distributed Ledger Technology"
],
"homepage": "https://github.com/hyperledger/cactus#readme",
"bugs": {
"url": "https://github.com/hyperledger/cactus/issues"
},
"repository": {
"type": "git",
"url": "git+https://github.com/hyperledger/cactus.git"
},
"license": "Apache-2.0",
"author": {
"name": "Hyperledger Cactus Contributors",
"email": "cactus@lists.hyperledger.org",
"url": "https://www.hyperledger.org/use/cactus"
},
"contributors": [
{
"name": "Please add yourself to the list of contributors",
"email": "your.name@example.com",
"url": "https://example.com"
},
{
"name": "Michal Bajer",
"email": "michal.bajer@fujitsu.com",
"url": "https://www.fujitsu.com/global/"
},
{
"name": "Tomasz Awramski",
"email": "tomasz.awramski@fujitsu.com",
"url": "https://www.fujitsu.com/global/"
}
],
"main": "dist/lib/main/typescript/index.js",
"module": "dist/lib/main/typescript/index.js",
"types": "dist/lib/main/typescript/index.d.ts",
"files": [
"dist/*"
],
"scripts": {
"codegen": "run-p 'codegen:*'",
"codegen:openapi": "npm run generate-sdk",
"generate-sdk": "openapi-generator-cli generate -i ./src/main/json/openapi.json -g typescript-axios -o ./src/main/typescript/generated/openapi/typescript-axios/ --reserved-words-mappings protected=protected",
"build": "npm run build-ts && npm run build:dev:backend:postbuild",
"build-ts": "tsc",
"build:dev:backend:postbuild": "npm run copy-sql && npm run copy-yarn-lock",
"copy-sql": "cp -raf ./src/main/sql ./dist/lib/main/",
"copy-yarn-lock": "cp -af ../../yarn.lock ./dist/yarn.lock"
},
"dependencies": {
"@ethersproject/abi": "5.7.0",
"@hyperledger/cactus-core": "1.2.0",
"@hyperledger/cactus-common": "1.2.0",
"pg": "8.8.0",
"run-time-error": "1.4.0",
"fast-safe-stringify": "2.1.1",
"sanitize-html": "2.7.0",
"web3-utils": "1.5.2",
"async-mutex": "0.4.0",
"uuid": "8.3.2"
},
"devDependencies": {
"@hyperledger/cactus-core-api": "1.2.0",
"@hyperledger/cactus-api-client": "1.2.0",
"@hyperledger/cactus-test-tooling": "1.2.0",
"@types/express": "4.17.13",
"@types/pg": "8.6.5",
"rxjs": "7.3.0",
"web3": "1.5.2",
"web3-eth": "1.5.2",
"web3-core": "1.5.2",
"jest-extended": "2.0.0"
},
"engines": {
"node": ">=10",
"npm": ">=6"
},
"publishConfig": {
"access": "public"
},
"watch": {}
}
Loading