Skip to content

Commit

Permalink
feat(persistence-fabric): add sample setup scripts, improve documenta…
Browse files Browse the repository at this point in the history
…tion

- Fix error when empty `transaction.actions` was received from a block.
- Copy schema SQL during build.
- Remove unique constraint on fabric transactions hash column (there
    are transactions with empty hash that would break it).
- Improve parsing of certificate subject/issuer attributes to handle more
    delimiters.
- Use new `install-fabric.sh` script (that is recommended for Fabric 2.5)
    instead of old bootstrap to fix some runtime issues that I've encountered.
- Add sample setup scripts. Simple can be used to run persistence against
    already running fabric ledger, complete will setup entire environment
    and run some basic operations to generate sample data.
- Improve documentation to include these new scripts and how to use them, fix
    smaller issues.

Signed-off-by: Michal Bajer <michal.bajer@fujitsu.com>
  • Loading branch information
outSH authored and petermetz committed Aug 13, 2024
1 parent 9ce9057 commit 9fef336
Show file tree
Hide file tree
Showing 12 changed files with 552 additions and 47 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ export function formatCactiFullBlockResponse(
const transaction = payload.data;

const transactionActions: FullBlockTransactionActionV1[] = [];
for (const action of transaction.actions) {
for (const action of transaction.actions ?? []) {
const actionPayload = action.payload;
const proposalPayload = actionPayload.chaincode_proposal_payload;
const invocationSpec = proposalPayload.input;
Expand Down
101 changes: 71 additions & 30 deletions packages/cactus-plugin-persistence-fabric/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,53 +25,94 @@ Clone the git repository on your local machine. Follow these instructions that w

### Prerequisites

#### Build

In the root of the project, execute the command to install and build the dependencies. It will also build this persistence plugin:

```sh
yarn run configure
```

### Usage
#### Hyperledger Fabric Ledger and Connector

Instantiate a new `PluginPersistenceFabric` instance:
This plugin requires a running Hyperledger Fabric ledger that you want to persist to a database. For testing purposes, you can use our [test fabric-all-in-one Docker image](../../tools/docker/fabric-all-in-one/README.md). To access the ledger you'll need your organization connection profile JSON and a wallet containing registered identity. If you are using our `fabric-all-in-one` image, you can run our [asset-transfer-basic-utils scripts](../../tools/docker/fabric-all-in-one/asset-transfer-basic-utils/README.md) to fetch Org1 connection profile from a docker and register new user to a localhost wallet.

```typescript
import { PluginPersistenceFabric } from "@hyperledger/cactus-plugin-persistence-fabric";
import { v4 as uuidv4 } from "uuid";
```shell
# Start the test ledger
docker compose -f tools/docker/fabric-all-in-one/docker-compose-v2.x.yml up
# Wait for it to start (status should become `healthy`)

const persistencePlugin = new PluginPersistenceFabric({
apiClient,
logLevel: "info",
instanceId,
connectionString: "postgresql://postgres:your-super-secret-and-long-postgres-password@localhost:5432/postgres",,
channelName: "mychannel",
gatewayOptions: {
identity: signingCredential.keychainRef,
wallet: {
keychain: signingCredential,
},
},
});
# Run asset-transfer-basic-utils scripts
cd tools/docker/fabric-all-in-one/asset-transfer-basic-utils
# Cleanup artifacts from previous runs
rm -fr wallet/ connection.json
# Fetch connection profile to `tools/docker/fabric-all-in-one/asset-transfer-basic-utils/connection.json`
# Enroll user using wallet under `tools/docker/fabric-all-in-one/asset-transfer-basic-utils/wallet`
npm install
CACTUS_FABRIC_ALL_IN_ONE_CONTAINER_NAME=fabric_all_in_one_testnet_2x ./setup.sh
```

// Initialize the connection to the DB
await persistencePlugin.onPluginInit();
Once you have an Fabric ledger ready, you need to start the [Ethereum Cacti Connector](../cactus-plugin-ledger-connector-fabric/README.md). We recommend running the connector on the same ApiServer instance as the persistence plugin for better performance and reduced network overhead. See the connector package README for more instructions, or check out the [setup sample scripts](./src/test/typescript/manual).

#### Supabase Instance

You need a running Supabase instance to serve as a database backend for this plugin.

### Setup Tutorials

We've created some sample scripts to help you get started quickly. All the steps have detailed comments on it so you can quickly understand the code.

#### Sample Setup

Location: [./src/test/typescript/manual/sample-setup.ts](./src/test/typescript/manual/sample-setup.ts)

This sample script can be used to set up `ApiServer` with the Fabric connector and persistence plugins to monitor and store ledger data in a database. You need to have a ledger running before executing this script.

To run the script you need to set the following environment variables:

- `FABRIC_CONNECTION_PROFILE_PATH`: Full path to fabric ledger connection profile JSON file.
- `FABRIC_CHANNEL_NAME`: Name of the channel we want to connect to (to store it's data).
- `FABRIC_WALLET_PATH` : Full path to wallet containing our identity (that can connect and observe specified channel).
- `FABRIC_WALLET_LABEL`: Name (label) of our identity in a wallet provided in FABRIC_WALLET_PATH

By default, the script will try to use our `supabase-all-in-one` instance running on localhost. This can be adjusted by setting PostgreSQL connection string in `SUPABASE_CONNECTION_STRING` environment variable (optional).

```shell
# Example assumes fabric-all-in-one was used. Adjust the variables accordingly.
FABRIC_CONNECTION_PROFILE_PATH=/home/cactus/tools/docker/fabric-all-in-one/asset-transfer-basic-utils/connection.json FABRIC_CHANNEL_NAME=mychannel FABRIC_WALLET_PATH=/home/cactus/tools/docker/fabric-all-in-one/asset-transfer-basic-utils/wallet FABRIC_WALLET_LABEL=appUser
node ./dist/lib/test/typescript/manual/sample-setup.js
```

Alternatively, import `PluginFactoryLedgerPersistence` from the plugin package and use it to create a plugin.
#### Complete Sample Scenario

Location: [./src/test/typescript/manual/common-setup-methods](./src/test/typescript/manual/common-setup-methods)

This script starts the test Hyperledger Fabric ledger for you and executes few transactions on a `basic` chaincode. Then, it synchronizes everything to a database and monitors for all new blocks. This script can also be used for manual, end-to-end tests of a plugin.

By default, the script will try to use our `supabase-all-in-one` instance running on localhost.

```shell
npm run complete-sample-scenario
```

Custom supabase can be set with environment variable `SUPABASE_CONNECTION_STRING`:

```shell
SUPABASE_CONNECTION_STRING=postgresql://postgres:your-super-secret-and-long-postgres-password@127.0.0.1:5432/postgres npm run complete-sample-scenario
```

### Usage

Instantiate a new `PluginPersistenceFabric` instance:

```typescript
import { PluginFactoryLedgerPersistence } from "@hyperledger/cactus-plugin-persistence-fabric";
import { PluginImportType } from "@hyperledger/cactus-core-api";
import { PluginPersistenceFabric } from "@hyperledger/cactus-plugin-persistence-fabric";
import { v4 as uuidv4 } from "uuid";

const factory = new PluginFactoryLedgerPersistence({
pluginImportType: PluginImportType.Local,
});

const persistencePlugin = await factory.create({
apiClient,
const persistencePlugin = new PluginPersistenceFabric({
apiClient: new FabricApiClient(apiConfigOptions),
logLevel: "info",
instanceId,
instanceId: "my-instance",
connectionString: "postgresql://postgres:your-super-secret-and-long-postgres-password@localhost:5432/postgres",,
channelName: "mychannel",
gatewayOptions: {
Expand Down
10 changes: 8 additions & 2 deletions packages/cactus-plugin-persistence-fabric/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -47,14 +47,18 @@
"dist/*"
],
"scripts": {
"build": "npm run build-ts",
"build": "npm run build-ts && npm run build:dev:backend:postbuild",
"build-ts": "tsc",
"build:dev:backend:postbuild": "npm run copy-sql && npm run copy-yarn-lock",
"codegen": "yarn run --top-level run-s 'codegen:*'",
"codegen:openapi": "npm run generate-sdk",
"copy-yarn-lock": "cp -af ../../yarn.lock ./dist/yarn.lock",
"complete-sample-scenario": "npm run build && node ./dist/lib/test/typescript/manual/complete-sample-scenario.js",
"copy-sql": "mkdir -p ./dist/lib/main/ && cp -Rfp ./src/main/sql ./dist/lib/main/",
"copy-yarn-lock": "mkdir -p ./dist/lib/ && cp -rfp ../../yarn.lock ./dist/yarn.lock",
"generate-sdk": "run-p 'generate-sdk:*'",
"generate-sdk:go": "openapi-generator-cli generate -i ./src/main/json/openapi.json -g go -o ./src/main/go/generated/openapi/go-client/ --git-user-id hyperledger --git-repo-id $(echo $npm_package_name | replace @hyperledger/ \"\" -z)/src/main/go/generated/openapi/go-client --package-name $(echo $npm_package_name | replace @hyperledger/ \"\" -z) --reserved-words-mappings protected=protected --ignore-file-override ../../openapi-generator-ignore",
"generate-sdk:typecript-axios": "openapi-generator-cli generate -i ./src/main/json/openapi.json -g typescript-axios -o ./src/main/typescript/generated/openapi/typescript-axios/ --reserved-words-mappings protected=protected --ignore-file-override ../../openapi-generator-ignore",
"sample-setup": "npm run build && node ./dist/lib/test/typescript/manual/sample-setup.js",
"watch": "npm-watch"
},
"dependencies": {
Expand All @@ -69,13 +73,15 @@
"uuid": "10.0.0"
},
"devDependencies": {
"@hyperledger/cactus-cmd-api-server": "2.0.0-rc.3",
"@hyperledger/cactus-plugin-keychain-memory": "2.0.0-rc.3",
"@hyperledger/cactus-test-tooling": "2.0.0-rc.3",
"@openapitools/openapi-generator-cli": "2.7.0",
"@types/express": "4.17.21",
"@types/pg": "8.6.5",
"body-parser": "1.20.2",
"express": "4.19.2",
"fabric-network": "2.5.0-snapshot.23",
"jest-extended": "4.0.1",
"rxjs": "7.8.1",
"socket.io": "4.6.2"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -112,10 +112,8 @@ ALTER TABLE fabric.transaction OWNER TO postgres;

ALTER TABLE ONLY fabric.transaction
ADD CONSTRAINT transaction_pkey PRIMARY KEY (id);
ALTER TABLE ONLY fabric.transaction
ADD CONSTRAINT transaction_hash_key UNIQUE (hash);

CREATE UNIQUE INDEX transaction_hash_unique_idx ON fabric.transaction USING btree (hash);
CREATE INDEX transaction_hash_idx ON fabric.transaction (hash);
--
-- Name: transaction_action; Type: TABLE; Schema: fabric; Owner: postgres
--
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -200,15 +200,18 @@ export default class PostgresDatabaseClient {
* @returns Map of cert attributes
*/
private certificateAttrsStringToMap(attrString: string): Map<string, string> {
const separatorSplitRegex = new RegExp(`[/,+;\n]`);

return new Map(
attrString.split("\n").map((a) => {
attrString.split(separatorSplitRegex).map((a) => {
const splitAttrs = a.split("=");
if (splitAttrs.length !== 2) {
throw new Error(
`Invalid certificate attribute string: ${attrString}`,
);
}
return splitAttrs as [string, string];
const [key, value] = splitAttrs;
return [key.trim(), value.trim()];
}),
);
}
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,171 @@
/**
* Common setup code for the persistence plugin with detailed comments on each step.
* Requires environment variable `SUPABASE_CONNECTION_STRING` to be set before running the script that includes this!
* If not provided, a localhost instance of supabase will be assumed.
*/

import process from "process";
import { v4 as uuidV4 } from "uuid";
import { PluginRegistry } from "@hyperledger/cactus-core";
import { PluginKeychainMemory } from "@hyperledger/cactus-plugin-keychain-memory";
import {
LogLevelDesc,
LoggerProvider,
Logger,
} from "@hyperledger/cactus-common";
import { Configuration } from "@hyperledger/cactus-core-api";
import {
ApiServer,
AuthorizationProtocol,
ConfigService,
} from "@hyperledger/cactus-cmd-api-server";
import { DiscoveryOptions, X509Identity } from "fabric-network";
import {
DefaultEventHandlerStrategy,
FabricApiClient,
PluginLedgerConnectorFabric,
} from "@hyperledger/cactus-plugin-ledger-connector-fabric";

import { PluginPersistenceFabric } from "../../../main/typescript";

//////////////////////////////////
// Constants
//////////////////////////////////

const SUPABASE_CONNECTION_STRING =
process.env.SUPABASE_CONNECTION_STRING ??
"postgresql://postgres:your-super-secret-and-long-postgres-password@127.0.0.1:5432/postgres";

const testLogLevel: LogLevelDesc = "info";
const sutLogLevel: LogLevelDesc = "info";

// Logger setup
const log: Logger = LoggerProvider.getOrCreate({
label: "common-setup-methods",
level: testLogLevel,
});

/**
* Common ApiServer instance, can be empty if setup was not called yet!
*/
let apiServer: ApiServer;

//////////////////////////////////
// Methods
//////////////////////////////////

/**
* Setup Cacti ApiServer instance containing Fabric Connector plugin (for accessing the fabric ledger)
* and Fabric Persistence plugin (for storing data read from ledger to the database).
*
* @param port Port under which an ApiServer will be started. Can't be 0.
* @param channelName Channel that we want to connect to.
* @param connectionProfile Fabric connection profile (JSON object, not a string!)
* @param userIdentity Signing identity to use to connect to the channel (object, not a string!)
*
* @returns `{ persistence, apiClient, signingCredential }`
*/
export async function setupApiServer(
port: number,
channelName: string,
connectionProfile: any,
userIdentity: X509Identity,
) {
// PluginLedgerConnectorFabric requires a keychain plugin to operate correctly, ensuring secure data storage.
// We will store our userIdentity in it.
// For testing and debugging purposes, we use PluginKeychainMemory, which stores all secrets in memory (remember: this is not secure!).
const keychainId = uuidV4();
const keychainEntryKey = "monitorUser";
const keychainPlugin = new PluginKeychainMemory({
instanceId: uuidV4(),
keychainId,
backend: new Map([[keychainEntryKey, JSON.stringify(userIdentity)]]),
logLevel: testLogLevel,
});
const signingCredential = {
keychainId,
keychainRef: keychainEntryKey,
};

// We create fabric connector instance with some default settings assumed.
const discoveryOptions: DiscoveryOptions = {
enabled: true,
asLocalhost: true,
};
const connector = new PluginLedgerConnectorFabric({
instanceId: uuidV4(),
pluginRegistry: new PluginRegistry({ plugins: [keychainPlugin] }),
sshConfig: {},
cliContainerEnv: {},
peerBinary: "/fabric-samples/bin/peer",
logLevel: sutLogLevel,
connectionProfile,
discoveryOptions,
eventHandlerOptions: {
strategy: DefaultEventHandlerStrategy.NetworkScopeAnyfortx,
commitTimeout: 300,
},
});

// Remember to initialize a plugin
await connector.onPluginInit();

// We need an `FabricApiClient` to access `PluginLedgerConnectorFabric` methods from our `PluginPersistenceFabric`.
const apiConfig = new Configuration({ basePath: `http://127.0.0.1:${port}` });
const apiClient = new FabricApiClient(apiConfig);

// We create persistence plugin, it will read data from fabric ledger through `apiClient` we've just created,
// and push it to PostgreSQL database accessed by it's SUPABASE_CONNECTION_STRING (read from the environment variable).
const persistence = new PluginPersistenceFabric({
channelName,
gatewayOptions: {
identity: signingCredential.keychainRef,
wallet: {
keychain: signingCredential,
},
},
apiClient,
logLevel: sutLogLevel,
instanceId: uuidV4(),
connectionString: SUPABASE_CONNECTION_STRING,
});
// Plugin initialization will check connection to the database and setup schema if needed.
await persistence.onPluginInit();

// The API Server is a common "container" service that manages our plugins (connector and persistence).
// We use a sample configuration with most security measures disabled for simplicity.
log.info("Create ApiServer...");
const configService = new ConfigService();
const cactusApiServerOptions = await configService.newExampleConfig();
cactusApiServerOptions.authorizationProtocol = AuthorizationProtocol.NONE;
cactusApiServerOptions.configFile = "";
cactusApiServerOptions.apiCorsDomainCsv = "*";
cactusApiServerOptions.apiTlsEnabled = false;
cactusApiServerOptions.apiPort = port;
const config = await configService.newExampleConfigConvict(
cactusApiServerOptions,
);

apiServer = new ApiServer({
config: config.getProperties(),
pluginRegistry: new PluginRegistry({ plugins: [connector, persistence] }),
});

const apiServerStartOut = await apiServer.start();
log.debug(`apiServerStartOut:`, apiServerStartOut);
// Our setup is operational now!

return { persistence, apiClient, signingCredential };
}

/**
* Cleanup all the resources allocated by our Api Server.
* Remember to call it before exiting!
*/
export async function cleanupApiServer() {
log.info("cleanupApiServer called.");

if (apiServer) {
await apiServer.shutdown();
}
}
Loading

0 comments on commit 9fef336

Please sign in to comment.