fix: import BNS v1 data during event replay (#1301)

* fix: import v1 data during replay

* fix: import names first, subdomains last

* feat: obtain genesis block data from tsv

* fix: v1 import tests

* fix: import route

* fix: api test

* fix: move to for await of

* docs: update README to reflect new replay

Author: rafaelcr

readme.md

@@ -98,19 +98,51 @@ For running offline mode set an environment variable `STACKS_API_MODE=offline`
 
 ## Event Replay
 
-The stacks-node is only able to emit events live as they happen. This poses a problem in the scenario where the stacks-blockchain-api needs to
-be upgraded and its database cannot be migrated to a new schema. One way to handle this upgrade is to wipe the stacks-blockchain-api's database
-and stacks-node working directory, and re-sync from scratch.
+The stacks-node is only able to emit events live as they happen. This poses a problem in the
+scenario where the stacks-blockchain-api needs to be upgraded and its database cannot be migrated to
+a new schema. One way to handle this upgrade is to wipe the stacks-blockchain-api's database and
+stacks-node working directory, and re-sync from scratch.
 
-Alternatively, an event-replay feature is available where the API records the HTTP POST requests from the stacks-node event emitter, then streams
-these events back to itself. Essentially simulating a wipe & full re-sync, but much quicker.
+Alternatively, an event-replay feature is available where the API records the HTTP POST requests
+from the stacks-node event emitter, then streams these events back to itself. Essentially simulating
+a wipe & full re-sync, but much quicker.
 
-The feature can be used via program args. For example, if there are breaking changes in the API's sql schema, like adding a new column which requires
-event's to be re-played, the following steps could be ran:
+The feature can be used via program args. For example, if there are breaking changes in the API's
+sql schema, like adding a new column which requires event's to be re-played, the following steps
+could be ran:
 
 ### Event Replay Instructions
 
-1. Ensure the API process is not running. When stopping the API, let the process exit gracefully so that any in-progress SQL writes can finish.
+#### V1 BNS Data
+
+**Optional but recommended** - If you want the V1 BNS data, there are going to be a few extra steps:
+
+1. Download BNS data:
+   ```shell
+   curl -L https://storage.googleapis.com/blockstack-v1-migration-data/export-data.tar.gz -o /stacks-node/bns/export-data.tar.gz
+   ```
+1. Extract it:
+   ```shell
+   tar -xzvf ./bns/export-data.tar.gz -C /stacks-node/bns/
+   ```
+1. Each file in `./bns` will have a corresponding `sha256` value. To Verify, run a script like the
+   following to check the sha256sum:
+
+    ```bash
+    for file in `ls /stacks-node/bns/* | grep -v sha256 | grep -v .tar.gz`; do
+        if [ $(sha256sum $file | awk {'print $1'}) == $(cat ${file}.sha256 ) ]; then
+            echo "sha256 Matched $file"
+        else
+            echo "sha256 Mismatch $file"
+        fi
+    done
+    ```
+1. Set the data's location as the value of `BNS_IMPORT_DIR` in your `.env` file.
+
+#### Export and Import
+
+1. Ensure the API process is not running. When stopping the API, let the process exit gracefully so
+   that any in-progress SQL writes can finish.
 1. Export event data to disk with the `export-events` command:
 
    ```shell
@@ -119,19 +151,25 @@ event's to be re-played, the following steps could be ran:
 1. Update to the new stacks-blockchain-api version.
 1. Perform the event playback using the `import-events` command:
 
-   **WARNING**: This will **drop _all_ tables** from the configured Postgres database, including any tables not automatically added by the API.
+   **WARNING**: This will **drop _all_ tables** from the configured Postgres database, including any
+   tables not automatically added by the API.
 
    ```shell
    node ./lib/index.js import-events --file /tmp/stacks-node-events.tsv --wipe-db --force
    ```
 
    This command has two modes of operation, specified by the `--mode` option:
-   * `archival` (default): The process will import and ingest *all* blockchain events that have happened since the first block.
-   * `pruned`: The import process will ignore some prunable events (mempool, microblocks) until the import block height has reached `chain tip - 256` blocks. This saves a considerable amount of time during import, but sacrifices some historical data. You can use this mode if you're mostly interested in running an API that prioritizes real time information.
-
-Alternatively, instead of performing the `export-events` command in step 1, an environmental variable can be set which enables events to be streamed to a file
-as they are received, while the application is running normally. To enable this feature, set the `STACKS_EXPORT_EVENTS_FILE` env var to the file path where
-events should be appended. Example:
+   * `archival` (default): The process will import and ingest *all* blockchain events that have
+     happened since the first block.
+   * `pruned`: The import process will ignore some prunable events (mempool, microblocks) until the
+     import block height has reached `chain tip - 256` blocks. This saves a considerable amount of
+     time during import, but sacrifices some historical data. You can use this mode if you're mostly
+     interested in running an API that prioritizes real time information.
+
+Alternatively, instead of performing the `export-events` command in step 1, an environmental
+variable can be set which enables events to be streamed to a file as they are received, while the
+application is running normally. To enable this feature, set the `STACKS_EXPORT_EVENTS_FILE` env var
+to the file path where events should be appended. Example:
 ```
 STACKS_EXPORT_EVENTS_FILE=/tmp/stacks-node-events.tsv
 ```

running_an_api.md

@@ -78,34 +78,14 @@ Since we'll need to create some files/dirs for persistent data we'll first creat
 We'll be using:
 
 ```bash
-$ mkdir -p ./stacks-node/{persistent-data/postgres,persistent-data/stacks-blockchain,bns,config}
+$ mkdir -p ./stacks-node/{persistent-data/postgres,persistent-data/stacks-blockchain,config}
 $ docker pull blockstack/stacks-blockchain-api \
     && docker pull blockstack/stacks-blockchain \
     && docker pull postgres:alpine
 $ docker network create stacks-blockchain > /dev/null 2>&1
 $ cd ./stacks-node
 ```
 
-**Optional but recommended**: If you need the v1 BNS data, there are going to be a few extra steps.
-
-1. Download the BNS data:  
-`curl -L https://storage.googleapis.com/blockstack-v1-migration-data/export-data.tar.gz -o ./bns/export-data.tar.gz`
-2. Extract the data:  
-`tar -xzvf ./bns/export-data.tar.gz -C ./bns/`
-3. Each file in `./bns` will have a corresponding `sha256` value.  
-
-To Verify, run a script like the following to check the sha256sum:
-
-```bash
-for file in `ls ./bns/* | grep -v sha256 | grep -v .tar.gz`; do
-    if [ $(sha256sum $file | awk {'print $1'}) == $(cat ${file}.sha256 ) ]; then
-        echo "sha256 Matched $file"
-    else
-        echo "sha256 Mismatch $file"
-    fi
-done
-```
-
 ## Postgres
 
 The `postgres:alpine` image can be run with default settings, the only requirement is that a password Environment Variable is set for the `postgres` user: `POSTGRES_PASSWORD=postgres`
@@ -161,16 +141,9 @@ STACKS_BLOCKCHAIN_API_PORT=3999
 STACKS_BLOCKCHAIN_API_HOST=0.0.0.0
 STACKS_CORE_RPC_HOST=stacks-blockchain
 STACKS_CORE_RPC_PORT=20443
-BNS_IMPORT_DIR=/bns-data
 API_DOCS_URL=https://docs.hiro.so/api
 ```
 
-**Note** that here we are importing the bns data with the env var `BNS_IMPORT`.  
-
-To Disable this import, simply comment the line: `#BNS_IMPORT_DIR=/bns-data`  
-
-***If you leave this enabled***: please allow several minutes for the one-time import to complete before continuing.
-
 The other Environment Variables to pay attention to:
 
 - `PG_HOST`: Set this to your **postgres** instance. In this guide, we'll be using a container named `postgres`.
@@ -184,7 +157,6 @@ docker run -d --rm \
     --name stacks-blockchain-api \
     --net=stacks-blockchain \
     --env-file $(pwd)/.env \
-    -v $(pwd)/bns:/bns-data \
     -p 3700:3700 \
     -p 3999:3999 \
     blockstack/stacks-blockchain-api

running_api_from_source.md

@@ -35,15 +35,15 @@ Since we'll need to create some files/dirs for persistent data,
 we'll first create a base directory structure and set some permissions:
 
 ```bash
-$ sudo mkdir -p /stacks-node/{persistent-data/stacks-blockchain,bns,config,binaries}
+$ sudo mkdir -p /stacks-node/{persistent-data/stacks-blockchain,config,binaries}
 $ sudo chown -R $(whoami) /stacks-node 
 $ cd /stacks-node
 ```
 
 ## Install Requirements
 
 ```bash
-$ PG_VERSION=12 \
+$ PG_VERSION=14 \
   && NODE_VERSION=16 \
   && sudo apt-get update \
   && sudo apt-get install -y \
@@ -65,26 +65,6 @@ $ PG_VERSION=12 \
     nodejs
 ```
 
-**Optional but recommended** - If you want the V1 BNS data, there are going to be a few extra steps:
-
-1. Download the BNS data:  
-`curl -L https://storage.googleapis.com/blockstack-v1-migration-data/export-data.tar.gz -o /stacks-node/bns/export-data.tar.gz`
-2. Extract the data:  
-`tar -xzvf ./bns/export-data.tar.gz -C /stacks-node/bns/`
-3. Each file in `./bns` will have a corresponding `sha256` value.
-
-To Verify, run a script like the following to check the sha256sum:
-
-```bash
-for file in `ls /stacks-node/bns/* | grep -v sha256 | grep -v .tar.gz`; do
-    if [ $(sha256sum $file | awk {'print $1'}) == $(cat ${file}.sha256 ) ]; then
-        echo "sha256 Matched $file"
-    else
-        echo "sha256 Mismatch $file"
-    fi
-done
-```
-
 ## postgres
 
 ### postgres permissions
@@ -127,8 +107,6 @@ $ git clone https://github.com/hirosystems/stacks-blockchain-api /stacks-node/st
 The stacks blockchain api requires several Environment Variables to be set in order to run properly.  
 To reduce complexity, we're going to create a `.env` file that we'll use for these env vars.  
 
-** Note: ** to enable BNS names, uncomment `BNS_IMPORT_DIR` in the below `.env` file. 
-
 Create a new file: `/stacks-node/stacks-blockchain-api/.env` with the following content:
 
 ```bash
@@ -148,7 +126,6 @@ STACKS_BLOCKCHAIN_API_PORT=3999
 STACKS_BLOCKCHAIN_API_HOST=0.0.0.0
 STACKS_CORE_RPC_HOST=localhost
 STACKS_CORE_RPC_PORT=20443
-#BNS_IMPORT_DIR=/stacks-node/bns
 EOF
 $ cd /stacks-node/stacks-blockchain-api && nohup node ./lib/index.js &
 ```

src/datastore/common.ts

@@ -439,7 +439,7 @@ export interface DataStoreAttachmentData {
   blockHeight: number;
 }
 
-export interface DataStoreSubdomainBlockData {
+export interface DataStoreBnsBlockData {
   index_block_hash: string;
   parent_index_block_hash: string;
   microblock_hash: string;
@@ -449,7 +449,7 @@ export interface DataStoreSubdomainBlockData {
 
 export interface DataStoreAttachmentSubdomainData {
   attachment?: DataStoreAttachmentData;
-  blockData?: DataStoreSubdomainBlockData;
+  blockData?: DataStoreBnsBlockData;
   subdomains?: DbBnsSubdomain[];
 }
 

src/datastore/postgres-store.ts

@@ -101,7 +101,7 @@ import {
   DbAssetEventTypeId,
   DbTxGlobalStatus,
   DataStoreAttachmentData,
-  DataStoreSubdomainBlockData,
+  DataStoreBnsBlockData,
   DataStoreAttachmentSubdomainData,
 } from './common';
 import {
@@ -2175,7 +2175,7 @@ export class PgDataStore
             );
             let isCanonical = true;
             let txIndex = -1;
-            const blockData: DataStoreSubdomainBlockData = {
+            const blockData: DataStoreBnsBlockData = {
               index_block_hash: '',
               parent_index_block_hash: '',
               microblock_hash: '',
@@ -7222,7 +7222,7 @@ export class PgDataStore
       // The `names` and `zonefiles` tables only track latest zonefile changes. We need to check
       // `nft_custody` for the latest name owner, but only for names that were NOT imported from v1
       // since they did not generate an NFT event for us to track.
-      if (nameZonefile.rows[0].registered_at !== 0) {
+      if (nameZonefile.rows[0].registered_at !== 1) {
         let value: Buffer;
         try {
           value = bnsNameCV(name);
@@ -7427,7 +7427,7 @@ export class PgDataStore
           names
         WHERE
           address = $1
-          AND registered_at = 0
+          AND registered_at = 1
           AND canonical = TRUE
           AND microblock_canonical = TRUE
         `,

src/event-replay/event-replay.ts

@@ -3,7 +3,8 @@ import * as fs from 'fs';
 import { cycleMigrations, dangerousDropAllTables, PgDataStore } from '../datastore/postgres-store';
 import { startEventServer } from '../event-stream/event-server';
 import { getApiConfiguredChainID, httpPostRequest, logger } from '../helpers';
-import { findTsvBlockHeight, getDbBlockHeight } from './helpers';
+import { findBnsGenesisBlockData, findTsvBlockHeight, getDbBlockHeight } from './helpers';
+import { importV1BnsNames, importV1BnsSubdomains, importV1TokenOfferingData } from '../import-v1';
 
 enum EventImportMode {
   /**
@@ -107,6 +108,8 @@ export async function importEventsFromTsv(
   if (eventImportMode === EventImportMode.pruned) {
     console.log(`Ignoring all prunable events before block height: ${prunedBlockHeight}`);
   }
+  // Look for the TSV's genesis block information for BNS import.
+  const tsvGenesisBlockData = await findBnsGenesisBlockData(resolvedFilePath);
 
   const db = await PgDataStore.connect({
     usageName: 'import-events',
@@ -122,6 +125,18 @@ export async function importEventsFromTsv(
     httpLogLevel: 'debug',
   });
 
+  await importV1TokenOfferingData(db);
+
+  // Import V1 BNS names first. Subdomains will be imported after TSV replay is finished in order to
+  // keep the size of the `subdomains` table small.
+  if (process.env.BNS_IMPORT_DIR) {
+    logger.info(`Using BNS export data from: ${process.env.BNS_IMPORT_DIR}`);
+    await importV1BnsNames(db, process.env.BNS_IMPORT_DIR, tsvGenesisBlockData);
+  } else {
+    logger.warn(`Notice: full BNS functionality requires 'BNS_IMPORT_DIR' to be set.`);
+  }
+
+  // Import TSV chain data
   const readStream = fs.createReadStream(resolvedFilePath);
   const rawEventsIterator = PgDataStore.getRawEventRequests(readStream, status => {
     console.log(status);
@@ -163,6 +178,9 @@ export async function importEventsFromTsv(
     }
   }
   await db.finishEventReplay();
+  if (process.env.BNS_IMPORT_DIR) {
+    await importV1BnsSubdomains(db, process.env.BNS_IMPORT_DIR, tsvGenesisBlockData);
+  }
   console.log(`Event import and playback successful.`);
   await eventServer.closeAsync();
   await db.close();

src/event-replay/helpers.ts

@@ -1,6 +1,15 @@
+import * as fs from 'fs';
+import * as readline from 'readline';
+import { decodeTransaction, TxPayloadTypeID } from 'stacks-encoding-native-js';
+import { DataStoreBnsBlockData } from '../datastore/common';
 import { PgDataStore } from '../datastore/postgres-store';
 import { ReverseFileStream } from './reverse-file-stream';
 
+export type BnsGenesisBlock = DataStoreBnsBlockData & {
+  tx_id: string;
+  tx_index: number;
+};
+
 /**
  * Traverse a TSV file in reverse to find the last received `/new_block` node message and return
  * the `block_height` reported by that event. Even though the block produced by that event might
@@ -26,6 +35,47 @@ export async function findTsvBlockHeight(filePath: string): Promise<number> {
   return blockHeight;
 }
 
+/**
+ * Traverse a TSV file to find the genesis block and extract its data so we can use it during V1 BNS
+ * import.
+ * @param filePath - TSV path
+ * @returns Genesis block data
+ */
+export async function findBnsGenesisBlockData(filePath: string): Promise<BnsGenesisBlock> {
+  const rl = readline.createInterface({
+    input: fs.createReadStream(filePath),
+    crlfDelay: Infinity,
+  });
+  for await (const line of rl) {
+    const columns = line.split('\t');
+    const eventName = columns[2];
+    if (eventName === '/new_block') {
+      const payload = JSON.parse(columns[3]);
+      // Look for block 1
+      if (payload.block_height === 1) {
+        for (const tx of payload.transactions) {
+          const decodedTx = decodeTransaction(tx.raw_tx);
+          // Look for the only token transfer transaction in the genesis block. This is the one
+          // that contains all the events, including all BNS name registrations.
+          if (decodedTx.payload.type_id === TxPayloadTypeID.TokenTransfer) {
+            rl.close();
+            return {
+              index_block_hash: payload.index_block_hash,
+              parent_index_block_hash: payload.parent_index_block_hash,
+              microblock_hash: payload.parent_microblock,
+              microblock_sequence: payload.parent_microblock_sequence,
+              microblock_canonical: true,
+              tx_id: decodedTx.tx_id,
+              tx_index: tx.tx_index,
+            };
+          }
+        }
+      }
+    }
+  }
+  throw new Error('BNS genesis block data not found');
+}
+
 /**
  * Get the current block height from the DB. We won't use the `getChainTip` method since that
  * adds some conversions from block hashes into strings that we're not interested in. We also can't