1.0.0 release

.vscode/launch.json

@@ -13,7 +13,7 @@
             "sourceMaps": true,
             "cwd": "${workspaceRoot}",
             "protocol": "inspector",
-            "console": "externalTerminal"
+            "console": "integratedTerminal"
         },
         {
             "type": "node",

README.md

@@ -1,9 +1,9 @@
 ![SLPDB](assets/slpdb_logo.png)
 
 # SLPDB Readme
-**Last Updated:** 2020-07-29
+**Last Updated:** 2021-01-03
 
-**Current SLPDB Version:** 1.0.0-beta-rc12
+**Current SLPDB Version:** 1.0.0
 
 * 1. [What is SLPDB?](#WhatisSLPDB)
 * 2. [Do you need to <u>install</u> SLPDB?](#DoyouneedtouinstalluSLPDB)
@@ -28,7 +28,6 @@
     * 7.1 [Parser Tests](#ParserTests)
     * 7.2 [Input Tests](#InputTests)
     * 7.3 [Regtest Network Tests](#E2ETests)
-* 8. [Change Log](#ChangeLog)
 
 
 
@@ -86,7 +85,7 @@ The services `SlpServe` and `SlpSockServer` return query results as a JSON objec
 
 ###  4.1. <a name='Prerequisites'></a>Prerequisites
 * Node.js 12
-* MongoDB 4.0+
+* MongoDB 4.4+
 * BitcoinABC, Bitcoin Cash Node, BitcoinUnlimited, BCHD, or other Bitcoin Cash full node with:
   * RPC-JSON (or gRPC) and 
   * ZeroMQ event notifications
@@ -348,75 +347,3 @@ The SLPJS npm library also passes all unit tests which test for the specified in
 ###  7.3. <a name='E2ETests'></a>End-to-End Tests
 
 A set of end-to-end tests have been created in order to ensure the expected behavior of SLPDB utilizing the bitcoin regtest network.  These tests simulate actual transaction activity using the bitcoin regtest test network and check for proper state in mongoDB and also check that zmq notifications are emitted.  The `tests` directory contains the end-to-end tests which can be run by following the instructions provided in the `regtest` directory.
-
-
-
-## 8. <a name='ChangeLog'></a>Change Log
-
-* 1.0.0
-    * Removed `utxos` and `addresses` collections, as this information can be queried from the `graphs` collection.  Updated README examples for queries which relied on now defunct `utxos` and `addresses` collections
-    * Added Topological sorting in block crawl to allow for token validation during block crawl
-    * Fixed issue where unconfirmed/confirmed transactions where initially writen to db without SLP property and then updated later after a subsequent validation step.
-    * Fixed issue where MINT baton address is wrong if not the same as token receiver address.
-    * Added graph pruning to actively reduce memory footprint
-    * Move mint baton status (mintBatonStatus) and txo out of statistics (mintBatonUtxo), these are now main property of the token document
-    * Update notification format for consistent value type, always returning string based numbers, no more Decimal128 in block notification
-    * Removed token stats properties: `block_last_active_send`, `block_last_active_mint`, `qty_valid_token_utxos`, `qty_valid_token_addresses`
-    * Removed token stats properties: `qty_token_minted`, `qty_token_burned`, `qty_token_circulating_supply`
-    * Removed UTXO status enums: `SPENT_INVALID_SLP`, `BATON_SPENT_INVALID_SLP`
-    * Renamed `qty_valid_txns_since_genesis` to `approx_txns_since_genesis`, since this can be corrupted if the block checkpoint is reset.
-    * Removed network dependency to rest.bitcoin.com
-    * Add env var to skip the initial full node sync check that happens only on startup
-
-* 0.15.6
-    * Bug fixes and improvements in slpjs
-
-* 0.15.5
-    * Various bug fixes and improvements
-    * NOTE: this version was not published
-
-* 0.15.4
-    * Added statuses collection
-    * Added env variables for disabling graph search and zmq publishing
-    * Caching improvements to speed up token processing
-    * Various fixes and improvements
-
-* 0.15.3
-    * Additional db cleanup needed in graphs collection from the issue fixed in 0.15.2
-    * Removed unneeded object initializers in FromDbObjects, moved to TokenGraph constructor
-
-* 0.15.2
-	* Fixed issue with address format occurring when restarting SLPDB
-	* Cleaned up readme file formatting (Jt)
-
-* 0.15.1
-	* Fix issue with minting baton status update on initiating token from scratch
-
-* 0.15.0
-	* Add Graph Search metadata (see example queries above for search examples)
-	* Start listening to blockchain updates before the startup process for improved data integrity during long startup periods
-	* Fixed bug where not all inputs were showing up in graph transactions
-	* Replaced getinfo RPC in favor of getBlockchainInfo
-	* Fixed issue with tokens collection items missing "block_created" data
-
-* 0.14.1
-	* Improved token burn detection after a new blocks is broadcast
-	* Updated to slpjs version 0.21.1 with caching bug fix.
-	* Updated updateTxnCollections() so that any valid transactions erroneously marked as "invalid" get updated.
-	* Refactored several methods to use destructured method parameters
-	* Other minor refactors and improvements
-
-* 0.14.0
-	* Added NFT1 support
-	* Improved token burn detection on new block event
-	* Prevent duplicate zmq transaction publish notifications
-	* Various bug fixes and improvements
-
-* 0.13.0
-	* Breaking Change: This change may impact any application using `TokenUtxoStatus` or the Graph collection's `graphTxn.outputs.status` property. A new enum type was added to `TokenUtxoStatus` called "EXCESS_INPUT_BURNED".  This status is applied to any transaction that has an input SLP quantity that is greater than output SLP quantity.  This enum label is used within the Graphs collection's `graphTxn.outputs.status` property.
-	* A new startup script called "reprocess" was added for the purpose of debugging.  Usage is `node index.js reprocess <tokenId>`.
-	* Fixed issue with block and mempool item cache used to ignore duplicate zmq notifications
-	* Fixed testnet config starting block (off by one)
-	* P2MS and P2PK output addresses are stored as `scriptPubKey:<hex>`
-	* Reduced number of RPC calls by subscribing to zmq 'rawtx' instead of 'hashtx'
-	* Other minor improvements

cache.ts

@@ -111,8 +111,9 @@ export class CacheMap<T, M> {
     }
 
     delete(key: T) {
-        if(this.map.delete(key))
+        if (this.map.delete(key)) {
             this.list = this.list.filter(k => k !== key);
+        }
     }
 
     toMap() {
@@ -121,8 +122,9 @@ export class CacheMap<T, M> {
 
     private shift(): T | undefined {
         let key = this.list.shift();
-        if(key)
+        if(key) {
             this.map.delete(key);
+        }
         return key;
     }
 

filters.ts

@@ -94,3 +94,5 @@ class _TokenFilter {
 
 // accessor to a singleton stack for filters
 export const TokenFilters = _TokenFilter.Instance;
+
+TokenFilters();

index.ts

@@ -30,7 +30,6 @@ let db = new Db({
     dbUrl: Config.db.url, 
     config: Config.db 
 });
-let filter = TokenFilters();
 let bit = new Bit(db);
 new SlpdbStatus(db, process.argv);
 
@@ -82,7 +81,7 @@ const daemon = {
         let currentHeight = await RpcClient.getBlockCount();
         tokenManager = new SlpGraphManager(db, currentHeight, network, bit);
         bit._slpGraphManager = tokenManager;
-        let pruningStack = PruneStack(tokenManager._tokens);
+        PruneStack(tokenManager._tokens);  // call instantiates singleton
 
         console.log('[INFO] Synchronizing SLPDB with BCH blockchain data...', new Date());
         console.time('[PERF] Initial Block Sync');

interfaces.ts

@@ -17,7 +17,6 @@ export interface GraphTxnOutput {
     vout: number;
     bchSatoshis: number|null;
     slpAmount: BigNumber;
-    isBurnCounted?: boolean;
     spendTxid: string | null;
     status: TokenUtxoStatus|BatonUtxoStatus;
     invalidReason: string | null;
@@ -26,7 +25,7 @@ export interface GraphTxnOutput {
 export interface GraphTxnInput {
     txid: string;
     vout: number;
-    slpAmount: BigNumber; 
+    slpAmount: BigNumber;
     address: string;
     bchSatoshis: number;
 }

package.json

@@ -1,6 +1,6 @@
 {
   "name": "slpdb",
-  "version": "1.0.0-beta-rc13",
+  "version": "1.0.0",
   "description": "Indexer for the Simple Ledger Protocol with real-time validation notifications.",
   "main": "index.js",
   "scripts": {
@@ -16,7 +16,6 @@
   "keywords": [
     "SLP"
   ],
-  "author": "James Cramer",
   "license": "MIT",
   "bugs": {
     "url": "https://github.com/simpleledger/SLPDB/issues"
@@ -47,7 +46,7 @@
     "level": "^5.0.1",
     "migrate-mongo": "^5.0.1",
     "mingo": "^2.5.3",
-    "mongodb": "^3.5.11",
+    "mongodb": "^3.5.6",
     "node-jq": "1.6.0",
     "os-utils": "0.0.14",
     "p-limit": "2.0.0",

rpc.ts

@@ -14,12 +14,11 @@ export class RpcClient {
     static transactionCache = new CacheMap<string, Buffer>(500000);
     static useGrpc: boolean | undefined;
     constructor({ useGrpc }: { useGrpc?: boolean }) {
-        if(useGrpc) {
+        if (useGrpc) {
             RpcClient.useGrpc = useGrpc;
-            if(Boolean(Config.grpc.url) && Config.grpc.certPath) {
+            if (Boolean(Config.grpc.url) && Config.grpc.certPath) {
                 grpc = new GrpcClient({ url: Config.grpc.url, rootCertPath: Config.grpc.certPath });
-            }
-            else {
+            } else {
                 grpc = new GrpcClient({ url: Config.grpc.url });
             }
         } else {
@@ -33,7 +32,7 @@ export class RpcClient {
     }
 
     static async getBlockCount(): Promise<number> {
-        if(this.useGrpc) {
+        if (this.useGrpc) {
             console.log("[INFO] gRPC: getBlockchainInfo");
             return (await grpc.getBlockchainInfo()).getBestHeight();
         }
@@ -42,7 +41,7 @@ export class RpcClient {
     }
 
     static async getBlockchainInfo(): Promise<BlockchainInfoResult> {
-        if(RpcClient.useGrpc) {
+        if (RpcClient.useGrpc) {
             console.log("[INFO] gRPC: getBlockchainInfo");
             let info = await grpc.getBlockchainInfo();
             return {
@@ -64,7 +63,7 @@ export class RpcClient {
     }
 
     static async getBlockHash(block_index: number, asBuffer=false): Promise<string|Buffer> {
-        if(RpcClient.useGrpc) {
+        if (RpcClient.useGrpc) {
             console.log("[INFO] gRPC: getBlockInfo (for getBlockHash)");
             let hash = Buffer.from((await grpc.getBlockInfo({ index: block_index })).getInfo()!.getHash_asU8().reverse());
             if(asBuffer) {
@@ -81,21 +80,23 @@ export class RpcClient {
     }
 
     static async getRawBlock(hash: string): Promise<string> {
-        if(RpcClient.useGrpc) {
+        if (RpcClient.useGrpc) {
             console.log("[INFO] gRPC: getRawBlock");
             return Buffer.from((await grpc.getRawBlock({ hash: hash, reversedHashOrder: true })).getBlock_asU8()).toString('hex')
         }
         return await rpc_retry.getBlock(hash, 0);
     }
 
     static async getBlockInfo({ hash, index }: { hash?: string, index?: number}): Promise<BlockHeaderResult> {
-        if(RpcClient.useGrpc) {
+        if (RpcClient.useGrpc) {
             console.log("[INFO] gRPC: getBlockInfo");
             let blockinfo: BlockInfo;
-            if(index)
+            if (index) {
                 blockinfo = (await grpc.getBlockInfo({ index: index })).getInfo()!;
-            else
+            } else {
                 blockinfo = (await grpc.getBlockInfo({ hash: hash, reversedHashOrder: true })).getInfo()!;
+            }
+
             return {
                 hash: Buffer.from(blockinfo.getHash_asU8().reverse()).toString('hex'),
                 confirmations: blockinfo.getConfirmations(),
@@ -117,8 +118,7 @@ export class RpcClient {
         if (index) {
             console.log("[INFO] JSON RPC: getBlockInfo/getBlockHash", index);
             hash = await rpc.getBlockHash(index);
-        }
-        else if (!hash) {
+        } else if (!hash) {
             throw Error("No index or hash provided for block")
         }
 
@@ -127,7 +127,7 @@ export class RpcClient {
     }
 
     static async getRawMemPool(): Promise<string[]> {
-        if(RpcClient.useGrpc) {
+        if (RpcClient.useGrpc) {
             console.log("[INFO] gRPC: getRawMemPool");
             return (await grpc.getRawMempool({ fullTransactions: false })).getTransactionDataList().map(t => Buffer.from(t.getTransactionHash_asU8().reverse()).toString('hex'))
         }
@@ -136,24 +136,26 @@ export class RpcClient {
     }
 
     static async getRawTransaction(hash: string, retryRpc=true): Promise<string> { 
-        if(RpcClient.transactionCache.has(hash)) {
+        if (RpcClient.transactionCache.has(hash)) {
             console.log("[INFO] cache: getRawTransaction");
             return RpcClient.transactionCache.get(hash)!.toString('hex');
         }
-        if(RpcClient.useGrpc) {
+
+        if (RpcClient.useGrpc) {
             console.log("[INFO] gRPC: getRawTransaction", hash);
             return Buffer.from((await grpc.getRawTransaction({ hash: hash, reversedHashOrder: true })).getTransaction_asU8()).toString('hex');
-        } 
+        }
+
         console.log("[INFO] JSON RPC: getRawTransaction", hash);
-        if(retryRpc) {
+        if (retryRpc) {
             return await rpc_retry.getRawTransaction(hash);
         } else {
             return await rpc.getRawTransaction(hash);
         }
     }
 
     static async getTransactionBlockHash(hash: string): Promise<string> {
-        if(RpcClient.useGrpc) {
+        if (RpcClient.useGrpc) {
             console.log("[INFO] gRPC: getTransaction", hash);
             let txn = await grpc.getTransaction({ hash: hash, reversedHashOrder: true });
             return Buffer.from(txn.getTransaction()!.getBlockHash_asU8().reverse()).toString('hex');
@@ -163,7 +165,7 @@ export class RpcClient {
     }
 
     static async getTxOut(hash: string, vout: number): Promise<TxOutResult|GetUnspentOutputResponse|null> {
-        if(RpcClient.useGrpc){
+        if (RpcClient.useGrpc){
             console.log("[INFO] gRPC: getTxOut", hash, vout);
             try {
                 let utxo = (await grpc.getUnspentOutput({ hash: hash, vout: vout, reversedHashOrder: true, includeMempool: true }));
@@ -177,7 +179,7 @@ export class RpcClient {
     }
 
     static async getMempoolInfo(): Promise<MempoolInfoResult|{}> {
-        if(RpcClient.useGrpc) {
+        if (RpcClient.useGrpc) {
             return {};
         }
         console.log("[INFO] JSON RPC: getMempoolInfo");

run-service.sh

@@ -0,0 +1,24 @@
+#!/bin/bash
+
+cd /root/SLPDB/
+
+echo "Compiling..."
+./node_modules/typescript/bin/tsc
+echo "Compiling done."
+
+echo "Checking for DB migrations..."
+export db_url=mongodb://127.0.0.1:27017
+./node_modules/migrate-mongo/bin/migrate-mongo.js up
+echo "Finished DB migrations."
+
+FLAG=./ctl/REPROCESS
+if [ -f "$FLAG" ]; then
+	echo "Found REPROCESS file flag"
+	echo "node --max_old_space_size=8192 ./index.js run --reprocess"
+        node --max_old_space_size=8192 ./index.js run --reprocess
+else
+	echo "Starting normally based on CMD"
+	echo "node --max_old_space_size=8192 ./index.js $@"
+	node --max_old_space_size=8192 ./index.js "$@"
+fi
+