Skip to content

Optimized block endpoint #1190

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Jun 21, 2022
Merged

Optimized block endpoint #1190

merged 4 commits into from
Jun 21, 2022

Conversation

MHassanTariq
Copy link
Contributor

Description

This PR optimizes /block endpoint. It changes the previous approach of individual queries for each block to 3 queries for blocks, txs, and microblocks.
For details refer to issue #1173

Type of Change

  • New feature
  • Bug fix
  • API reference/documentation update
  • Other

Checklist

  • Code is commented where needed
  • Unit test coverage for new or modified code paths
  • npm run test passes
  • Changelog is updated
  • Tag 1 of @rafaelcr or @zone117x for review

@MHassanTariq MHassanTariq self-assigned this May 31, 2022
@github-actions
Copy link

github-actions bot commented May 31, 2022

@github-actions github-actions bot temporarily deployed to pull request May 31, 2022 11:05 Inactive
@coveralls
Copy link

coveralls commented May 31, 2022

Coverage Status

Coverage decreased (-18.6%) to 57.902% when pulling aec28ab on opmitize-block-endpoint into 8ee1da8 on develop.

@MHassanTariq MHassanTariq marked this pull request as ready for review May 31, 2022 13:01
@MHassanTariq MHassanTariq linked an issue May 31, 2022 that may be closed by this pull request
@MHassanTariq MHassanTariq changed the title refactor: optimized block endpoint Optimized block endpoint Jun 1, 2022
@MHassanTariq MHassanTariq requested a review from zone117x June 6, 2022 11:22
@github-actions github-actions bot temporarily deployed to pull request June 6, 2022 11:22 Inactive
@github-actions github-actions bot temporarily deployed to commit June 6, 2022 11:22 Inactive
@codecov-commenter
Copy link

codecov-commenter commented Jun 6, 2022

Codecov Report

Merging #1190 (bd88f6d) into develop (8ee1da8) will decrease coverage by 29.79%.
The diff coverage is 9.75%.

❗ Current head bd88f6d differs from pull request most recent head aec28ab. Consider uploading reports for the commit aec28ab to get more accurate results

@@             Coverage Diff              @@
##           develop    #1190       +/-   ##
============================================
- Coverage    76.61%   46.82%   -29.80%     
============================================
  Files          104      104               
  Lines         9725     9758       +33     
  Branches      2092     2091        -1     
============================================
- Hits          7451     4569     -2882     
- Misses        2164     4750     +2586     
- Partials       110      439      +329     
Impacted Files Coverage Δ
src/datastore/common.ts 100.00% <ø> (ø)
src/datastore/pg-store.ts 42.64% <0.00%> (-49.65%) ⬇️
src/api/controllers/db-controller.ts 36.87% <20.00%> (-43.51%) ⬇️
src/api/init.ts 77.29% <50.00%> (-2.51%) ⬇️
src/api/routes/block.ts 28.35% <50.00%> (-62.06%) ⬇️
src/datastore/offline-dummy-store.ts 0.00% <0.00%> (-100.00%) ⬇️
src/api/routes/search.ts 10.98% <0.00%> (-83.52%) ⬇️
src/api/routes/bns/names.ts 14.44% <0.00%> (-81.12%) ⬇️
src/api/rosetta-validate.ts 10.14% <0.00%> (-78.27%) ⬇️
src/api/routes/address.ts 12.40% <0.00%> (-75.59%) ⬇️
... and 38 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8ee1da8...aec28ab. Read the comment docs.

@MHassanTariq MHassanTariq requested a review from zone117x June 15, 2022 19:08
Copy link
Member

@zone117x zone117x left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needs a rebase, otherwise lgtm.
@rafaelcr could you also give this a review?

@MHassanTariq MHassanTariq force-pushed the opmitize-block-endpoint branch from e89df7a to aec28ab Compare June 20, 2022 13:58
@github-actions github-actions bot temporarily deployed to commit June 20, 2022 14:00 Inactive
@github-actions github-actions bot temporarily deployed to pull request June 20, 2022 14:00 Inactive
@MHassanTariq MHassanTariq requested a review from rafaelcr June 20, 2022 14:34
@MHassanTariq MHassanTariq force-pushed the opmitize-block-endpoint branch from bd88f6d to aec28ab Compare June 21, 2022 08:21
@github-actions github-actions bot temporarily deployed to pull request June 21, 2022 08:23 Inactive
@github-actions github-actions bot temporarily deployed to commit June 21, 2022 08:23 Inactive
Copy link
Collaborator

@rafaelcr rafaelcr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@rafaelcr rafaelcr merged commit 943e2d1 into develop Jun 21, 2022
@rafaelcr rafaelcr deleted the opmitize-block-endpoint branch June 21, 2022 14:01
blockstack-devops pushed a commit that referenced this pull request Sep 13, 2022
## [6.0.0-beta.1](v5.0.0...v6.0.0-beta.1) (2022-09-13)

### ⚠ BREAKING CHANGES

* mark breaking change

### Features

* add `transaction_count` for `microblocks_accepted` in block ([#1162](#1162)) ([78d7d9c](78d7d9c))
* add API version in response header ([#1216](#1216)) ([1e998db](1e998db))
* mark breaking change ([669fd0d](669fd0d))
* mempool stats endpoint and prometheus metrics ([#1241](#1241)) ([9482238](9482238))
* refactor pg classes, switch to postgres.js ([#1148](#1148)) ([3ff4177](3ff4177)), closes [#1168](#1168)
* send nft updates through websocket channels ([#1218](#1218)) ([920a7b8](920a7b8))

### Bug Fixes

* consolidate db migrations ([#1314](#1314)) ([d6bdf9f](d6bdf9f))
* included query params in redirecting to prefix 0x in tx endpoint ([#1205](#1205)) ([664cce7](664cce7))
* incorrect websocket/socket.io transaction updates ([#1197](#1197)) ([8ee1da8](8ee1da8))
* mobx breakage by locking package dependencies ([#1206](#1206)) ([5f8bc9f](5f8bc9f))
* optimize `getMicroblocks` query ([#1179](#1179)) ([7691109](7691109))
* optimize block endpoint ([#1190](#1190)) ([943e2d1](943e2d1))
* remove duplicate txs in microblock responses ([#1167](#1167)) ([15c0c11](15c0c11))
* tests ([1c1fd16](1c1fd16))
* upsert nft and ft metadata ([#1193](#1193)) ([c4eec5d](c4eec5d))
@blockstack-devops
Copy link
Contributor

🎉 This PR is included in version 6.0.0-beta.1 🎉

The release is available on:

Your semantic-release bot 📦🚀

blockstack-devops pushed a commit that referenced this pull request Oct 6, 2022
## [6.0.0](v5.0.1...v6.0.0) (2022-10-06)

### ⚠ BREAKING CHANGES

* remove deprecated `/nft_events` endpoint (#1329)
* mark breaking change

### Features

* add `transaction_count` for `microblocks_accepted` in block ([#1162](#1162)) ([78d7d9c](78d7d9c))
* add API version in response header ([#1216](#1216)) ([1e998db](1e998db))
* add owner index to subdomains table ([#1331](#1331)) ([a6c5e12](a6c5e12))
* add token_type metadata for rosetta ft operations ([#1332](#1332)) ([09af27b](09af27b))
* mark breaking change ([669fd0d](669fd0d))
* mempool stats endpoint and prometheus metrics ([#1241](#1241)) ([9482238](9482238))
* refactor pg classes, switch to postgres.js ([#1148](#1148)) ([3ff4177](3ff4177)), closes [#1168](#1168)
* send nft updates through websocket channels ([#1218](#1218)) ([920a7b8](920a7b8))

### Bug Fixes

* consolidate db migrations ([#1314](#1314)) ([d6bdf9f](d6bdf9f))
* event_observer_requests json writes ([#1334](#1334)) ([465aa0b](465aa0b))
* included query params in redirecting to prefix 0x in tx endpoint ([#1205](#1205)) ([664cce7](664cce7))
* incorrect websocket/socket.io transaction updates ([#1197](#1197)) ([8ee1da8](8ee1da8))
* mobx breakage by locking package dependencies ([#1206](#1206)) ([5f8bc9f](5f8bc9f))
* optimize `getMicroblocks` query ([#1179](#1179)) ([7691109](7691109))
* optimize block endpoint ([#1190](#1190)) ([943e2d1](943e2d1))
* refresh materialized views concurrently in new pg format ([#1324](#1324)) ([20b284f](20b284f))
* remove duplicate txs in microblock responses ([#1167](#1167)) ([15c0c11](15c0c11))
* remove live tsv append ([#1315](#1315)) ([e2a1247](e2a1247))
* retry pg connection on new library code ([#1326](#1326)) ([35db939](35db939))
* support multiple BNS name events in the same transaction ([#1337](#1337)) ([1edb256](1edb256))
* tests ([1c1fd16](1c1fd16))
* upgrade stacks node versions to 2.05.0.3.0 ([#1328](#1328)) ([e30636e](e30636e))
* upsert nft and ft metadata ([#1193](#1193)) ([c4eec5d](c4eec5d))

### Reverts

* Revert "chore!: remove deprecated `/nft_events` endpoint (#1329)" (#1343) ([c537ee4](c537ee4)), closes [#1329](#1329) [#1343](#1343)

### Miscellaneous Chores

* remove deprecated `/nft_events` endpoint ([#1329](#1329)) ([65bb4e5](65bb4e5))
@blockstack-devops
Copy link
Contributor

🎉 This PR is included in version 6.0.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

blockstack-devops pushed a commit that referenced this pull request Nov 22, 2022
# [4.2.0-stacks-2.1.1](v4.1.1...v4.2.0-stacks-2.1.1) (2022-11-22)

### Bug Fixes

* included query params in redirecting to prefix 0x in tx endpoint ([#1205](#1205)) ([664cce7](664cce7))
* incorrect websocket/socket.io transaction updates ([#1197](#1197)) ([8ee1da8](8ee1da8))
* mobx breakage by locking package dependencies ([#1206](#1206)) ([5f8bc9f](5f8bc9f))
* optimize `getMicroblocks` query ([#1179](#1179)) ([7691109](7691109))
* optimize block endpoint ([#1190](#1190)) ([943e2d1](943e2d1))
* remove duplicate txs in microblock responses ([#1167](#1167)) ([15c0c11](15c0c11))
* report locked STX balance correctly when multiple event entries exist from stack-increase and stack-extend operations ([#1363](#1363)) ([d06beb3](d06beb3))
* upsert nft and ft metadata ([#1193](#1193)) ([c4eec5d](c4eec5d))

### Features

* add `transaction_count` for `microblocks_accepted` in block ([#1162](#1162)) ([78d7d9c](78d7d9c))
* add API version in response header ([#1216](#1216)) ([1e998db](1e998db))
* include `stx-transfer-memo` results in `/address/<recipient>/stx_inbound` endpoint ([#1289](#1289)) ([cb5ca93](cb5ca93))
* include Stacks 2.1 `stx-transfer-memo` in Rosetta token transfer operations ([#1290](#1290)) ([c7cc900](c7cc900))
* mempool stats endpoint and prometheus metrics ([#1241](#1241)) ([9482238](9482238))
* refactor pg classes, switch to postgres.js ([#1148](#1148)) ([3ff4177](3ff4177)), closes [#1168](#1168)
* send nft updates through websocket channels ([#1218](#1218)) ([920a7b8](920a7b8))
* Stacks 2.1 PoX-2 support in Rosetta ([#1339](#1339)) ([e04bf64](e04bf64))
* store Stacks block miner address value from new rewards payload introduced in stacks-network/stacks-core#3373 ([#1413](#1413)) ([e34fab5](e34fab5))
* support for Stacks 2.1 `coinbase-pay-to-alt-recipient` txs ([#1283](#1283)) ([7e5752a](7e5752a))
* support for Stacks 2.1 `versioned-smart-contract` transactions ([#1279](#1279)) ([4ff1a91](4ff1a91)), closes [#1280](#1280)
* support for Stacks 2.1 stx-transfer event memo field ([#1285](#1285)) ([67332ce](67332ce))
@blockstack-devops
Copy link
Contributor

🎉 This PR is included in version 4.2.0-stacks-2.1.1 🎉

The release is available on:

Your semantic-release bot 📦🚀

zone117x added a commit that referenced this pull request Dec 23, 2022
* feat: support for Stacks 2.1 `versioned-smart-contract` transactions (#1279)

* dev: use latest `next` branch (Stacks 2.1) in docker-compose / dev environment

* chore: setup for testing with a local stacks-node v2.1 binary

* feat: support for versioned-smart-contract txs (Stacks 2.1)

* chore: update `stacks-encoding-native-js` dep to latest alpha

* fix: smart_contracts table missing the `clarity_version` column

* chore: test fixes

* chore: update test & dev env, and test suites to Stacks 2.1 image (#1280)

* fix: update several test suites to work with Stacks 2.1 docker image

* chore: move rosetta network endpoint tests to after more time-sensitive construction tests

* fix: attempt fix for `tests-rosetta-cli-data`

* chore: package.json fix

* fix: attempt fix for `tests-rosetta-cli-construction`

* chore: move test config into directory

* feat: separate suite for RPC request tests

* chore: update .github/workflows/ci.yml

Co-authored-by: Rafael Cárdenas <rafael@rafaelcr.com>

* chore: move btc faucet tests into own suite

* ci: add btc-faucet test to CI

Co-authored-by: Rafael Cárdenas <rafael@rafaelcr.com>

* chore: remove debug console.log

* chore: tests for versioned-smart-contact txs

* chore: update mocknet config to bootstrap to epoch 2.1

* chore: fix docker-compose background task logging in vscode debugger

* chore: remove local stacks-node debug and testing changes

Co-authored-by: Rafael Cárdenas <rafael@rafaelcr.com>

* feat: support for Stacks 2.1 `coinbase-pay-to-alt-recipient` txs (#1283)

* feat: support Stacks 2.1 `coinbase-pay-to-alt` tx types

* fix: add a `coinbase-to-alt` tx handling in a few more areas

* chore: fix coinbase in mempool table schema

* chore: update unit tests with new coinbase payload responses

* test: add tests for ingesting new coinbase tx payloads and generating expected API responses

* docs: better description for coinbase alt recipient

* feat: support for Stacks 2.1 stx-transfer event memo field (#1285)

* feat: support for Stacks 2.1 stx-transfer event memo field (the `stx-transfer-memo` Clarity2 function)

* fix: remove null memo field from different asset transfer events

* test: unit tests for Stacks 2.1 stx-transfer event memo field

* feat: include `stx-transfer-memo` results in `/address/<recipient>/stx_inbound` endpoint (#1289)

* feat: include Stacks 2.1 `stx-transfer-memo` in Rosetta token transfer operations (#1290)

* chore: cleanup Rosetta operation type strings

* feat: include `memo` from stx-transfer-memo events in Rosetta token transfer operations

* chore: lint fix

* chore: Rosetta tests for memo in stx-transfer-memo events

* fix: Rosetta memo strings should be decoded from hex

* chore: add `coinbase-pay-to-alt` tx type test for Rosetta parsing

* chore: upgrade to stacks.js v4.x (#1295)

* chore: upgrade to stacks.js v4.x

* chore: hardcode tx fees in rosetta tests

* chore: hardcode tx fees in token tests

* chore: hardcode tx fees in rosetta-cli-data tests

* chore: remove bugged fees in token tests

* chore: remove bugged fees in rosetta-cli-data tests

* chore: hardcode tx fees in rosetta-cli-data tests

* chore: fixes for bns tests

* feat: Stacks 2.1 PoX-2 support in Rosetta (#1339)

* feat: working setup for local krypton stacks-node miner with regtest bitcoind and auto-mining script

* feat: detect when stx mining starts for faster krypton stacks-node startup

* feat: use stacsk2.1-transition image for krypton debug deployment

* feat: update `/extended/v1/debug/broadcast/stack` endpoint to work with PoX-2

* chore: use PoX address decoding code from the WIP @stacks/stacking lib

* chore: update Rosetta PoX address parsing function

* chore(rosetta): skip extra /v2/info request when constructing stack-stx operation in Stacks 2.1

* feat: option in debug endpoint to use Rosetta to construct and submit Stacking transactions

* chore: simplify internal rosetta calls

* fix: Rosetta bug where parsing Stacking operations fails when the contract ABI is missing

* feat: Rosetta support for parsing PoX-2 contract calls

* feat: Rosetta support for parsing new Bitcoin address formats

* chore: remove no longer used testing configs and scripts

* chore: remove no longer used testing configs and scripts

* chore: debug page cleanup

* chore: configs for launching Stacks2.1-transition environment, and launching directly into Stacks2.1

* chore: task naming nit

* test: initial Stacks2.1 test suite

* chore: vscode debugger for 2.1 test suite

* ci: run 2.1 tests

* ci: possible fix for mapping host.docker.internal on linux

* chore: bitcoin address decoding without depending on try/catches for version detection

* chore: pin stacks2.1 images to hash

* chore: comment fix

* fix: report locked STX balance correctly when multiple event entries exist from stack-increase and stack-extend operations (#1363)

* chore: lint fixes

* chore: test moving migration files to non-ts source

* chore: rename migration file exts to js

* chore: fix pg migrations failing from certain jest scripts

* chore: cleanup jest.config.2.1.js

* chore: lint fix (revert)

* test: ensure locked stx balance event data has expected values after pox-2 stack-increase operation

* test: ensure stx unlock height event data has expected value after pox-2 stack-extend operation

* fix: report locked STX balance correctly when multiple event entries exist from stack-increase and stack-extend operations

* chore: refactor PoX-2 operation steps into separate tests

* chore: test scaffolding for pox-2 rewards

* chore: lint fixes

* test: validate Stacking slots and rewards after PoX-2 operations

* chore: update to bitcoin-v0.21.1 in stacks-2.1 test suite

* test: add additional 1 block padding to unlock height waiter for test flakiness

* chore(stacks-2.1): upgrade to bitcoinjs-lib-v6.x, initial P2TR work (#1367)

* chore: upgrade to bitcoinjs-lib-v6.x which has P2TR support

* feat: P2TR bitcoin address util

* chore: PoX-2 integration testing for segwit addresses (#1372)

* feat: functions for generating Bitcoin addresses in all formats supported by Stacks 2.1

* test: segwit PoX-2 integration tests

* test: e2e tests for PoX-2 segwit (P2WPKH) support

* chore: test PoX rewards against bitcoin json-rpc results

* test: e2e tests for PoX-2 rewards address formats p2pkh, p2sh, p2wpkh, p2wsh

* test: use more robust bitcoin `listtransactions` RPC method for validating PoX-2 rewards

* chore: update plain P2SH address generation to create a more standard P2SH(P2PKH) rather than 1-of-1 multisig address

* chore: temp skip segwit tests until `feat/native-segwit` stacks-blockchain branch is merged

* chore: Revert "temp skip segwit tests until `feat/native-segwit` stacks-blockchain branch is merged" -- too many other dependent changes

* chore: filter bitcoin rpc `listtransactions` result for only the address being tested

* ci: don't persist config in checkout

* feat: store Stacks block miner address value from new rewards payload introduced in stacks-network/stacks-core#3373 (#1413)

* chore: bump to latest Stacks 2.1 `next` branch (#1449)

* ci: configure semrel branches for 2.1 prerelease packages

* chore(release): 4.2.0-stacks-2.1.1 [skip ci]

# [4.2.0-stacks-2.1.1](v4.1.1...v4.2.0-stacks-2.1.1) (2022-11-22)

### Bug Fixes

* included query params in redirecting to prefix 0x in tx endpoint ([#1205](#1205)) ([664cce7](664cce7))
* incorrect websocket/socket.io transaction updates ([#1197](#1197)) ([8ee1da8](8ee1da8))
* mobx breakage by locking package dependencies ([#1206](#1206)) ([5f8bc9f](5f8bc9f))
* optimize `getMicroblocks` query ([#1179](#1179)) ([7691109](7691109))
* optimize block endpoint ([#1190](#1190)) ([943e2d1](943e2d1))
* remove duplicate txs in microblock responses ([#1167](#1167)) ([15c0c11](15c0c11))
* report locked STX balance correctly when multiple event entries exist from stack-increase and stack-extend operations ([#1363](#1363)) ([d06beb3](d06beb3))
* upsert nft and ft metadata ([#1193](#1193)) ([c4eec5d](c4eec5d))

### Features

* add `transaction_count` for `microblocks_accepted` in block ([#1162](#1162)) ([78d7d9c](78d7d9c))
* add API version in response header ([#1216](#1216)) ([1e998db](1e998db))
* include `stx-transfer-memo` results in `/address/<recipient>/stx_inbound` endpoint ([#1289](#1289)) ([cb5ca93](cb5ca93))
* include Stacks 2.1 `stx-transfer-memo` in Rosetta token transfer operations ([#1290](#1290)) ([c7cc900](c7cc900))
* mempool stats endpoint and prometheus metrics ([#1241](#1241)) ([9482238](9482238))
* refactor pg classes, switch to postgres.js ([#1148](#1148)) ([3ff4177](3ff4177)), closes [#1168](#1168)
* send nft updates through websocket channels ([#1218](#1218)) ([920a7b8](920a7b8))
* Stacks 2.1 PoX-2 support in Rosetta ([#1339](#1339)) ([e04bf64](e04bf64))
* store Stacks block miner address value from new rewards payload introduced in stacks-network/stacks-core#3373 ([#1413](#1413)) ([e34fab5](e34fab5))
* support for Stacks 2.1 `coinbase-pay-to-alt-recipient` txs ([#1283](#1283)) ([7e5752a](7e5752a))
* support for Stacks 2.1 `versioned-smart-contract` transactions ([#1279](#1279)) ([4ff1a91](4ff1a91)), closes [#1280](#1280)
* support for Stacks 2.1 stx-transfer event memo field ([#1285](#1285)) ([67332ce](67332ce))

* feat: ingestion and querying for new PoX-2 events (#1403)

* test: Rosetta PoX-2 rewards and unlock events

* feat: progress on parsing and writing PoX2 events to postgres

* feat: sql insert for pox2 event data

* feat: reading pox2 event data from db, sql constraints on named pox event types

* feat: db-controller and initial GET route for reading & parsing pox2 events

* chore: progress on e2e testing for pox-2 delegation functionality

* chore: e2e testing for pox-2 delegation commit operation

* chore: e2e testing for pox-2 delegate-stack-increase

* chore: e2e testing for pox-2 delegate-stack-extend

* chore: e2e testing for pox-2 handle-unlock (missed reward slot)

* chore: lint fix, possible test fix

* chore: validation tests for each of the pox2-event payloads

* chore: potential test fixes

* chore: test fixes num 2

* chore: additional flaky test fixes

* feat: use pox2-events table to calculate account STX locked amount in /extended/v1/address balance endpoints

* chore: Rosetta integration tests for using segwit address in stacking operation

* test: ensure stacks RPC reports expected locked balance after Rosetta lock op

* chore: bump to latest pox2-events stacks-node image

* chore: move pox2-event parsing code into separate file

* ci: bump stacks2.1 test suite time to account for longer block mining interval from latest stacks-node changes

* chore: speed up tests with better "wait for pox cycle" logic

* ci: test fixes

* fix: adjustments to `handle-unlock` processing

* feat: add support for latest additions to pox2: `stack-aggregation-commit-indexed` and `stack-aggregation-increase`

* chore: integration tests for `stack-aggregation-commit-indexed` and `stack-aggregation-increase` pox2 event handling

* feat: store pox_v1_unlock_height as singleton in pg

* chore: initial setup for stacks-2.1 transition test suite

* chore: fix pox_state migrations

* feat: make account balance drop pox1-locked state after pox_v1_unlock_height is reached

* fix: rosetta block unlock events were limited to one account

* fix: off-by-one bug causing accounts to be reported as unlocked one block too early

* feat: add burn_block_height to db chainTip and /extended/status

* chore: fix datastore-tests from new status endpoint field

* chore: fix datastore-tests from new status endpoint field, second pass

* fix: adjust pox2_events query to handle special case with `handle-unlock`

* feat: add `contract_name` processing for the `stx_lock_event`, see stacks-network/stacks-core#3413

* chore: bump to latest stacks-network/stacks-core#3415

* chore: add account balance test for consistency between RPC and API balances throughout pox transition

* ci: matrix for stacks2.1-transition tests

* chore: start of Rosetta pox transition period tests

* chore: test Rosetta account balance endpoint in pox-2 stacking test suite

* ci: use unique account key for pox-2 rosetta test suite

* chore: test for pox_v1_unlock_height in Rosetta pox transition tests

* chore: refactor giant pox-2 test file into multiple files, run separately in ci

* feat: generate synthetic Rosetta stx unlock operations for the pox_v1_unlock_height block

* feat: generate Rosetta unlock operations from pox2 auto-unlock events

* fix: store locked balance from `handle-unlock` event rather than resetting to zero

* chore: remove todos

* chore: remove todos

* chore: remove no-longer correct sanity check

* chore: address PR feedback, composite index for pox2_events ordering

* fix: add default stx faucet tx fee if estimate not available (#1456)

* test: add integration test for stx faucet requests

* fix: add default stx faucet tx fee if estimate not available

* ci: disable generated doc type file test for now

* fix: add automatic pox switching to rosetta (#1454)

* refactor: add minor changes to rosetta

* chore: switch to stacks.js v6 and refactor helpers

* fix: update rosetta construction for 2.1

* test: refactor pox transition tests

* test: add btc address format rosetta test

* build: npm audit

* chore: lint fix json files not in src

* chore: fix exports

* test: add rosetta stacking for each block height

* test: add double stacking test

* chore: revert back to original suggested_fee_multiplier behavior

* chore: remove dynamic contract from dummy data

* chore: remove todos

* test: fix test address

* test: skip failing tests

* test: add check to rosetta contract tests

* chore: update to latest node next branch for testing

Co-authored-by: janniks <janniks@users.noreply.github.com>

* chore(release): 7.0.0-stacks-2.1.1 [skip ci]

# [7.0.0-stacks-2.1.1](v7.0.0-beta.1...v7.0.0-stacks-2.1.1) (2022-11-29)

### Bug Fixes

* add automatic pox switching to rosetta ([#1454](#1454)) ([ad7e492](ad7e492))
* add default stx faucet tx fee if estimate not available ([#1456](#1456)) ([eeeffd0](eeeffd0))

### Features

* ingestion and querying for new PoX-2 events ([#1403](#1403)) ([1936ba6](1936ba6))

* ci: re-enable docs-lint

* chore: fix post-merge lint errors

* fix: use new `this.sqlTransaction(...)` in pox2 db queries

* ci: re-enable docs-lint

* chore: post-merge fixes

* ci: add missing gh job for `event-replay` test suite

* ci: increase timeout for 2.1 test suite

* chore: split up pox-2-rosetta tests (#1461)

* chore: split up pox-2-rosetta tests

* chore: split up pox-2-rosetta test suites into smaller units

* feat: [Stacks 2.1] Support new "block 0" boot events (#1476)

* chore: split up pox-2-rosetta tests

* chore: split up pox-2-rosetta test suites into smaller units

* feat: new regtest standalone image with 2.1 node

* chore: store postgres data in same volume

* chore: add configurable heights for each epoch

* feat: arg to toggle if docker image is bootstraps the chainstate

* chore: block height adjustments

* chore: block heights that work with epoch 2.0 and 2.05 receipts

* chore: fix bash script error

* feat: store and forward boot events

* chore: allow pg server connections

* chore: do not error when skipping bootstrap

* chore: update to latest stacks2.1 images

* chore: add epoch config for mocknet node

* ci: attempt fix for host.docker.internal for tests using stacks mocknet image

* ci: fix tokens and rosetta mocknet mode tests

* feat: finish ingesting "block 0" data once the first real block is received

* test: implement test suite to ensure "block-zero" is ingested correctly

* test: re-enable double-stacking test suite

* ci: add workflow dispatch for standalone docker image build [skip ci]

* ci: allow use of existing stacks-node binaries from docker context

* ci: allow stacks-api version tag to be set, require git sha args

* ci: add workflow dispatch for standalone docker image build [skip ci]

* chore: bump to latest stacks-blockchain `next` branch

* chore(release): 7.0.0-stacks-2.1.2 [skip ci]

## [7.0.0-stacks-2.1.2](v7.0.0-stacks-2.1.1...v7.0.0-stacks-2.1.2) (2022-12-21)

### ⚠ BREAKING CHANGES

* remove deprecated `/nft_events` endpoint (#1329)
* mark breaking change
* optimize tables and improve canonical treatment of BNS data (#1287)

### Features

* [Stacks 2.1] Support new "block 0" boot events ([#1476](#1476)) ([3d1b8f6](3d1b8f6))
* add ENV configs for DB close and API shutdown timeouts ([#1366](#1366)) ([444f008](444f008))
* add indexes for index_block_hash on BNS tables ([#1304](#1304)) ([bbf4b2d](bbf4b2d))
* add owner index to subdomains table ([#1331](#1331)) ([a6c5e12](a6c5e12))
* add token_type metadata for rosetta ft operations ([#1332](#1332)) ([09af27b](09af27b))
* **agg-paging-limits:** aggregated all paging query limits ([#1401](#1401)) ([0203d36](0203d36)), closes [#1379](#1379) [#1379](#1379)
* configurable pg connection lifetime and idle timeouts ([#1355](#1355)) ([46ccf06](46ccf06))
* mark breaking change ([669fd0d](669fd0d))
* optimize tables and improve canonical treatment of BNS data ([#1287](#1287)) ([1f64818](1f64818))

### Bug Fixes

* add memos to send-many-memo rosetta STX transfer operations ([#1389](#1389)) ([0a552b8](0a552b8))
* add owner index on subdomains table ([#1323](#1323)) ([c9c6d05](c9c6d05))
* add postgres connection error checking for ECONNRESET code ([03a1896](03a1896))
* bump version ([3863cce](3863cce))
* catch cache controller db errors ([#1368](#1368)) ([f15df41](f15df41))
* catch pg exceptions on queries outside of express ([#1348](#1348)) ([1f07b85](1f07b85))
* consolidate db migrations ([#1314](#1314)) ([d6bdf9f](d6bdf9f))
* detect name transfers and renewals in special circumstances ([#1303](#1303)) ([cd381a9](cd381a9))
* disable faucet endpoints on mainnet ([#1425](#1425)) ([b79b9b4](b79b9b4))
* event_observer_requests json writes ([#1334](#1334)) ([465aa0b](465aa0b))
* filter BNS processing for successful txs only ([#1309](#1309)) ([6a12936](6a12936))
* get rosetta latest block from chain_tip view ([#1445](#1445)) ([ad386d3](ad386d3))
* guarantee db is empty before performing a replay ([#1374](#1374)) ([ef8e7a9](ef8e7a9))
* guard against empty lists before querying postgres ([#1345](#1345)) ([6c88a16](6c88a16))
* handle pg exceptions on web socket transmitter ([#1353](#1353)) ([2e6448d](2e6448d))
* handle postgres dns lookup error ([#1433](#1433)) ([e00efd4](e00efd4))
* handle postgres.js connection timeouts ([#1424](#1424)) ([4a2a342](4a2a342))
* handle websocket messages with a priority queue ([#1427](#1427)) ([f0cb01c](f0cb01c))
* import BNS v1 data during event replay ([#1301](#1301)) ([bc59817](bc59817))
* lint docs ci dependencies ([#1458](#1458)) ([90d0c7b](90d0c7b))
* log PoisonMicroblock tx instead rather than throwing ([#1379](#1379)) ([cee6352](cee6352))
* refresh materialized views concurrently ([#1270](#1270)) ([057c541](057c541))
* refresh materialized views concurrently in new pg format ([#1324](#1324)) ([20b284f](20b284f))
* refresh materialized views in their own pg connection ([#1356](#1356)) ([9433d3c](9433d3c))
* remove duplicate tx socket updates inside db transactions ([#1360](#1360)) ([60c185d](60c185d))
* remove live tsv append ([#1315](#1315)) ([e2a1247](e2a1247))
* retry pg connection on new library code ([#1326](#1326)) ([35db939](35db939))
* revert to 404 error code on bns name errors ([#1440](#1440)) ([cdc039c](cdc039c))
* skip migrations on read-only start ([#1351](#1351)) ([1d32261](1d32261))
* sql transaction consistency ([#1410](#1410)) ([01e26d9](01e26d9))
* support multiple BNS name events in the same transaction ([#1337](#1337)) ([1edb256](1edb256))
* tests ([1c1fd16](1c1fd16))
* update testnet send-many-memo contract id ENV ([#1420](#1420)) ([45ea24d](45ea24d))
* upgrade stacks node versions to 2.05.0.3.0 ([#1328](#1328)) ([e30636e](e30636e))
* use new `this.sqlTransaction(...)` in pox2 db queries ([27102da](27102da))

### Reverts

* Revert "chore!: remove deprecated `/nft_events` endpoint (#1329)" (#1343) ([c537ee4](c537ee4)), closes [#1329](#1329) [#1343](#1343)

### Miscellaneous Chores

* remove deprecated `/nft_events` endpoint ([#1329](#1329)) ([65bb4e5](65bb4e5))

* chore: bump stacks-native-encoding-js version (#1495)

* chore: bump stacks-native-encoding-js version

* chore: revert auto-formatted line changes

* chore: bump stacks-native-encoding-js version

* Merge branch 'develop' into stacks-2.1

# Conflicts:
#	src/event-stream/event-server.ts

Co-authored-by: Rafael Cárdenas <rafael@rafaelcr.com>
Co-authored-by: CharlieC3 <2747302+CharlieC3@users.noreply.github.com>
Co-authored-by: semantic-release-bot <semantic-release-bot@martynus.net>
Co-authored-by: janniks <6362150+janniks@users.noreply.github.com>
Co-authored-by: janniks <janniks@users.noreply.github.com>
blockstack-devops pushed a commit that referenced this pull request Jan 10, 2023
## [7.0.0-beta.1](v6.2.1...v7.0.0-beta.1) (2023-01-10)

### ⚠ BREAKING CHANGES

* a sync from genesis is required to use with a Stacks v2.1-rc node

### Features

* **agg-paging-limits:** aggregated all paging query limits ([#1401](#1401)) ([0203d36](0203d36)), closes [#1379](#1379) [#1379](#1379)
* Stacks 2.1 support ([#1498](#1498)) ([dcbdfb9](dcbdfb9)), closes [#1279](#1279) [#1280](#1280) [#1283](#1283) [#1285](#1285) [#1289](#1289) [#1290](#1290) [#1295](#1295) [#1339](#1339) [#1363](#1363) [#1367](#1367) [#1372](#1372) [#1413](#1413) [#1449](#1449) [#1205](#1205) [#1197](#1197) [#1206](#1206) [#1179](#1179) [#1190](#1190) [#1167](#1167) [#1363](#1363) [#1193](#1193) [#1162](#1162) [#1216](#1216) [#1289](#1289) [#1290](#1290) [#1241](#1241) [#1168](#1168) [#1218](#1218) [#1339](#1339) [#1413](#1413) [#1283](#1283) [#1280](#1280) [#1285](#1285) [#1403](#1403) [#1456](#1456) [#1454](#1454) [#1454](#1454) [#1456](#1456) [#1403](#1403) [#1461](#1461) [#1476](#1476) [#1329](#1329) [#1287](#1287) [#1476](#1476) [#1366](#1366) [#1304](#1304) [#1331](#1331) [#1332](#1332) [#1379](#1379) [#1379](#1379) [#1355](#1355) [#1287](#1287) [#1389](#1389) [#1323](#1323) [#1368](#1368) [#1348](#1348) [#1314](#1314) [#1303](#1303) [#1425](#1425) [#1334](#1334) [#1309](#1309) [#1445](#1445) [#1374](#1374) [#1345](#1345) [#1353](#1353) [#1433](#1433) [#1424](#1424) [#1427](#1427) [#1301](#1301) [#1458](#1458) [#1379](#1379) [#1270](#1270) [#1324](#1324) [#1356](#1356) [#1360](#1360) [#1315](#1315) [#1326](#1326) [#1440](#1440) [#1351](#1351) [#1410](#1410) [#1337](#1337) [#1420](#1420) [#1328](#1328) [#1329](#1329) [#1343](#1343) [#1329](#1329) [#1495](#1495)

### Bug Fixes

* add bnsImportUpdate to event emitter to fix BNS import test ([#1491](#1491)) ([2f9cb0c](2f9cb0c))
* guarantee db is empty before performing a replay ([#1374](#1374)) ([ef8e7a9](ef8e7a9))
* lint docs ci dependencies ([#1458](#1458)) ([90d0c7b](90d0c7b))
* make query limits backwards compatible ([#1509](#1509)) ([a0cebf5](a0cebf5))
* reorg txs by inserting txs that are missing from the mempool table ([#1429](#1429)) ([a512511](a512511))
* synthetic tx parsing for pox2 bitcoin-ops ([#1505](#1505)) ([720dc87](720dc87))

### Miscellaneous Chores

* support for Stacks 2.1 ([e88ec29](e88ec29))
blockstack-devops pushed a commit that referenced this pull request Jan 10, 2023
## [7.0.0-beta.1](v6.2.1...v7.0.0-beta.1) (2023-01-10)

### ⚠ BREAKING CHANGES

* a sync from genesis is required to use with a Stacks v2.1-rc node

### Features

* **agg-paging-limits:** aggregated all paging query limits ([#1401](#1401)) ([0203d36](0203d36)), closes [#1379](#1379) [#1379](#1379)
* Stacks 2.1 support ([#1498](#1498)) ([dcbdfb9](dcbdfb9)), closes [#1279](#1279) [#1280](#1280) [#1283](#1283) [#1285](#1285) [#1289](#1289) [#1290](#1290) [#1295](#1295) [#1339](#1339) [#1363](#1363) [#1367](#1367) [#1372](#1372) [#1413](#1413) [#1449](#1449) [#1205](#1205) [#1197](#1197) [#1206](#1206) [#1179](#1179) [#1190](#1190) [#1167](#1167) [#1363](#1363) [#1193](#1193) [#1162](#1162) [#1216](#1216) [#1289](#1289) [#1290](#1290) [#1241](#1241) [#1168](#1168) [#1218](#1218) [#1339](#1339) [#1413](#1413) [#1283](#1283) [#1280](#1280) [#1285](#1285) [#1403](#1403) [#1456](#1456) [#1454](#1454) [#1454](#1454) [#1456](#1456) [#1403](#1403) [#1461](#1461) [#1476](#1476) [#1329](#1329) [#1287](#1287) [#1476](#1476) [#1366](#1366) [#1304](#1304) [#1331](#1331) [#1332](#1332) [#1379](#1379) [#1379](#1379) [#1355](#1355) [#1287](#1287) [#1389](#1389) [#1323](#1323) [#1368](#1368) [#1348](#1348) [#1314](#1314) [#1303](#1303) [#1425](#1425) [#1334](#1334) [#1309](#1309) [#1445](#1445) [#1374](#1374) [#1345](#1345) [#1353](#1353) [#1433](#1433) [#1424](#1424) [#1427](#1427) [#1301](#1301) [#1458](#1458) [#1379](#1379) [#1270](#1270) [#1324](#1324) [#1356](#1356) [#1360](#1360) [#1315](#1315) [#1326](#1326) [#1440](#1440) [#1351](#1351) [#1410](#1410) [#1337](#1337) [#1420](#1420) [#1328](#1328) [#1329](#1329) [#1343](#1343) [#1329](#1329) [#1495](#1495)

### Bug Fixes

* add bnsImportUpdate to event emitter to fix BNS import test ([#1491](#1491)) ([2f9cb0c](2f9cb0c))
* guarantee db is empty before performing a replay ([#1374](#1374)) ([ef8e7a9](ef8e7a9))
* lint docs ci dependencies ([#1458](#1458)) ([90d0c7b](90d0c7b))
* make query limits backwards compatible ([#1509](#1509)) ([a0cebf5](a0cebf5))
* reorg txs by inserting txs that are missing from the mempool table ([#1429](#1429)) ([a512511](a512511))
* synthetic tx parsing for pox2 bitcoin-ops ([#1505](#1505)) ([720dc87](720dc87))

### Miscellaneous Chores

* support for Stacks 2.1 ([e88ec29](e88ec29))
blockstack-devops pushed a commit that referenced this pull request Feb 7, 2023
## [7.0.0](v6.3.4...v7.0.0) (2023-02-07)

### ⚠ BREAKING CHANGES

* support for upcoming Stacks 2.1 features, event-replay required;
* a sync from genesis is required to use with a Stacks v2.1-rc node

### Features

* [Stacks 2.1] `delegate-stx` Bitcoin-op parsing ([#1527](#1527)) ([ea01587](ea01587))
* **agg-paging-limits:** aggregated all paging query limits ([#1401](#1401)) ([0203d36](0203d36)), closes [#1379](#1379) [#1379](#1379)
* Stacks 2.1 support ([#1498](#1498)) ([dcbdfb9](dcbdfb9)), closes [#1279](#1279) [#1280](#1280) [#1283](#1283) [#1285](#1285) [#1289](#1289) [#1290](#1290) [#1295](#1295) [#1339](#1339) [#1363](#1363) [#1367](#1367) [#1372](#1372) [#1413](#1413) [#1449](#1449) [#1205](#1205) [#1197](#1197) [#1206](#1206) [#1179](#1179) [#1190](#1190) [#1167](#1167) [#1363](#1363) [#1193](#1193) [#1162](#1162) [#1216](#1216) [#1289](#1289) [#1290](#1290) [#1241](#1241) [#1168](#1168) [#1218](#1218) [#1339](#1339) [#1413](#1413) [#1283](#1283) [#1280](#1280) [#1285](#1285) [#1403](#1403) [#1456](#1456) [#1454](#1454) [#1454](#1454) [#1456](#1456) [#1403](#1403) [#1461](#1461) [#1476](#1476) [#1329](#1329) [#1287](#1287) [#1476](#1476) [#1366](#1366) [#1304](#1304) [#1331](#1331) [#1332](#1332) [#1379](#1379) [#1379](#1379) [#1355](#1355) [#1287](#1287) [#1389](#1389) [#1323](#1323) [#1368](#1368) [#1348](#1348) [#1314](#1314) [#1303](#1303) [#1425](#1425) [#1334](#1334) [#1309](#1309) [#1445](#1445) [#1374](#1374) [#1345](#1345) [#1353](#1353) [#1433](#1433) [#1424](#1424) [#1427](#1427) [#1301](#1301) [#1458](#1458) [#1379](#1379) [#1270](#1270) [#1324](#1324) [#1356](#1356) [#1360](#1360) [#1315](#1315) [#1326](#1326) [#1440](#1440) [#1351](#1351) [#1410](#1410) [#1337](#1337) [#1420](#1420) [#1328](#1328) [#1329](#1329) [#1343](#1343) [#1329](#1329) [#1495](#1495)

### Bug Fixes

* add bnsImportUpdate to event emitter to fix BNS import test ([#1491](#1491)) ([2f9cb0c](2f9cb0c))
* build rosetta with node 16 ([654b64f](654b64f))
* datastore tests ([bb96507](bb96507))
* guarantee db is empty before performing a replay ([#1374](#1374)) ([ef8e7a9](ef8e7a9))
* lint docs ci dependencies ([#1458](#1458)) ([90d0c7b](90d0c7b))
* make query limits backwards compatible ([#1509](#1509)) ([a0cebf5](a0cebf5))
* prevent token metadata processor from blocking api launch ([#1514](#1514)) ([63da7e1](63da7e1))
* reorg txs by inserting txs that are missing from the mempool table ([#1429](#1429)) ([a512511](a512511))
* synthetic tx parsing for pox2 bitcoin-ops ([#1505](#1505)) ([720dc87](720dc87))
* test tx types ([11b9013](11b9013))
* use correct `pox-addr` arg while parsing `stack-stx` bitcoin-op [#415](#415) ([#1533](#1533)) ([ab14ad5](ab14ad5))
* use pg bigint for `pox_v1_unlock_height` column ([#1521](#1521)) ([d3fd685](d3fd685))

### Miscellaneous Chores

* note for Stacks 2.1 support and major version bump ([d27f956](d27f956))
* support for Stacks 2.1 ([e88ec29](e88ec29))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

blocks endpoint is pretty slow consistently
6 participants