Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Wallet TX Count and time indexing #888

Merged
merged 33 commits into from
Nov 20, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
b01c806
node-http: Add get median time to node http.
nodech Mar 4, 2024
c38e4b9
chain: add getEntries, similar to getHashes.
nodech Mar 13, 2024
f233b76
walletdb: Write time with blockmeta record. Update wdb version to 3.
nodech Mar 18, 2024
666c412
wdb: add median time past.
nodech Mar 18, 2024
63776fc
wdb: pass tx index and median time to addTX.
nodech Mar 19, 2024
eb610d8
wdb-txdb: Add pagination indexes.
nodech Mar 20, 2024
40f7d7f
wdb: Add tx history queries for the count and time indexes.
nodech Mar 26, 2024
4c11a5c
wdb: Pass mtime to the TXRecord.
nodech Mar 27, 2024
0f08c66
wallet: switch zap to new indexes.
nodech Mar 28, 2024
dc2471c
wallet-http: Change parameters for getHistory and add pagination.
nodech Mar 28, 2024
671494d
wallet-http: Update getPending, change parameters and add pagination.
nodech Mar 30, 2024
062d1f6
wallet: deprecate getLast and getRange. Update hsw-cli history and
nodech Mar 30, 2024
c6c495e
test: use nodeCtx in wallet rpc.
nodech Mar 30, 2024
de79842
wallet-rpc: add new methods for listing history.
nodech Mar 30, 2024
6aeceed
wdb: Add timeCache to the interactive rescan callback.
nodech Aug 29, 2024
66ebb3b
wdb: Update wdb version and fix txdb layout.x
nodech Sep 4, 2024
46f099f
wallet: add getBlockHeader method to node and nullclients.
nodech Sep 10, 2024
ac20476
txdb: call nowFn once in insert.
nodech Sep 21, 2024
9832a3d
wdb: merge getMedianTime and getMedianTimeTip.
nodech Sep 21, 2024
03aeaf3
wdb: calculate pagination count indexes per wallet instead of per block.
nodech Sep 22, 2024
9d21181
wdb: Add tx pagination migration from v3 to v4.
nodech Sep 22, 2024
a03ddce
wdb-migrations: add block time cache.
nodech Sep 22, 2024
4283253
wdb-migration: verify time entries. Add more comments for the migrati…
nodech Sep 23, 2024
9165139
wdb-migration: skip already migrated parts in TXCountTimeIndex.
nodech Sep 23, 2024
ce04d65
wdb-migration: remove unnecessary indexes from TXCountTimeIndex.
nodech Sep 23, 2024
fc00570
txdb: reduce lifetime of the layout.x.
nodech Oct 2, 2024
15e9646
wdb-migration: handle non-wallet outputs and fix medianTime.
nodech Oct 9, 2024
a354485
wallet-http: fix unconfirmed tx list json.
nodech Oct 25, 2024
ab6292a
wdb-migration: handle not-owned txs that are not being fully tracked.
nodech Oct 28, 2024
904756e
wdb-migration: resume CountAndTime migration after interruption.
nodech Oct 28, 2024
a914b01
txdb: move count and time indexes under the same prefix. (O).
nodech Nov 18, 2024
0d0cc1f
wdb-migration: handle edge case when wdb.addBlock is interrupted.
nodech Nov 19, 2024
6558b45
wallet: fix getBlockHeader for NodeClient.
nodech Nov 19, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
txdb: call nowFn once in insert.
  • Loading branch information
nodech committed Sep 21, 2024
commit ac2047646d8e9696b4b35a7752d59573f5c8709d
6 changes: 4 additions & 2 deletions lib/blockchain/migrations.js
Original file line number Diff line number Diff line change
Expand Up @@ -351,6 +351,8 @@ class MigrateBlockStore extends AbstractMigration {
this.ldb = options.ldb;
this.blocks = options.db.blocks;
this.layout = MigrateBlockStore.layout();

this.batchWriteSize = 10000;
}

/**
Expand Down Expand Up @@ -407,7 +409,7 @@ class MigrateBlockStore extends AbstractMigration {
await this.blocks.writeBlock(hash, value);
parent.del(key);

if (++total % 10000 === 0) {
if (++total % this.batchWriteSize === 0) {
await parent.write();
this.logger.debug('Migrated up %d blocks.', total);
parent = this.ldb.batch();
Expand Down Expand Up @@ -441,7 +443,7 @@ class MigrateBlockStore extends AbstractMigration {
await this.blocks.writeUndo(hash, value);
parent.del(key);

if (++total % 10000 === 0) {
if (++total % this.batchWriteSize === 0) {
await parent.write();
this.logger.debug('Migrated up %d undo blocks.', total);
parent = this.ldb.batch();
Expand Down
20 changes: 11 additions & 9 deletions lib/wallet/txdb.js
Original file line number Diff line number Diff line change
Expand Up @@ -1178,13 +1178,15 @@ class TXDB {
// we need to add that information.
// TODO: This can be skipped if TX is coinbase, but even now
// it will be properly cleaned up on erase.
await this.addTimeAndCountIndexUnconfirmedUndo(b, hash);
await this.addTimeAndCountIndexUnconfirmedUndo(b, hash, wtx.mtime);

await this.addBlockMap(b, height);
await this.addBlock(b, tx.hash(), block);
} else {
// Add indexing for unconfirmed transactions.
await this.addCountAndTimeIndexUnconfirmed(b, state.accounts, hash);
await this.addCountAndTimeIndexUnconfirmed(b, state.accounts, hash,
wtx.mtime);

await this.addTXMap(b, hash);
}

Expand Down Expand Up @@ -1920,11 +1922,11 @@ class TXDB {
* @private
* @param {Batch} b
* @param {Hash} hash - Transaction hash.
* @param {Number} time - Transaction time.
* @returns {Promise}
*/

async addTimeAndCountIndexUnconfirmedUndo(b, hash) {
const time = this.nowFn();
async addTimeAndCountIndexUnconfirmedUndo(b, hash, time) {
const count = await this.getLatestUnconfirmedTXCount();

b.put(layout.e.encode(hash), fromU32(time));
Expand Down Expand Up @@ -1972,16 +1974,16 @@ class TXDB {
* @param {Batch} b
* @param {Map<Number, Balance>} accounts
* @param {Hash} hash - Transaction hash.
* @param {Number} time - Transaction time.
* @returns {Promise}
*/

async addCountAndTimeIndexUnconfirmed(b, accounts, hash) {
async addCountAndTimeIndexUnconfirmed(b, accounts, hash, time) {
const count = await this.getLatestUnconfirmedTXCount();

b.put(layout.z.encode(count.height, count.index), hash);
b.put(layout.y.encode(hash), count.encode());

const time = this.nowFn();
b.put(layout.e.encode(hash), fromU32(time));
b.put(layout.w.encode(time, count.index, hash));

Expand Down Expand Up @@ -2121,11 +2123,11 @@ class TXDB {
b.put(layout.y.encode(hash), count.encode());

const time = blockextra.medianTime;
b.put(layout.g.encode(time, height, index, options.hash));
b.put(layout.g.encode(time, height, index, hash));

for (const [acct] of accounts) {
b.put(layout.Z.encode(acct, height, index), hash);
b.put(layout.G.encode(acct, time, height, index, options.hash));
b.put(layout.G.encode(acct, time, height, index, hash));
}
}

Expand Down Expand Up @@ -4628,7 +4630,7 @@ class BlockRecord extends bio.Struct {
/**
* Instantiate wallet block from serialized tip data.
* @private
* @param {Buffer} data
* @param {bio.BufferReader} br
*/

read(br) {
Expand Down
17 changes: 11 additions & 6 deletions test/wallet-test.js
Original file line number Diff line number Diff line change
Expand Up @@ -3975,15 +3975,20 @@ describe('Wallet', function() {
for (let i = 0; i < 2; i++) {
for (const account of [DEFAULT, ALT]) {
const mtx = await dummyTX(wallet, account);
// this increments/calls nowFn twice. One for
// wtx creation and another for unconfirmed index.
// this increments/calls nowFn once.
await wdb.addTX(mtx.toTX());
hashes.push(mtx.hash());
}
}

// zap will also call nowFn once. (0 - 3 time is incremented by first two)
const zapped = await wallet.zap(-1, time - 3);
// time will be 4 (4 txs), if we want to zap oldest 2 txs,
// zap will call nowFn once more, so time will be 5.
// First 2 txs have time 0 and 1. Zap accepts second argument
// age, which is time - age. So, we need to pass time - 1.
// e.g. time - 1 = 4. Internal timer will be 5 (nowFn increment).
// Age becomes: 5 - 4 = 1. So, zap will zap all txs with age 1
// - so first 2 txs.
const zapped = await wallet.zap(-1, time - 1);
assert.strictEqual(zapped.length, 2);

const txsAfterZap = await wallet.listUnconfirmed(-1, {
Expand All @@ -4008,8 +4013,8 @@ describe('Wallet', function() {
}
}

// two transactions from default
const zapped = await wallet.zap(DEFAULT, time - 5);
// two transactions from default (calculation above.)
const zapped = await wallet.zap(DEFAULT, time - 3);
assert.strictEqual(zapped.length, 2);

const txsAfterZap = await wallet.listUnconfirmed(DEFAULT, {
Expand Down
Loading