Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimized address service mempool index size #352

Merged
merged 9 commits into from
Nov 3, 2015
Merged

Conversation

braydonf
Copy link
Contributor

Closes #324

New features:

  • significant memory footprint improvements
  • address service now keeps mempool index with levelup that can be configured to use memdown or leveldown (default)
  • the mempoolSpentIndex has been kept to be memory only and the key will be 37 bytes
  • mempool address index size should be roughly half the size (binary encoding instead of utf-8)
  • mempool indexes are more accurately kept in sync by using tx and txleave events from bitcoind and will nolonger make an expensive query for all transactions in the mempool

Includes breaking API changes:

  • address service resetMempoolIndex method have been removed
  • db service runAllMempoolIndexes has been removed

Braydon Fuller added 4 commits October 30, 2015 18:36
- Adds default to store a large portion of the mempool index in leveldb
- Includes an option to use memdown to have the mempool index in-memory
Replaced with using `tx` and `txleave` to manage the state of the mempool indexes.
@braydonf braydonf changed the title WIP: Optimize memory address index usage Optimized address service mempool index size Nov 2, 2015
@kleetus
Copy link
Contributor

kleetus commented Nov 3, 2015

LGTM

if (this.node.network === Networks.livenet) {
this.mempoolIndexPath = this.node.datadir + '/bitcore-addressmempool.db';
} else if (this.node.network === Networks.testnet) {
this.mempoolIndexPath = this.node.datadir + '/testnet3/bitcore-addressmempool.db';
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would be nice not to have these prefixes hardcoded

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Which prefixes?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, instead of having to write this same code block multiple times, it could be DRY'ed up. Not a show stopper, we can clean it up later.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The prefixes I was talking about were {livenet: '', testnet: 'testnet3', regtest: 'regtest'}.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since there is some shared code, it could be useful to have this.node.networks.getBaseDataDir() or at least a utility function on node, we can spawn an issue and add it in later, since this should also touch the db.js code.

@pnagurny
Copy link
Contributor

pnagurny commented Nov 3, 2015

LGTM

pnagurny added a commit that referenced this pull request Nov 3, 2015
Optimized address service mempool index size
@pnagurny pnagurny merged commit 02620a5 into bitpay:master Nov 3, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants