chainweb-data stores and serves data from the Kadena Public Blockchain in a
form optimized for lookups and analysis by humans. With this reindexed data we
can easily determine mining statistics and confirm transaction contents.
chainweb-data requires Postgres.
chainweb-data can be built with either cabal, stack, or
Nix. Building with nix is the most
predictable because it is a single build command once you’ve installed Nix.
This process will go significantly faster if you set up the Kadena nix cache
as described
here.
Building with cabal or stack will probably require a little more knowledge
of the Haskell ecosystem.
git clone https://github.com/kadena-io/chainweb-data cd chainweb-data nix-build
By default, chainweb-data will attempt to connect to Postgres via the
following values:
| Field | Value |
|---|---|
| Host | localhost |
| Port | 5432 |
| User | postgres |
| Pass | Empty |
| DB Name | postgres |
You can alter these defaults via command line flags, or via a Postgres Connection String.
Assuming you had set up Postgres, done a createdb chainweb-data, and had
configured user permissions for a user named joe, the following would connect
to a local Postgres database at port 5432:
chainweb-data <command> --service-host=<node> --p2p-host=<node> --dbuser=joe --dbname=chainweb-data
chainweb-data <command> --service-host=<node> --p2p-host=<node> --dbstring="host=localhost port=5432..."
chainweb-data syncs its data from a running chainweb-node. The node’s
P2P address is specified with the --p2p-host command. The node’s
Service address is specified with the --service-host command.
If custom ports are used, you can specify them with --service-port and --p2p-port
chainweb-data <command> --service-host=foo.chainweb.com --p2p-host=foo.chainweb.com ...
chainweb-data also needs some special node configuration. The server
command needs headerStream and some of the other stats and information made
available requires allowReadsInLocal. Increasing the throttling settings on
the node also makes the backfill and gaps operations dramatically faster.
chainweb:
allowReadsInLocal: true
headerStream: true
throttling:
global: 1000
You can find an example node config in this repository in [node-config-for-chainweb-data.yaml](node-config-for-chainweb-data.yaml).
When running chainweb-data for the first time you should run chainweb-data
server -m (with the necessary DB and node options of course). This will create
the database and start filling the database with blocks. Wait a couple minutes,
then hit Ctrl-c to stop the server process. Next run chainweb-data fill
--disable-indexes (again with the necessary options). If you want to leave the
server running while you run fill, you should not use --disable-indexes.
This will take longer but will allow the server to stay running. After the
fill operation finishes, you can run server again with the -f option and it
will automatically run fill once a day to populate the DB with missing blocks.
listen fetches live data from a chainweb-node whose headerStream
configuration value is true.
> chainweb-data listen --service-host=foo.chainweb.com --p2p-host=foo.chainweb.com --dbuser=joe --dbname=chainweb-data DB Tables Initialized 28911337084492566901513774
As a new block comes in, its chain number is printed as a single digit.
listen will continue until you stop it.
server is just like listen but also runs an HTTP server that serves a
few endpoints for doing common queries.
/txs/recentgets a list of recent transactions/txs/search?search=foo&limit=20&offset=40searches for transactions containing the stringfoo/txs/tx?requestkey=<request-key>gets the details of a transaction with the given request key/txs/events?search=foo&limit=20&offset=40gets the details of a transaction with the given request key/statsreturns a few stats such as transaction count and coins in circulation/coinsreturns just the coins in circulation
For more detailed information, see the API definition here.
fill fills in missing blocks. This command used to be called gaps but it has
been improved to encompass all block filling operations.
> chainweb-data fill --service-host=foo.chainweb.com --p2p-host=foo.chainweb.com --dbuser=joe --dbname=chainweb-data
Deprecated: The backfill command is deprecated and will be removed in future
releases. Use the fill command instead.
backfill rapidly fills the database downward from the lowest block height it
can find for each chain.
Note: If your database is empty, you must fetch at least one block for each
chain first via listen before doing backfill! If backfill detects any
empty chains, it won’t proceed.
> chainweb-data backfill --service-host=foo.chainweb.com --p2p-host=foo.chainweb.com --dbuser=joe --dbname=chainweb-data DB Tables Initialized Backfilling... [INFO] Processed blocks: 1000. Progress sample: Chain 9, Height 361720 [INFO] Processed blocks: 2000. Progress sample: Chain 4, Height 361670
backfill will stop when it reaches height 0.
Deprecated: The backfill command is deprecated and will be removed in future
releases. Use the fill command instead.
gaps fills in missing blocks that may have been missed during listen or
backfill. Such gaps will naturally occur if you turn listen off or use
single.
> chainweb-data gaps --service-host=foo.chainweb.com --p2p-host=foo.chainweb.com --dbuser=joe --dbname=chainweb-data DB Tables Initialized [INFO] Processed blocks: 1000. Progress sample: Chain 9, Height 361624 [INFO] Processed blocks: 2000. Progress sample: Chain 9, Height 362938 [INFO] Filled in 2113 missing blocks.
single allows you to sync a block at any location in the blockchain.
> chainweb-data single --chain=0 --height=200 --service-host=foo.chainweb.com --p2p-host=foo.chainweb.com --dbuser=joe --dbname=chainweb-data DB Tables Initialized [INFO] Filled in 1 blocks.
Note: Even though you specified a single chain/height pair, you might see it report that it filled in more than one block. This is expected, and will occur when orphans/forks are present at that height.