The library allows an application receive real-time notifications or historical data from an EOSIO blockchain, such as Telos, EOS, WAX, Lynxchain, Europechain, and many others.
This development is sponsored by Telos blockchain community as part of Worker Proposal #95.
Chronicle is a
software package for receiving and decoding the data flow that is
exported by state_history_plugin of nodeos, the blockchain node
daemon developed by Block One.
State history contains all data about transaction traces and rows added or deleted in smart contract memory. It exports this data in binary form, and the binary flow needs to be decoded using the contract ABI. That's what Chronicle is doing, and it exports the data as JSON ojects.
The chronicle-consumer module lets you build a consumer process for
Chronicle export and react on blockchain updates as your application
needs.
"tests" folder in chronicle-consumer repository and the examples repository provide practical examples on using the library.
npm install chronicle-consumer
Chronicle sets a default number of unacknowledged blocks to 1000,
which due to asynchronous manner of nodejs will generate a large queue
of pending messages within the nodejs process. It is recommended to
use a lower value in Chronicle configuration, such as
exp-ws-max-unack = 200.
The module defines an event emitter class, and your application subscribes to events of interest.
The consumer process opens a Websocket server on a specified port, and this server accepts only one connection from Chronicle. If you need to process multiple streams of Chronicle data, you need to create multiple server instances using different TCP port numbers.
The module allows one of two modes of operation: synchronous and asynchronous. Default mode is synchronous, so that events are emitted synchronously and are not expected to fire any asynchronous tasks.
In asynchronous mode, event handlers are expected to return promise
objects which would eventually resolve. The handler of ackBlock
event should return a promise that resolves when all asynchronous
tasks resolve. See tests/async_slow.js for a reference.
If asynchronous processing is potentially taking a significant time,
such as writing to a slow database, the nodejs process may run out of
memory due to too many pending requests. In this case, it is
recommended to specify a higher memory limits in nodejs process, as
follows: node --max-old-space-size=4096 app.js.
Synchronous mode:
const ConsumerServer = require('chronicle-consumer');
const server = new ConsumerServer({port: 8899});
server.on('fork', function(data) {
let block_num = data['block_num'];
console.log('fork: ' + block_num);
});
server.on('tx', function(data) {
let tx_printed = false;
let trace = data.trace;
if(trace.status == 'executed') {
for(let i=0; i< trace.action_traces.length; i++) {
let atrace = trace.action_traces[i];
if(atrace.receipt.receiver == atrace.act.account) {
if(atrace.act.name == 'transfer') {
if(!tx_printed) {
console.log('tx: ' + trace.id);
tx_printed = true;
}
let d = atrace.act.data;
console.log(' ' + atrace.act.account + ' ' +
d.from + '->' +
d.to + ': ' +
d.quantity);
}
}
}
}
});
server.start();
Asynchronous mode, with random timers:
const ConsumerServer = require('chronicle-consumer');
function _delay() {return Math.floor(Math.random()*10000);}
const setTimeoutPromise = util.promisify(setTimeout);
var pendingTasks = new Array();
const server = new ConsumerServer({port: 8899, async: true});
server.on('fork', function(data) {
let block_num = data['block_num'];
console.log('fork: ' + block_num);
return Promise.all(pendingTasks).then(() => {
pendingTasks = new Array();
console.log('fork: ' + block_num + ' all pending tasks finished');
});
});
server.on('tx', function(data) {
pendingTasks.push(setTimeoutPromise(_delay(), data).then((data) => {
let trace = data.trace;
if(trace.status == 'executed') {
let msg = 'tx: ' + trace.id + ' ';
for(let i=0; i< trace.action_traces.length; i++) {
let atrace = trace.action_traces[i];
if(atrace.receipt.receiver == atrace.act.account) {
msg += atrace.act.name + ' ';
}
}
console.log(msg);
}
}));
});
server.on('blockCompleted', function(data) {
pendingTasks.push(setTimeoutPromise(_delay(), data).then((data) => {
console.log('block completed: ' + data['block_num']);
}));
});
server.on('ackBlock', function(bnum) {
console.log('ack: ' + bnum);
return Promise.all(pendingTasks).then(() => {
pendingTasks = new Array();
console.log('ack: ' + bnum + ' all pending tasks finished');
});
});
The constructor takes an object representing options for the
server. Only the port option is mandatory, indicating the TCP port
tpo listen to.
host defines the IP address to bind the server to. Default is
0.0.0.0 (meaning Chronicle can connect from anywhere). It is
recommended to secure this connection, and use 127.0.0.1 if
applicable.
ackEvery defines how frequently the process sends acknowledgements
to Chronicle, in blocks. All blocks that have not been acknowledged
are discarded by Chronicle on next restart. By default, Chronicle
pauses sending the data as soon as 1000 blocks have not been
acknowledged. It's not recommended to set ackEvery too low, as it
adds overhead in processing. Values of 10 to 100 are
reasonable. Default value is 100. It is also recommended to lower the
default setting in Chronicle, such as exp-ws-max-unack = 200.
interactive, if set to true, turns the consumer in nteractive
mode. Correspondingly, Chronicle process should also be started in
interactive mode. In this mode, instead of a constant stream of
blo0ckchain data, the process needs to request a block range from
Chronicle, by using async requestBlocks(start, end) method.
async, if set to true, the emitter works in asynchronous mode
(using Emittery module).
The constructor does not start the websocket server. Once you
subscribe to events using on method, you need to call the start()
method.
The start() method creates a websocket server that should listen on
the specified TCP port.
This method closes the websocket server.
The method is only applicable to interactive mode, and it requests a range of blocks from Chronicle. The process does not need to wait for full delivery of block data, and multiple ranges can be requested in a row.
In synchronous mode, all events are emitted sequentially, in order to guarantee the exactly the same order of execution as transactions are executed on the blockchain.
In asynchronous mode, the order of execution cannot be guaranteed, so
the program should take into account that transactions and table
deltas will be mixed up between the blocks and within each block. Also
blockCompleted event, although emitted, does not have any sense in
asynchronous mode.
It is important to understand the workflow: the consumer process acknowledges block numbers to Chronicle, and it must acknowledge only the blocks that have been properly processed. If the data is written to a database, all data needs to be committed prior to sending the acknowledgement.
It is also important to understand forks in EOSIO blockchains: due to geographical distance and network conditions, the handover between block producers may result in one or several blocks re-produced by the next BP in the schedule. This creates a microfork: the block that was previously signed is no longer valid, and a new block with the same number is generated. Once 2/3+1 producers have validated a block, it becomes irreversible and final.
Also when Chronicle process starts, it emits an explicit FORK event indicating that it may have erased blocks that were not acknowledged, and a new block data is coming.
Event data: an opject with the following fields:
-
block_num: block number that is invalidated, as well as all blocks past this one. -
depth: informative field indicating the number of blocks being invalidated. Do not rely on this number. -
fork_reason: informative field indicating why the event was emitted:network(fork occurred in the EOSIO network),restart(explicit fork on Chronicle restart),resync(full resync from the genesis). -
last_irreversible: last irreversible block number.
Upon receiving a fork event, the application must erase all
previously received data for the fork block and all blocks past
it. Subsequent events will have the new data for the block number that
was indicated in the fork event.
In asynchronous mode, the event handler must return a promise that resolves when the data cleanup finishes.
It is recommended to disable block events by using skip-block-events = true in Chronicle configuration. The event data is a full JSON
representation of a raw blockchain block. This data is bulky and
CPU-intensive, and rarely used.
The event represents a single transaction with a full trace of its all inline actions.
Event data: an opject with the following fields:
-
block_num: block number. -
block_timestamp: block timestamp in UTC time zone in EOSIO timestamp notation. -
trace: full transaction trace, similar to that returned by Hyperion or legacy history plugin. It includes all inline actions and receipts. Important members are:-
idcontaining the transaction identifier, -
statuswhich is set toexecutedif a transaction was successful, -
action_traceswhich is an array of action traces,
-
It is practical to ignore notifications caused by
require_recipient() call in the smart contracts. In order to skip
them, you only process action traces where recipient is equal to the
contract account:
if(atrace.receipt.receiver == atrace.act.account) {
....
}
The event indicates that a smart contract has updated its ABI, with the full copy of ABI in the event data.
The event indicates that a smart contract has removed ots ABI.
The event indicates that there was an error processing the ABI in Chronicle. It typically indicates that invalid data types are used in it.
The event indicates that a table row is added, modified or deleted in a smart contract within a specified block. Unfortunately there is no way to associate it with a transaction ID.
The event data is an object as follows:
-
block_num: block number. -
block_timestamp: block timestamp in UTC time zone in EOSIO timestamp notation. -
added: boolean. If set tofalse, the row has been deleted. Otherwise it was inserted or updated. -
kvo: key-value object with the following fields:-
code: contract account, -
scope: table scope in name representation, -
table: table name, -
primary_key: 64-bit integer primary key, identfying the row, -
payer: account name charged for the row memory, -
value: an object representing the row structure. If ABI is invalid or unavailable, value is presented as a hexademical string with raw byte contents.
-
The event is reporting internal errors if they occur during the processing of state history.
The event indicates that Chronicle has paused receiving the data from state history plugin because there were no acknowledgements for blocks, exceeding the maximum number of unacknowledged blocks.
The application normally does not need to subscribe to this event.
In synchronous mode, this is the last in the sequence of events that are emitted for a specific block number. It is guaranteed that there will not be new events for this block, unless there is a fork.
In asynchronous mode this event does not make any sense, and the application is not expected to subscribe to it.
Data fields:
-
block_num: block number. -
block_timestamp: block timestamp in UTC time zone in EOSIO timestamp notation. -
block_id: 32-byte block identifyer (SHA256 of block contents). -
last_irreversible: last irreversible block number.
These events indicate updates in account information that is not part of contract data.
permission reports an update in public key or permission definition
for an account.
permissionLink indicates that a particular permission is allowed to
execute an action on a smart contract.
accMetadata contains code_hash field which indicates that either a
new smart contract code is uploaded, or the code was deleted.
The event data is the block number that is about to be acknowledged back to Chronicle.
In synchronous mode, the application should commit all written data to nonvolatile storage (or commit data to an SQL server and finish a transaction).
In asynchronous mode, the handler should return a promise that resolves only when all the previously received data is processed and committed.
The events indicate that a Chronicle process has opened or closed a websocket connection to the consumer process. The event data is an object with fields as follows:
remoteAddress: the IPv4 or IPv6 address of the Chronicle process;remoteFamily:IPv4orIPv6;remotePort: TCP port number of remote side.
Building a new module from sources on Ubuntu
apt install -y curl
curl -fsSL https://deb.nodesource.com/setup_18.x | sudo -E bash -
sudo apt-get install -y nodejs
sudo npm install -g typescript
curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -
echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list
sudo apt update && sudo apt install -y --no-install-recommends yarn
git clone https://github.com/EOSChronicleProject/chronicle-consumer-npm.git /opt/src/chronicle-consumer-npm
cd /opt/src/chronicle-consumer-npm
yarn install
yarn run build
Copyright 2020 cc32d9@gmail.com
The MIT License (MIT)
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.