Rosetta Constructor DSL Support

This PR adds support for rosetta-specifications@v1.4.5, rosetta-sdk-go@v0.5.1, and adds support the new Rosetta Constructor DSL (Ethereum example).

Configuration Changes

// ReconciliationCoverage is used to add conditions
// to reconciliation coverage for exiting `check:data`.
// All provided conditions must be satisfied before
// the end condition is considered satisfied.
//
// If FromTip, Tip, Height, and AccountCount are not provided,
// `check:data` will halt as soon as coverage surpasses
// Coverage.
type ReconciliationCoverage struct {
	// Coverage is some value [0.0, 1.0] that represents
	// the % of accounts reconciled.
	Coverage float64 `json:"coverage"`

	// FromTip is a boolean indicating if reconciliation coverage
	// should only be measured from tip (i.e. reconciliations
	// performed at or after tip was reached).
	FromTip bool `json:"from_tip,omitempty"`

	// Tip is a boolean indicating that tip must be reached
	// before reconciliation coverage is considered valid.
	Tip bool `json:"tip,omitempty"`

	// Index is an int64 indicating the height that must be
	// reached before reconciliation coverage is considered valid.
	Index *int64 `json:"height,omitempty"`

	// AccountCount is an int64 indicating the number of accounts
	// that must be observed before reconciliation coverage is considered
	// valid.
	AccountCount *int64 `json:"account_count,omitempty"`
}

Changelog

• [configuration] Support advanced reconciliation end condition #154
• rosetta-sdk-go@v0.5.0 Update #152
• Replace broken link for commit guidelines #151
• Fix printing of reconciliation coverage percentage #150

Execution Error Logging

This release greatly improves logging when check:construction fails during Workflow execution!

Changelog

• Prepare for v0.5.7 release #149
• Add ability to generate cpu and mem profiling #145
• [Cmd] Fix Error on Exit #148
• logger: Fix possible panic after completion #146
• [Processor] Verbose Construction API Logging #144

Status Endpoint for check:*

This release enables rosetta-cli to serve status queries over an HTTP endpoint (#143 has details on how this is done). This is useful for teams building dashboards to monitor testing runs of either check:data or check:construction.

Changelog

• Expose a Status Endpoint for check:* #143

Progress Logging

This release logs sync progress and in-use memory stats while running check:data. You can see an example of what this looks like in the image below:

Changelog

• [check:construction] Wait to start check:construction until reaching tip #142
• Log Blocks Synced Per Second #141

[BUG] Account Import

This release fixes a bug related to importing edwards25519 accounts. You can read more about the fix in this PR. Props to @metajack for uncovering this!

Changelog

• [BUG] Account Import Issue #140

Automatic Storage Pruning

This PR adds support for automatic storage pruning and for making HTTP requests during workflow execution (for use in algorithmic fauceting).

Configuration File Changes

New Fields

• pruning_disabled
  -> If set to true, storage pruning will be disabled.

Changelog

• Add support for storage pruning #139

Automated Fund Return Support

This release adds support for automatic fund return when reaching an end condition. You can read more about this feature in #119!

Configuration File Changes

New Fields

• max_retries (default = 5)
  -> it is not possible to set the number of retries instead of setting retry_elapsed_time

Changelog

• Support Automated Fund Return #135
• [Configuration] Support Max Retries #134

Rosetta-Specifications V1.4.4 Support

This release includes rosetta-sdk-go updates that enable rosetta-specifications@v1.4.4 support and "dry runs" during transaction construction (to get an estimate of the "suggested fee").

Configuration File Changes

New Fields

• max_online_connections / max_offline_connections
  -> it is now possible to limit the total number of connections opened
• max_sync_concurrency
  -> sync concurrency is now handled automatically (based on block size and available connections)
• historical_balance_enabled
  -> If left nil, we now auto-populate this using the value in NetworkOptionsResponse.Allow.HistoricalBalanceLookup

Removed Fields

• disable_memory_limit
  -> we no longer support running without a memory limit as new db optimizations have significantly improved performance
• historical_balance_disabled
  -> see the new historical_balance_enabled field
• transaction_concurrency
  -> this is now set automatically
• sync_concurrency
  -> this is now set dynamically, see max_sync_concurrency to set a limit

Changelog

• Update to v0.4.2 of rosetta-sdk-go #133
• Infer Historical Balance Lookup #131
• improved error handling and test coverage #130
• [rosetta-sdk-go] Update to v0.4.1 #129
• Update readme to remove install from source #126
• Binary Install Script #125

Improved Construction API Testing

This release introduces the new and improved Construction API testing suite. You can read more about it here:

• #112
• https://community.rosetta-api.org/t/feedback-request-automated-construction-api-testing-improvements/146

You can find examples of how to use this new testing suite here:

• Bitcoin
• Ethereum

Changelog

• [BUG] Support syncing on non-default network #123
• Construction End Conditions #122
• Integrate Construction API 2.0 #120
• Generate binaries #121
• ZSTD Train Command #118
• Prefunded Accounts #117
• [BUG] Ensure block synced before starting Construction API Testing Loop #114
• [BUG] Patch Statefulsyncer #113
• Assert statefulsyncer.Logger interface #111
• Move statefulsyncer, storage, and utils to rosetta-sdk-go #110
• Support for importing prefunded accounts #105
• Use types.Coin in CoinStorage #109
• [Enhancement] Optimize Coin Storage #108
• Don't Omit Anything in CheckDataResults #107
• Allow Multiple End Conditions #106
• End Condition Logging + File Output #103
• Reconciler End Condition #102
• Prepare for reconciler end condition #101

Sync Optimization and Omitted Block Support

This release includes major syncing optimizations (30% reduction in sync time + 23% less memory usage) and adds support for syncing blockchains that omit blocks.

Warning

This release upgrades the rosetta-cli database from dgraph-io/badger@v1.6.1 to dgraph-io/badger@v2.0.3. Unfortunately, badger v1.6.x and v2.x are data incompatible and you will need to delete any previously used data directory when upgrading to rosetta-cli@v0.4.1.

Configuration Changes

• block_concurrency => sync_concurrency

Changelog

• Support Omitted Blocks #98
• Export Internal Packages + Miscellaneous Bug Fixes #93
• fix a typo in configuration comment #96
• fix check:data with bootstrap balance resume panic #95
• Enhanced end condition #89
• 30% Reduction in Sync Time #88