core/vm, eth: add standard json tracing into filesystem dumps

[holiman]

This PR adds debug methods to dump out traces from blocks and/or individual transactions.

When @karalabe implemented tracing, the intended usecase was to enable long-running tracers that could stream results back via a subscription channel. However, when investigating e..g consensus errors and/or bad blocks, that format is not very good.

• The existing generic tracer framework collects all result data (per block) into a big blob in memory, before streaming. For a large trace, such as the 205K steps in the recent Ropsten-fork, the full output from the tracer was 2.4GB, which can easily crash many nodes.

• The existing callTracer does not confirm to std-json, which is

  • jsonl not json, to allow for streaming output
  • Uses op for string-representation of opcode, instead of op/opName for numeric/string, and other simlar small differences.

This PR enables dumping a tx from a given block:

> debug.standardTraceBlockToFile("0x0bbe9f1484668a2bf159c63f0cf556ed8c8282f99e3ffdb03ad2175a863bca63", {txHash:"0x4049f61ffbb0747bb88dc1c85dd6686ebf225a3c10c282c45a8e0c644739f7e9", disableMemory:true})
["/tmp/block_0x0bbe9f14-14-0x4049f61f-099048234"]

Or all txs from a block:

> debug.standardTraceBlockToFile("0x0bbe9f1484668a2bf159c63f0cf556ed8c8282f99e3ffdb03ad2175a863bca63", {disableMemory:true})
["/tmp/block_0x0bbe9f14-0-0xb4502ea7-409046657", "/tmp/block_0x0bbe9f14-1-0xe839be8f-954614764", "/tmp/block_0x0bbe9f14-2-0xc6e2052f-542255195", "/tmp/block_0x0bbe9f14-3-0x01b7f3fe-209673214", "/tmp/block_0x0bbe9f14-4-0x0f290422-320999749", "/tmp/block_0x0bbe9f14-5-0x2dc0fb80-844117472", "/tmp/block_0x0bbe9f14-6-0x35542da1-256306111", "/tmp/block_0x0bbe9f14-7-0x3e199a08-086370834", "/tmp/block_0x0bbe9f14-8-0x87778b88-194603593", "/tmp/block_0x0bbe9f14-9-0xbcb081ba-629580052", "/tmp/block_0x0bbe9f14-10-0xc254381a-578605923", "/tmp/block_0x0bbe9f14-11-0xcc434d58-405931366", "/tmp/block_0x0bbe9f14-12-0xce61967d-874423181", "/tmp/block_0x0bbe9f14-13-0x05a20b35-267153288", "/tmp/block_0x0bbe9f14-14-0x4049f61f-606653767", "/tmp/block_0x0bbe9f14-15-0x46d473d2-614457338", "/tmp/block_0x0bbe9f14-16-0x35cf5500-411906321", "/tmp/block_0x0bbe9f14-17-0x79222961-278569788", "/tmp/block_0x0bbe9f14-18-0xad84e7b1-095032683", "/tmp/block_0x0bbe9f14-19-0x4bd48260-019097038", "/tmp/block_0x0bbe9f14-20-0x1517411d-292624085", "/tmp/block_0x0bbe9f14-21-0x6857e350-971385904", "/tmp/block_0x0bbe9f14-22-0xbe3ae2ca-236639695"]

Files are created in a temp-location, with the naming standard block_<blockhash:4>-<txindex>-<txhash:4>-<random suffix>. Each opcode immediately streams to file, with no in-geth buffering aside from whatever buffering the os normally does.

It also adds some more info when regenerating historical state, namely, the reexec-number if required historical state is not avaiable is encountered, so a user can experiment with increasing that setting. It also prints out the remaining block until it reaches target:

INFO [10-15|13:48:25.263] Regenerating historical state            block=2385959 target=2386012 remaining=53   elapsed=3m30.990537767s
INFO [10-15|13:48:33.342] Regenerating historical state            block=2386012 target=2386012 remaining=0    elapsed=3m39.070073163s
INFO [10-15|13:48:33.343] Historical state regenerated             block=2386012 elapsed=3m39.070454362s nodes=10.03mB preimages=652.08kB
INFO [10-15|13:48:33.352] Wrote trace                              file=/tmp/block_0x14490c57-0-0xfbbd6d91-715824834
INFO [10-15|13:48:33.352] Wrote trace                              file=/tmp/block_0x14490c57-1-0x71076194-187462969
INFO [10-15|13:48:34.421] Wrote trace file=/tmp/block_0x14490c57-2-0x3f4263fe-056924484

There are two new methods added:

debug.standardTraceBlockToFile(blockhash, options)
debug.standardTraceBadBlockToFile(blockhash, options)

Where options is

type StdTraceConfig struct {
	*vm.LogConfig
	Reexec *uint64
	TxHash *common.Hash
}

One more thing: it also changes the existing debug_traceBadBlock to take a block hash instead of index-of-badblocks. A hash is less ambiguous and more natural choice.

[holiman]

Travis only has a spurious swarm failure

[holiman]

Rebased to fix conflicts. PTAL

[holiman]

LGTM

[karalabe]

LGTM too