Extending the Toolkit to support shadow fork tool

[Federico2014]

Background

Shadow Fork is a technique that allows developers to create and test new blockchain protocols or modifications without affecting the main chain. It can be seen as a sandbox environment that simulates the current state of the main chain, enabling developers to experiment and validate changes in custom settings. This approach provides a more realistic testing environment than launching a new testnet. As a result, we can obtain performance metrics in real-world conditions without impacting the operation of the standard network.

Shadow forks are important tools for web3 developers. As explained here, Ethereum has conducted the Merge upgrade testing through the shadow fork. Many of the development tools in the Ethereum ecosystem have supported the shadow fork, such as the Hardhat network, and Foundry Anvil. There are also other relevant projects. Shadow can implement advanced on-chain data retrieval through shadow fork. BuildBear has built a professional Dapp development and testing platform with a shadow fork sandbox environment.

However, there are no relevant shadow fork tools in TRON ecosystem, which may negatively influence developers, as shown in the #6126 . So it will be necessary to design the shadow fork tool for the TRON network.

Rationale

Shadow fork tool can bring many potential benefits to protocol and application developers, especially for testing in the simulated production environment. The use cases include:

• Testnet or Mainnet upgrade testing

• Smart contracts testing against the production environment

• Integrating with existing tools, such as Tronbox

• Local development testing with mainnet state

Specification

To implement the shadow fork tool in the TRON network, the base idea is:

• Obtain the current state of the chain, through node syncing or downloading the archived data

• Use the shadow fork tool to modify the SR and other data of the current state to produce blocks locally

• Launch the fullnode against the modified state to execute the development and testing

Test Specification

The test should guarantee the tool can complete the shadow fork successfully.

Scope Of Impact

The feature is to implement the shadow fork within the Toolkit and does not influence the FullNode.

Implementation

As shown in the following figure, we have designed an explicit scheme to implement the state fork tool in the TRON network. The main steps are:

1. Obtain the state data of the main chain. There are three possible ways:

• Download the light node state data, and then sync to the specified block number;

• Download the full node state data, and then sync to the specified block number;

• Sync to the specified block number directly.

We can modify the node.shutdown option to make the fullnode sync to the specified block number.

2. Run the DBFork tool in the Toolkit to modify some data to launch the node and produce blocks locally. The modified data includes:

• erase the historical witnesses and active witnesses

• Write new witnesses to the state and update new active witnesses

• Write new balance for new addresses

• set the new latesteBlockHeaderTimestamp to avoid the delay in producing blocks

• set the new maintenanceTimeInterval and nextMaintenanceTime if needed.

3. Launch the full node against the modified state. To launch the node smoothly, we may need to change some parameters:

needSyncCheck = false
minParticipation = 0
node.p2p.versioon != 11111

To produce the blocks, we also need to configure the private key of the witness and run the fullnode with the --witness parameter, which is similar to the mainnet.

4. If another node wants to join the shadow fork network, it must execute the above steps or copy the state data from the first shadow fork node directly.

Here is a brief introduction to the shadow fork scheme. We are willing to implement this feature and contribute to it.

[GrothenDI]

A question about 'Write new balance for new addresses', Where are the new addresses from during shadow fork?

[Federico2014]

@GrothenDI It is similar to a faucet. The developers can modify the balance for their addresses for testing.

[angrynurd]

part (1)
per my understanding of ethereum's shadow fork testing process, I just compare it with what this issue describes, and I have a few questions which may contribute to the issue. as below:

1.Network Isolation:

Ethereum's shadow fork testnet operates in complete isolation to ensure test activities do not affect the mainnet. Has the TRON proposal considered sufficient isolation measures to prevent test network behavior from leaking to mainnet?

2.Security and Testing Standards:

Ethereum implements strict security testing standards and specifications during shadow fork testing. Does the TRON proposal include clear security testing standards and processes, and how will these standards be enforced?

3.Post-Test Data Management:

After Ethereum shadow fork testing concludes, test network data is typically discarded. Does the TRON proposal provide clear guidance on handling post-test data, and how to ensure test environment rollback and cleanup?

[angrynurd]

part (2)

• Missing handling strategy for new blocks and transactions during sync
• how to make sure the data consistency before and after the shadow fork
• lacks a proper snapshot mechanism for mainnet state(reuse the java--tron dayly snapshot？)

[Sunny6889]

It seems like private network with real transaction data

[Sunny6889]

How about making it easier to add other fullnodes?

[Federico2014]

Hi, @endiaoekoe, thank you for your suggestions. Here are some explains:

1. For the shadow fork network, we can configure the node.p2p.versioon != 11111 to implement the network isolation.
2. The DBfork tool is mainly used for local development testing, which may not need strict security standards. If you have any good suggestions about the security specifications, please tell us.
3. The test data is managed locally by the developers and can be discarded after the testing.
4. After the first node completes the shadow fork, the other can join and sync the blocks from the first node, they will form a local testnet. They won't sync the blocks and transactions from the mainnet.
5. From the step2 in the implementation described above, it can make sure the data consistency.
6. The mainnet state can reuse the java-tron daily snapshot.

[Federico2014]

How about making it easier to add other fullnodes?

Yeah, it is private network with mainnet state data. But what do mean to add other fullnodes? As described above, If other node wants to join the shadow fork network, it must execute the above steps or copy the state data from the first shadow fork node directly. Then they can form a ordinary local testnet, which can sync and produce blocks.

[blockketten]

The current proposal for enabling fork testing with Tron RPC nodes revolves around enabling syncing historical state, which seems to be the missing functionality preventing us from fork testing with Foundry currently. Given that many development teams—including ours—already use frameworks like Hardhat and Foundry for testing and simulation, we’re curious about the decision to build a separate implementation specific to Tron toolkits rather than designing the functionality to integrate with these more widely adopted tools.

Could you share more about the rationale behind this approach and whether there are plans to enable fork test compatibility with Hardhat or Foundry in the future? It seems like such compatibility could significantly lower the barrier to adoption for developers and accelerate broader ecosystem development for Tron.

[Federico2014]

@blockketten Considering the difference between TRON and Ethereum, we believe the best choice to implement the shadow fork is within the Toolkit currently. We will do more investigation later and try to implement the shadow fork function compatible with Hardhat and Foundry.

[317787106]

@Federico2014 After the node's syncing completes at certain height, you should backup the database like this:

java -jar Toolkit.jar db copy output-directory/database backup-output-directory/database

Once the database is written, it cannot be rolled back. And you may need to test multiple times. The backup precedure only cost several seconds.

And you can also stop service of StatisticManager to skip missing check. So you don't need to write latest timestamp.

[Federico2014]

@317787106 Thank you for your suggestions. Backup is quite necessary. For the fullnode data, it may take up too much space. The developers should decide whether to backup the database.
I have checked there is no switch config to stop the service of StatisticManager. Modifying the FullNode code will be inconvenient for practical use.

[317787106]

@317787106 Thank you for your suggestions. Backup is quite necessary. For the fullnode data, it may take up too much space. The developers should decide whether to backup the database. I have checked there is no switch config to stop the service of StatisticManager. Modifying the FullNode code will be inconvenient for practical use.

@Federico2014 This backup is not same as copy and it only takes up very little space. You can try it.

[simbadMarino]

The current proposal for enabling fork testing with Tron RPC nodes revolves around enabling syncing historical state, which seems to be the missing functionality preventing us from fork testing with Foundry currently. Given that many development teams—including ours—already use frameworks like Hardhat and Foundry for testing and simulation, we’re curious about the decision to build a separate implementation specific to Tron toolkits rather than designing the functionality to integrate with these more widely adopted tools.

Could you share more about the rationale behind this approach and whether there are plans to enable fork test compatibility with Hardhat or Foundry in the future? It seems like such compatibility could significantly lower the barrier to adoption for developers and accelerate broader ecosystem development for Tron.

@Federico2014 , I second @blockketten proposal; we have to make this as compatible as possible with other tools. Otherwise, we are limiting TRON's interoperability. Thanks for following up on this :)

[Federico2014]

@simbadMarino Compatibility with existing tools is undoubtedly the best choice, but it is indeed a challenge. The basic tests for Hardhat and Foundry compatibility with the TRON network are still not feasible, not to mention the shadow fork test. Due to the differences in transaction data structure and storage model between the TRON network and Ethereum, achieving complete compatibility would require a groundbreaking upgrade to the TRON network, which may lead to unpredictable consequences. We will conduct a more thorough analysis of the differences later and evaluate the feasibility of implementing compatibility. Thank you so much for your suggestions.