change "manual close" to use VIRTUAL_TIME

[MonsieurNicolas]

manual close mode is used when testing, and can be used to replay transactions or debug issues;

there are a too many "gotcha's" that come with the way manual close works right now (for example, if the caller attempts to close too fast, some failures occur, or certain timeouts trigger).

Best option seems to be:

• to use VIRTUAL_TIME
• move the clock forward by 5 seconds when closing
• allow to set the clock to an arbitrary date (optional parameter to manualclose, probably in the same unit than the close time used in the ledger header).
  • This allows to replay transaction sets exactly the same way than found on a live network for example.

[rokopt]

A couple of notes:

• We should only use VIRTUAL_TIME when RUN_STANDALONE is specified in addition to MANUAL_CLOSE. In that case, we can allow manualclose with parameters to override the defaults (such as the next ledger sequence number and close time). There are networked supercluster tests that rely on parameter-less manualclose.

• When we do use VIRTUAL_TIME as a result of both MANUAL_CLOSE and RUN_STANDALONE being set, we must also validate that automatic maintenance is off (AUTOMATIC_MAINTENANCE_COUNT/AUTOMATIC_MAINTENANCE_PERIOD are 0). Otherwise, we end up trimming history in a tight loop around ExternalQueue::deleteOldEntries().

[rokopt]

Here's an example config file that I've been using to test VIRTUAL_TIME induced by MANUAL_CLOSE and RUN_STANDALONE (see PR #2696 ):

NODE_SEED="SA7FGJMMUIHNE3ZPI2UO5I632A7O5FBAZTXFAIEVFA4DSSGLHXACLAIT"
NODE_HOME_DOMAIN="developer-testing.stellar.org"
NODE_IS_VALIDATOR=true
NETWORK_PASSPHRASE="Standalone 'network'"
DATABASE="sqlite3:///tmp/developer-testing.db"

MANUAL_CLOSE=true
RUN_STANDALONE=true
AUTOMATIC_MAINTENANCE_PERIOD=0
AUTOMATIC_MAINTENANCE_COUNT=0
UNSAFE_QUORUM=true

[[HOME_DOMAINS]]
HOME_DOMAIN="developer-testing.stellar.org"
QUALITY="MEDIUM"

# History archives

# Stellar.org history store
[HISTORY.sdf1]
get="curl -sf http://history.stellar.org/prd/core-live/core_live_001/{0} -o {1}"

[HISTORY.sdf2]
get="curl -sf http://history.stellar.org/prd/core-live/core_live_002/{0} -o {1}"

[HISTORY.sdf3]
get="curl -sf http://history.stellar.org/prd/core-live/core_live_003/{0} -o {1}"

[rokopt]

See also commit d04230d1384901759d88b5878f0465334fe16dc9 for another source of continual cranking and 100% CPU use when in virtual time induced by MANUAL_CLOSE and RUN_STANDALONE: the Herder's re-broadcast timer.

[rokopt]

I believe that this issue could now be closed through PR #2696 .