Loadgen stall

[mhofman]

Describe the bug

Under fairly heavy loadgen load, the chain or solo seem to sometimes no longer make progress loadgen tasks, even after a restart of the solo and loadgen client.

This has so far only been reproduced on a fairly old version of the sdk (6dc7152 / agoricdev-17), and only on the benchmark machine (my personal laptop never encountered such an issue).

To Reproduce

Steps to reproduce the behavior:

1. Connect to benchmark machine
2. docker run -e SDK_REVISION=6dc7152 loadgen-runner --stages=26 --stage.duration=60 --stage.loadgen.vault.interval=12 --stage.loadgen.amm.interval=12 --stage.loadgen.amm.wait=6 --stage.loadgen.vault.limit=10 --stage.loadgen.amm.limit=10
3. Observe chain getting into a state where no loadgen tasks occur (empty blocks)

Expected behavior

Forward progress on pending tasks

Platform Environment

• Loadgen runner Docker image (roughly at Timeline of loadgen stats testnet-load-generator#38)
• is there anything special/unusual about your platform? No
• what version of the Agoric-SDK are you using? 6dc7152

Additional context

I did not see any reported errors, but it's possible some rejection happened and wasn't handled. I haven't ruled out a loadgen solo agent bug not catching an error, but a cursory look indicate that all results are awaited or returned. A restart of the solo and loadgen client does not unblock anything, with the loadgen deploy script never reaching the ready state.

Screenshots

The full logs with chain storage captured by the runner can be found on the benchmark machine at /mnt/volume_sfo3_03/manual-phase45-fix/manual-6dc7152-with-storage

[Tartuffo]

@mhofman has not seen this occur again after a medium number of runs. will close if we get more confidence that it is not an issue.

[mhofman]

Closing, we can reopen if needed