How to performance-test a contract? Available tools?

[dckc]

It's required by the checklist:

• Mainnet 2 Deployment Checklist #8280

Perform performance and stress testing on the smart contract deployed on our testnet. Build test tools and scripts to simulate high loads, monitor resource usage, and identify bottlenecks. Document results, including performance metrics, failure points, and recommendations for improvement.

Any suggestions on how to do it? Are there any available tools?

cc @tgrecojs

[dckc]

Benchmarking Contract operations at the SwingSet VM Level

One somewhat mature set of tools for benchmarking smart contract operations runs within a SwingSet VM, but not using the cosmos-sdk consensus layer:

• The Benchmarkerator

There's a somewhat extended write-up:

• Running the benchmark for vault creation

[dckc]

end-to-end testing with transactions and queries

The ultimate test is to run an actual blockchain, submit transactions, and run queries.

For example, we have a shell script that tests minting a KREAd character:

KREAD_ITEM_OFFER=$(mktemp -t kreadItem.XXX)
node ./generate-kread-item-request.mjs > $KREAD_ITEM_OFFER
agops perf satisfaction --from $GOV1ADDR --executeOffer $KREAD_ITEM_OFFER --keyring-backend=test

agd query vstorage data published.wallet.$GOV1ADDR.current  -o json >& gov1.out
name=`jq '.value | fromjson | .values[2] | fromjson | .body[1:] | fromjson | .purses[1].balance.value.payload[0][0].name ' gov1.out`
test_val $name \"ephemeral_Ace\" "found KREAd character"

-- https://github.com/Agoric/agoric-3-proposals/blob/main/proposals/71%3Aupgrade-14/create-kread-item-test.sh

agops perf satisfaction is a test tool that submits a smart-wallet transaction and polls until the offer is satisfied.

Before KREAd went into production, we generated various levels of load using scripts like that and looked at various metrics. I think we ran them not just in an a3p context, but on an actual multi-validator-node test network in a kubernetes cluster, and we looked at the output using tools such as datadog. @toliaqat is there any more detail to share in this context?

[dckc]

end-to-end testing APIs

There are a few other tools for submitting transactions and making queries in the @agoric/synthetic-chain package.

See also:

• testing API to extend ava functional tests to end-to-end context #8963

... including recent multichain-testing work such as...

    const wdUser1 = await provisionSmartWallet(agoricAddr, {
      BLD: 100_000n,
      IST: 100_000n,
    });
    t.log(`provisioning agoric smart wallet for ${agoricAddr}`);

    const doOffer = makeDoOffer(wdUser1);

    const brands = await vstorageClient.queryData(
      'published.agoricNames.brand',
    );

-- https://github.com/Agoric/agoric-sdk/blob/master/multichain-testing/test/send-anywhere.test.ts

[erights]

Attn @gibson042

[dckc]

Here's a goal from internal discussion. I don't expect that this is self-explanatory, but I'll go ahead and share it...

• tool which submits txns at a fixed rate, and records completion time, and displays latency and acceptance rate, and some kind of "how busy is the chain" metric like block times or duration of the cosmic-swingset-begin-block -> cosmic-swingset-end-block-finish elapsed time
• run that at maybe 1txn/min for 10-15min, then increase to 2/min, then 3/min, etc, and plot the curves

we should learn how many computrons are spent to do one of those offers, to guess what our 65Mc limit will do to the scheduling.

Once we deploy to a multi-node net, we should see how it compares against the actual wallclock time needed by those validators.