When running the chain-index and PAB in containers PAB eventually starts returning empty lists for UTXOs

[eselkin]

Summary

Background: We are running the PAB with plutus-apps tagged at commit: 6e3f6a59d64f6d4cd9d38bf263972adaf4f7b244

When we start the PAB after the sync of chain-index, we attempt to make several transactions from contract endpoints in the PAB. At the start all transactions seem to work. Payments get registered on the node and in the wallet. However after some time the PAB starts getting incorrect information like utxos being an empty list (when the previous transactions to pay succeeded (checked on testnet explorer, with wallet server, and with cardano-cli against the node we are running).

The last message in the chain index for a recent run is:

chain-index-1  | [chain-index:Info:41] [2022-05-03 16:45:17.50 UTC] Syncing (100.00%). Processed 253 blocks, 0 rollbacks in the last 5s. Current tip is Point(Slot 57224786, BlockId 7961364aba8b8c51a31d4b08d76a9b9c3e362279cd2fd88d69bd6a94e0012532)

Generally for the utxos to appear we need to stop the chain index and PAB, then restart the chain-index and the PAB. This is not a deployable solution. We are testing now using the node and wallet in containers and the chain index running in a normal process as well as the PAB to see if there are similar results.

supplements:
Chain index config
config.json

PAB config
pab-config.yml
Note urls for connection to chain-index are using the local docker network we create

Steps to reproduce the behavior

0. Start node with docker compose
1. Build containers for chain-index and PAB from the root image agocorona/ubuntu-nix as in the plutus-apps repo's Dockerfile.
2. Derive the nix-shell from plutus-apps at commit 6e3f6a59d64f6d4cd9d38bf263972adaf4f7b244 in the container build steps.
3. Use nix shell from derivation above to build our PAB instance which has plutus-apps tagged at this version.
4. Start chain index from docker compose
5. Wait for sync
6. Start PAB from docker compose (note above URLs in config - wallet-server and chain-index hostnames are on local docker network)
   Note: We have been using a mounted shared volume for PAB, Node, Wallet Server, and Chain index which contains configs, node.socket. All permissions on the shared volume are for the user running the processes in each container.

Actual Result

Errored results after some time on PAB (see summary)
Empty UTXO list when queried after some time

Expected Result

UTXOs with values and datum attached as in cardano cli

Describe the approach you would take to fix this

Uncertain

System info

OS: Ubuntu
Version: 20.04

OS: Debian
Version: 10 and 11
Plutus-apps commit: 6e3f6a5

[eselkin]

When we leave the PAB running, seems like it runs out of memory (probably what was happening in the containers causing restarts).

We tested using:
{ "tag" : "Point"
, "pointBlockId" : "9b60243d55dc836305e93728816d50c4654ecf8c612d2ff2f46cca68a0688fc4"
, "pointSlot" : { "getSlot" : 57612105 }
}
And the PAB instance seems to use 0.01-0.05% more available memory every couple of minutes. When we used a pointSlot that was about 2 days old it used up all available memory. In a cloud instance with 64GB of memory it could not initialize and pegged the system's memory. Swap could alleviate this, but that's an untenable solution.

[ghost]

Thanks for the issue!

Yes, indeed, PAB uses a lot of memory to sync the chain. It's being investigated under #420.