CCIP-2003 Improved fetching Commit Reports from database

[mateusz-sekara]

Motivation

Current implementation fetches all the CommitReports within the permissionlessExecThreshold from database and filter out result set based on the snoozedRoots. It means that when everything is executed, we keep fetching all roots to memory only to discover that we don't need that data. During high traffic spikes, this is an unnecessary waste of resources in the CL node and database as the dataset size grows in time till it reaches permissionlessExec window (8 hours of logs for prod).

Example load run in which you can see how it changes over time during steady traffic of messages

Solution

There are at least two solutions to make it more performant

1. Pass executed roots to the database query to fetch only unexpired roots from the database. Unexpired roots can be determined from the snoozedRoots cache. This is probably the most accurate approach because it always fetches the minimal required set from the database.

You can think of the query like this

select * from evm.logs
where address = :addr
and event_sig = :event_sig
and substring(data, X, Y) not in (:executed_roots_array)
and block_timestamp > :permissionlessThresholdWindow

2. Make permissionlessThreshold filter "sliding". Commit Reports are executed sequentially for most of the cases, so we can keep track of timestamps of the oldest fully executed report. Instead of inventing a new query for LogPoller we can achieve a lot based on the assumption that execution is sequential. Whenever report is markedAsExecuted we keep bumping block_timestamp filter. We already have that information in cache and indirectly rely on that when filtering out snoozedRoots

and block_timestamp > :oldest_executed_timestamp 

Solution 2 is easier to implement, doesn't require any changes to LogPoller (and query perf testing) and can be handled in the plugin. We already cache snoozedRoots so we can reuse that to move filtering to the database by narrowing the block_timestamp search window. Assuming that reports are executed sequentially, it should be as accurate as solution 1.

Load testing solution 2 (the same configuration, but shorter test duration)

[github-actions]

I see you updated files related to core. Please run pnpm changeset in the root directory to add a changeset.

[dimkouv]

I would prefer not to use this lib.
We could have our own simple implementation or use a different type.

[dimkouv]

If we have our own simple implementation you can also have the rootsQueueMu defined there.

[mateusz-sekara]

It was already in the deps so I've decided to reuse it. Do you have any better ideas? I'm not sure about reimplementing, sounds like reinventing the wheel.

[mateusz-sekara]

Alternative implementation would be this https://github.com/elliotchance/orderedmap (more GH stars)