Description
What was wrong?
Filters suck.
- Sometimes the server loses the
filter_id - You don't really have any clarity to what is actually happening under the hood on the server.
- You can't use filters with infura
- They don't work if you're load balanced across multiple nodes.
How can it be fixed?
I propose we implement a new opt-in middleware to handle log filters locally.
Consider the following generator.
from cytoolz import (
merge,
)
def log_filter(web3, filter_params):
# TODO: handle pending/latest/earliest and cases where these are not included in the filter params.
from_block = filter_params['fromBlock']
to_block = filter_params['toBlock']
# TODO: handle dynamic block ranges to enforce reasonable request sizes.
# TODO: handle things like `latest` and query what the latest block number is from the last block number that we checked.
for block_number in range(from_block, to_block + 1):
params = merge(filter_params, {'fromBlock': block_number, 'toBlock': block_number})
logs = web3.eth.getLogs(params)
yield from logsThis log_filter generator can be used under the hood in place of using the node based filters to handle all filter logic locally, only relying on the node for eth_getLogs which is stateless.
The middleware would then resemble the following.
def filter_middleware(make_request, web3):
filters = {}
filter_id_counter = itertools.count()
def middleware(method, params):
if method == 'eth_newFilter':
filter_id = next(filter_id_counter)
filter = log_filter(web3, params[0])
filters[filter_id] = filter
return {'result': filter_id}
elif method == 'eth_getFilterChanges' or method == 'eth_getFilterLogs':
# do appropriate logic to return logs.
else:
return make_request(method, params)The middleware creates and tracks these generator functions locally, retrieving them when requests are made to retrieve the logs for a given filter and using them to return the appropriate log entries.
Each time a new filter is created, the filter backend will create one of these generators for it and keep track of it internally. Then, any calls to eth_getFilterChanges can just return the next(...) for the appropriate filter generator. We should be able to do something similar with eth_getFilterLogs by re-generating the original filter generator and then returning all of the values up-to-the-current state of the generator.