Implement an optimization for get
and exist
to skip holders by their timestamp
#702
Labels
get
and exist
to skip holders by their timestamp
#702
The current implementation reads data from all holders, which can be slow when there are many records for the desired key. We can implement an optimization that skips holders whose maximum record timestamp in that holder is less than the record timestamp already read. The proper implementation requires tracking max and min record timestamp within holder and tracked by issue #694. But we can implement it faster in simplified and unsafe way.
The problem is that now the holder's
end_timestamp
is calculated asstart_timestamp + self.settings.timestamp_period_as_secs()
. If suddenly the value oftimestamp_period
in the config changes, the calculation will be incorrect.With proposing unsafe optimization, we can try to estimate the maximum period of creation of existed holders, multiply it by some safe coefficient, and also adjust it by the value of the new config parameter.
First, we should introduce new config parameter
skip_holders_by_timestamp_step_when_reading
. By default it should beNone
(means that optimization is turned off), but it can be configured with the human readable time period (e.g.1d
).Procedure of calculating of safe timestamp step:
skip_holders_by_timestamp_step_when_reading
isNone
then returnNone
2.1. Copy
start_timestamp
fromholders
collection to a vector2.2. Order vector by ascending
2.3. Add current timestamp to an end
2.4. Find maximum difference between 2 timestamps in that vector
max(settings.skip_holders_by_timestamp_step_when_reading, max(2 * max_period_from_existed_holders, 5 * settings.timestamp_period)) + 1h
Then within
get
andexist
functions we can implement smart skipping by compare timestamp of already read record andend_timestamp
of current holder with addition of safe timestamp step (https://github.com/qoollo/bob/blob/master/bob-backend/src/pearl/group.rs#L264):This issue should be done after #621 will be merged
The text was updated successfully, but these errors were encountered: