You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Background
In many cases certain portion of data should be reacquired during incremental loading. Ie we want to always capture last 7 days of data when getting daily analytics report. Or we want to refresh slack message replies and we have a moving window of 7 days for that.
Technically we would always pass start_date +/- the lag to the function accepting incremental
Requirements
add new optional filed to Incremental class that will hold the lag.
lag should be expressed as float and will be interpreted depending on the type of the cursor. for datetimes it is a lag value in seconds, for any other type use + / - operator depending on the last_value_func
we support only min and max for last_value_func. for customs we do not have "+" operator defined.
The text was updated successfully, but these errors were encountered:
Hi, I was just looking for a way to add lag to the incremental process (for example, some distributed sources can have unsync datetimes where an "older" record is added seconds before the last cursor)
Background
In many cases certain portion of data should be reacquired during incremental loading. Ie we want to always capture last 7 days of data when getting daily analytics report. Or we want to refresh slack message replies and we have a moving window of 7 days for that.
Technically we would always pass start_date +/- the lag to the function accepting incremental
Requirements
Incremental
class that will hold the lag.last_value_func
min
andmax
forlast_value_func
. for customs we do not have "+" operator defined.The text was updated successfully, but these errors were encountered: