Support to customize incremental compare operators for sql_table

[marius-sb1]

Feature description

The sql_table source/resource is hard coded to use the operator ge (>=) when comparing data with incremental state (or le (>=) depending on the incremental function/sort order), we would like the option to configure it to use gt (>) / lt in stead.

Are you a dlt user?

Yes, I run dlt in production.

Use case

We have tables which are loaded seldom, but with large volumes, all with the same incremental key value. We would like to run dlt on top of these often, but not use resource re-loading the same data every time no new data has been added.

Proposed solution

An option to the sql_table source/resource whether to use classic (e.g. operator.ge) or aggressive (operator.gt) incremental filtering on the extracted sql-table.

Related issues

No response

[rudolfix]

@marius-sb1 we should definitely support it. right now you could use query_adapter_callback to rewrite the query before it is called. this requires knowledge of SQLAlchemy expressions so it is not optimal.

@steinitzu could you take a look at this? my take would be to use gt and lt when deduplication is disabled (ie. via setting primary_key to (). we can also add explicit option to the signature.