feat: Fallback to include whole table in replication if where clauses unsupported #2367
+165
−9
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #2360
This will make the running Electric fallback to replicating whole tables if it receives any shapes with unsupported where clauses (e.g. enums, varchar with
IN
checks, user-defined data types in general, and who knows what else).There is no "recovery" mechanism to return to row-filtering as the Postgres error does not allow for an easy way to check which where clause caused the issue - once we go to relation-only filtering we stay there, like we would if an active shape had no where clause or if we were in PG14.
Ideally we would detect where clauses that are unsupported at the relation filter processing level, so we can fine tune that, but until then this fallback makes sure that Electric works even if an unsupported where clause is provided.
As discussed in this Discord thread, we could also have a configuration flag and better errors to avoid this sort of radical fallback, but we opted for an "always works" approach here. IIRC some benchmarking had shown that our filtering is fast enough that the PG level filtering might not be as important anyway, although the limiting of transmitted data is definitely nice (despite several issues we have with not being able to limit columns replicated etc).
This is related with #1778 , and I'm also referencing #1831 as we had encountered many limitations to row filtering which has led to this proposed change.
We can definitely improve this by detecting unsupported where clauses, checking filter diffs to know what caused the issue and reverting back after, periodically attempting to revert back to row-filtering, and an array of different approaches, but this allows all where clauses to be accepted and Electric to adjust accordingly.