You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Occasionally huge transactions to public.pg_temp_nnn (where nnn could be any number) tables appear in logical replication data. We don't need these tables, and their size causes our pipeline process to die from OOM.
Is there a way to filter these tables? I've tried using wild cards in the filter-tables option but the wildcards appear to only work as a replacement for the entire schema or table name, not part of it.
SELECT * FROM pg_logical_slot_peek_changes(
'databricks_wal2json', NULL, 100,
'add-tables', 'public.*'
returns all tables in public, while
SELECT * FROM pg_logical_slot_peek_changes(
'databricks_wal2json', NULL, 100,
'add-tables', 'public.s*'
returns nothing even though several tables start with the letter s.
Is there a way to filter records from tables that begin with pg_ or some other recommended approach for skipping these temporary table transactions?
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
Occasionally huge transactions to
public.pg_temp_nnn
(wherennn
could be any number) tables appear in logical replication data. We don't need these tables, and their size causes our pipeline process to die from OOM.Is there a way to filter these tables? I've tried using wild cards in the
filter-tables
option but the wildcards appear to only work as a replacement for the entire schema or table name, not part of it.returns all tables in
public
, whilereturns nothing even though several tables start with the letter s.
Is there a way to filter records from tables that begin with
pg_
or some other recommended approach for skipping these temporary table transactions?Beta Was this translation helpful? Give feedback.
All reactions