You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: src/connections/reverse-etl/system.md
+31
Original file line number
Diff line number
Diff line change
@@ -16,6 +16,37 @@ For Segment to compute the data changes within your warehouse, Segment needs to
16
16
> warning ""
17
17
> There may be cost implications to having Segment query your warehouse tables.
18
18
19
+
## Reverse ETL schema
20
+
When using Reverse ETL with Segment, several system tables are created within the `__segment_reverse_etl` schema in your warehouse. These tables are crucial for managing the sync process efficiently and tracking state information. Below are the details of the system tables in this schema:
21
+
22
+
### Records table
23
+
24
+
`records_<subscription_id>` table is located within the ` __segment_reverse_etl` schema.
25
+
26
+
This table contains two key columns:
27
+
28
+
-`record_id`: A unique identifier for each record.
29
+
-`checksum`: A checksum value that is used to detect changes to a record since the last sync.
30
+
The records table helps in determining new and updated rows by comparing the checksum values during each sync. If a record’s checksum changes, it indicates that the record has been modified and should be included in the next sync. This ensures that only the necessary updates are processed, reducing the amount of data transferred.
31
+
32
+
### Checkpoint table
33
+
34
+
The `checkpoints_<subscription_id>` tables are located within the __segment_reverse_etl schema.
35
+
36
+
This table contains the following columns:
37
+
38
+
-`source_id`: Identifies the source from which the data is being synced.
39
+
-`model_id`: Identifies the specific model or query that is used to pull data.
40
+
-`checkpoint`: Stores a timestamp value that represents the last sync point for a particular model.
41
+
42
+
The checkpoints table is used for timestamp-based checkpointing between syncs. This enables Segment to track the last successful sync for each model and avoid duplicating data when syncing, ensuring incremental and efficient data updates.
43
+
44
+
### Important Considerations
45
+
46
+
Do not modify or delete these tables. Altering or deleting the records and checkpoints tables can cause unpredictable behavior in the sync process. These tables are essential for maintaining the integrity of data during Reverse ETL operations.
47
+
State management: The `__segment_reverse_etl` schema and its associated tables (records and checkpoints) manage the state of each sync, ensuring that only necessary data changes are synced and that the sync process can resume where it left off.
48
+
49
+
19
50
## Limits
20
51
To provide consistent performance and reliability at scale, Segment enforces default use and rate limits for Reverse ETL.
0 commit comments