Migration of a connection's Jobs to Timeline events

[nataliekwong]

Connection Timeline

As of v1.1.0, we have released the Connection Timeline feature.

This is a big change to our UI - allowing you to review not only your sync history, but also related connection activities to give you additional transparency. The "Timeline" replaces "Job History" for all connections.

Backfilling

If you are upgrading to v1.1.0 and beyond, you may want to backfill your existing job history into the new Timeline. This guide explains how to add those jobs to the Timeline.

Backfilling can be done by calling this migration endpoint: /v1/connections/events/backfill
This is a permission controlled API which allows users to configure the request with the time window of jobs they want to backfill.

Example request:

{
  "connectionId": "123e4567-e89b-12d3-a456-426614174000",
  "createdAtStart": "2024-09-01T00:00:00Z",
  "createdAtEnd": "2024-09-30T23:59:59Z"
}

Users can customize the time frame parameters (start time and end time) of jobs to be backfilled. Please note that as of now, we do not support deduplication in the time event table. If you are running the migration multiple times, or running the migration with a time window has any overlap with the existing events, duplicate events will be backfilled.

Only admin users have the permission to fulfill this request.

Cloud users

Cloud users have already been migrated to the Timeline and backfilled with data from 2024-06-20 and on. To see additional jobs created before this date, reach out to Airbyte Support for assistance in additional backfilling.

Supported Events

We support backfill jobs including sync, refresh, and clear (formerly reset). Using the final job status (succeeded, cancelled, failed, incomplete), we will log the following events on the Timeline:

• Sync Cancelled
• Sync Succeeded
• Sync Failed
• Sync Incomplete
• Refresh Succeeded
• Refresh Failed
• Refresh Incomplete
• Refresh Cancelled
• Clear Succeded
• Clear Failed
• Clear Incomplete
• Clear Cancelled

Notes

We save manually started events (x_started events), but these events will not be backfilled because we cannot determine which jobs were manually triggered by just looking at Jobs table. Similarly, when a job was manually cancelled, we don’t know who performed the cancellation by just looking at the Jobs table. Therefore, when we backfill the x_cancelled events, the associated user ID will be missing (i.e. always null).

[stevenmurphy12]

Hi @nataliekwong (cc @keyihuang), this appears to be working though not quite as expected. The connectionId appears to be getting ignored.

For context, I'm upgrading from Airbyte v0.53.0 to v1.2.0.

When I run the following curl, I see activity in the airbyte-server log

curl -i -X POST http://localhost:9000/api/v1/connections/events/backfill \
-H "Content-Type: application/json" \
-d '{
  "connectionId": "de8977e1-c7ee-49ff-9632-317651a77d91"
  "createdAtStart": "2024-03-01T00:00:00Z",
  "createdAtEnd": "2024-12-01T00:00:00Z"
}'

2024-12-05 13:28:59,210 [io-executor-thread-8]    INFO    i.a.c.s.h.ConnectionsHandler(backfillConnectionEvents):1294 - Backfilled events from 2024-03-01T00:00Z to 2024-12-01T00:00Z for connection de8977e1-c7ee-49ff-9632-317651a77d91
2024-12-05 13:29:01,866 [io-executor-thread-8]    INFO    i.a.c.s.h.ConnectionsHandler(backfillConnectionEvents):1301 - Verify listing jobs. 265 jobs found.

When I run the same curl command but with a different connection ID, I get the same number of jobs found (265)

2024-12-05 13:39:33,656 [io-executor-thread-7]    INFO    i.a.c.s.h.ConnectionsHandler(backfillConnectionEvents):1294 - Backfilled events from 2024-03-01T00:00Z to 2024-12-01T00:00Z for connection 5c9ff6ad-7442-4e7e-b778-a6d236d7a805
2024-12-05 13:39:35,241 [io-executor-thread-7]    INFO    i.a.c.s.h.ConnectionsHandler(backfillConnectionEvents):1301 - Verify listing jobs. 265 jobs found.

Finally, if I provide no connection ID at all, I get the same 265 jobs found

curl -i -X POST http://localhost:9000/api/v1/connections/events/backfill \
-H "Content-Type: application/json" \
-d '{
  "createdAtStart": "2024-03-01T00:00:00Z",
  "createdAtEnd": "2024-12-01T00:00:00Z"
}'

2024-12-05 13:43:30,277 [io-executor-thread-6]    INFO    i.a.c.s.h.ConnectionsHandler(backfillConnectionEvents):1294 - Backfilled events from 2024-03-01T00:00Z to 2024-12-01T00:00Z for connection null
2024-12-05 13:43:31,587 [io-executor-thread-6]    INFO    i.a.c.s.h.ConnectionsHandler(backfillConnectionEvents):1301 - Verify listing jobs. 265 jobs found.

Looking at the code, this appears to be as expected, as the connectionId is only used for logging purposes - https://github.com/airbytehq/airbyte-platform/blob/6a703b02985007005fac0dd7294e278a4a098e22/airbyte-commons-server/src/main/java/io/airbyte/commons/server/handlers/ConnectionsHandler.java#L1291

This (bug?) would actually be ideal, as it means I can migrate all jobs at once, as opposed to creating a curl command for each connection individually. This test was run against a staging instance of Airbyte, before I run it against our prod Airbyte, could you confirm this theory is correct (ie the connection ID is effectively ignored)

Thank you

[keyihuang]

Hi,
Thanks for reporting!
The connectionId was added to the request body only to resolve an auth issue. It will not impact the job migration. As you can see, it is only referenced in a printing log.
I will verify if the auth issue has been fixed. If so, we do not need to pass the connectionId in the request. And I will go ahead and remove the connectionId from the request body and the log statement. Also update this guide doc.
Thanks again!

[stevenmurphy12]

I appreciate the update, thank you!