Database Sources (Postgres) implements Progress Bar protocol - AirbyteEstimateTraceMessage

[bleonard]

Implement the AirbyteEstimateTraceMessage messages described in #18875

Here is an example PR for another source: #19197

The goal is to emit the predicated total number of rows/bytes in the sync (per stream).

[bleonard]

We can't do bytes, but we can do rows for Postgres Standard.

So in the AbstractJdbcSource, we'll do SELECT COUNT(*) FROM table WHERE cursor > value (or without the WHERE if not incremental or there is no cursor) and output that as the row_count

It's not clear to me at the moment @evantahler if we do this for all streams at the begniing or for each sttream as we start to work on it. Let's assume at the beginning so it can be a full lenght of the whole sync.

To support CDC, we can talk about updating the protocol to take a percentage value.

[evantahler]

Why can't you get an estimate to the size of a postgres table?

• There are fancy ways to compute the size of the rows in question directly
• Or, you could get a count of all rows in the table (10,000) and ask postgres for the fast table size (select pg_relation_size('table_name')) and devide to get an average size of row... and then multiply that by the # rows you are sending (considering the offset)

Either way, the feature is implemented such that:

• you don't have to send the estimates at the beginning - you can send them whenever
• It's better to send the estimates at the beginning because we won't display the progress bar or time-remaining estimate until we get an estimate for every stream that's being synced.
  • The estimates are based on rows remaining, rather than bytes remaining

👍 on future protocol updates

[bleonard]

Cool. Maybe we can send bytes then. We'll look into it, but it does look like we want to do it per stream at the beginning. I'm curious what happens if it turns out we don't have access to a random stream. Let's say we want to then do that one last. Do we still have to send some number of rows (0) to make it show the progress bar? @evantahler

[evantahler]

Do we still have to send some number of rows (0) to make it show the progress bar?

I think so