Experiment with Sentry performance monitoring for BigQuery destination

[tuliren]

Acceptance criteria

1. Introduce Sentry SDK to warehouse destination connector
   • Let's start with BigQuery destination.
   • It can either be a pure Java SDK, or Log4j SDK.
   • Compare the two different ways to install Sentry (or any other approach if there is more), and determine which way is better.
2. Collect exceptions with the Sentry client.
   • If there is a way to automatically collect them all (e.g. through the Log4j SDK), let's try that.
   • Otherwise, let's try to collect one or two frequent exceptions, and see if it is useful.
     • We can systematically determine which exceptions to collect in the future.
3. Add custom instrumentation to track the integration runner.
   • It is unclear what information Sentry will collect.
   • Determine whether this is useful.
4. Add JDBC instrumentation to track the JDBC communication.
   • Again it is unclear what exactly will be sent to Sentry.
   • Determine whether this is helpful.
5. Add documentation to Sentry configuration for Java-based connectors.

I have created a new Sentry project: test-java-connector. Feel free to use this one, or create new testing project if necessary.

[sherifnada]

@tuliren Could we actually switch this to be focused on destinations? With the recent timeline change to getting snowflake in GA, it would be very helpful to do this experiment for snowflake so we can start collecting metrics about it asap.

[tuliren]

I have updated this issue to start with BigQuery, instead of databases.

[tuliren]

@alexandr-shegeda, @oustynova, can this issue be prioritized starting next week (2022-01-10)? We just realized that it is more important than we previously thought. If this project cannot be prioritized any time soon, I can work on it.

[tuliren]

@alexandertsukanov, any update or ETA of this issue?

[alexandertsukanov]

@tuliren Hi Liren!
I created the ticket to get the access for the cloud to test how logging works at the cloud https://github.com/airbytehq/airbyte-cloud/issues/830.
Also, I consider to use Log4j SDK for logging as I think it will allow us to make logging more flexible. In local dev environment I was successful with that.

However, I Have a couple of questions:

• Regarding https://docs.sentry.io/platforms/java/performance/instrumentation/custom-instrumentation/. Do we know what part of Airbyte do we need to track with transactions? (Do you want to track performance of IntegrationRunner(destination).run(args) by its-self or something more deeper?)
• Do we know what DSL for Sentry will be used in cloud?
• You say in point airbytehq/airbyte-platform-internal#3: "Add JDBC instrumentation to track the JDBC communication". As far as I know BigQuery doesn't use JDBC underneath, correct me if I am wrong. So, Should I make PR only for BigQuery destination and BigQuery destination denormailzed in scope of this task?

Thanks.

[tuliren]

I consider to use Log4j SDK for logging as I think it will allow us to make logging more flexible. In local dev environment I was successful with that.

Cool.

Regarding https://docs.sentry.io/platforms/java/performance/instrumentation/custom-instrumentation/. Do we know what part of Airbyte do we need to track with transactions? (Do you want to track performance of IntegrationRunner(destination).run(args) by its-self or something more deeper?)

You can start with instrumenting the IntegrationRunner#run method and see what it gives us.

Do we know what DSL for Sentry will be used in cloud?

What do you mean by DSL?

You say in point checkin catalogs #3: "Add JDBC instrumentation to track the JDBC communication". As far as I know BigQuery doesn't use JDBC underneath, correct me if I am wrong. So, Should I make PR only for BigQuery destination and BigQuery destination denormailzed in scope of this task?

Oh, right. My bad. Is it too late to switch to Snowflake destination? If not, the SnowflakeInsertDestination and SnowflakeInternalStagingDestination would be a good candidate. It's a JDBC destination. If it is too much trouble to switch, you can ignore this bullet point for now.

[alexandertsukanov]

@tuliren
What do you mean by DSL?

• Sorry my bad. I mean DSN, here we go https://docs.sentry.io/platforms/java/configuration/
  From my point of view, we should use some Airbyte application at Sentry to consume logs from Airbyte connectors. Correct me if I am wrong. (At least, in Cloud env) Off course, for basic skeleton I can use my own https://sentry.io/organizations/test-company-g4/, but it doesn't looks like the working solution.

Oh, right. My bad. Is it too late to switch to Snowflake destination? If not, the SnowflakeInsertDestination and SnowflakeInternalStagingDestination would be a good candidate. It's a JDBC destination. If it is too much trouble to switch, you can ignore this bullet point for now.,

• I am finishing with BigQuery and BigQuery denormilized. I can whether to add integration with the Sentry for Snowflake in this PR (I don't think it will take much time) or in scope of other task.

Also, should we have some filter for sensitive information? (Some keywords we should ignore in logs, etc).

[alexandertsukanov]

I believe we can should use Performance Metrics, at least in IntegrationRunner#run, as its provide us useful info about about Duration Breakdown here is an example:

You can give me your email I can provide you an access to my test-application, after you can see which info we have in Performance metrics. Thanks.

[tuliren]

DSN

You can use yours for now. We will figure out these variables to inject into the cloud later.

I am finishing with BigQuery and BigQuery denormilized. I can whether to add integration with the Sentry for Snowflake in this PR (I don't think it will take much time) or in scope of other task.

Cool. It would be very helpful if you can add the integration for Snowflake as well.

should we have some filter for sensitive information?

Secrets should already be removed from the logs. There may be some edge cases. We can look into this later.

You can give me your email I can provide you an access to my test-application, after you can see which info we have in Performance metrics.

This looks nice. Thanks. I will DM you my email.

[tuliren]

Current status

• Sentry integration is working.
• Using Sentry Log4j seems to mess up with routine logging. Let's use the more generic Sentry Java SDK for now.

Remaining TODOs

• [P0] Debug Sentry exception emission
• [P0] Add connector version as a tag.
• [P1] Add more granular transaction to at least measure runtime of each command.
• [P1] Add a release tag to Sentry.
• [P1] Publish the release tag to Sentry in CI.
• [P1] Move the environment variable to .env.
• [P2] Set an environment to distinguish production from dev