feat: add capability for writing to bigquery #993

joseph-sentry · 2025-01-07T21:59:10Z

this PR:

creates the lower level glue between our code and BQ in the form of the BigQueryService
creates an abstraction for persisting testruns and implements it for both BQ and PG

codecov · 2025-01-07T22:05:49Z

Codecov Report

Attention: Patch coverage is 71.47436% with 89 lines in your changes missing coverage. Please review.

Project coverage is 97.74%. Comparing base (e1d597d) to head (0efb81c).
Report is 5 commits behind head on main.

✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
services/bigquery.py	25.80%	46 Missing ⚠️
services/tests/test_bigquery.py	40.54%	22 Missing ⚠️
ta_storage/pg.py	88.54%	11 Missing ⚠️
generated_proto/testrun/ta_testrun_pb2.py	72.22%	5 Missing ⚠️
ta_storage/bq.py	90.24%	4 Missing ⚠️
ta_storage/base.py	85.71%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #993      +/-   ##
==========================================
- Coverage   97.96%   97.74%   -0.22%     
==========================================
  Files         446      454       +8     
  Lines       36045    36368     +323     
==========================================
+ Hits        35311    35548     +237     
- Misses        734      820      +86

Flag	Coverage Δ
integration	`42.01% <38.46%> (-0.02%)`	⬇️
unit	`90.46% <71.47%> (-0.18%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

⚠️ Impact Analysis from Codecov is deprecated and will be sunset on Jan 31 2025. See more

codecov-staging · 2025-01-07T22:05:56Z

Codecov Report

Attention: Patch coverage is 71.47436% with 89 lines in your changes missing coverage. Please review.

✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
services/bigquery.py	25.80%	46 Missing ⚠️
services/tests/test_bigquery.py	40.54%	22 Missing ⚠️
ta_storage/pg.py	88.54%	11 Missing ⚠️
generated_proto/testrun/ta_testrun_pb2.py	72.22%	5 Missing ⚠️
ta_storage/bq.py	90.24%	4 Missing ⚠️
ta_storage/base.py	85.71%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

codecov-qa · 2025-01-07T22:06:05Z

❌ 1 Tests Failed:

Tests completed	Failed	Passed	Skipped
1	1	0	0

View the top 1 failed tests by shortest run time

ta_storage/tests/test_pg.py::ta_storage.tests.test_pg

Stack Traces | 0s run time

ImportError while importing test module '.../ta_storage/tests/test_pg.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
.../local/lib/python3.13/importlib/__init__.py:88: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ta_storage/tests/test_pg.py:3: in <module>
    from ta_storage.pg import PGDriver
ta_storage/pg.py:4: in <module>
    from msgpack import unpackb
E   ModuleNotFoundError: No module named 'msgpack'

To view more test analytics, go to the Test Analytics Dashboard
📢 Thoughts on this report? Let us know!

codecov-public-qa · 2025-01-07T22:06:13Z

❌ 1 Tests Failed:

Tests completed	Failed	Passed	Skipped
1	1	0	0

View the top 1 failed tests by shortest run time

ta_storage/tests/test_pg.py::::ta_storage.tests.test_pg

Stack Traces | 0s run time

No failure message available

To view more test analytics, go to the Test Analytics Dashboard
📢 Thoughts on this report? Let us know!

github-actions · 2025-01-07T22:06:24Z

✅ All tests successful. No failed tests were found.

📣 Thoughts on this report? Let Codecov know! | Powered by Codecov

this is an abstraction to help us use bigquery for test analytics data this abstraction allows us to write data to bigquery using the Storage Write API and query it using SQL statements as strings, it also handles creating the connection to bigquery for us the use protocol buffers is a consequence of using the Storage Write API feat: create TADriver abstraction this commit creates the TADriver abstraction which (for now) exposes a write_testruns method, which will take care of persisting a list of testruns to some specific storage this commit implements this interface for Postgres and for Bigquery the idea is that we will start by persisting testruns to both, then eventually we add a method to the interface for retrieving the aggregates at which point it will be safe for our cloud offering to switch over to only bigquery for storage of TA testruns.

protobuf/sample.proto

protobuf/ta_testrun.proto

requirements.in

ta_storage/base.py

ta_storage/pg.py

ta_storage/bq.py

ta_storage/pg.py

Swatinem · 2025-01-08T10:36:22Z

services/bigquery.py

+        batch_commit_write_streams_request.parent = parent
+        batch_commit_write_streams_request.write_streams = [write_stream.name]
+
+        self.write_client.batch_commit_write_streams(batch_commit_write_streams_request)


Is the complexity specific for batch_commit_write_streams? I mean wholy sh…, this API is convoluted and overly complex.

Can you point to the docs related to inserting / batch-writing to bigquery?

https://cloud.google.com/bigquery/docs/write-api-batch

services/tests/test_bigquery.py

joseph-sentry marked this pull request as ready for review January 7, 2025 21:59

joseph-sentry requested a review from a team January 7, 2025 21:59

joseph-sentry force-pushed the joseph/bq branch from e47e13c to b863e0f Compare January 7, 2025 22:17

fix: add helper functions to pg.py

0b8e968

Swatinem reviewed Jan 8, 2025

View reviewed changes

fix: address feedback

0efb81c

Swatinem approved these changes Jan 9, 2025

View reviewed changes

joseph-sentry added this pull request to the merge queue Jan 9, 2025

Merged via the queue into main with commit 78ce991 Jan 9, 2025
14 of 27 checks passed

joseph-sentry deleted the joseph/bq branch January 9, 2025 17:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add capability for writing to bigquery #993

feat: add capability for writing to bigquery #993

joseph-sentry commented Jan 7, 2025

codecov bot commented Jan 7, 2025 •

edited

Loading

codecov-staging bot commented Jan 7, 2025 •

edited by codecov-notifications bot

Loading

codecov-qa bot commented Jan 7, 2025 •

edited

Loading

codecov-public-qa bot commented Jan 7, 2025

github-actions bot commented Jan 7, 2025 •

edited

Loading

Swatinem Jan 8, 2025

joseph-sentry Jan 8, 2025

feat: add capability for writing to bigquery #993

feat: add capability for writing to bigquery #993

Conversation

joseph-sentry commented Jan 7, 2025

codecov bot commented Jan 7, 2025 • edited Loading

Codecov Report

codecov-staging bot commented Jan 7, 2025 • edited by codecov-notifications bot Loading

Codecov Report

codecov-qa bot commented Jan 7, 2025 • edited Loading

❌ 1 Tests Failed:

codecov-public-qa bot commented Jan 7, 2025

❌ 1 Tests Failed:

github-actions bot commented Jan 7, 2025 • edited Loading

Swatinem Jan 8, 2025

Choose a reason for hiding this comment

joseph-sentry Jan 8, 2025

Choose a reason for hiding this comment

codecov bot commented Jan 7, 2025 •

edited

Loading

codecov-staging bot commented Jan 7, 2025 •

edited by codecov-notifications bot

Loading

codecov-qa bot commented Jan 7, 2025 •

edited

Loading

github-actions bot commented Jan 7, 2025 •

edited

Loading