feat(database monitor): handles multi-region and connection pooling #3685

iainsproat · 2024-12-12T21:27:52Z

Description & motivation

This addresses two features with the existing database monitor:

It could not monitor databases where a connection pool was present (and the database name in the connection string did not match the actual database name): https://linear.app/speckle/issue/WEB-2338/fix-connection-monitor-pod
It did not monitor databases in additional regions if the server is multi-region capable: https://linear.app/speckle/issue/WEB-2142/multi-region-db-connection-health

Changes:

rewritten in typescript. This allows us to utilise the common functionality for multi-regions in the shared package. It also removes one of the only Python components in this repository, ensuring consistent language.
Uses consistent logging as other components. Inconsistent logging in Python caused a mess in aggregated logging platforms.
Optionally expects the multi region config to include databaseName as a paramater. This allows it to gather data on the database size. If it does not exist, will log a warning and continue.

To-do before merge:

recreate validation on testing4 & testing5

Screenshots:

Validation of changes:

Checklist:

My pull request follows the guidelines in the Contributing guide?
My pull request does not duplicate any other open Pull Requests for the same update/change?
My commits are related to the pull request and do not amend unrelated code or documentation.
My code follows a similar style to existing code.
I have added appropriate tests.
I have updated or added relevant documentation.

References

- rewritten in typescript

packages/database-monitor/src/observability/prometheusMetrics.ts

github-actions · 2024-12-12T21:38:34Z

📸 Preview service has generated an image.

github-actions · 2024-12-12T21:43:04Z

📸 Preview service has generated an image.

packages/database-monitor/src/observability/prometheusMetrics.ts

iainsproat · 2024-12-12T21:49:38Z

packages/database-monitor/src/observability/prometheusMetrics.ts

+        GROUP BY status;
+      `
+    )
+    const remainingWebhookStatus = new Set(Array(4).keys())


NOTE: This hard codes the domain logic that there are only 4 webhook status. If it ever changes this breaks.

iainsproat · 2024-12-12T21:49:46Z

packages/database-monitor/src/observability/prometheusMetrics.ts

+        GROUP BY "previewStatus";
+        `)
+
+    const remainingPreviewStatus = new Set(Array(4).keys())


NOTE: This hard codes the domain logic that there are only 4 webhook status. If it ever changes this breaks.

iainsproat · 2024-12-12T21:50:27Z

packages/database-monitor/src/observability/prometheusMetrics.ts

+  return {
+    start: () => {
+      const intervalId = setInterval(() => {
+        void (async () => {


NOTE: setInterval cannot be async, so we can't await the call. (or is there a better way of achieving this?)

iainsproat · 2024-12-12T21:50:55Z

packages/database-monitor/src/observability/prometheusMetrics.ts

+      register: prometheusClient.register,
+      collectionPeriodMilliseconds: databaseMonitorCollectionPeriodSeconds() * 1000,
+      config: {
+        getDbClients,


Do we want to pass in getDbClients from elsewhere?
(minor change, can be in a different PR if at all)

Will probably adjust this if & when we add tests to better help dependency injection in test environments, but no need for now.

github-actions · 2024-12-12T21:59:01Z

📸 Preview service has generated an image.

github-actions · 2024-12-12T21:59:41Z

📸 Preview service has generated an image.

…tric with the metric definition

github-actions · 2024-12-13T10:50:27Z

📸 Preview service has generated an image.

…from shared

github-actions · 2024-12-13T12:22:23Z

📸 Preview service has generated an image.

… for now

github-actions · 2024-12-13T12:30:27Z

📸 Preview service has generated an image.

github-actions · 2024-12-13T13:56:26Z

📸 Preview service has generated an image.

github-actions · 2024-12-13T14:18:56Z

📸 Preview service has generated an image.

github-actions · 2024-12-13T14:46:49Z

📸 Preview service has generated an image.

github-actions · 2024-12-13T15:11:47Z

📸 Preview service has generated an image.

github-actions · 2024-12-13T16:55:44Z

📸 Preview service has generated an image.

github-actions · 2024-12-13T17:25:19Z

📸 Preview service has generated an image.

github-actions · 2024-12-13T18:05:52Z

📸 Preview service has generated an image.

github-actions · 2024-12-13T20:58:33Z

📸 Preview service has generated an image.

- as database monitor tries to monitor all configured databases, there may be unregistered databases which are not yet migrated and do not have the required tables - many of the queries are multi-region and now query all database clients

github-actions · 2024-12-14T15:08:52Z

📸 Preview service has generated an image.

github-actions · 2024-12-16T16:02:39Z

📸 Preview service has generated an image.

github-actions · 2024-12-16T16:16:56Z

📸 Preview service has generated an image.

github-actions · 2024-12-16T16:34:23Z

📸 Preview service has generated an image.

…eature flag

github-actions · 2024-12-16T17:41:37Z

📸 Preview service has generated an image.

github-actions · 2024-12-16T17:49:07Z

📸 Preview service has generated an image.

github-actions · 2024-12-16T18:18:58Z

📸 Preview service has generated an image.

iainsproat added 2 commits December 12, 2024 21:26

feat(database monitor): handles multi-region and connection pooling

f0b411e

- rewritten in typescript

Restore changes to unrelated components

264c4f2

iainsproat commented Dec 12, 2024

View reviewed changes

packages/database-monitor/src/observability/prometheusMetrics.ts Outdated Show resolved Hide resolved

iainsproat commented Dec 12, 2024

View reviewed changes

iainsproat added 2 commits December 12, 2024 21:52

Fix for testing

c9e5758

Merge branch 'main' into iain/db-monitor-pooling

9c959c3

iainsproat added 2 commits December 13, 2024 09:39

Merge branch 'main' into iain/db-monitor-pooling

4ada3b5

refactor prometheus metrics to keep the logic for collecting for a me…

3825b8e

…tric with the metric definition

iainsproat added 2 commits December 13, 2024 11:31

Reintroduce zod and znv as they are peer dependencies of Environment …

0a41730

…from shared

pass in databasename from config, simplify types

417ca63

We don't have any tests, simplify the PR by removing the unused files…

64f606e

… for now

iainsproat marked this pull request as ready for review December 13, 2024 12:26

iainsproat requested a review from gjedlicska December 13, 2024 12:26

Merge branch 'main' into iain/db-monitor-pooling

ce9c6f7

fix(server): correct type in notifications helper

8bbfde5

iainsproat added 2 commits December 13, 2024 14:31

ci(publish): update publish job

ef5921e

rename to ensure consistent image name with existing database monitor

bb8355d

iainsproat added 2 commits December 13, 2024 15:05

Do not gitignore some bin directories

b24c23b

Include bin/www.js file

e99316f

fix helm chart indentation

3059df4

Metrics host must be externally exposed

3506d2f

iainsproat added 2 commits December 13, 2024 19:38

Express logging should be debug

89f924f

Fix issue with NaN in metrics, and allow self-signed CA certs

e2ad346

iainsproat added 2 commits December 13, 2024 21:08

Provide details about connections

cc464d8

alemagio previously approved these changes Dec 16, 2024

View reviewed changes

iainsproat added 3 commits December 16, 2024 13:42

Merge branch 'main' into iain/db-monitor-pooling

f26cb64

Refactor to place metrics in separate files

efe2416

feat(metrics): more metrics related to replications

4261e26

iainsproat dismissed alemagio’s stale review via 4261e26 December 16, 2024 15:53

tidying up new metrics

ad18135

Allow database monitor to configure connection timeouts

a69175b

iainsproat added 2 commits December 16, 2024 17:35

Use aiven_extras to get replication & subscription, and protect via f…

32ecc36

…eature flag

track errors

f647bed

Improve error handling where aiven_extras is not yet enabled

ccd6e30

alemagio approved these changes Dec 17, 2024

View reviewed changes

iainsproat merged commit 0115e65 into main Dec 17, 2024
42 of 44 checks passed

iainsproat deleted the iain/db-monitor-pooling branch December 17, 2024 09:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(database monitor): handles multi-region and connection pooling #3685

feat(database monitor): handles multi-region and connection pooling #3685

iainsproat commented Dec 12, 2024 •

edited

Loading

github-actions bot commented Dec 12, 2024

github-actions bot commented Dec 12, 2024

iainsproat Dec 12, 2024 •

edited

Loading

iainsproat Dec 12, 2024 •

edited

Loading

iainsproat Dec 12, 2024 •

edited

Loading

iainsproat Dec 12, 2024 •

edited

Loading

iainsproat Dec 13, 2024

github-actions bot commented Dec 12, 2024

github-actions bot commented Dec 12, 2024

github-actions bot commented Dec 13, 2024

github-actions bot commented Dec 13, 2024

github-actions bot commented Dec 13, 2024

github-actions bot commented Dec 13, 2024

github-actions bot commented Dec 13, 2024

github-actions bot commented Dec 13, 2024

github-actions bot commented Dec 13, 2024

github-actions bot commented Dec 13, 2024

github-actions bot commented Dec 13, 2024

github-actions bot commented Dec 13, 2024

github-actions bot commented Dec 13, 2024

github-actions bot commented Dec 14, 2024

github-actions bot commented Dec 16, 2024

github-actions bot commented Dec 16, 2024

github-actions bot commented Dec 16, 2024

github-actions bot commented Dec 16, 2024

github-actions bot commented Dec 16, 2024

github-actions bot commented Dec 16, 2024

feat(database monitor): handles multi-region and connection pooling #3685

feat(database monitor): handles multi-region and connection pooling #3685

Conversation

iainsproat commented Dec 12, 2024 • edited Loading

Description & motivation

Changes:

To-do before merge:

Screenshots:

Validation of changes:

Checklist:

References

github-actions bot commented Dec 12, 2024

github-actions bot commented Dec 12, 2024

iainsproat Dec 12, 2024 • edited Loading

Choose a reason for hiding this comment

iainsproat Dec 12, 2024 • edited Loading

Choose a reason for hiding this comment

iainsproat Dec 12, 2024 • edited Loading

Choose a reason for hiding this comment

iainsproat Dec 12, 2024 • edited Loading

Choose a reason for hiding this comment

iainsproat Dec 13, 2024

Choose a reason for hiding this comment

github-actions bot commented Dec 12, 2024

github-actions bot commented Dec 12, 2024

github-actions bot commented Dec 13, 2024

github-actions bot commented Dec 13, 2024

github-actions bot commented Dec 13, 2024

github-actions bot commented Dec 13, 2024

github-actions bot commented Dec 13, 2024

github-actions bot commented Dec 13, 2024

github-actions bot commented Dec 13, 2024

github-actions bot commented Dec 13, 2024

github-actions bot commented Dec 13, 2024

github-actions bot commented Dec 13, 2024

github-actions bot commented Dec 13, 2024

github-actions bot commented Dec 14, 2024

github-actions bot commented Dec 16, 2024

github-actions bot commented Dec 16, 2024

github-actions bot commented Dec 16, 2024

github-actions bot commented Dec 16, 2024

github-actions bot commented Dec 16, 2024

github-actions bot commented Dec 16, 2024

iainsproat commented Dec 12, 2024 •

edited

Loading

iainsproat Dec 12, 2024 •

edited

Loading

iainsproat Dec 12, 2024 •

edited

Loading

iainsproat Dec 12, 2024 •

edited

Loading

iainsproat Dec 12, 2024 •

edited

Loading