Add support for structured logs (#323)

* wip on refactoring docs

* wip

* initial structured logs impl

* structured log docs

* create logger package

* add news entry for structured logging

* add logger package to dockerfile and cleanup

* add gh workflow for catching broken links

* further wip

* fix

* further wip on docs

* review feedback

* remove logger dep from mcp package

* fix build errors

* add back auth_url warning

* fix sidebar title consistency

---------

Co-authored-by: bkellam <bshizzle1234@gmail.com>

Author: msukkari

Dockerfile

@@ -42,11 +42,13 @@ COPY ./packages/db ./packages/db
 COPY ./packages/schemas ./packages/schemas
 COPY ./packages/crypto ./packages/crypto
 COPY ./packages/error ./packages/error
+COPY ./packages/logger ./packages/logger
 
 RUN yarn workspace @sourcebot/db install
 RUN yarn workspace @sourcebot/schemas install
 RUN yarn workspace @sourcebot/crypto install
 RUN yarn workspace @sourcebot/error install
+RUN yarn workspace @sourcebot/logger install
 # ------------------------------------
 
 # ------ Build Web ------
@@ -89,6 +91,7 @@ COPY --from=shared-libs-builder /app/packages/db ./packages/db
 COPY --from=shared-libs-builder /app/packages/schemas ./packages/schemas
 COPY --from=shared-libs-builder /app/packages/crypto ./packages/crypto
 COPY --from=shared-libs-builder /app/packages/error ./packages/error
+COPY --from=shared-libs-builder /app/packages/logger ./packages/logger
 
 # Fixes arm64 timeouts
 RUN yarn workspace @sourcebot/web install
@@ -128,6 +131,7 @@ COPY --from=shared-libs-builder /app/packages/db ./packages/db
 COPY --from=shared-libs-builder /app/packages/schemas ./packages/schemas
 COPY --from=shared-libs-builder /app/packages/crypto ./packages/crypto
 COPY --from=shared-libs-builder /app/packages/error ./packages/error
+COPY --from=shared-libs-builder /app/packages/logger ./packages/logger
 RUN yarn workspace @sourcebot/backend install
 RUN yarn workspace @sourcebot/backend build
 
@@ -209,6 +213,7 @@ COPY --from=shared-libs-builder /app/packages/db ./packages/db
 COPY --from=shared-libs-builder /app/packages/schemas ./packages/schemas
 COPY --from=shared-libs-builder /app/packages/crypto ./packages/crypto
 COPY --from=shared-libs-builder /app/packages/error ./packages/error
+COPY --from=shared-libs-builder /app/packages/logger ./packages/logger
 
 # Configure dependencies
 RUN apk add --no-cache git ca-certificates bind-tools tini jansson wget supervisor uuidgen curl perl jq redis postgresql postgresql-contrib openssl util-linux unzip

docs/docs.json

@@ -74,7 +74,8 @@
                                     "docs/configuration/auth/roles-and-permissions"
                                 ]
                             },
-                            "docs/configuration/transactional-emails"
+                            "docs/configuration/transactional-emails",
+                            "docs/configuration/structured-logging"
                         ]
                     },
                     {

docs/docs/configuration/auth/overview.mdx

@@ -2,6 +2,8 @@
 title: Overview
 ---
 
+<Warning>If you're deploying Sourcebot behind a domain, you must set the [AUTH_URL](/docs/configuration/environment-variables) environment variable.</Warning>
+
 Sourcebot has built-in authentication that gates access to your organization. OAuth, email codes, and email / password are supported. 
 
 The first account that's registered on a Sourcebot deployment is made the owner. All other users who register must be [approved](/docs/configuration/auth/overview#approving-new-members) by the owner.
@@ -40,8 +42,6 @@ See [transactional emails](/docs/configuration/transactional-emails) for more de
 
 ## Enterprise Authentication Providers
 
-<Warning>If you're deploying Sourcebot behind a domain, you must set the [AUTH_URL](/docs/configuration/environment-variables) environment variable to use these providers.</Warning>
-
 The following authentication providers require an [enterprise license](/docs/license-key) to be enabled.
 
 By default, a new user registering using these providers must have their join request accepted by the owner of the organization to join. To allow a user to join automatically when

docs/docs/configuration/auth/roles-and-permissions.mdx

@@ -1,5 +1,6 @@
 ---
 title: Roles and Permissions
+sidebarTitle: Roles and permissions
 ---
 
 <Note>Looking to sync permissions with your identify provider? We're working on it - [reach out](https://www.sourcebot.dev/contact) to us to learn more</Note>

docs/docs/configuration/environment-variables.mdx

@@ -27,6 +27,8 @@ The following environment variables allow you to configure your Sourcebot deploy
 | `SMTP_CONNECTION_URL` | `-` | <p>The url to the SMTP service used for sending transactional emails. See [this doc](/docs/configuration/transactional-emails) for more info.</p> |
 | `SOURCEBOT_ENCRYPTION_KEY` | Automatically generated at startup if no value is provided. Generated using `openssl rand -base64 24` | <p>Used to encrypt connection secrets and generate API keys.</p> |
 | `SOURCEBOT_LOG_LEVEL` | `info` | <p>The Sourcebot logging level. Valid values are `debug`, `info`, `warn`, `error`, in order of severity.</p> |
+| `SOURCEBOT_STRUCTURED_LOGGING_ENABLED` | `false` | <p>Enables/disable structured JSON logging. See [this doc](/docs/configuration/structured-logging) for more info.</p> |
+| `SOURCEBOT_STRUCTURED_LOGGING_FILE` | - | <p>Optional file to log to if structured logging is enabled</p> | 
 | `SOURCEBOT_TELEMETRY_DISABLED` | `false` | <p>Enables/disables telemetry collection in Sourcebot. See [this doc](/docs/overview.mdx#telemetry) for more info.</p> |
 | `TOTAL_MAX_MATCH_COUNT` | `100000` | <p>The maximum number of matches per query</p> |
 | `ZOEKT_MAX_WALL_TIME_MS` | `10000` | <p>The maximum real world duration (in milliseconds) per zoekt query</p> | 

docs/docs/configuration/structured-logging.mdx

@@ -0,0 +1,39 @@
+---
+title: Structured logging
+---
+
+By default, Sourcebot will output logs to the console in a human readable format. If you'd like Sourcebot to output structured JSON logs, set the following env vars:
+
+- `SOURCEBOT_STRUCTURED_LOGGING_ENABLED` (default: `false`): Controls whether logs are in a structured JSON format
+- `SOURCEBOT_STRUCTURED_LOGGING_FILE`: If structured logging is enabled and this env var is set, structured logs will be written to this file (ex. `/data/sourcebot.log`)
+
+### Structured log schema
+```json
+{
+    "$schema": "http://json-schema.org/draft-07/schema#",
+    "type": "object",
+    "title": "SourcebotLog",
+    "properties": {
+        "level": {
+            "type": "string",
+            "description": "The log level (error, warning, info, debug)"
+        },
+        "service": {
+            "type": "string",
+            "description": "The Sourcebot component that generated the log"
+        },
+        "message": {
+            "type": "string",
+            "description": "The log message"
+        },
+        "status": {
+            "type": "string",
+            "description": "The same value as the level field added for datadog support"
+        },
+        "timestamp": {
+            "type": "string",
+            "description": "The timestamp of the log in ISO 8061 format"
+        }
+    }
+}
+```

docs/docs/connections/overview.mdx

@@ -43,7 +43,7 @@ A JSON configuration file is used to specify connections. For example:
 
 Configuration files must conform to the [JSON schema](#schema-reference).
 
-When running Sourcebot, this file must be mounted in a volume that is accessible to the container, with it's path specified in the `CONFIG_PATH` environment variable. For example:
+When running Sourcebot, this file must be mounted in a volume that is accessible to the container, with its path specified in the `CONFIG_PATH` environment variable. For example:
 
 ```bash
 docker run \

docs/docs/deployment-guide.mdx

@@ -53,6 +53,9 @@ Watch this 1:51 minute video to get a quick overview of how to deploy Sourcebot
     </Step>
 
     <Step title="Launch your instance">
+        <Warning>If you're deploying Sourcebot behind a domain, you must set the [AUTH_URL](/docs/configuration/environment-variables) environment variable.</Warning>
+
+
         In the same directory as `config.json`, run the following command to start your instance:
 
         ``` bash

docs/docs/features/agents/review-agent.mdx

@@ -1,6 +1,6 @@
 ---
 title: AI Code Review Agent
-sidebarTitle: AI Code Review Agent
+sidebarTitle: AI code review agent
 ---
 
 <Note>

packages/backend/package.json

@@ -23,15 +23,14 @@
     },
     "dependencies": {
         "@gitbeaker/rest": "^40.5.1",
-        "@logtail/node": "^0.5.2",
-        "@logtail/winston": "^0.5.2",
         "@octokit/rest": "^21.0.2",
         "@sentry/cli": "^2.42.2",
         "@sentry/node": "^9.3.0",
         "@sentry/profiling-node": "^9.3.0",
         "@sourcebot/crypto": "workspace:*",
         "@sourcebot/db": "workspace:*",
         "@sourcebot/error": "workspace:*",
+        "@sourcebot/logger": "workspace:*",
         "@sourcebot/schemas": "workspace:*",
         "@t3-oss/env-core": "^0.12.0",
         "@types/express": "^5.0.0",
@@ -51,7 +50,6 @@
         "prom-client": "^15.1.3",
         "simple-git": "^3.27.0",
         "strip-json-comments": "^5.0.1",
-        "winston": "^3.15.0",
         "zod": "^3.24.3"
     }
 }