Add support for Sentry monitoring

Includes addition to documentation.

README.md

@@ -1,10 +1,20 @@
 # RADAR-Gateway
 
-REST Gateway to the Apache Kafka, similar to the REST Proxy provided by Confluent. In addition, it does authentication and authorization, content validation and decompression if needed. It is available as a [docker image](https://hub.docker.com/r/radarbase/radar-gateway).
+REST Gateway to the Apache Kafka, similar to the REST Proxy provided by Confluent. In addition, it does authentication
+and authorization, content validation and decompression if needed. It is available as
+a [docker image](https://hub.docker.com/r/radarbase/radar-gateway).
+
+<!-- TOC -->
+* [RADAR-Gateway](#radar-gateway)
+  * [Configuration](#configuration)
+  * [Usage](#usage)
+  * [Sentry monitoring](#sentry-monitoring)
+<!-- TOC -->
 
 ## Configuration
 
-The [RADAR-Auth] library is used for authentication and authorization of users. Refer to the documentation there for a full description of the configuration options.
+The [RADAR-Auth] library is used for authentication and authorization of users. Refer to the documentation there for a
+full description of the configuration options.
 
 ## Usage
 
@@ -21,22 +31,60 @@ TOPIC=test
 docker-compose exec kafka-1 kafka-topics --create --topic $TOPIC --bootstrap-server kafka-1:9092
 ```
 
-Now the gateway is accessible through <http://localhost:8090/radar-gateway/> and the [ManagementPortal] is available through <http://localhost:8080/managementportal/>
+Now the gateway is accessible through <http://localhost:8090/radar-gateway/> and the [ManagementPortal] is available
+through <http://localhost:8080/managementportal/>
 
-The access token should be generated by the aforementioned Management portal. The access token is a JWT (JSON Web Token) that should contain the `MEASUREMENT.CREATE` scope for resource `res_gateway`, and list all applicable sources to submit data for. The gateway does content validation for posted data. It requires to use the Avro format with JSON serialization, using the `application/vnd.kafka.avro.v1+json` or `application/vnd.kafka.avro.v2+json` media types, as described in the [REST Proxy documentation].  It also requires messages to have both a key and a value with schemas. The key should have a `userId` and `sourceId` field. The `userId` should match the `sub` field in the OAuth2 JWT access token. That JWT should also contain a `sources` array claim which should contain the given `sourceId`. Sources can be added in the ManagementPortal or be generated by the app dynamically and then registered with the ManagementPortal.
+The access token should be generated by the aforementioned Management portal. The access token is a JWT (JSON Web Token)
+that should contain the `MEASUREMENT.CREATE` scope for resource `res_gateway`, and list all applicable sources to submit
+data for. The gateway does content validation for posted data. It requires to use the Avro format with JSON
+serialization, using the `application/vnd.kafka.avro.v1+json` or `application/vnd.kafka.avro.v2+json` media types, as
+described in the [REST Proxy documentation]. It also requires messages to have both a key and a value with schemas. The
+key should have a `userId` and `sourceId` field. The `userId` should match the `sub` field in the OAuth2 JWT access
+token. That JWT should also contain a `sources` array claim which should contain the given `sourceId`. Sources can be
+added in the ManagementPortal or be generated by the app dynamically and then registered with the ManagementPortal.
 
 Now you can access the gateway:
+
 ```shell
 TOKEN=<access token from management portal>
 curl -H "Authorization: Bearer $TOKEN" http://localhost:8090/radar-gateway/topics
 ```
 
-Data compressed with GZIP is decompressed if the `Content-Encoding: gzip` header is present. With `curl`, use the `-H "Content-Encoding: gzip" --data-binary @data.json.gz` flags. It can be activated in `radar-commons` Java `RestClient` by setting `RestClient.Builder.gzipCompression(true)`. Likewise it accepts Apple LZFSE encoded data by adding the header `Content-Encoding: lzfse`.
+Data compressed with GZIP is decompressed if the `Content-Encoding: gzip` header is present. With `curl`, use the
+`-H "Content-Encoding: gzip" --data-binary @data.json.gz` flags. It can be activated in `radar-commons` Java
+`RestClient` by setting `RestClient.Builder.gzipCompression(true)`. Likewise it accepts Apple LZFSE encoded data by
+adding the header `Content-Encoding: lzfse`.
 
 Otherwise, it accepts all the same Avro messages and headers as specified in the Kafka [REST Proxy documentation].
 
-Finally, the gateway accepts a custom binary format for data ingestion. The data must follow the binary Avro serialization of the [RecordSet schema](https://github.com/RADAR-base/RADAR-Schemas/blob/master/commons/kafka/record_set.avsc). Data in this format can be posted by using the content type `application/vnd.radarbase.avro.v1+binary`. It will construct an `ObservationKey` based on the user data in the `RecordSet`, and read the binary data values using the schema version provided in the `RecordSet`. This data sending mode can be activated in Java by using radar-commons `RestSender.Builder.useBinaryContent(true)`. Using binary mode has the added benefit of having a much more efficient GZIP encoding for many datasets.
+Finally, the gateway accepts a custom binary format for data ingestion. The data must follow the binary Avro
+serialization of
+the [RecordSet schema](https://github.com/RADAR-base/RADAR-Schemas/blob/master/commons/kafka/record_set.avsc). Data in
+this format can be posted by using the content type `application/vnd.radarbase.avro.v1+binary`. It will construct an
+`ObservationKey` based on the user data in the `RecordSet`, and read the binary data values using the schema version
+provided in the `RecordSet`. This data sending mode can be activated in Java by using radar-commons
+`RestSender.Builder.useBinaryContent(true)`. Using binary mode has the added benefit of having a much more efficient
+GZIP encoding for many datasets.
 
 [REST Proxy documentation]: https://docs.confluent.io/current/kafka-rest/api.html
+
 [RADAR-Auth]: https://github.com/RADAR-base/ManagementPortal/tree/master/radar-auth
+
 [ManagementPortal]: https://github.com/RADAR-base/ManagementPortal
+
+## Sentry monitoring
+
+To enable Sentry monitoring:
+
+1. Set a `SENTRY_DSN` environment variable that points to the desired Sentry DSN.
+2. (Optional) Set the `SENTRY_LOG_LEVEL` environment variable to control the minimum log level of events sent to Sentry.
+   The default log level for Sentry is `ERROR`. Possible values are `TRACE`, `DEBUG`, `INFO`, `WARN`, and `ERROR`.
+
+For further configuration of Sentry via environmental variables see [here](https://docs.sentry.io/platforms/java/configuration/#configuration-via-the-runtime-environment). For instance:
+
+```
+SENTRY_LOG_LEVEL: 'ERROR'
+SENTRY_DSN: 'https://000000000000.ingest.de.sentry.io/000000000000'
+SENTRY_ATTACHSTACKTRACE: true
+SENTRY_STACKTRACE_APP_PACKAGES: io.confluent.connect.jdbc
+```

build.gradle.kts

@@ -15,15 +15,4 @@ radarRootProject {
     projectVersion.set(Versions.project)
 }
 
-subprojects {
-    apply(plugin = "org.radarbase.radar-kotlin")
-
-    radarKotlin {
-        kotlinVersion.set(Versions.kotlin)
-        javaVersion.set(Versions.java)
-        log4j2Version.set(Versions.log4j2)
-        slf4jVersion.set(Versions.slf4j)
-    }
-}
-
 

buildSrc/src/main/kotlin/Versions.kt

@@ -8,10 +8,9 @@ object Versions {
 
     const val ktor = "2.3.10"
     const val radarJersey = "0.11.1"
-    const val radarCommons = "1.1.2"
+    const val radarCommons = "1.1.3"
     const val radarSchemas = "0.8.9"
     const val jackson = "2.15.3"
-    const val slf4j = "2.0.13"
     const val log4j2 = "2.23.1"
     const val lzfse = "0.1.1"
     const val radarAuth = "2.1.1"

docker-compose.yml

@@ -79,3 +79,8 @@ services:
         - "127.0.0.1:8090:8090"
       volumes:
         - ./gateway.yml:/etc/radar-gateway/gateway.yml
+      # environment:
+      #   SENTRY_LOG_LEVEL: 'ERROR'
+      #   SENTRY_DSN: 'https://000000000000.ingest.de.sentry.io/000000000000'
+      #   SENTRY_ATTACHSTACKTRACE: true
+      #   SENTRY_STACKTRACE_APP_PACKAGES: org.radarbase.gateway

radar-gateway/build.gradle.kts

@@ -3,6 +3,7 @@ import java.time.Duration
 plugins {
     application
     kotlin("plugin.serialization") version Versions.kotlin
+    id("org.radarbase.radar-kotlin") version Versions.radarCommons
     id("com.avast.gradle.docker-compose") version Versions.dockerCompose
 }
 
@@ -95,3 +96,9 @@ dependencies {
     integrationTestImplementation("org.radarbase:radar-schemas-commons:${Versions.radarSchemas}")
     integrationTestImplementation("org.radarbase:radar-commons-testing:${Versions.radarCommons}")
 }
+
+radarKotlin {
+    javaVersion.set(Versions.java)
+    log4j2Version.set(Versions.log4j2)
+    sentryEnabled.set(true)
+}

radar-gateway/src/main/resources/log4j2.xml

@@ -0,0 +1,39 @@
+<?xml version="1.0" encoding="UTF-8" ?>
+<!--
+  ~ /*
+  ~  *  Copyright 2024 The Hyve
+  ~  *
+  ~  *  Licensed under the Apache License, Version 2.0 (the "License");
+  ~  *  you may not use this file except in compliance with the License.
+  ~  *  You may obtain a copy of the License at
+  ~  *
+  ~  *    http://www.apache.org/licenses/LICENSE-2.0
+  ~  *
+  ~  *  Unless required by applicable law or agreed to in writing, software
+  ~  *  distributed under the License is distributed on an "AS IS" BASIS,
+  ~  *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  ~  *  See the License for the specific language governing permissions and
+  ~  *  limitations under the License.
+  ~  */
+  -->
+
+<configuration status="INFO">
+    <appenders>
+        <Console name="Console" target="SYSTEM_OUT">
+            <PatternLayout
+                    pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n"
+            />
+        </Console>
+        <!-- For Sentry to work the DSN must be set via SENTRY_DSN environment variable
+             When SENTRY_DSN is empty string, the Sentry SDK is disabled -->
+        <Sentry name="Sentry" debug="false"/>
+    </appenders>
+
+    <loggers>
+        <root level="INFO">
+            <appender-ref ref="Console" />
+            <!-- Note that the Sentry logging threshold is at ERROR level by default -->
+            <appender-ref ref="Sentry" level="${env:SENTRY_LOG_LEVEL:-ERROR}" />
+        </root>
+    </loggers>
+</configuration>