diff --git a/README.md b/README.md index a0f3bb6..4e0ae28 100644 --- a/README.md +++ b/README.md @@ -1,10 +1,20 @@ # RADAR-Gateway -REST Gateway to the Apache Kafka, similar to the REST Proxy provided by Confluent. In addition, it does authentication and authorization, content validation and decompression if needed. It is available as a [docker image](https://hub.docker.com/r/radarbase/radar-gateway). +REST Gateway to the Apache Kafka, similar to the REST Proxy provided by Confluent. In addition, it does authentication +and authorization, content validation and decompression if needed. It is available as +a [docker image](https://hub.docker.com/r/radarbase/radar-gateway). + + +* [RADAR-Gateway](#radar-gateway) + * [Configuration](#configuration) + * [Usage](#usage) + * [Sentry monitoring](#sentry-monitoring) + ## Configuration -The [RADAR-Auth] library is used for authentication and authorization of users. Refer to the documentation there for a full description of the configuration options. +The [RADAR-Auth] library is used for authentication and authorization of users. Refer to the documentation there for a +full description of the configuration options. ## Usage @@ -21,22 +31,60 @@ TOPIC=test docker-compose exec kafka-1 kafka-topics --create --topic $TOPIC --bootstrap-server kafka-1:9092 ``` -Now the gateway is accessible through and the [ManagementPortal] is available through +Now the gateway is accessible through and the [ManagementPortal] is available +through -The access token should be generated by the aforementioned Management portal. The access token is a JWT (JSON Web Token) that should contain the `MEASUREMENT.CREATE` scope for resource `res_gateway`, and list all applicable sources to submit data for. The gateway does content validation for posted data. It requires to use the Avro format with JSON serialization, using the `application/vnd.kafka.avro.v1+json` or `application/vnd.kafka.avro.v2+json` media types, as described in the [REST Proxy documentation]. It also requires messages to have both a key and a value with schemas. The key should have a `userId` and `sourceId` field. The `userId` should match the `sub` field in the OAuth2 JWT access token. That JWT should also contain a `sources` array claim which should contain the given `sourceId`. Sources can be added in the ManagementPortal or be generated by the app dynamically and then registered with the ManagementPortal. +The access token should be generated by the aforementioned Management portal. The access token is a JWT (JSON Web Token) +that should contain the `MEASUREMENT.CREATE` scope for resource `res_gateway`, and list all applicable sources to submit +data for. The gateway does content validation for posted data. It requires to use the Avro format with JSON +serialization, using the `application/vnd.kafka.avro.v1+json` or `application/vnd.kafka.avro.v2+json` media types, as +described in the [REST Proxy documentation]. It also requires messages to have both a key and a value with schemas. The +key should have a `userId` and `sourceId` field. The `userId` should match the `sub` field in the OAuth2 JWT access +token. That JWT should also contain a `sources` array claim which should contain the given `sourceId`. Sources can be +added in the ManagementPortal or be generated by the app dynamically and then registered with the ManagementPortal. Now you can access the gateway: + ```shell TOKEN= curl -H "Authorization: Bearer $TOKEN" http://localhost:8090/radar-gateway/topics ``` -Data compressed with GZIP is decompressed if the `Content-Encoding: gzip` header is present. With `curl`, use the `-H "Content-Encoding: gzip" --data-binary @data.json.gz` flags. It can be activated in `radar-commons` Java `RestClient` by setting `RestClient.Builder.gzipCompression(true)`. Likewise it accepts Apple LZFSE encoded data by adding the header `Content-Encoding: lzfse`. +Data compressed with GZIP is decompressed if the `Content-Encoding: gzip` header is present. With `curl`, use the +`-H "Content-Encoding: gzip" --data-binary @data.json.gz` flags. It can be activated in `radar-commons` Java +`RestClient` by setting `RestClient.Builder.gzipCompression(true)`. Likewise it accepts Apple LZFSE encoded data by +adding the header `Content-Encoding: lzfse`. Otherwise, it accepts all the same Avro messages and headers as specified in the Kafka [REST Proxy documentation]. -Finally, the gateway accepts a custom binary format for data ingestion. The data must follow the binary Avro serialization of the [RecordSet schema](https://github.com/RADAR-base/RADAR-Schemas/blob/master/commons/kafka/record_set.avsc). Data in this format can be posted by using the content type `application/vnd.radarbase.avro.v1+binary`. It will construct an `ObservationKey` based on the user data in the `RecordSet`, and read the binary data values using the schema version provided in the `RecordSet`. This data sending mode can be activated in Java by using radar-commons `RestSender.Builder.useBinaryContent(true)`. Using binary mode has the added benefit of having a much more efficient GZIP encoding for many datasets. +Finally, the gateway accepts a custom binary format for data ingestion. The data must follow the binary Avro +serialization of +the [RecordSet schema](https://github.com/RADAR-base/RADAR-Schemas/blob/master/commons/kafka/record_set.avsc). Data in +this format can be posted by using the content type `application/vnd.radarbase.avro.v1+binary`. It will construct an +`ObservationKey` based on the user data in the `RecordSet`, and read the binary data values using the schema version +provided in the `RecordSet`. This data sending mode can be activated in Java by using radar-commons +`RestSender.Builder.useBinaryContent(true)`. Using binary mode has the added benefit of having a much more efficient +GZIP encoding for many datasets. [REST Proxy documentation]: https://docs.confluent.io/current/kafka-rest/api.html + [RADAR-Auth]: https://github.com/RADAR-base/ManagementPortal/tree/master/radar-auth + [ManagementPortal]: https://github.com/RADAR-base/ManagementPortal + +## Sentry monitoring + +To enable Sentry monitoring: + +1. Set a `SENTRY_DSN` environment variable that points to the desired Sentry DSN. +2. (Optional) Set the `SENTRY_LOG_LEVEL` environment variable to control the minimum log level of events sent to Sentry. + The default log level for Sentry is `ERROR`. Possible values are `TRACE`, `DEBUG`, `INFO`, `WARN`, and `ERROR`. + +For further configuration of Sentry via environmental variables see [here](https://docs.sentry.io/platforms/java/configuration/#configuration-via-the-runtime-environment). For instance: + +``` +SENTRY_LOG_LEVEL: 'ERROR' +SENTRY_DSN: 'https://000000000000.ingest.de.sentry.io/000000000000' +SENTRY_ATTACHSTACKTRACE: true +SENTRY_STACKTRACE_APP_PACKAGES: io.confluent.connect.jdbc +``` diff --git a/build.gradle.kts b/build.gradle.kts index 1bb7b1c..569683f 100644 --- a/build.gradle.kts +++ b/build.gradle.kts @@ -15,15 +15,4 @@ radarRootProject { projectVersion.set(Versions.project) } -subprojects { - apply(plugin = "org.radarbase.radar-kotlin") - - radarKotlin { - kotlinVersion.set(Versions.kotlin) - javaVersion.set(Versions.java) - log4j2Version.set(Versions.log4j2) - slf4jVersion.set(Versions.slf4j) - } -} - diff --git a/buildSrc/src/main/kotlin/Versions.kt b/buildSrc/src/main/kotlin/Versions.kt index f1e5d37..f81ab37 100644 --- a/buildSrc/src/main/kotlin/Versions.kt +++ b/buildSrc/src/main/kotlin/Versions.kt @@ -8,10 +8,9 @@ object Versions { const val ktor = "2.3.10" const val radarJersey = "0.11.1" - const val radarCommons = "1.1.2" + const val radarCommons = "1.1.3" const val radarSchemas = "0.8.9" const val jackson = "2.15.3" - const val slf4j = "2.0.13" const val log4j2 = "2.23.1" const val lzfse = "0.1.1" const val radarAuth = "2.1.1" diff --git a/docker-compose.yml b/docker-compose.yml index e740894..858ec98 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -79,3 +79,8 @@ services: - "127.0.0.1:8090:8090" volumes: - ./gateway.yml:/etc/radar-gateway/gateway.yml + # environment: + # SENTRY_LOG_LEVEL: 'ERROR' + # SENTRY_DSN: 'https://000000000000.ingest.de.sentry.io/000000000000' + # SENTRY_ATTACHSTACKTRACE: true + # SENTRY_STACKTRACE_APP_PACKAGES: org.radarbase.gateway diff --git a/radar-gateway/build.gradle.kts b/radar-gateway/build.gradle.kts index 35c9a3a..65a1da7 100644 --- a/radar-gateway/build.gradle.kts +++ b/radar-gateway/build.gradle.kts @@ -3,6 +3,7 @@ import java.time.Duration plugins { application kotlin("plugin.serialization") version Versions.kotlin + id("org.radarbase.radar-kotlin") version Versions.radarCommons id("com.avast.gradle.docker-compose") version Versions.dockerCompose } @@ -95,3 +96,9 @@ dependencies { integrationTestImplementation("org.radarbase:radar-schemas-commons:${Versions.radarSchemas}") integrationTestImplementation("org.radarbase:radar-commons-testing:${Versions.radarCommons}") } + +radarKotlin { + javaVersion.set(Versions.java) + log4j2Version.set(Versions.log4j2) + sentryEnabled.set(true) +} diff --git a/radar-gateway/src/main/resources/log4j2.xml b/radar-gateway/src/main/resources/log4j2.xml new file mode 100644 index 0000000..71d9c20 --- /dev/null +++ b/radar-gateway/src/main/resources/log4j2.xml @@ -0,0 +1,39 @@ + + + + + + + + + + + + + + + + + + + +