diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 000000000..eb0794c7c --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,145 @@ + +# Contributing to RADAR-Kubernetes + +First off, thanks for taking the time to contribute! ❤️ + +All types of contributions are encouraged and valued. See the [Table of Contents](#table-of-contents) for different ways to help and details about how this project handles them. Please make sure to read the relevant section before making your contribution. It will make it a lot easier for us maintainers and smooth out the experience for all involved. The community looks forward to your contributions. 🎉 + +> And if you like the project, but just don't have time to contribute, that's fine. There are other easy ways to support the project and show your appreciation, which we would also be very happy about: +> - Star the project +> - Tweet about it +> - Refer this project in your project's readme +> - Mention the project at local meetups and tell your friends/colleagues + + +## Table of Contents + +- [I Have a Question](#i-have-a-question) +- [I Want To Contribute](#i-want-to-contribute) + - [Reporting Bugs](#reporting-bugs) + - [Suggesting Enhancements](#suggesting-enhancements) + - [Your First Code Contribution](#your-first-code-contribution) + - [Improving The Documentation](#improving-the-documentation) +- [Styleguides](#styleguides) + - [Commit Messages](#commit-messages) +- [Join The Project Team](#join-the-project-team) + + + +## I Have a Question + +> If you want to ask a question, we assume that you have read the available [Documentation](https://github.com/RADAR-base/RADAR-Kubernetes/blob/main/README.md). + +Before you ask a question, it is best to search for existing [Issues](https://github.com/RADAR-base/RADAR-Kubernetes/issues) and [Discussions](https://github.com/RADAR-base/RADAR-Kubernetes/discussions) that might help you. In case you have found a suitable issue and still need clarification, you can write your question in this issue. + +If you then still feel the need to ask a question and need clarification, we recommend the following: + +- Open a [Discussion](https://github.com/RADAR-base/RADAR-Kubernetes/discussions/new/choose). +- Provide as much context as you can about what you're running into. +- Provide project and platform versions (nodejs, npm, etc), depending on what seems relevant. + +We will then take care of the issue as soon as possible. + +## I Want To Contribute + +> ### Legal Notice +> When contributing to this project, you must agree that you have authored 100% of the content, that you have the necessary rights to the content and that the content you contribute may be provided under the project license. + +### Reporting Bugs + + +#### Before Submitting a Bug Report + +A good bug report shouldn't leave others needing to chase you up for more information. Therefore, we ask you to investigate carefully, collect information and describe the issue in detail in your report. Please complete the following steps in advance to help us fix any potential bug as fast as possible. + +- Make sure that you are using the latest version. +- Determine if your bug is really a bug and not an error on your side e.g. using incompatible environment components/versions (Make sure that you have read the [documentation](https://github.com/RADAR-base/RADAR-Kubernetes/blob/main/README.md). If you are looking for support, you might want to check [this section](#i-have-a-question)). +- To see if other users have experienced (and potentially already solved) the same issue you are having, check if there is not already a bug report existing for your bug or error in the [bug tracker](https://github.com/RADAR-base/RADAR-Kubernetes/issues?q=label%3Abug). + + +#### How Do I Submit a Good Bug Report? + +> You must never report security related issues, vulnerabilities or bugs including sensitive information to the issue tracker, or elsewhere in public. Instead sensitive bugs must be sent by email to radar-base@kcl.ac.uk and radar-base@thehyve.nl. + +We use GitHub issues to track bugs and errors. If you run into an issue with the project: + +- Open an [Issue](https://github.com/RADAR-base/RADAR-Kubernetes/issues/new). +- Explain the behavior you would expect and the actual behavior. +- Please provide as much context as possible and describe the *reproduction steps* that someone else can follow to recreate the issue on their own. This usually includes your code. For good bug reports you should isolate the problem and create a reduced test case. +- Provide the information you collected in the previous section. + +Once it's filed: + +- The project team will label the issue accordingly. +- A team member will try to reproduce the issue with your provided steps. If there are no reproduction steps or no obvious way to reproduce the issue, the team will ask you for those steps and mark the issue as `needs-repro`. Bugs with the `needs-repro` tag will not be addressed until they are reproduced. + + +### Suggesting Enhancements + +This section guides you through submitting an enhancement suggestion for RADAR-Kubernetes, **including completely new features and minor improvements to existing functionality**. Following these guidelines will help maintainers and the community to understand your suggestion and find related suggestions. + + +#### Before Submitting an Enhancement + +- Make sure that you are using the latest version. +- Read the [documentation](https://github.com/RADAR-base/RADAR-Kubernetes/blob/main/README.md) carefully and find out if the functionality is already covered, maybe by an individual configuration. +- Perform a [search](https://github.com/RADAR-base/RADAR-Kubernetes/issues) to see if the enhancement has already been suggested. If it has, add a comment to the existing issue instead of opening a new one. +- Find out whether your idea fits with the scope and aims of the project. Keep in mind that we want features that will be useful to the majority of our users and not just a small subset. + + +#### How Do I Submit a Good Enhancement Suggestion? + +Enhancement suggestions are tracked as [GitHub issues](https://github.com/RADAR-base/RADAR-Kubernetes/issues). + +- Use a **clear and descriptive title** for the issue to identify the suggestion. +- Provide a **step-by-step description of the suggested enhancement** in as many details as possible. +- **Describe the current behavior** and **explain which behavior you expected to see instead** and why. At this point you can also tell which alternatives do not work for you. +- **Explain why this enhancement would be useful** to most RADAR-Kubernetes users. You may also want to point out the other projects that solved it better and which could serve as inspiration. + +### Your First Code Contribution + +In order to contribute to the repository, first, you need to make sure that you have: +- Read the README.md +- Have a good knowledge of Kubernetes, Helm and Helmfile as listed in [How to get started with tools around RADAR-Kubernetes](https://radar-base.atlassian.net/wiki/spaces/RAD/pages/2731638785/How+to+get+started+with+tools+around+RADAR-Kubernetes) +- Have a working installation of RADAR-Base for testing your changes + +Then you can make a new fork or branch and make your changes there and after you have tested it and added neccessarry documentation create a pull request to `dev` branch. We test the installation automatically on supported Kubernetes versions so make sure that the Github actions run successfully. The changes will be reviewed and once merged, they'll be released in the next cycle. +If you're changing an existing code, make sure that it is either backwards compatible or the documentation shows a clear path of applying the changes without breaking the existing installations. + + +#### Adding a new component to RADAR-Kuberentes +In order to add a new component you first need to add its helm chart to [radar-helm-charts)](https://github.com/RADAR-base/radar-helm-charts) repository. Refer to contributing guidelines of that repository for more information. Once the chart has been added you need to: +- Add a helmfile for it in `helmfile.d` directory. The helmfiles are seperated in a modular way in order to avoid having a huge file and also installing certain components in order. Have a look at the current helmfiles and if your component is related to one of them add your component in that file other file create a new file. If your component is a dependency to other components, like Kafka or PostgreSQL prefix the file name with a smaller number so it will be installed first, but if it's a standalone component, the prefix number can be higher. +- Add release to helmfile. Depending on the helm chart this can mostly be copy pasted from other releases and change names to your component. If you've added custom values files in `etc` directory make sure to reference them in the release. +- Add a basic configuration of it to `etc/base.yaml` which should include at least `_install`, `_chart_version` and `_extra_timeout` values. In order to keep the `base.yaml` short, only add configurations that the user will most likely to change during installation. +- If your component is dealing with credentials, the values in the helm charts that refer to that has to be added to `etc/base-secrets.yaml` file. + - If the credentials isn't something external and can be auto-generated be sure to add it to `bin/generate-secrets`, following examples of the current credentials +- If the user has to input a file to the helm chart, add the relavant key to the `base.yaml.gotmpl` file. +- If the component that you're adding is an external component and you want it to have some default configuration, create a folder with its name in `etc` directory and add the default configuration there in a YAML file and refer to that configuration in the helmfile of the component. + +#### Testing the changes +In order to test the changes locally you can use helmfile command to install the component in your cluster. You can make installation faster if you only select your component to install: +``` +helmfile apply --file helmfile.d/name-of-the-helmfile.yaml --selector name=name-of-the-component +``` +You can also use other the helmfile commands like `helmfile template` and `helmfile diff` to see what is being applied to the cluster. + + +### Improving The Documentation + +Feel free to make a PR for any part of the documentation that is missing or isn't clear. If the documentation in question is in the wiki send an email to radar-base@kcl.ac.uk and radar-base@thehyve.nl so we can create an account for you to edit the documentation. + + +## Join The Project Team + +It's highly recommended to join the [RADAR-Base slack community](https://docs.google.com/forms/d/e/1FAIpQLScKNZ-QonmxNkekDMLLbP-b_IrNHyDRuQValBy1BAsLOjEFpg/viewform) in order to be involved with the community. You can joing #radar-kubernetes to discuss with other developers and attend weekly development meetings. + + +## Attribution +This guide is based on the **contributing-gen**. [Make your own](https://github.com/bttger/contributing-gen)! diff --git a/README.md b/README.md index bac89b236..a4cc030ba 100644 --- a/README.md +++ b/README.md @@ -11,48 +11,51 @@ The Kubernetes stack of RADAR-base platform. ## Table of contents - - + - [About](#about) - [Status](#status) - - [Compatibility](#compatibility) - - [Disclaimer](#disclaimer) - [Prerequisites](#prerequisites) - - [Hosting Infrastructure](#hosting-infrastructure) - - [Local machine](#local-machine) + * [Knowledge requirements ](#knowledge-requirements) + * [Software Compatibility](#software-compatibility) + * [Hosting ](#hosting) + * [Third party services](#third-party-services) + * [Local machine](#local-machine) - [Installation](#installation) - - [Prepare](#prepare) - - [Configure](#configure) - - [Project Structure](#project-structure) - - [Install](#install) - - [Install RADAR-Kubernetes on your cluster.](#install-radar-kubernetes-on-your-cluster) - - [Monitor and verify the installation process.](#monitor-and-verify-the-installation-process) - - [Ensure Kafka cluster is functional and RADAR-base topics are loaded](#ensure-kafka-cluster-is-functional-and-radar-base-topics-are-loaded) - - [Troubleshoot](#troubleshoot) - - [Optional](#optional) + * [Prepare](#prepare) + * [Project Structure](#project-structure) + * [Configure](#configure) + * [Install](#install) +- [Usage and accessing the applications](#usage-and-accessing-the-applications) +- [Troubleshooting](#troubleshooting) - [Upgrade instructions](#upgrade-instructions) - - [Upgrade to RADAR-Kubernetes version 1.0.0](#upgrade-to-radar-kubernetes-version-100) -- [Usage](#usage) - - [Accessing the applications](#accessing-the-applications) + * [Upgrade to RADAR-Kubernetes version 1.1.x](#upgrade-to-radar-kubernetes-version-11x) + * [Upgrade to RADAR-Kubernetes version 1.0.0](#upgrade-to-radar-kubernetes-version-100) - [Volume expansion](#volume-expansion) - [Uninstall](#uninstall) - [Update charts](#update-charts) - [Feedback and Contributions](#feedback-and-contributions) - + ## About RADAR-base is an open-source platform designed to support remote clinical trials by collecting continuous data from wearables and mobile applications. RADAR-Kubernetes enables installing the RADAR-base platform onto Kubernetes clusters. RADAR-base platform can be used for wide range of use-cases. Depending on the use-case, the selection of applications need to be installed can vary. Please read the [component overview and breakdown](https://radar-base.atlassian.net/wiki/spaces/RAD/pages/2673967112/Component+overview+and+breakdown) to understand the role of each component and how components work together. -RADAR-Kubernetes setup uses [Helm 3](https://github.com/helm/helm) charts to package necessary Kubernetes resources for each component and [helmfile](https://github.com/roboll/helmfile) to modularize and deploy Helm charts of the platform on a Kubernetes cluster. This setup is designed to be a lightweight way to install and configure the RADAR-base components. The original images or charts may provide more and granular configurations. Please visit the `README` of respective charts in [radar-helm-charts](https://github.com/RADAR-base/radar-helm-charts) to understand the configurations and visit the main repository for in depth knowledge. +RADAR-Kubernetes setup uses [Helm](https://github.com/helm/helm) charts to package necessary Kubernetes resources for each component and [helmfile](https://github.com/roboll/helmfile) to modularize and deploy Helm charts of the platform on a Kubernetes cluster. This setup is designed to be a lightweight way to install and configure the RADAR-base components. The original images or charts may provide more and granular configurations. Please visit the `README` of respective charts in [radar-helm-charts](https://github.com/RADAR-base/radar-helm-charts) to understand the configurations and visit the main repository for in depth knowledge. ## Status RADAR-Kubernetes is one of the youngest project of RADAR-base and will be the **long term supported form of deploying the platform**. Even though, RADAR-Kubernetes is being used in few production environments, it is still in its early stage of development. We are working on improving the set up and documentation to enable RADAR-base community to make use of the platform. -### Compatibility +## Prerequisites + +### Knowledge requirements + +This documentation assumes familiarity with all referenced Kubernetes concepts, utilities, and procedures and familiarity with Helm charts and helmfile, depending on your environment you might need to have knowledege of other hosting infrastructure such as DNS and mail servers as well. +While this documentation will provide guidance for installing and configuring RADAR-base platform on a Kubernetes cluster and tries to make is as simple and possible, it is not a replacement for the detailed knowledege of the tools that have been used. If you are not familiar with these tools, we strongly recommend you to get familiar with these tools. Here is a [list of useful links](https://radar-base.atlassian.net/wiki/spaces/RAD/pages/2731638785/How+to+get+started+with+tools+around+RADAR-Kubernetes) to get started. + +### Software Compatibility Currently RADAR-Kubernetes is tested and supported on following component versions: | Component | Version | @@ -67,21 +70,26 @@ Currently RADAR-Kubernetes is tested and supported on following component versio It's possible to install RADAR-Kubernetes on different version of tools as well, but you might encounter compatibility issues. Make sure that `kubectl` version matches or it's higher than Kubernetes or K3s version that you're using. For other tools such as Git or Java the version, as long as it's not very old, it's not very impactful. -### Disclaimer - -This documentation assumes familiarity with all referenced Kubernetes concepts, utilities, and procedures and familiarity with Helm charts and helmfile. While this documentation will provide guidance for installing and configuring RADAR-base platform on a Kubernetes cluster, it is not a replacement for the official detailed documentation or tutorial of Kubernetes, Helm or Helmfile. If you are not familiar with these tools, we strongly recommend you to get familiar with these tools. Here is a [list of useful links](https://radar-base.atlassian.net/wiki/spaces/RAD/pages/2731638785/How+to+get+started+with+tools+around+RADAR-Kubernetes) to get started. - -## Prerequisites - -### Hosting Infrastructure +### Hosting +Kubernetes can be installed on wide varaity of platforms and in turn you can install RADAR-Base on most places that Kuberentes can run. However your infrastructure needs to have a set up requirements listed below: | Component | Description | Required | | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------- | | Kubernetes cluster | An infrastructure with working installation of Kubernetes services. Read [this article](https://radar-base.atlassian.net/wiki/spaces/RAD/pages/2744942595?draftShareId=e09429e8-38c8-4b71-955d-5df8de94b694) for available options. Minimum requirements for a single node: 8 vCPU's, 32 GB memory, 200 GB storage. Minimum requirements for a cluster: 3 nodes with 3 vCPUs, 16 GB memory, 100 GB storage each and 200 GB shared storage. | Required | | DNS Server | Some applications are only accessible via HTTPS and it's essential to have a DNS server via providers like GoDaddy, Route53, etc | Required | | SMTP Server | RADAR-Base needs an SMTP server to send registration email to researchers and participants. | Required | +| Whitelisted acces to ports 80 and 443 | We use Let's Encrypt to create SSL certificates and in the default configuration we use HTTP challenge. This means that the RADAR-Base installation needs to be visible to Let's Encrypt servers for the verification, so make sure these ports are white listed in your firewall. If you want to have a private installation you should change Let's Encrypt configuration to use DNS challenge. | Required | | Object storage | An external object storage allows RADAR-Kubernetes to backup cluster data such as manifests, application configuration and data via Velero to a backup site. You can also send the RADAR-Base output data to this object storage, which can provider easier management and access compared to bundled Minio server inside RADAR-Kubernetes. | Optional | -| Managed services | RADAR-Kubernetes includes all necessary components to run the platform as a standalone application. However, you can also opt to use managed services such as with the platform, e.g. Confluent cloud for Kafka and schema registry, Postgres DB for storage, Azure blob storage or AWS S3 instead of minio. | Optional | +| Managed services | RADAR-Kubernetes includes all necessary components to run the platform as a standalone application. However, you can also opt to use managed services such as with the platform, e.g. Confluent cloud for Kafka and schema registry, Postgres DB for storage, Azure blob storage or AWS S3 instead of Minio. If you're using a managed object storage that you have to pay per request (such as AWS S3), it's recommeneded that to install the Minio just for the `radar-intermediate-storage` since the applications send a lot of API calls to that bucket. | Optional | + +In order to have a simple single node Kubernetes server you can run these commands on a Linux server: +```shell +curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION="v1.26.3+k3s1" K3S_KUBECONFIG_MODE="644" INSTALL_K3S_SYMLINK="skip" sh -s - --disable traefik --disable-helm-controller +``` + + +### Third party services +Depending on which components you've enabled you might need credentials for Fitbit, REDCap, Google Firebase, etc. You need to provide them in order for the respective component to work properly. ### Local machine @@ -121,26 +129,21 @@ The following tools should be installed in your local machine to install the RAD It is recommended make a private clone of this repository, if you want to version control your configurations and/or share with other people. -**You must keep `etc/secrets.yaml` secure and confidential once you have started installing the platform** and the best practice to share your platform configurations is by **sharing the encrypted version of `etc/secrets.yaml`**. - -### Configure +**You must keep `etc/secrets.yaml` secure and confidential once you have started installing the platform** and the best practice to share your platform configurations is by **sharing the encrypted version of `etc/secrets.yaml`, this can be done via the combination of [sops](https://github.com/getsops/sops) and [helm-secrets](https://github.com/jkroepke/helm-secrets) but it's outside the scope of this document**. -#### Project Structure +### Project Structure -- [/bin](bin): Contains initialization scripts -- [/etc](etc): Contains configurations for some Helm charts. -- [/secrets](secrets): Contains secrets configuration for helm charts. -- [/helmfile.d](helmfile.d): Contains Helmfiles for modular deployment of the platform -- [environments.yaml](environments.yaml): Defines current environment files in order to be used by helmfile. Read more about `bases` [here](https://github.com/roboll/helmfile/blob/master/docs/writing-helmfile.md). +- `bin/`: Contains initialization scripts. +- `etc/`: Contains configurations for the Helm charts. +- `helmfile.d/`: Contains Helmfiles for modular deployment of the platform. +- `environments.yaml/`: Defines current environment files in order to be used by helmfile and where to find the configuration files. - `etc/production.yaml`: Production helmfile template to configure and install RADAR-base components. Inspect the file to enable, disable and configure components required for your use case. The default helmfile enables all core components that are needed to run RADAR-base platform with pRMT and aRMT apps. If you're not sure which components you want to enable you can refer to wiki for [an overview and breakdown on RADAR-Base components and their roles](https://radar-base.atlassian.net/wiki/spaces/RAD/pages/2673967112/Component+overview+and+breakdown). -- `etc/production.yaml.gotmpl`: Change setup parameters that require Go templating, such as reading input files +- `etc/production.yaml.gotmpl`: Some helm charts need an external file during installation, you should put those files in the specifed path and uncomment the respective lines. - `etc/secrets.yaml`: Passwords and client secrets used by the installation. -1. Configure the [environments.yaml](environments.yaml) to use the files that you have created by copying the template files. - ```shell - nano environments.yaml # use the files you just created - ``` -2. Configure the `etc/production.yaml`. Optionally, you can also enable or disable other components that are configured otherwise by default. +### Configure + +1. Configure the `etc/production.yaml`. Make sure to read the comments in the file and change the values that are relevant to your installation. You at least want to change the `server_name` and `management_portal.smtp` configuration. Optionally, you can also enable or disable other components that are configured otherwise by default. ```shell nano etc/production.yaml # Change setup parameters and configurations @@ -148,17 +151,20 @@ It is recommended make a private clone of this repository, if you want to versio When doing a clean install, you are advised to change the `postgresql`, `radar_appserver_postgresql` `radar_upload_postgresql` image tags to the latest PostgreSQL version. Likewise, the timescaledb image tag should use the latest timescaledb version. PostgreSQL passwords and major versions cannot easily be updated after installation. -3. In `etc/production.yaml.gotmpl` file, change setup parameters that require Go templating, such as reading input files and selecting an option for the `keystore.p12` +3. In `etc/production.yaml.gotmpl` file, change setup parameters for charts that are reading input files. You most likely just want to put the file in the default location specified in the file and uncomment the respective lines. Make sure to remove both `#` and `{{/*` from the line in order to uncomment it. ```shell nano etc/production.yaml.gotmpl ``` -4. In `etc/secrets.yaml` file, change any passwords, client secrets or API credentials like for Fitbit or Garmin Connect. +4. (Optional) If you are installing `radar-appserver`, it needs to be authorized with the Google Firebase also used by the aRMT / Questionnaire app. In Firebase, go to _Project settings_ -> _Service accounts_ and download a Firebase Admin SDK private key. Store the generated key as `etc/radar-appserver/firebase-adminsdk.json` and uncomment the respective section in `etc/production.yaml.gotmpl`. + +5. In `etc/secrets.yaml` file, change any passwords, client secrets or API credentials like for Fitbit or Garmin Connect. After the installation you can find login credentials to the components in this file. Be sure to keep it private. ```shell nano etc/secrets.yaml ``` + ### Install Once all configuration files are ready, the RADAR-Kubernetes can be deployed on a Kubernetes cluster. @@ -169,12 +175,12 @@ Once all configuration files are ready, the RADAR-Kubernetes can be deployed on helmfile sync --concurrency 1 ``` -The `helmfile sync` will synchronize all the Kubernetes resources defined in the helmfiles with your Kubernetes cluster. Having `--concurrency 1` will make sure components are installed in required order. Depending on your cluster specification, this may take a few minutes when installed for the first time. +The `helmfile sync` will synchronize all the Kubernetes resources defined in the helmfiles with your Kubernetes cluster. Having `--concurrency 1` will make sure components are installed in required order. Depending on your cluster specification, this may take around 30 minutes when installed for the first time. | :exclamation: Note | | :----------------- | -| Installing the stack with `--concurrency 1` may make the installation slower. However, it is necessary because some components such as `kube-prometheus-stack` and `kafka-init` (aka `catalog-server`) should be installed in their specified order. If you've forgotten to use this flag, then the installation may not be successful. To continue, follow [Uninstallation](#uninstall) steps to clean up the Kubernetes cluster before you can try again. +| Installing the stack with `--concurrency 1` may make the installation slower. However, it is necessary because some components such as `kube-prometheus-stack` and `catalog-server`, should be installed in their specified order. If you've forgotten to use this flag, then the installation may not be successful. To continue, follow [Uninstallation](#uninstall) steps to clean up the Kubernetes cluster before you can try again. Graylog and fluent-bit services in the `graylog` namespace will not immediately be operational, first it needs an input source defined. Log into `graylog.` with the Graylog credentials. Then navigate to _System_ -> _Inputs_, select GELF TCP in the dropdown and _Launch new input_. Set it as a global input on port 12222. @@ -182,9 +188,9 @@ Graylog and fluent-bit services in the `graylog` namespace will not immediately Once the installation is done or in progress, you can check the status using `kubectl get pods`. -If the installation has been successful, you should see an output similar to the list below. +If the installation has been successful, you should see an output similar to the list below. However depending on which components that you've enabled for installation this list and be longer or shorter. -``` +```shell ➜ kubectl get pods NAME READY STATUS RESTARTS AGE catalog-server-5c6767cbd8-dc6wc 1/1 Running 0 8m21s @@ -197,14 +203,12 @@ cp-schema-registry-7968b7c554-sz6w2 2/2 Running 0 8m cp-zookeeper-0 2/2 Running 0 8m21s cp-zookeeper-1 2/2 Running 0 8m21s cp-zookeeper-2 2/2 Running 0 8m21s -kafka-manager-6858986866-qhphj 1/1 Running 0 8m21s management-portal-56cd7f88c6-vmqfk 1/1 Running 0 8m21s minio-0 1/1 Running 0 8m21s minio-1 1/1 Running 0 8m21s minio-2 1/1 Running 0 8m21s minio-3 1/1 Running 0 8m21s nginx-ingress-controller-748f5b5b88-9j882 1/1 Running 0 8m21s -nginx-ingress-default-backend-659bd647bd-kk922 1/1 Running 0 8m21s postgresql-0 3/3 Running 0 8m21s radar-fitbit-connector-594d8b668c-h8m4d 2/2 Running 0 8m21s radar-gateway-5c4b8c8645-c8zrh 2/2 Running 0 8m21s @@ -225,7 +229,7 @@ timescaledb-postgresql-0 3/3 Running 0 8m If you have enabled monitoring, logging and HTTPS you should see these as well: -``` +```shell ➜ kubectl -n monitoring get pods NAME READY STATUS RESTARTS AGE kube-prometheus-stack-grafana-674bb6887f-2pgxh 2/2 Running 0 8m29s @@ -234,10 +238,6 @@ kube-prometheus-stack-operator-7d456878d7-bwrsx 1/1 Running 0 kube-prometheus-stack-prometheus-node-exporter-84n2m 1/1 Running 0 8m29s kube-prometheus-stack-prometheus-node-exporter-h5kgc 1/1 Running 0 8m29s kube-prometheus-stack-prometheus-node-exporter-p6mkb 1/1 Running 0 8m29s -kube-prometheus-stack-prometheus-node-exporter-tmsk7 1/1 Running 0 8m29s -kube-prometheus-stack-prometheus-node-exporter-vvk6d 1/1 Running 0 8m29s -kube-prometheus-stack-prometheus-node-exporter-wp2t7 1/1 Running 0 8m29s -kube-prometheus-stack-prometheus-node-exporter-zsls7 1/1 Running 0 8m29s prometheus-kube-prometheus-stack-prometheus-0 2/2 Running 1 8m21s prometheus-kube-prometheus-stack-prometheus-1 2/2 Running 1 8m21s prometheus-kube-prometheus-stack-prometheus-2 2/2 Running 1 8m21s @@ -250,12 +250,6 @@ elasticsearch-master-2 1/1 Running 0 8m21s fluentd-6jmhn 1/1 Running 0 8m21s fluentd-9lc2g 1/1 Running 0 8m21s fluentd-cfzqv 1/1 Running 0 8m21s -fluentd-g88cr 1/1 Running 0 8m21s -fluentd-ks5zx 1/1 Running 0 8m21s -fluentd-mdg8p 1/1 Running 0 8m21s -fluentd-qnn8b 1/1 Running 0 8m21s -fluentd-x4vjd 1/1 Running 0 8m21s -fluentd-zwzfw 1/1 Running 0 8m21s graylog-0 1/1 Running 0 8m21s graylog-1 1/1 Running 0 8m21s mongodb-mongodb-replicaset-0 2/2 Running 0 8m21s @@ -269,11 +263,11 @@ cert-manager-cainjector-75b6bc7b8b-dv2js 1/1 Running 0 8m21s cert-manager-webhook-8444c4bc77-jhzgb 1/1 Running 0 8m21s ``` -Other ways to ensure that installation have been successful is to check application logs for errors and exceptions. +In most cases seeing `1/1` or `2/2` in `READY` column and `Running` in `STATUS` column indicates that the application is running and healthy. Other ways to ensure that installation have been successful is to check application logs for errors and exceptions. #### Ensure Kafka cluster is functional and RADAR-base topics are loaded -```bash +```shell ➜ kubectl exec -it cp-kafka-0 -c cp-kafka-broker -- kafka-topics --bootstrap-server localhost:9092 --list | wc -l 273 ``` @@ -300,31 +294,75 @@ pod=$(kubectl get pods --selector=app=cp-schema-registry -o jsonpath="{.items[0] kubectl exec -it $pod -c cp-schema-registry-server -- sh -c "$command --topic $topic $args" ``` -#### Troubleshoot +## Usage and accessing the applications + +In order to access to the applications first you need to find the IP address that Nginx service is listening to and then point the domain that you've specified in `server_name` variable to this IP address via a DNS server (e.g. [Route53](https://aws.amazon.com/route53/), [Cloudflare](https://www.cloudflare.com/dns/), [Bind](https://www.isc.org/bind/)) or [`hosts` file]() in your local machine. + +> For this guide we assume that you've set `server_name` to "k8s.radar-base.org" and SSL is enabled. Please replace it with a DNS domain under your control. + +You can see details of Nginx service with following command: -If an application doesn't become fully ready installation will not be successful. In this case, you should investigate the root cause by investigating the relevant component. +```shell +➜ kubectl get service nginx-ingress-controller +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE +nginx-ingress-controller LoadBalancer 10.100.237.75 XXXX.eu-central-1.elb.amazonaws.com 80:31046/TCP,443:30932/TCP 1h +``` + +- If you're using a cloud provider you need to point the value in `EXTERNAL-IP` column (in this example `XXXX.eu-central-1.elb.amazonaws.com`) to `k8s.radar-base.org` domain in your DNS server. +- Some of the RADAR-base applications are accesible through sub-domains and you need to configure the DNS server to allow access to those applications. The easy way to do this is to create two wildcard CNAME records: + ``` + *.k8s.radar-base.org IN CNAME k8s.radar-base.org + *.*.k8s.radar-base.org IN CNAME k8s.radar-base.org + ``` +- If you're not using a cloud provider you need to use a load balancer to expose `31046` and `30932` ports (will be different in your setup) to a IP address and then point `k8s.radar-base.org` domain to that IP address. +- For development and testing purposes you can run `sudo kubectl port-forward svc/nginx-ingress-controller 80:80 443:443` which will forward Nginx service ports to your local machine and you can have access to applications after adding `127.0.0.1 k8s.radar-base.org` to your `hosts` file. + +Now when you go to this IP address you should see a home page with a few links to applications that are installed in the cluster: + +``` +https://k8s.radar-base.org +``` + +**Note:** If you have enabled the SSL you might see invalid certificate error when you try to access to the websites, in this case wait a couple of minutes until `cert-manager` issues those certificates. + +Now you can head over to the [Management Portal](https://radar-base.atlassian.net/wiki/spaces/RAD/pages/49512484/Management+Portal) guide for next steps. + + +## Troubleshooting + +If an application doesn't become fully ready, installation will not be successful. In this case, you should investigate the root cause by investigating the relevant component. It's suggested to run the following command when `helmfile sync` command is running so you can keep an eye on the installation: +```shell +# on linux +watch kubectl get pods + +# on other platforms +kubectl get pods --watch +``` +This can help you to findout potential issues faster. + +It is suggested to change value of `atomicInstall` to `false` in `etc/production.yaml` file during the installation. This will help troubleshooting potential installation issues easier since it will leave the broken components in place for further inspection, be sure to enable this flag after the installation to prevent broken components causing distruption in case of a faulty update. Some useful commands for troubleshooting a component are mentioned below. -1. Describe a pod to understand current status +- Describe a pod to understand current status: ```shell kubectl describe pods ``` -2. Investigate the logs of the pod +- Investigate the logs of the pod: ```shell kubectl logs ``` -To check last few lines +- To check last few lines: ```shell kubectl logs --tail 100 ``` -To continue monitoring the logs +- To continuously monitor the logs: ```shell kubectl logs -f @@ -335,11 +373,10 @@ For more information on how `kubectl` can be used to manage a Kubernetes applica |--------------------| | For most of the components, you can reinstall them without additional actions. However, for some components such as `kube-prometheus-stack` and `kafka-init`, you may need to remove everything before trying again.| -If you have enabled monitoring you should also check **Prometheus** to see if there are any alerts. In next section there is a guide on how to connect to Prometheus. +Once you've solved the issue, you need to run the `helmfile sync` command again. -#### Optional +If you have enabled monitoring you should also check **Prometheus** to see if there are any alerts. In next section there is a guide on how to connect to Prometheus. -If you are installing `radar-appserver`, it needs to be authorized with the Google Firebase also used by the aRMT / Questionnaire app. In Firebase, go to _Project settings_ -> _Service accounts_ and download a Firebase Admin SDK private key. Store the generated key as `etc/radar-appserver/firebase-adminsdk.json`. ## Upgrade instructions @@ -412,7 +449,7 @@ helmfile -f helmfile.d/20-grafana.yaml apply If installed, to upgrade `radar-upload-postgresql`, uncomment the `production.yaml` line `radar_upload_postgresql.primary.existingClaim: "data-radar-upload-postgresql-postgresql-0"`. Then run -``` +```shell kubectl delete secrets radar-upload-postgresql kubectl delete statefulsets radar-upload-postgresql-postgresql helmfile -f helmfile.d/20-upload.yaml apply @@ -460,61 +497,11 @@ helmfile -f helmfile.d/20-s3.yaml apply Delete the redis stateful set (this will not delete the data on the volume) -``` +```shell kubectl delete statefulset redis-master helmfile -f helmfile.d/20-s3.yaml sync --concurrency 1 ``` -## Usage - -### Accessing the applications - -In order to access to the applications first you need to find the IP address that Nginx service is listening to and then point the domain that you've specified in `server_name` variable to this IP address via a DNS server (e.g. [Route53](https://aws.amazon.com/route53/), [Cloudflare](https://www.cloudflare.com/dns/), [Bind](https://www.isc.org/bind/)) or [`hosts` file]() in your local machine. - -> For this guide we assume that you've set `server_name` to "k8s.radar-base.org" and SSL is enabled. Please replace it with a DNS domain under your control. - -You can see details of Nginx service with following command: - -``` -➜ kubectl get service nginx-ingress-controller -NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE -nginx-ingress-controller LoadBalancer 10.100.237.75 XXXX.eu-central-1.elb.amazonaws.com 80:31046/TCP,443:30932/TCP 1h -``` - -- If you're using a cloud provider you need to point the value in `EXTERNAL-IP` column (in this example `XXXX.eu-central-1.elb.amazonaws.com`) to `k8s.radar-base.org` domain in your DNS server. -- RADAR-base uses a few subdomains, which need to be present in DNS for the certificate manager to get an SSL certificate. The easy way to do this is to create a wildcard CNAME record: - ``` - *.k8s.radar-base.org IN CNAME k8s.radar-base.org - ``` - Alternatively, create each DNS entry manually: - ``` - dashboard.k8s.radar-base.org IN CNAME k8s.radar-base.org - graylog.k8s.radar-base.org IN CNAME k8s.radar-base.org - alertmanager.k8s.radar-base.org IN CNAME k8s.radar-base.org - s3.k8s.radar-base.org IN CNAME k8s.radar-base.org - prometheus.k8s.radar-base.org IN CNAME k8s.radar-base.org - grafana.k8s.radar-base.org IN CNAME k8s.radar-base.org - ``` - With the latter method, when a new subdomain is needed for a new service, also a new CNAME entry needs to be created. -- If you're not using a cloud provider you need to use a load balancer to expose `31046` and `30932` ports (will be different in your setup) to a IP address and then point `k8s.radar-base.org` domain to that IP address. -- For development and testing purposes you can run `sudo kubectl port-forward svc/nginx-ingress-controller 80:80 443:443` which will forward Nginx service ports to your local machine and you can have access to applications after adding `127.0.0.1 k8s.radar-base.org` to your `hosts` file. - -Now depending on your setup you should have access to following URLs: - -``` -https://k8s.radar-base.org/managementportal -https://s3.k8s.radar-base.org -https://k8s.radar-base.org/upload -https://k8s.radar-base.org/rest-sources/authorizer -https://k8s.radar-base.org/kafkamanager/ -https://graylog.k8s.radar-base.org # Log management -https://prometheus.k8s.radar-base.org # Monitoring stack -https://alertmanager.k8s.radar-base.org -https://grafana.k8s.radar-base.org -``` - -**Note:** If you have enabled the SSL you might see invalid certificate error when you try to access to the websites, in this case wait a couple of minutes until `cert-manager` issues those certificates. - ## Volume expansion If want to resize a volumes after its initialization you need to make sure that it's supported by its underlying volume plugin: @@ -525,7 +512,7 @@ https://www.jeffgeerling.com/blog/2019/expanding-k8s-pvs-eks-on-aws ## Uninstall -If you want to remove the RADAR-base from your cluster you and use following command to delete the applications from cluster: +If you can spin up a new Kubernetes cluster in a few mintues it's generally suggested to recreate the cluster since the installation creates various components that might need to be manually removed. If that's not an option you can run following commands to delete the applications from cluster: ```shell helmfile destroy @@ -541,6 +528,7 @@ kubectl delete ValidatingWebhookConfiguration prometheus-admission kubectl delete crd certificaterequests.cert-manager.io certificates.cert-manager.io challenges.acme.cert-manager.io clusterissuers.cert-manager.io issuers.cert-manager.io orders.acme.cert-manager.io kubectl delete pvc --all +kubectl delete pv --all kubectl -n cert-manager delete secrets letsencrypt-prod kubectl -n default delete secrets radar-base-tls kubectl -n monitoring delete secrets radar-base-tls @@ -550,11 +538,12 @@ kubectl -n monitoring delete secrets radar-base-tls To find any updates to the Helm charts that are listed in the repository, run -``` +```shell bin/chart-updates ``` ## Feedback and Contributions -Enabling RADAR-base community to use RADAR-Kubernetes is important for us. If you have troubles setting up the platform using provided instructions, you can create an issue with exact details to reproduce and the expected behavior. +Enabling RADAR-base community to use RADAR-Kubernetes is important for us. If you have troubles setting up the platform using provided instructions, you can create an dicussion with exact details to reproduce and the expected behavior. You can also reach out to the RADAR-base community via RADAR-base Slack on **[radar-kubernetes channel](https://radardevelopment.slack.com/archives/C021AGGESC9)**. The RADAR-base developers support the community on a voluntary basis and will pick up your requests as time permits. +If you'd like to contribute to this project, please checkout [CONTRIBUTING.md](https://github.com/RADAR-base/RADAR-Kubernetes/blob/main/CONTRIBUTING.md) file. diff --git a/bin/generate-secrets b/bin/generate-secrets old mode 100755 new mode 100644 index 0e5ee7320..f7746aeec --- a/bin/generate-secrets +++ b/bin/generate-secrets @@ -68,6 +68,7 @@ insert_secret ".management_portal.oauth_clients.radar_redcap_integrator.client_s insert_secret ".management_portal.oauth_clients.radar_fitbit_connector.client_secret" insert_secret ".management_portal.oauth_clients.radar_appconfig.client_secret" insert_secret ".management_portal.oauth_clients.radar_push_endpoint.client_secret" +insert_secret ".management_portal.oauth_clients.radar_data_dashboard_backend.client_secret" insert_secret \ ".radar_appserver_postgresql.global.postgresql.auth.postgresPassword" \ diff --git a/etc/base-secrets.yaml b/etc/base-secrets.yaml index 6e422b31d..03868990d 100644 --- a/etc/base-secrets.yaml +++ b/etc/base-secrets.yaml @@ -86,6 +86,8 @@ management_portal: client_secret: secret radar_push_endpoint: client_secret: secret + radar_data_dashboard_backend: + client_secret: secret smtp: password: secret @@ -120,7 +122,7 @@ oura_api_secret: "secret" radar_rest_sources_backend: postgres: password: secret -# --------------------------------------------------------- 20-grafana.yaml --------------------------------------------------------- +# --------------------------------------------------------- 20-dashboard.yaml --------------------------------------------------------- timescaledb_password: secret grafana_password: secret grafana_metrics_password: secret diff --git a/etc/base.yaml b/etc/base.yaml index 3bd0a5c6b..f66373588 100644 --- a/etc/base.yaml +++ b/etc/base.yaml @@ -270,11 +270,10 @@ radar_rest_sources_backend: garmin: enable: "false" -# --------------------------------------------------------- 20-grafana.yaml --------------------------------------------------------- +# --------------------------------------------------------- 20-dashboard.yaml --------------------------------------------------------- timescaledb_username: postgres -timescaledb_db_name: grafana-metrics -grafana_metrics_username: postgres +timescaledb_db_name: data-dashboard timescaledb: _install: true @@ -295,6 +294,8 @@ timescaledb: # Uncomment when upgrading #existingClaim: "data-timescaledb-postgresql-0" +grafana_metrics_username: postgres + radar_grafana: _install: true _chart_version: 6.26.8 @@ -303,6 +304,11 @@ radar_grafana: env: GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH: /var/lib/grafana/dashboards/allprojects/home.json +data_dashboard_backend: + _install: false + _chart_version: 0.1.0 + replicaCount: 1 + radar_jdbc_connector: _install: true _chart_version: 0.5.1 @@ -312,6 +318,10 @@ radar_jdbc_connector: # Change the list of topics if you have dashboards that read other data or if you don't have certain topics available on your cluster. topics: android_phone_relative_location, android_phone_battery_level, connect_fitbit_intraday_heart_rate, connect_fitbit_intraday_steps +kafka_data_transformer: + _install: false + _chart_version: 0.3.1 + # --------------------------------------------------------- 20-ingestion.yaml --------------------------------------------------------- radar_gateway: diff --git a/etc/base.yaml.gotmpl b/etc/base.yaml.gotmpl index ec1d6aa56..b6167ec51 100644 --- a/etc/base.yaml.gotmpl +++ b/etc/base.yaml.gotmpl @@ -1,7 +1,7 @@ # Remove below Go comment to enable management_portal reading -# management_portal: +management_portal: # read unencrypted keystore - # {{/* keystore: {{ readFile "../etc/management-portal/keystore.p12" | b64enc | quote }} */}} + keystore: {{ readFile "../etc/management-portal/keystore.p12" | b64enc | quote }} # read encrypted keystore # {{/* keystore: {{ exec "sops" (list "-d" "../secrets/management-portal/keystore.sops.p12") | b64enc | quote }} */}} # @@ -27,9 +27,15 @@ radar_grafana: # google_application_credentials: {{ readFile "../etc/radar-appserver/firebase-adminsdk.json" | quote }} #*/}} -# Remove below Go comment to read the queries.sql and set the queries -# in the ksql_server -#ksql_server: +# If data transformation of kafka topic data is needed, please remove the Go template comments and yaml comments. +# Make sure to reference a ksql transformation file that contains the required transformation logic. +# The files below are transform the data from the questionnaire_response and questionnaire_app_events topics to the +# ksql_observations topic, used by the data-dashboard-backend. If using the data-dashboard-backend, please make sure +# to uncomment the relevant ksql transformer files. +# Note: never remove the _base_observations_stream.sql file. +# kafka_data_transformer: # ksql: # queries: | -# {{/*- readFile "cp-ksql-server/queries.sql" | nindent 8 */}} \ No newline at end of file +# {{/* - readFile "../etc/cp-ksql-server/_base_observations_stream.sql" | nindent 8 */}} +# {{/* - readFile "../etc/cp-ksql-server/questionnaire_response_observations.sql" | nindent 8 */}} +# {{/* - readFile "../etc/cp-ksql-server/questionnaire_app_events_observations.sql" | nindent 8 */}} diff --git a/etc/cp-ksql-server/README.md b/etc/cp-ksql-server/README.md new file mode 100644 index 000000000..474116e7d --- /dev/null +++ b/etc/cp-ksql-server/README.md @@ -0,0 +1,41 @@ +# kafka-data transformer (KSQLDB) + +Reference: https://docs.ksqldb.io/ + +The KSQLDB Kafka data transformer is able to register Consumer/Producers to Kafka that transform data in a topic and +publish the results to another topic. + +The provided KSQLDB _questionnaire_response_observations.sql_ and _questionnaire_app_events_observation.sql_ SQL files +transform, respectively, the _questionnaire_response_ and _questionnaire_app_event_ topics and publish the data to the +_ksql_observations_ topic. The _ksql_observations_ topic is consumed by the Kafka-JDBC-connector used for the by the +RADAR-base Data Dashboard backend service (see: [20-data-dashboard.yaml](../../helmfile.d/20-dashboard.yaml)). + +When transformation of other topics is required, new SQL files can be added to this directory. These new files should be +referenced in the _kafka_data_transformer -> ksql -> queries_ section of the `etc/base.yaml.gotmpl` file. New KSQLDB SQL +files should transform towards the following format of the _ksql_observations_ topic: + +``` + TOPIC KEY: + PROJECT: the project identifier + SOURCE: the source identifier + SUBJECT: the subject/study participant identifier + TOPIC VALUE: + TOPIC: the topic identifier + CATEGORY: the category identifier (optional) + VARIABLE: the variable identifier + DATE: the date of the observation + END_DATE: the end date of the observation (optional) + TYPE: the type of the observation (STRING, STRING_JSON, INTEGER, DOUBLE) + VALUE_TEXTUAL: the textual value of the observation (optional, must be set when VALUE_NUMERIC is NULL) + VALUE_NUMERIC: the numeric value of the observation (optional, must be set when VALUE_TEXTUAL is NULL) +``` + +New messages are added to the _ksql_observations_ topic by inserting into the _observations_ stream (see [_base_observations_stream.sql](_base_observations_stream.sql)): + +``` +INSERT INTO observations +SELECT +... +PARTITION BY q.projectId, q.userId, q.sourceId +EMIT CHANGES; +``` \ No newline at end of file diff --git a/etc/cp-ksql-server/_base_observations_stream.sql b/etc/cp-ksql-server/_base_observations_stream.sql new file mode 100644 index 000000000..28236dd53 --- /dev/null +++ b/etc/cp-ksql-server/_base_observations_stream.sql @@ -0,0 +1,20 @@ +SET 'auto.offset.reset' = 'earliest'; + +-- Register the 'ksql_observations' topic (is created when not exists). +CREATE STREAM observations ( + PROJECT VARCHAR KEY, -- 'KEY' means that this field is part of the kafka message key + SUBJECT VARCHAR KEY, + SOURCE VARCHAR KEY, + `TOPIC` VARCHAR, + CATEGORY VARCHAR, + VARIABLE VARCHAR, + DATE TIMESTAMP, + END_DATE TIMESTAMP, + TYPE VARCHAR, + VALUE_NUMERIC DOUBLE, + VALUE_TEXTUAL VARCHAR +) WITH ( + kafka_topic = 'ksql_observations', + partitions = 3, + format = 'avro' +); diff --git a/etc/cp-ksql-server/queries.sql b/etc/cp-ksql-server/queries.sql deleted file mode 100644 index e69de29bb..000000000 diff --git a/etc/cp-ksql-server/questionnaire_app_event_observations.sql b/etc/cp-ksql-server/questionnaire_app_event_observations.sql new file mode 100644 index 000000000..dd98674ec --- /dev/null +++ b/etc/cp-ksql-server/questionnaire_app_event_observations.sql @@ -0,0 +1,30 @@ +CREATE STREAM questionnaire_app_event ( + projectId VARCHAR KEY, -- 'KEY' means that this field is part of the kafka message key + userId VARCHAR KEY, + sourceId VARCHAR KEY, + questionnaireName VARCHAR, + eventType VARCHAR, + time DOUBLE, + metadata MAP +) WITH ( + kafka_topic = 'questionnaire_app_event', + partitions = 3, + format = 'avro' +); + +INSERT INTO observations +SELECT + q.projectId AS PROJECT, + q.userId AS SUBJECT, + q.sourceId AS SOURCE, + 'questionnaire_app_event' as `TOPIC`, + CAST(NULL as VARCHAR) as CATEGORY, + q.questionnaireName as VARIABLE, + FROM_UNIXTIME(CAST(q.time * 1000 AS BIGINT)) as DATE, + CAST(NULL as TIMESTAMP) as END_DATE, + 'STRING_JSON' as TYPE, + CAST(NULL as DOUBLE) as VALUE_NUMERIC, + TO_JSON_STRING(q.metadata) as VALUE_TEXTUAL +FROM questionnaire_app_event q +PARTITION BY q.projectId, q.userId, q.sourceId -- this sets the fields in the kafka message key +EMIT CHANGES; diff --git a/etc/cp-ksql-server/questionnaire_response_observations.sql b/etc/cp-ksql-server/questionnaire_response_observations.sql new file mode 100644 index 000000000..ccb38cabf --- /dev/null +++ b/etc/cp-ksql-server/questionnaire_response_observations.sql @@ -0,0 +1,82 @@ +CREATE STREAM questionnaire_response ( + projectId VARCHAR KEY, -- 'KEY' means that this field is part of the kafka message key + userId VARCHAR KEY, + sourceId VARCHAR KEY, + time DOUBLE, + timeCompleted DOUBLE, + timeNotification DOUBLE, + name VARCHAR, + version VARCHAR, + answers ARRAY, startTime DOUBLE, endTime DOUBLE>> +) WITH ( + kafka_topic = 'questionnaire_response', + partitions = 3, + format = 'avro' +); + +CREATE STREAM questionnaire_response_exploded +AS SELECT + EXPLODE(TRANSFORM(q.answers, a => a->questionId)) as VARIABLE, + FROM_UNIXTIME(CAST(q.time * 1000 AS BIGINT)) as DATE, + q.projectId, + q.userId, + q.sourceId, + 'questionnaire_response' as `TOPIC`, + q.name as CATEGORY, + CAST(NULL as TIMESTAMP) as END_DATE, + -- WARNING!!! The cast from VARCHAR (string) to DOUBLE will throw an JAVA exception if the string is not a number. + -- This does not mean that the message will be lost. The value will be present in the VALUE_TEXTUAL_OPTIONAL field. + EXPLODE(TRANSFORM(q.answers, a => COALESCE(a->value->double, CAST(a->value->int as DOUBLE), CAST(a->value->string as DOUBLE)))) as VALUE_NUMERIC, + EXPLODE(TRANSFORM(q.answers, a => CASE + WHEN a->value->int IS NOT NULL THEN 'INTEGER' + WHEN a->value->double IS NOT NULL THEN 'DOUBLE' + ELSE NULL + END)) as TYPE, + -- Note: When cast to double works for the string value, the VALUE_TEXTUAL_OPTIONAL will also be set. + EXPLODE(TRANSFORM(q.answers, a => a->value->string)) as VALUE_TEXTUAL_OPTIONAL +FROM questionnaire_response q +EMIT CHANGES; + +INSERT INTO observations +SELECT + q.projectId as PROJECT, + q.sourceId as SOURCE, + q.userId as SUBJECT, + `TOPIC`, CATEGORY, VARIABLE, DATE, END_DATE, + CASE + WHEN TYPE IS NULL AND VALUE_NUMERIC IS NOT NULL THEN 'DOUBLE' -- must have been derived from a string cast + WHEN TYPE IS NULL AND VALUE_NUMERIC IS NULL THEN 'STRING' + ELSE TYPE -- keep the original type when TYPE is not NULL + END as TYPE, + VALUE_NUMERIC, + CASE + WHEN VALUE_NUMERIC IS NOT NULL THEN NULL -- When cast to double has worked for the string value, set VALUE_TEXTUAL to NULL. + ELSE VALUE_TEXTUAL_OPTIONAL + END as VALUE_TEXTUAL +FROM questionnaire_response_exploded q +PARTITION BY q.projectId, q.userId, q.sourceId -- this sets the fields in the kafka message key +EMIT CHANGES; + +-- TODO: exploding the 'select:' questions is not yet fully designed. +-- I keep the code here for future reference. +-- Multi-select questionnaire questions are stored as a single 'value' string with the +-- names of the selected options separated by comma's. Multiselect questions are prefixed +-- by 'select:' in the questionId. +-- When 'questionId' is like 'select:%' create a new stream with the select options. +-- The options in the value field split commas and added as separate VARIABLE records. +-- The VALUE_NUMERIC is set to 1 and VALUE_TEXTUAL is set to NULL. +-- INSERT INTO observations +-- SELECT +-- EXPLODE(SPLIT(VALUE_TEXTUAL, ',')) as VARIABLE, +-- PROJECT, SOURCE, SUBJECT, `TOPIC`, CATEGORY, DATE, END_DATE, +-- 'INTEGER' as TYPE, +-- CAST(1 as DOUBLE) VALUE_NUMERIC, +-- CAST(NULL as VARCHAR) as VALUE_TEXTUAL +-- FROM questionnaire_response_observations +-- WHERE +-- VARIABLE IS NOT NULL +-- AND VARIABLE LIKE 'select:%' +-- AND VALUE_TEXTUAL IS NOT NULL +-- AND VALUE_TEXTUAL != '' +-- PARTITION BY SUBJECT, PROJECT, SOURCE +-- EMIT CHANGES; diff --git a/etc/data-dashboard-backend/values.yaml b/etc/data-dashboard-backend/values.yaml new file mode 100644 index 000000000..7d2f41f66 --- /dev/null +++ b/etc/data-dashboard-backend/values.yaml @@ -0,0 +1,23 @@ +ingress: + enabled: true + annotations: + cert-manager.io/cluster-issuer: letsencrypt-prod + nginx.ingress.kubernetes.io/enable-cors: "true" + className: nginx + hosts: + - host: localhost + tls: + secretName: radar-base-data-dashboard + hosts: + - localhost +path: /api +jdbc: + url: jdbc:postgresql://timescaledb-postgresql-hl:5432/data-dashboard + dialect: org.hibernate.dialect.PostgreSQLDialect + user: postgres + password: secret +managementPortal: + url: http://management-portal:8080/managementportal + clientId: radar_data_dashboard_backend + clientSecret: secret +jwtResourceName: res_DataDashboardAPI diff --git a/etc/timescaledb/values.yaml b/etc/timescaledb/values.yaml index 81b4d5af6..882b5b201 100644 --- a/etc/timescaledb/values.yaml +++ b/etc/timescaledb/values.yaml @@ -39,7 +39,7 @@ auth: postgresPassword: "" ## @param auth.database Name for a custom database to create ## - database: grafana-metrics + database: data-dashboard ## @param architecture PostgreSQL architecture (`standalone` or `replication`) ## architecture: standalone diff --git a/helmfile.d/20-grafana.yaml b/helmfile.d/20-dashboard.yaml similarity index 69% rename from helmfile.d/20-grafana.yaml rename to helmfile.d/20-dashboard.yaml index abc59601b..881922143 100644 --- a/helmfile.d/20-grafana.yaml +++ b/helmfile.d/20-dashboard.yaml @@ -61,6 +61,38 @@ releases: - name: "grafana\\.ini.auth\\.generic_oauth.token_url" value: "https://{{ .Values.server_name }}/managementportal/oauth/token" + - name: data-dashboard-backend + chart: radar/data-dashboard-backend + version: {{ .Values.data_dashboard_backend._chart_version }} + installed: {{ .Values.data_dashboard_backend._install }} + values: + - "../etc/data-dashboard-backend/values.yaml" + - {{ .Values.data_dashboard_backend | toYaml | indent 8 | trim }} + set: + - name: serverName + value: {{ .Values.server_name }} + - name: ingress.hosts + values: + - data.{{ .Values.server_name }} + - name: jdbc.user + value: {{ .Values.timescaledb_username }} + - name: jdbc.password + value: {{ .Values.timescaledb_password }} + - name: jdbc.url + value: {{ dig "jdbc" "url" (printf "jdbc:postgresql://timescaledb-postgresql-hl:5432/%s" .Values.timescaledb_db_name) .Values.data_dashboard_backend }} + - name: managementPortal.clientSecret + value: {{ .Values.management_portal.oauth_clients.radar_data_dashboard_backend.client_secret }} + + - name: kafka-data-transformer + chart: cp-radar/cp-ksql-server + version: {{ .Values.kafka_data_transformer._chart_version }} + timeout: {{ add .Values.base_timeout .Values.kafka_data_transformer._extra_timeout }} + wait: false + installed: {{ .Values.kafka_data_transformer._install }} + values: + - "../etc/cp-ksql-server/values.yaml" + - {{ .Values.kafka_data_transformer | toYaml | indent 8 | trim }} + - name: radar-jdbc-connector chart: radar/radar-jdbc-connector version: {{ .Values.radar_jdbc_connector._chart_version }}