[CrateDB] Add support for data acquisition and data export

.env

@@ -6,6 +6,11 @@ MOSQUITTO_VERSION=2.0
 MOSQUITTO_MQTT_PORT=1883
 MOSQUITTO_WS_PORT=9001
 
+# CrateDB
+CRATEDB_VERSION=latest
+CRATEDB_HTTP_PORT=4200
+CRATEDB_POSTGRESQL_PORT=5432
+
 # InfluxDB
 INFLUXDB_VERSION=1.8
 INFLUXDB_HTTP_PORT=8086

.github/workflows/tests.yml

@@ -24,12 +24,19 @@ jobs:
         os: [ ubuntu-20.04 ] # , macos-latest, windows-latest ]
         python-version: [ "3.7", "3.8", "3.9", "3.10", "3.11" ]
         mosquitto-version: [ "2.0" ]
+        cratedb-version: [ "5.3" ]
         influxdb-version: [ "1.8" ]
         grafana-version: [ "7.5.17", "8.5.27", "9.5.7", "10.0.3" ]
 
     # https://docs.github.com/en/free-pro-team@latest/actions/guides/about-service-containers
     services:
 
+      cratedb:
+        image: crate:${{ matrix.cratedb-version }}
+        ports:
+          - 4200:4200
+          - 5432:5432
+
       influxdb:
         image: influxdb:${{ matrix.influxdb-version }}
         ports:
@@ -52,7 +59,7 @@ jobs:
       OS: ${{ matrix.os }}
       PYTHON: ${{ matrix.python-version }}
 
-    name: Python ${{ matrix.python-version }}, Grafana ${{ matrix.grafana-version }}, Mosquitto ${{ matrix.mosquitto-version }}, InfluxDB ${{ matrix.influxdb-version }}
+    name: Py ${{ matrix.python-version }}, Grafana ${{ matrix.grafana-version }}, Mosquitto ${{ matrix.mosquitto-version }}, InfluxDB ${{ matrix.influxdb-version }}, CrateDB ${{ matrix.cratedb-version }}
     steps:
 
       - name: Acquire sources

CHANGES.rst

@@ -27,6 +27,7 @@ in progress
 - [docs] Refactor "decoders" section to "integrations", and improve index/overview page
 - [export] Improve export capabilities by adding parameters ``sort``, ``direction``,
   ``limit``, and ``scalar``. Thanks, @ClemensGruber.
+- [CrateDB] Add support for data acquisition and data export
 
 
 .. _kotori-0.27.0:

CONTRIBUTORS.rst

@@ -14,39 +14,52 @@ contributed to Kotori in one way or another.
 * Anne-Julia Scheuermann <annejulia@gmail.com>
 * Carolin Johannsen <caro@hiverize.org>
 * Clemens Gruber <clemens@hiveeyes.org>
+* Georg Traar <georg@crate.io>
 * Georges Gagliano <https://github.com/ggeorges39>
 * Géraldine Farcy-Merlo <https://mellisphera.com/>
+* Hernan Cianfagna <hernan.cianfagna@crate.io>
 * Jan Hoffmann <j.hoffmann@isarengineering.de>
 * Jens Schmelkus <jens@schmelkus.net>
 * Karsten Harazim <email@karstenharazim.de>
 * Lorenzo Pons <https://mellisphera.com/>
+* Marija Selakovic <marija@crate.io>
 * Markus Euskirchen <m@euse.de>
 * Matthias Büchner <m.buechner@gmail.com>
 * Matthias Mehldau <wetter@hiveeyes.org>
 * Michael Haberler <mah@mah.priv.at>
+* Niklas Schmidtmer <niklas@crate.io>
 * Richard Pobering <einsiedlerkrebs@ginnungagap.org>
 * Roh <incognito>
 * Rui Alexandre Teixeira Sousa Pinto <https://github.com/RuiPinto96>
 * Sebastian Donner <info@thunder-gfx.de>
 * Sebastian Henneke <s.henneke@isarengineering.de>
+* Sebastian Utz <sebastian@crate.io>
 * Smilie <smilie@posteo.de>
 * Thorsten Kluß <tox@uni-bremen.de>
+* Walter Behmann <walter@crate.io>
 * Weef <incognito>
 
-Thanks a bunch!
+Thanks a stack.
 
 
 ****************
 Acknowledgements
 ****************
+
 Kotori would not have been possible without all the amazing people
-working on the infrastructure the software is leveraging.
+working on the infrastructure the software is leveraging and building
+upon.
 
 - Glyph Lefkowitz et al. for conceiving, building and maintaining the
   Twisted network programming framework.
+- Dr. Andy Stanford-Clark and Arlen Nipper for inventing the MQTT
+  messaging protocol.
 - Roger Light et al. for conceiving, building and maintaining the
   Mosquitto MQTT broker.
 - Torkel Ödegaard and his team for creating and maintaining Grafana.
+- Bernd Dorn, Bernd Rössl, Jodok Batlogg, Bernhard Kuzel, Mathias
+  Fußenegger, Sebastian Utz, et al., for conceiving and maintaining
+  CrateDB.
 - Paul Dix and his team for creating and maintaining InfluxDB.
 - The PostgreSQL and MongoDB developers for creating and maintaining
   their database systems.
@@ -59,4 +72,4 @@ working on the infrastructure the software is leveraging.
   to put everything together.
 
 Thank you so much for providing such great infrastructure
-components and resources to the community! You know who you are.
+components and resources to the community. You know who you are.

README.rst

@@ -76,6 +76,10 @@ Kotori
         :target: https://github.com/grafana/grafana
         :alt: Supported Grafana versions
 
+  .. image:: https://img.shields.io/badge/CrateDB-4.2%20--%205.x-blue.svg
+        :target: https://github.com/crate/crate
+        :alt: Supported CrateDB versions
+
   .. image:: https://img.shields.io/badge/InfluxDB-1.6%2C%201.7%2C%201.8-blue.svg
         :target: https://github.com/influxdata/influxdb
         :alt: Supported InfluxDB versions
@@ -105,7 +109,7 @@ Details
 
 Kotori takes the role of the `data historian`_ component within a `SCADA`_ / `MDE`_
 system, exclusively built upon industry-grade `free and open-source software`_
-like Grafana_, Mosquitto_, or InfluxDB_. It is written in Python_,
+like Grafana_, Mosquitto_, CrateDB_, or InfluxDB_. It is written in Python_,
 and uses the Twisted_ networking library.
 
 The best way to find out what you can do with Kotori, is by looking at
@@ -222,6 +226,7 @@ The project is licensed under the terms of the GNU AGPL license, see `LICENSE`_.
 
 .. _Autobahn: https://crossbar.io/autobahn/
 .. _contributors: https://getkotori.org/docs/project/contributors.html
+.. _CrateDB: https://github.com/crate/crate
 .. _Create an issue: https://github.com/daq-tools/kotori/issues/new
 .. _data historian: https://en.wikipedia.org/wiki/Operational_historian
 .. _development sandbox: https://getkotori.org/docs/setup/sandbox.html

doc/source/_meta.rst

@@ -16,5 +16,5 @@
     :keywords lang=es: adquisición de datos, gráficas, conjunto de herramientas, telemetría
     :keywords lang=fr: acquisition de données, graphiques, boîte à outils, télémétrie
     :keywords:         m2m, iot, mqtt, wamp, http, rest, udp, binary struct,
-                       Kotori, Grafana, InfluxDB, Mosquitto, MongoDB, XMPP, SMTP
+                       Kotori, Grafana, CrateDB, InfluxDB, Mosquitto, MongoDB, XMPP, SMTP
 

doc/source/_resources.rst

@@ -170,6 +170,8 @@
 .. _Raumfahrtagentur: http://www.raumfahrtagentur.org/
 
 .. NEW
+.. _CrateDB: https://github.com/crate/crate
+.. _CrateDB reference documentation: https://crate.io/docs/crate/reference/
 .. _curl: https://en.wikipedia.org/wiki/CURL
 .. _Flux data scripting language: https://docs.influxdata.com/flux/
 .. _Funky v3: https://harizanov.com/product/funky-v3/
@@ -178,12 +180,15 @@
 .. _InfluxDB line protocol: https://docs.influxdata.com/influxdb/v1.8/write_protocols/line_protocol_reference/
 .. _InfluxDB storage engine: https://docs.influxdata.com/influxdb/v2.7/reference/internals/storage-engine/
 .. _LoRaWAN: https://en.wikipedia.org/wiki/LoRa#LoRaWAN
+.. _Lucene: https://lucene.apache.org/
 .. _MongoDB manual: https://www.mongodb.com/docs/manual/
 .. _MongoDB Wire Protocol: https://www.mongodb.com/docs/manual/reference/mongodb-wire-protocol/
 .. _OpenXC: https://openxcplatform.com/
 .. _OpenXC for Python: http://python.openxcplatform.com/
 .. _multi-tenant: https://en.wikipedia.org/wiki/Multitenancy
+.. _near real-time: https://en.wikipedia.org/wiki/Near_real-time#Near_real-time
 .. _NodeUSB: https://web.archive.org/web/20210621192219/http://www.nodeusb.com/
+.. _PostgreSQL wire protocol: https://www.postgresql.org/docs/16/protocol.html
 .. _trunking: https://en.wikipedia.org/wiki/Trunking
 .. _webhook: https://en.wikipedia.org/wiki/Webhook
 .. _webhooks: https://en.wikipedia.org/wiki/Webhook

doc/source/about/scenarios.rst

@@ -85,8 +85,8 @@ We are standing on the shoulders of giants:
 - Leverage the open infrastructure based on Twisted_ - an event-driven networking engine -
   to implement custom software components.
 - Listen and talk M2M_ using the *MQ Telemetry Transport* connectivity protocol and software bus (MQTT_).
-- Store data points into InfluxDB_, a leading open source time series database suitable
-  for realtime analytics and sensor data storage.
+- Store data points into CrateDB_, InfluxDB_, or other open source time series databases
+  suitable for realtime analytics and sensor data storage.
 - Automate dashboard management in the context of data arriving on different data channels
   using Grafana_, an open source, feature rich metrics dashboard and graph editor.
 - Make Browser clients first-class citizens of the underpinning software bus framework

doc/source/about/technologies.rst

@@ -26,14 +26,16 @@ Infrastructure components
 
 - Kotori_, a data acquisition, graphing and telemetry toolkit
 - Grafana_, a graph and dashboard builder for visualizing time series metrics
-- InfluxDB_, a time-series database
+- CrateDB_, a time-series database ¹
+- InfluxDB_, a time-series database ¹
 - Mosquitto_, an MQTT message broker
-- MongoDB_, a document store (optional) ¹²
+- MongoDB_, a document store (optional) ²³
 
-| ¹ MongoDB is only required when doing CSV data acquisition, so it is completely
+| ¹ Kotori can either use CrateDB or InfluxDB as timeseries database.
+| ² MongoDB is only required when doing CSV data acquisition, so it is completely
 | optional for regular operations of Kotori.
-| ² As MongoDB - strictly speaking - stopped being free software recently (2018/2019),
-| it will probably be phased out gradually and replaced by PostgreSQL_.
+| ³ As MongoDB - strictly speaking - stopped being free software recently (2018/2019),
+| it will probably be phased out gradually and replaced by PostgreSQL_ or CrateDB_.
 
 
 Supported protocols