diff --git a/CHANGES.rst b/CHANGES.rst index 668fb540..2db1d36e 100644 --- a/CHANGES.rst +++ b/CHANGES.rst @@ -25,6 +25,7 @@ in progress a single TTN Application. Thanks, @thiasB, @einsiedlerkrebs, and @ClemensGruber. - [core] Fix error when connecting to MQTT broker without authentication credentials - [docs] Refactor "decoders" section to "integrations", and improve index/overview page +- [CrateDB] Add support for data acquisition and data export .. _kotori-0.27.0: diff --git a/README.rst b/README.rst index d7ef2a13..958b8b43 100644 --- a/README.rst +++ b/README.rst @@ -76,6 +76,10 @@ Kotori :target: https://github.com/grafana/grafana :alt: Supported Grafana versions + .. image:: https://img.shields.io/badge/CrateDB-4.2%20--%205.x-blue.svg + :target: https://github.com/crate/crate + :alt: Supported CrateDB versions + .. image:: https://img.shields.io/badge/InfluxDB-1.6%2C%201.7%2C%201.8-blue.svg :target: https://github.com/influxdata/influxdb :alt: Supported InfluxDB versions @@ -105,7 +109,7 @@ Details Kotori takes the role of the `data historian`_ component within a `SCADA`_ / `MDE`_ system, exclusively built upon industry-grade `free and open-source software`_ -like Grafana_, Mosquitto_, or InfluxDB_. It is written in Python_, +like Grafana_, Mosquitto_, CrateDB_, or InfluxDB_. It is written in Python_, and uses the Twisted_ networking library. The best way to find out what you can do with Kotori, is by looking at @@ -222,6 +226,7 @@ The project is licensed under the terms of the GNU AGPL license, see `LICENSE`_. .. _Autobahn: https://crossbar.io/autobahn/ .. _contributors: https://getkotori.org/docs/project/contributors.html +.. _CrateDB: https://github.com/crate/crate .. _Create an issue: https://github.com/daq-tools/kotori/issues/new .. _data historian: https://en.wikipedia.org/wiki/Operational_historian .. _development sandbox: https://getkotori.org/docs/setup/sandbox.html diff --git a/doc/source/_meta.rst b/doc/source/_meta.rst index 515723cd..f1f81541 100644 --- a/doc/source/_meta.rst +++ b/doc/source/_meta.rst @@ -16,5 +16,5 @@ :keywords lang=es: adquisición de datos, gráficas, conjunto de herramientas, telemetría :keywords lang=fr: acquisition de données, graphiques, boîte à outils, télémétrie :keywords: m2m, iot, mqtt, wamp, http, rest, udp, binary struct, - Kotori, Grafana, InfluxDB, Mosquitto, MongoDB, XMPP, SMTP + Kotori, Grafana, CrateDB, InfluxDB, Mosquitto, MongoDB, XMPP, SMTP diff --git a/doc/source/_resources.rst b/doc/source/_resources.rst index 0931f922..61e4f004 100644 --- a/doc/source/_resources.rst +++ b/doc/source/_resources.rst @@ -170,6 +170,7 @@ .. _Raumfahrtagentur: http://www.raumfahrtagentur.org/ .. NEW +.. _CrateDB: https://github.com/crate/crate .. _curl: https://en.wikipedia.org/wiki/CURL .. _Flux data scripting language: https://docs.influxdata.com/flux/ .. _Funky v3: https://harizanov.com/product/funky-v3/ @@ -183,6 +184,7 @@ .. _OpenXC: https://openxcplatform.com/ .. _OpenXC for Python: http://python.openxcplatform.com/ .. _multi-tenant: https://en.wikipedia.org/wiki/Multitenancy +.. _near real-time: https://en.wikipedia.org/wiki/Near_real-time#Near_real-time .. _NodeUSB: https://web.archive.org/web/20210621192219/http://www.nodeusb.com/ .. _trunking: https://en.wikipedia.org/wiki/Trunking .. _webhook: https://en.wikipedia.org/wiki/Webhook diff --git a/doc/source/about/scenarios.rst b/doc/source/about/scenarios.rst index edcfa346..8eeb20c6 100644 --- a/doc/source/about/scenarios.rst +++ b/doc/source/about/scenarios.rst @@ -85,8 +85,8 @@ We are standing on the shoulders of giants: - Leverage the open infrastructure based on Twisted_ - an event-driven networking engine - to implement custom software components. - Listen and talk M2M_ using the *MQ Telemetry Transport* connectivity protocol and software bus (MQTT_). -- Store data points into InfluxDB_, a leading open source time series database suitable - for realtime analytics and sensor data storage. +- Store data points into CrateDB_, InfluxDB_, or other open source time series databases + suitable for realtime analytics and sensor data storage. - Automate dashboard management in the context of data arriving on different data channels using Grafana_, an open source, feature rich metrics dashboard and graph editor. - Make Browser clients first-class citizens of the underpinning software bus framework diff --git a/doc/source/about/technologies.rst b/doc/source/about/technologies.rst index 62ed8abe..19e4f62b 100644 --- a/doc/source/about/technologies.rst +++ b/doc/source/about/technologies.rst @@ -26,14 +26,16 @@ Infrastructure components - Kotori_, a data acquisition, graphing and telemetry toolkit - Grafana_, a graph and dashboard builder for visualizing time series metrics -- InfluxDB_, a time-series database +- CrateDB_, a time-series database ¹ +- InfluxDB_, a time-series database ¹ - Mosquitto_, an MQTT message broker -- MongoDB_, a document store (optional) ¹² +- MongoDB_, a document store (optional) ²³ -| ¹ MongoDB is only required when doing CSV data acquisition, so it is completely +| ¹ Kotori can either use CrateDB or InfluxDB as timeseries database. +| ² MongoDB is only required when doing CSV data acquisition, so it is completely | optional for regular operations of Kotori. -| ² As MongoDB - strictly speaking - stopped being free software recently (2018/2019), -| it will probably be phased out gradually and replaced by PostgreSQL_. +| ³ As MongoDB - strictly speaking - stopped being free software recently (2018/2019), +| it will probably be phased out gradually and replaced by PostgreSQL_ or CrateDB_. Supported protocols diff --git a/doc/source/database/cratedb.rst b/doc/source/database/cratedb.rst new file mode 100644 index 00000000..c8e2e315 --- /dev/null +++ b/doc/source/database/cratedb.rst @@ -0,0 +1,18 @@ +.. include:: ../_resources.rst + +.. _database-cratedb: + +####### +CrateDB +####### + +.. admonition:: + + .. todo:: Content on this page may need an update. + +***** +About +***** + +`CrateDB`_ is a distributed and scalable SQL database for storing and analyzing massive +amounts of data in `near real-time`_. It is PostgreSQL-compatible, and based on Lucene. diff --git a/doc/source/database/index.md b/doc/source/database/index.md index b3d15aa9..c5f4622d 100644 --- a/doc/source/database/index.md +++ b/doc/source/database/index.md @@ -14,6 +14,7 @@ with Kotori. Adding more adapters is possible. :maxdepth: 1 :hidden: +cratedb influxdb mongodb ``` @@ -24,6 +25,34 @@ mongodb :padding: 0 +:::::{grid-item-card} +::::{grid} 2 +:margin: 0 +:padding: 0 + +:::{grid-item} +:columns: 8 +#### [](#database-cratedb) + +CrateDB is a distributed and scalable SQL database for storing and analyzing massive +amounts of data in near real-time. It is PostgreSQL-compatible, and based on Lucene. + + +Categories: timeseries-database, multi-modal database + +::: +:::{grid-item} +:columns: 4 +{bdg-primary-line}`eth` {bdg-primary-line}`wifi` {bdg-primary-line}`http` {bdg-primary-line}`postgresql` + +{bdg-success-line}`sql` {bdg-success-line}`blob` + +{bdg-secondary-line}`amd64` {bdg-secondary-line}`arm64` +::: +:::: +::::: + + :::::{grid-item-card} ::::{grid} 2 :margin: 0 diff --git a/doc/source/development/releasing/foundation.rst b/doc/source/development/releasing/foundation.rst index 8ce7102a..b51010e3 100644 --- a/doc/source/development/releasing/foundation.rst +++ b/doc/source/development/releasing/foundation.rst @@ -34,6 +34,24 @@ Download packages scp mosquitto*.deb workbench@packages.example.org:/srv/packages/organizations/elmyra/foss/aptly/public/incoming +******* +CrateDB +******* +- https://github.com/crate/crate +- https://crate.io/docs/crate/tutorials/ + +Download packages +================= +:: + + # Download amd64 packages + wget https://cdn.crate.io/downloads/apt/stable/pool/main/c/crate/crate_5.3.2-1~buster_amd64.deb + wget https://cdn.crate.io/downloads/apt/stable/pool/main/c/crate/crate_5.3.2-1~bullseye_amd64.deb + + # Upload to "incoming" directory + scp crate_*.deb workbench@packages.example.org:/srv/packages/organizations/elmyra/foss/aptly/public/incoming + + ******** InfluxDB ******** @@ -100,7 +118,7 @@ Publish packages # Add packages to repository aptly repo add -config=$APTLY_CONFIG -remove-files=true $APTLY_REPOSITORY \ - $PACKAGES_INCOMING/influxdb*.deb $PACKAGES_INCOMING/grafana*.deb $PACKAGES_INCOMING/*mosquitto*.deb + $PACKAGES_INCOMING/crate*.deb $PACKAGES_INCOMING/influxdb*.deb $PACKAGES_INCOMING/grafana*.deb $PACKAGES_INCOMING/*mosquitto*.deb # Publish repository aptly publish update -config=$APTLY_CONFIG -gpg-key=2543A838 -passphrase=esp $APTLY_DISTRIBUTION diff --git a/doc/source/development/releasing/index.rst b/doc/source/development/releasing/index.rst index 00fdc425..266acc9f 100644 --- a/doc/source/development/releasing/index.rst +++ b/doc/source/development/releasing/index.rst @@ -4,7 +4,7 @@ Releasing and packaging This section describes how to cut a new release, how to build and publish Kotori packages and how to add foundation packages -like InfluxDB and Grafana to the same package repositories. +like CrateDB, InfluxDB, and Grafana to the same package repositories. .. toctree:: :maxdepth: 1 diff --git a/doc/source/development/systemd.rst b/doc/source/development/systemd.rst index 2cfaad28..b28a93f6 100644 --- a/doc/source/development/systemd.rst +++ b/doc/source/development/systemd.rst @@ -169,12 +169,12 @@ systemd troubleshooting :: - systemctl status influxdb mosquitto grafana-server kotori + systemctl status crate influxdb mosquitto grafana-server kotori :: - systemctl list-unit-files kotori* influx* grafana* mosquitto* + systemctl list-unit-files kotori* crate* influx* grafana* mosquitto* :: diff --git a/doc/source/development/tests.rst b/doc/source/development/tests.rst index 977ca8e2..282976e4 100644 --- a/doc/source/development/tests.rst +++ b/doc/source/development/tests.rst @@ -11,7 +11,7 @@ The tests are mostly full integration tests. They are testing the whole system and the interactions between the subsystems. Messages will get published to the MQTT bus by shelling out to ``mosquitto_pub``. -After that, InfluxDB will be checked to contain the right data and Grafana will +After that, the database will be checked to contain the right data and Grafana will be checked to be accurately provisioned. While the shell-out can well be optimized for efficiency, it is also pleasant @@ -32,12 +32,12 @@ Install some needed packages:: Foundation services =================== -The test suite will assume running instances of Mosquitto, InfluxDB, MongoDB +The test suite will assume running instances of Mosquitto, CrateDB, InfluxDB, MongoDB and Grafana and fire up an in-process instance of Kotori to complement these. Please have a look at :ref:`setup-docker` in order to get the complementing services up and running in a quick and ad hoc manner. -Run Mosquitto, InfluxDB, MongoDB and Grafana as Docker containers:: +Run Mosquitto, CrateDB, InfluxDB, MongoDB and Grafana as Docker containers:: make start-foundation-services diff --git a/doc/source/handbook/acquisition/error-signalling.rst b/doc/source/handbook/acquisition/error-signalling.rst index 3df6357b..bc37186c 100644 --- a/doc/source/handbook/acquisition/error-signalling.rst +++ b/doc/source/handbook/acquisition/error-signalling.rst @@ -48,13 +48,24 @@ the system will respond with a MQTT "response" on the corresponding topic with s "description": "Error processing MQTT message \"{\"value\": 42.42\" from topic \"mqttkit-1/testdrive/area-42/node-1/data.json\"." } -Database error -============== -When sending a payload with an existing field already seeded in a different data type:: +Database errors +=============== +When sending a payload with an existing field already materialized using a different data type:: echo '{"value": "invalid"}' | mosquitto_pub -h kotori.example.org -t mqttkit-1/testdrive/area-42/node-1/data.json -l -the system will respond with:: +the system will respond correspondingly. + +CrateDB:: + + mqttkit-1/testdrive/area-42/node-1/error.json { + "type": "", + "message": "SQLParseException[Cannot cast object element `humidity` with value `invalid` to type `double precision`]", + "description": "Error processing MQTT message \"b'{\"temperature\": 42.84, \"humidity\": \"invalid\"}'\" from topic \"mqttkit-2/foo/bar/1/data.json\".", + "timestamp": "2023-06-20T12:00:08+00:00" + } + +InfluxDB:: mqttkit-1/testdrive/area-42/node-1/error.json { "message": "400: {\"error\":\"field type conflict: input field \\\"value\\\" on measurement \\\"area_42_node_1_sensors\\\" is type string, already exists as type float dropped=1\"}\n", @@ -73,11 +84,4 @@ and then sending an invalid payload like:: echo '2017-05-01 22:39:09,invalid' | http POST http://kotori.example.org/api/mqttkit-1/testdrive/area-42/node-1/data Content-Type:text/csv -The system will also respond over MQTT with:: - - mqttkit-1/testdrive/area-42/node-1/error.json { - "message": "400: {\"error\":\"field type conflict: input field \\\"value\\\" on measurement \\\"area_42_node_1_sensors\\\" is type string, already exists as type float dropped=1\"}\n", - "type": "", - "description": "Error processing MQTT message \"{\"time\": \"2017-05-01 22:40:09\", \"value\": \"invalid\"}\" from topic \"mqttkit-1/testdrive/area-42/node-1/data.json\"." - } - +The system will also respond over MQTT correspondingly, like outlined above. diff --git a/doc/source/handbook/index.rst b/doc/source/handbook/index.rst index e729020d..77ac46ba 100644 --- a/doc/source/handbook/index.rst +++ b/doc/source/handbook/index.rst @@ -49,4 +49,5 @@ Usage usage/kotori usage/grafana + usage/cratedb usage/influxdb diff --git a/doc/source/handbook/usage/cratedb.rst b/doc/source/handbook/usage/cratedb.rst new file mode 100644 index 00000000..05cd134b --- /dev/null +++ b/doc/source/handbook/usage/cratedb.rst @@ -0,0 +1,14 @@ +.. include:: ../../_resources.rst + +.. _cratedb-handbook: + +################ +CrateDB handbook +################ + +.. admonition:: + + .. todo:: Content on this page may need an update. + +This section of the documentation will give a short and concise summary about +how to operate and work with CrateDB. diff --git a/doc/source/integration/tts-ttn.rst b/doc/source/integration/tts-ttn.rst index 426d9d4a..f406164f 100644 --- a/doc/source/integration/tts-ttn.rst +++ b/doc/source/integration/tts-ttn.rst @@ -45,6 +45,7 @@ Overview subgraph backend direction LR + Kotori --> CrateDB Kotori --> InfluxDB Kotori --> Grafana end diff --git a/doc/source/setup/docker.rst b/doc/source/setup/docker.rst index 2e5aaf17..cebcbd64 100644 --- a/doc/source/setup/docker.rst +++ b/doc/source/setup/docker.rst @@ -26,7 +26,7 @@ its configuration at :ref:`getting-started`. Introduction ************ -This section outlines how to conveniently run Mosquitto, InfluxDB, +This section outlines how to conveniently run Mosquitto, CrateDB, InfluxDB, MongoDB, Grafana and Kotori using Docker. The repository provides two files ``docker-compose.yml`` and ``.env``. They @@ -48,7 +48,7 @@ Those images are published to Docker Hub. Please note that this Docker Compose configuration is primarily suited for evaluation and development purposes. As it either disables authentication - or uses insecure authentication credentials for Mosquitto, InfluxDB, + or uses insecure authentication credentials for Mosquitto, CrateDB, InfluxDB, and Grafana, it is not prepared for production setups. @@ -56,7 +56,7 @@ Those images are published to Docker Hub. Prerequisites ************* -This will give you Mosquitto, InfluxDB, MongoDB, Grafana, an improved +This will give you Mosquitto, CrateDB, InfluxDB, MongoDB, Grafana, an improved Grafana map panel plugin, and a command alias for invoking Kotori. In order to invoke the auxiliary services, run:: @@ -94,6 +94,34 @@ Testdrive This is a basic test walkthrough, to check if data is correctly routed from the telemetry message bus to the database. +CrateDB +======= + +This example uses CrateDB as timeseries-database. + +Invoke Kotori:: + + kotori --config /etc/kotori/docker/docker-cratedb.ini + +Publish single reading using MQTT:: + + export CHANNEL_TOPIC=sensorwan-cratedb/foo/bar/1/data.json + docker run \ + --network kotori_default \ + -it --rm eclipse-mosquitto \ + mosquitto_pub -d -h mosquitto -t $CHANNEL_TOPIC -m '{"temperature": 42.84, "humidity": 83.1}' + +Check if reading has been stored in CrateDB:: + + docker run \ + --network kotori_default \ + -it --rm crate \ + crash --hosts cratedb -c 'SELECT * FROM sensorwan_cratedb_foo.bar_1_sensors;' + +Go to Grafana and visit the dashboard just created:: + + open "http://localhost:3000/?orgId=1&search=open&query=sensorwan-cratedb" + InfluxDB ======== diff --git a/doc/source/setup/index.rst b/doc/source/setup/index.rst index ec3f1f4a..adcfabb2 100644 --- a/doc/source/setup/index.rst +++ b/doc/source/setup/index.rst @@ -16,7 +16,7 @@ from Docker Hub, from the Python Package Index (PyPI) or from the Git repository. For running Kotori in a full configuration, you will need some other -infrastructure services like Mosquitto_, InfluxDB_, Grafana_ and optionally +infrastructure services like Mosquitto_, CrateDB_, InfluxDB_, Grafana_ and optionally MongoDB_. Have fun and enjoy your data acquisition! @@ -62,7 +62,7 @@ Details ¹ When choosing to install from the Debian package repository, you will also be -able to receive appropriate Debian packages for Mosquitto, InfluxDB and Grafana +able to receive appropriate Debian packages for Mosquitto, CrateDB, InfluxDB, and Grafana through the `DaqZilla package repository `_. This makes it easy to setup the complete DAQ system from a single package source. diff --git a/doc/source/setup/linux-arch.rst b/doc/source/setup/linux-arch.rst index 93cf7ffa..ef3c870f 100644 --- a/doc/source/setup/linux-arch.rst +++ b/doc/source/setup/linux-arch.rst @@ -59,6 +59,7 @@ MongoDB is provided through the Arch User Repository (AUR):: cd mongodb-tools-bin makepkg -si +.. todo:: Add walkthrough how to install CrateDB on Arch Linux. Kotori ====== diff --git a/doc/source/setup/linux-debian.rst b/doc/source/setup/linux-debian.rst index 60021b74..7083c1ed 100644 --- a/doc/source/setup/linux-debian.rst +++ b/doc/source/setup/linux-debian.rst @@ -102,11 +102,11 @@ Install Kotori together with all recommended and suggested packages:: apt install --install-recommends kotori -InfluxDB and Grafana are not always enabled and started automatically, +CrateDB, InfluxDB, and Grafana are not always enabled and started automatically, so ensure they are running by invoking:: - systemctl enable mosquitto influxdb grafana-server - systemctl start mosquitto influxdb grafana-server + systemctl enable mosquitto crate influxdb grafana-server + systemctl start mosquitto crate influxdb grafana-server Notes for ARM machines ====================== @@ -127,3 +127,5 @@ These are the log files at a glance where system messages might appear:: tail -F /var/log/kotori/*.log /var/log/grafana/*.log /var/log/influxdb/*.log /var/log/mosquitto/*.log journalctl -f -u influxdb + +.. todo:: Add where CrateDB stores its log. diff --git a/doc/source/setup/macos.rst b/doc/source/setup/macos.rst index b6cc9299..9dd9cd10 100644 --- a/doc/source/setup/macos.rst +++ b/doc/source/setup/macos.rst @@ -63,6 +63,8 @@ instructions:: brew info mosquitto influxdb grafana | grep Caveats -A8 +.. todo:: Add walkthrough how to install CrateDB on macOS. + Install Kotori:: brew install python diff --git a/doc/source/setup/sandbox.rst b/doc/source/setup/sandbox.rst index 98987a81..3f235b56 100644 --- a/doc/source/setup/sandbox.rst +++ b/doc/source/setup/sandbox.rst @@ -31,13 +31,13 @@ Install some needed packages:: Foundation services =================== -You will need InfluxDB_, Grafana_, Mosquitto_ and optionally MongoDB_. +You will need Grafana_, Mosquitto_, one of CrateDB_ or InfluxDB_, and optionally MongoDB_. For installing them on your workstation, you might want to have a look at the :ref:`setup-docker`. When running Linux, you can also install the infrastructure on your local workstation natively like :ref:`setup-debian`. -The most easy way is to run Mosquitto, InfluxDB, MongoDB and Grafana as Docker +The most easy way is to run Mosquitto, CrateDB, InfluxDB, MongoDB and Grafana as Docker containers:: make start-foundation-services diff --git a/doc/source/setup/security.rst b/doc/source/setup/security.rst index b88f0556..314539e9 100644 --- a/doc/source/setup/security.rst +++ b/doc/source/setup/security.rst @@ -6,6 +6,11 @@ Secure the installation Close ports =========== +CrateDB +------- + +.. todo:: Outline how to protect CrateDB's public listening ports. + InfluxDB -------- @@ -39,6 +44,14 @@ Grafana Enable authentication ===================== +CrateDB +------- + +.. todo:: + + Outline how to enable authentication with CrateDB, and how to amend the default + credentials. + InfluxDB -------- Purpose: Enable auth-only access to InfluxDB. diff --git a/etc/docker/docker-cratedb.ini b/etc/docker/docker-cratedb.ini new file mode 100644 index 00000000..e09a6bcb --- /dev/null +++ b/etc/docker/docker-cratedb.ini @@ -0,0 +1,56 @@ +; #################################################### +; Kotori configuration with Docker Compose and CrateDB +; #################################################### + +; Services need to be invoked using Docker Compose. +; docker-compose up + + +; ================ +; General settings +; ================ + +; http server +[kotori] + +; TODO: Refactor to [http] section, or, even better, into the channel configuration. +http_listen = localhost +http_port = 24642 + + +; ===================== +; Connectivity settings +; ===================== + +; MQTT bus adapter +[mqtt] +host = mosquitto +#port = 1883 +username = kotori +password = kotori + +; Storage adapter +[cratedb] +host = cratedb +; port = 4200 +; username = crate +; password = + +; User interface +[grafana] +host = grafana +#port = 3000 +username = admin +password = admin + + +; ================ +; Channel settings +; ================ + +[sensorwan-cratedb] +enable = true +type = application +realm = sensorwan-cratedb +mqtt_topics = sensorwan-cratedb/# +application = kotori.daq.application.mqttkit:mqttkit_application diff --git a/etc/test/cratedb.ini b/etc/test/cratedb.ini index b0bbc714..60343ed1 100644 --- a/etc/test/cratedb.ini +++ b/etc/test/cratedb.ini @@ -3,9 +3,9 @@ ; ###################################### -; ========================================== -; General settings -; ========================================== +; ================ +; General settings +; ================ ; http server [kotori]