Skip to content

Commit

Permalink
[CrateDB] Add documentation across the board
Browse files Browse the repository at this point in the history
  • Loading branch information
amotl committed Jun 20, 2023
1 parent ce1fbbb commit fcd4379
Show file tree
Hide file tree
Showing 25 changed files with 238 additions and 41 deletions.
1 change: 1 addition & 0 deletions CHANGES.rst
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ in progress
a single TTN Application. Thanks, @thiasB, @einsiedlerkrebs, and @ClemensGruber.
- [core] Fix error when connecting to MQTT broker without authentication credentials
- [docs] Refactor "decoders" section to "integrations", and improve index/overview page
- [CrateDB] Add support for data acquisition and data export


.. _kotori-0.27.0:
Expand Down
7 changes: 6 additions & 1 deletion README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,10 @@ Kotori
:target: https://github.com/grafana/grafana
:alt: Supported Grafana versions

.. image:: https://img.shields.io/badge/CrateDB-4.2%20--%205.x-blue.svg
:target: https://github.com/crate/crate
:alt: Supported CrateDB versions

.. image:: https://img.shields.io/badge/InfluxDB-1.6%2C%201.7%2C%201.8-blue.svg
:target: https://github.com/influxdata/influxdb
:alt: Supported InfluxDB versions
Expand Down Expand Up @@ -105,7 +109,7 @@ Details

Kotori takes the role of the `data historian`_ component within a `SCADA`_ / `MDE`_
system, exclusively built upon industry-grade `free and open-source software`_
like Grafana_, Mosquitto_, or InfluxDB_. It is written in Python_,
like Grafana_, Mosquitto_, CrateDB_, or InfluxDB_. It is written in Python_,
and uses the Twisted_ networking library.

The best way to find out what you can do with Kotori, is by looking at
Expand Down Expand Up @@ -222,6 +226,7 @@ The project is licensed under the terms of the GNU AGPL license, see `LICENSE`_.

.. _Autobahn: https://crossbar.io/autobahn/
.. _contributors: https://getkotori.org/docs/project/contributors.html
.. _CrateDB: https://github.com/crate/crate
.. _Create an issue: https://github.com/daq-tools/kotori/issues/new
.. _data historian: https://en.wikipedia.org/wiki/Operational_historian
.. _development sandbox: https://getkotori.org/docs/setup/sandbox.html
Expand Down
2 changes: 1 addition & 1 deletion doc/source/_meta.rst
Original file line number Diff line number Diff line change
Expand Up @@ -16,5 +16,5 @@
:keywords lang=es: adquisición de datos, gráficas, conjunto de herramientas, telemetría
:keywords lang=fr: acquisition de données, graphiques, boîte à outils, télémétrie
:keywords: m2m, iot, mqtt, wamp, http, rest, udp, binary struct,
Kotori, Grafana, InfluxDB, Mosquitto, MongoDB, XMPP, SMTP
Kotori, Grafana, CrateDB, InfluxDB, Mosquitto, MongoDB, XMPP, SMTP

2 changes: 2 additions & 0 deletions doc/source/_resources.rst
Original file line number Diff line number Diff line change
Expand Up @@ -170,6 +170,7 @@
.. _Raumfahrtagentur: http://www.raumfahrtagentur.org/

.. NEW
.. _CrateDB: https://github.com/crate/crate
.. _curl: https://en.wikipedia.org/wiki/CURL
.. _Flux data scripting language: https://docs.influxdata.com/flux/
.. _Funky v3: https://harizanov.com/product/funky-v3/
Expand All @@ -183,6 +184,7 @@
.. _OpenXC: https://openxcplatform.com/
.. _OpenXC for Python: http://python.openxcplatform.com/
.. _multi-tenant: https://en.wikipedia.org/wiki/Multitenancy
.. _near real-time: https://en.wikipedia.org/wiki/Near_real-time#Near_real-time
.. _NodeUSB: https://web.archive.org/web/20210621192219/http://www.nodeusb.com/
.. _trunking: https://en.wikipedia.org/wiki/Trunking
.. _webhook: https://en.wikipedia.org/wiki/Webhook
Expand Down
4 changes: 2 additions & 2 deletions doc/source/about/scenarios.rst
Original file line number Diff line number Diff line change
Expand Up @@ -85,8 +85,8 @@ We are standing on the shoulders of giants:
- Leverage the open infrastructure based on Twisted_ - an event-driven networking engine -
to implement custom software components.
- Listen and talk M2M_ using the *MQ Telemetry Transport* connectivity protocol and software bus (MQTT_).
- Store data points into InfluxDB_, a leading open source time series database suitable
for realtime analytics and sensor data storage.
- Store data points into CrateDB_, InfluxDB_, or other open source time series databases
suitable for realtime analytics and sensor data storage.
- Automate dashboard management in the context of data arriving on different data channels
using Grafana_, an open source, feature rich metrics dashboard and graph editor.
- Make Browser clients first-class citizens of the underpinning software bus framework
Expand Down
12 changes: 7 additions & 5 deletions doc/source/about/technologies.rst
Original file line number Diff line number Diff line change
Expand Up @@ -26,14 +26,16 @@ Infrastructure components

- Kotori_, a data acquisition, graphing and telemetry toolkit
- Grafana_, a graph and dashboard builder for visualizing time series metrics
- InfluxDB_, a time-series database
- CrateDB_, a time-series database ¹
- InfluxDB_, a time-series database ¹
- Mosquitto_, an MQTT message broker
- MongoDB_, a document store (optional) ¹²
- MongoDB_, a document store (optional) ²³

| ¹ MongoDB is only required when doing CSV data acquisition, so it is completely
| ¹ Kotori can either use CrateDB or InfluxDB as timeseries database.
| ² MongoDB is only required when doing CSV data acquisition, so it is completely
| optional for regular operations of Kotori.
| ² As MongoDB - strictly speaking - stopped being free software recently (2018/2019),
| it will probably be phased out gradually and replaced by PostgreSQL_.
| ³ As MongoDB - strictly speaking - stopped being free software recently (2018/2019),
| it will probably be phased out gradually and replaced by PostgreSQL_ or CrateDB_.

Supported protocols
Expand Down
18 changes: 18 additions & 0 deletions doc/source/database/cratedb.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
.. include:: ../_resources.rst

.. _database-cratedb:

#######
CrateDB
#######

.. admonition::

.. todo:: Content on this page may need an update.

*****
About
*****

`CrateDB`_ is a distributed and scalable SQL database for storing and analyzing massive
amounts of data in `near real-time`_. It is PostgreSQL-compatible, and based on Lucene.
29 changes: 29 additions & 0 deletions doc/source/database/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ with Kotori. Adding more adapters is possible.
:maxdepth: 1
:hidden:
cratedb
influxdb
mongodb
```
Expand All @@ -24,6 +25,34 @@ mongodb
:padding: 0


:::::{grid-item-card}
::::{grid} 2
:margin: 0
:padding: 0

:::{grid-item}
:columns: 8
#### [](#database-cratedb)

CrateDB is a distributed and scalable SQL database for storing and analyzing massive
amounts of data in near real-time. It is PostgreSQL-compatible, and based on Lucene.

<small>
<strong>Categories:</strong> timeseries-database, multi-modal database
</small>
:::
:::{grid-item}
:columns: 4
{bdg-primary-line}`eth` {bdg-primary-line}`wifi` {bdg-primary-line}`http` {bdg-primary-line}`postgresql`

{bdg-success-line}`sql` {bdg-success-line}`blob`

{bdg-secondary-line}`amd64` {bdg-secondary-line}`arm64`
:::
::::
:::::


:::::{grid-item-card}
::::{grid} 2
:margin: 0
Expand Down
20 changes: 19 additions & 1 deletion doc/source/development/releasing/foundation.rst
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,24 @@ Download packages
scp mosquitto*.deb workbench@packages.example.org:/srv/packages/organizations/elmyra/foss/aptly/public/incoming


*******
CrateDB
*******
- https://github.com/crate/crate
- https://crate.io/docs/crate/tutorials/

Download packages
=================
::

# Download amd64 packages
wget https://cdn.crate.io/downloads/apt/stable/pool/main/c/crate/crate_5.3.2-1~buster_amd64.deb
wget https://cdn.crate.io/downloads/apt/stable/pool/main/c/crate/crate_5.3.2-1~bullseye_amd64.deb

# Upload to "incoming" directory
scp crate_*.deb workbench@packages.example.org:/srv/packages/organizations/elmyra/foss/aptly/public/incoming


********
InfluxDB
********
Expand Down Expand Up @@ -100,7 +118,7 @@ Publish packages

# Add packages to repository
aptly repo add -config=$APTLY_CONFIG -remove-files=true $APTLY_REPOSITORY \
$PACKAGES_INCOMING/influxdb*.deb $PACKAGES_INCOMING/grafana*.deb $PACKAGES_INCOMING/*mosquitto*.deb
$PACKAGES_INCOMING/crate*.deb $PACKAGES_INCOMING/influxdb*.deb $PACKAGES_INCOMING/grafana*.deb $PACKAGES_INCOMING/*mosquitto*.deb

# Publish repository
aptly publish update -config=$APTLY_CONFIG -gpg-key=2543A838 -passphrase=esp $APTLY_DISTRIBUTION
2 changes: 1 addition & 1 deletion doc/source/development/releasing/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ Releasing and packaging

This section describes how to cut a new release, how to build
and publish Kotori packages and how to add foundation packages
like InfluxDB and Grafana to the same package repositories.
like CrateDB, InfluxDB, and Grafana to the same package repositories.

.. toctree::
:maxdepth: 1
Expand Down
4 changes: 2 additions & 2 deletions doc/source/development/systemd.rst
Original file line number Diff line number Diff line change
Expand Up @@ -169,12 +169,12 @@ systemd troubleshooting

::

systemctl status influxdb mosquitto grafana-server kotori
systemctl status crate influxdb mosquitto grafana-server kotori


::

systemctl list-unit-files kotori* influx* grafana* mosquitto*
systemctl list-unit-files kotori* crate* influx* grafana* mosquitto*


::
Expand Down
6 changes: 3 additions & 3 deletions doc/source/development/tests.rst
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ The tests are mostly full integration tests. They are testing the whole system
and the interactions between the subsystems.

Messages will get published to the MQTT bus by shelling out to ``mosquitto_pub``.
After that, InfluxDB will be checked to contain the right data and Grafana will
After that, the database will be checked to contain the right data and Grafana will
be checked to be accurately provisioned.

While the shell-out can well be optimized for efficiency, it is also pleasant
Expand All @@ -32,12 +32,12 @@ Install some needed packages::
Foundation services
===================

The test suite will assume running instances of Mosquitto, InfluxDB, MongoDB
The test suite will assume running instances of Mosquitto, CrateDB, InfluxDB, MongoDB
and Grafana and fire up an in-process instance of Kotori to complement these. Please
have a look at :ref:`setup-docker` in order to get the complementing services
up and running in a quick and ad hoc manner.

Run Mosquitto, InfluxDB, MongoDB and Grafana as Docker containers::
Run Mosquitto, CrateDB, InfluxDB, MongoDB and Grafana as Docker containers::

make start-foundation-services

Expand Down
28 changes: 16 additions & 12 deletions doc/source/handbook/acquisition/error-signalling.rst
Original file line number Diff line number Diff line change
Expand Up @@ -48,13 +48,24 @@ the system will respond with a MQTT "response" on the corresponding topic with s
"description": "Error processing MQTT message \"{\"value\": 42.42\" from topic \"mqttkit-1/testdrive/area-42/node-1/data.json\"."
}

Database error
==============
When sending a payload with an existing field already seeded in a different data type::
Database errors
===============
When sending a payload with an existing field already materialized using a different data type::

echo '{"value": "invalid"}' | mosquitto_pub -h kotori.example.org -t mqttkit-1/testdrive/area-42/node-1/data.json -l

the system will respond with::
the system will respond correspondingly.

CrateDB::

mqttkit-1/testdrive/area-42/node-1/error.json {
"type": "<class 'crate.client.exceptions.ProgrammingError'>",
"message": "SQLParseException[Cannot cast object element `humidity` with value `invalid` to type `double precision`]",
"description": "Error processing MQTT message \"b'{\"temperature\": 42.84, \"humidity\": \"invalid\"}'\" from topic \"mqttkit-2/foo/bar/1/data.json\".",
"timestamp": "2023-06-20T12:00:08+00:00"
}

InfluxDB::

mqttkit-1/testdrive/area-42/node-1/error.json {
"message": "400: {\"error\":\"field type conflict: input field \\\"value\\\" on measurement \\\"area_42_node_1_sensors\\\" is type string, already exists as type float dropped=1\"}\n",
Expand All @@ -73,11 +84,4 @@ and then sending an invalid payload like::

echo '2017-05-01 22:39:09,invalid' | http POST http://kotori.example.org/api/mqttkit-1/testdrive/area-42/node-1/data Content-Type:text/csv

The system will also respond over MQTT with::

mqttkit-1/testdrive/area-42/node-1/error.json {
"message": "400: {\"error\":\"field type conflict: input field \\\"value\\\" on measurement \\\"area_42_node_1_sensors\\\" is type string, already exists as type float dropped=1\"}\n",
"type": "<class 'influxdb.exceptions.InfluxDBClientError'>",
"description": "Error processing MQTT message \"{\"time\": \"2017-05-01 22:40:09\", \"value\": \"invalid\"}\" from topic \"mqttkit-1/testdrive/area-42/node-1/data.json\"."
}

The system will also respond over MQTT correspondingly, like outlined above.
1 change: 1 addition & 0 deletions doc/source/handbook/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -49,4 +49,5 @@ Usage

usage/kotori
usage/grafana
usage/cratedb
usage/influxdb
14 changes: 14 additions & 0 deletions doc/source/handbook/usage/cratedb.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
.. include:: ../../_resources.rst

.. _cratedb-handbook:

################
CrateDB handbook
################

.. admonition::

.. todo:: Content on this page may need an update.

This section of the documentation will give a short and concise summary about
how to operate and work with CrateDB.
1 change: 1 addition & 0 deletions doc/source/integration/tts-ttn.rst
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ Overview

subgraph backend
direction LR
Kotori --> CrateDB
Kotori --> InfluxDB
Kotori --> Grafana
end
Expand Down
34 changes: 31 additions & 3 deletions doc/source/setup/docker.rst
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ its configuration at :ref:`getting-started`.
Introduction
************

This section outlines how to conveniently run Mosquitto, InfluxDB,
This section outlines how to conveniently run Mosquitto, CrateDB, InfluxDB,
MongoDB, Grafana and Kotori using Docker.

The repository provides two files ``docker-compose.yml`` and ``.env``. They
Expand All @@ -48,15 +48,15 @@ Those images are published to Docker Hub.

Please note that this Docker Compose configuration is primarily suited for
evaluation and development purposes. As it either disables authentication
or uses insecure authentication credentials for Mosquitto, InfluxDB,
or uses insecure authentication credentials for Mosquitto, CrateDB, InfluxDB,
and Grafana, it is not prepared for production setups.


*************
Prerequisites
*************

This will give you Mosquitto, InfluxDB, MongoDB, Grafana, an improved
This will give you Mosquitto, CrateDB, InfluxDB, MongoDB, Grafana, an improved
Grafana map panel plugin, and a command alias for invoking Kotori.

In order to invoke the auxiliary services, run::
Expand Down Expand Up @@ -94,6 +94,34 @@ Testdrive
This is a basic test walkthrough, to check if data is correctly routed from the
telemetry message bus to the database.

CrateDB
=======

This example uses CrateDB as timeseries-database.

Invoke Kotori::

kotori --config /etc/kotori/docker/docker-cratedb.ini

Publish single reading using MQTT::

export CHANNEL_TOPIC=sensorwan-cratedb/foo/bar/1/data.json
docker run \
--network kotori_default \
-it --rm eclipse-mosquitto \
mosquitto_pub -d -h mosquitto -t $CHANNEL_TOPIC -m '{"temperature": 42.84, "humidity": 83.1}'

Check if reading has been stored in CrateDB::

docker run \
--network kotori_default \
-it --rm crate \
crash --hosts cratedb -c 'SELECT * FROM sensorwan_cratedb_foo.bar_1_sensors;'

Go to Grafana and visit the dashboard just created::

open "http://localhost:3000/?orgId=1&search=open&query=sensorwan-cratedb"

InfluxDB
========

Expand Down
4 changes: 2 additions & 2 deletions doc/source/setup/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ from Docker Hub, from the Python Package Index (PyPI) or from the Git
repository.

For running Kotori in a full configuration, you will need some other
infrastructure services like Mosquitto_, InfluxDB_, Grafana_ and optionally
infrastructure services like Mosquitto_, CrateDB_, InfluxDB_, Grafana_ and optionally
MongoDB_.

Have fun and enjoy your data acquisition!
Expand Down Expand Up @@ -62,7 +62,7 @@ Details


¹ When choosing to install from the Debian package repository, you will also be
able to receive appropriate Debian packages for Mosquitto, InfluxDB and Grafana
able to receive appropriate Debian packages for Mosquitto, CrateDB, InfluxDB, and Grafana
through the `DaqZilla package repository <https://packages.elmyra.de/elmyra/foss/debian/>`_.
This makes it easy to setup the complete DAQ system from a single package source.

Expand Down
1 change: 1 addition & 0 deletions doc/source/setup/linux-arch.rst
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,7 @@ MongoDB is provided through the Arch User Repository (AUR)::
cd mongodb-tools-bin
makepkg -si

.. todo:: Add walkthrough how to install CrateDB on Arch Linux.

Kotori
======
Expand Down
Loading

0 comments on commit fcd4379

Please sign in to comment.