Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[dbquery] Initial contribution #8780

Merged
merged 49 commits into from
Oct 17, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
32ce395
Initial commit
lujop Sep 26, 2020
93acd40
Implement reconnect attempts
lujop Sep 26, 2020
c60896e
Minor documentation changes and fix trigger channel name
lujop Oct 10, 2020
7974e3a
Fix NPE bug initializing ThingActions
lujop Oct 10, 2020
ac330ae
Implement query actions and another fixes
lujop Oct 12, 2020
a275713
Update parameters and correct channel
lujop Oct 16, 2020
98a55ad
Fix formatting and forgot part on previous commit
lujop Oct 17, 2020
5874268
Improve documentation
lujop Oct 17, 2020
4e75253
Add javadoc comment and license to all classes
lujop Oct 17, 2020
99f4dbf
Code cleanup
lujop Oct 17, 2020
e7ee6b2
Untrack unused i18n file
lujop Oct 17, 2020
f219a4a
Fix log level for query actions trace information
lujop Oct 17, 2020
41e7827
Add dbquery addon to bundles pom
lujop Oct 17, 2020
21c0995
Temporary remove mqtt bindings that make travis build to fail
lujop Oct 17, 2020
b4ac7b5
Fix formatting issue
lujop Oct 17, 2020
06802a8
Revert "Temporary remove mqtt bindings that make travis build to fail"
lujop Oct 18, 2020
2a6fec8
Code clean up from static analysis
lujop Oct 18, 2020
ba2ae9f
Merge branch 'main' into dbquery_initial_submission
Skinah Feb 25, 2021
c7efbcf
Update code to be compatible with 3.1.0
lujop Mar 5, 2021
6f0c114
Merge branch 'main' into dbquery_initial_submission
lujop May 23, 2021
4afe2af
Requested PR changes
lujop May 23, 2021
c5ec59e
Update bundles/org.openhab.binding.dbquery/src/main/java/org/openhab/…
lujop Aug 19, 2021
a05e81a
Update bundles/org.openhab.binding.dbquery/README.md
lujop Aug 19, 2021
318e4e7
Update bundles/org.openhab.binding.dbquery/README.md
lujop Aug 19, 2021
89a4ac0
Update bundles/org.openhab.binding.dbquery/README.md
lujop Aug 19, 2021
dae8934
Update bundles/org.openhab.binding.dbquery/README.md
lujop Aug 19, 2021
2fb00ab
Update bundles/org.openhab.binding.dbquery/README.md
lujop Aug 19, 2021
143ac55
Update bundles/org.openhab.binding.dbquery/README.md
lujop Aug 19, 2021
fdf75a0
Update bundles/org.openhab.binding.dbquery/README.md
lujop Aug 19, 2021
8513271
Update bundles/org.openhab.binding.dbquery/README.md
lujop Aug 19, 2021
c0abd28
Update bundles/org.openhab.binding.dbquery/README.md
lujop Aug 19, 2021
729b1a2
Update bundles/org.openhab.binding.dbquery/README.md
lujop Aug 19, 2021
ef8d6d0
Update bundles/org.openhab.binding.dbquery/README.md
lujop Aug 19, 2021
701583f
Update bundles/org.openhab.binding.dbquery/README.md
lujop Aug 19, 2021
24d9a18
Update bundles/org.openhab.binding.dbquery/README.md
lujop Aug 19, 2021
0ea5efe
Update bundles/org.openhab.binding.dbquery/src/main/java/org/openhab/…
lujop Aug 19, 2021
2a696b0
Apply suggestions from code review
lujop Aug 22, 2021
e900721
Suggestions from code review
lujop Aug 22, 2021
255e561
Merge remote-tracking branch 'openhab/main' into dbquery_initial_subm…
lujop Aug 22, 2021
7a9ec0b
Update parent version
lujop Aug 23, 2021
0abdf2f
Update bundles/org.openhab.binding.dbquery/src/main/resources/OH-INF/…
lujop Sep 8, 2021
dd2ec87
Update bundles/org.openhab.binding.dbquery/src/main/resources/OH-INF/…
lujop Sep 8, 2021
e5cc8c1
Update bundles/org.openhab.binding.dbquery/src/main/resources/OH-INF/…
lujop Sep 8, 2021
2f58d26
Update bundles/org.openhab.binding.dbquery/src/main/resources/OH-INF/…
lujop Sep 8, 2021
e1453e0
Update bundles/org.openhab.binding.dbquery/src/main/resources/OH-INF/…
lujop Sep 8, 2021
b175337
Changes asked in PR review
lujop Sep 22, 2021
0895e03
Update bundles/org.openhab.binding.dbquery/README.md
lujop Sep 30, 2021
cf1155a
README documentation imporovements
lujop Sep 30, 2021
fc2be94
Fix format issue
lujop Sep 30, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CODEOWNERS
Validating CODEOWNERS rules …
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@
/bundles/org.openhab.binding.dali/ @rs22
/bundles/org.openhab.binding.danfossairunit/ @pravussum
/bundles/org.openhab.binding.darksky/ @cweitkamp
/bundles/org.openhab.binding.dbquery/ @lujop
/bundles/org.openhab.binding.deconz/ @openhab/add-ons-maintainers
/bundles/org.openhab.binding.denonmarantz/ @jwveldhuis
/bundles/org.openhab.binding.digiplex/ @rmichalak
Expand Down
5 changes: 5 additions & 0 deletions bom/openhab-addons/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -286,6 +286,11 @@
<artifactId>org.openhab.binding.darksky</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.openhab.addons.bundles</groupId>
<artifactId>org.openhab.binding.dbquery</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.openhab.addons.bundles</groupId>
<artifactId>org.openhab.binding.deconz</artifactId>
Expand Down
210 changes: 210 additions & 0 deletions bundles/org.openhab.binding.dbquery/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,210 @@
# DBQuery Binding

This binding allows creating items from the result of native database queries.
It currently only supports InfluxDB 2.X.

You can use the addon in any situation where you want to create an item from a native query.
The source of the query can be any supported database, and doesn't need to be the one you use as the persistence service in openHAB.
Some use cases can be:

- Integrate a device that stores its data in a database
- Query derived data from you openHAB persistence, for example with Influx2 tasks you can process your data to create a new one
- Bypass limitations of current openHAB persistence queries

## Supported Things

There are two types of supported things: `influxdb2` and a `query`.
For each different database you want to connect to, you must define a `Bridge` thing for that database.
Then each `Bridge` can define as many `Query` things that you want to execute.

## Thing Configuration

### Bridges

#### influxdb2

Defines a connection to an Influx2 database and allows creating queries on it.

| Parameter | Required | Description |
|--------------|----------|----------------------------------------- |
| url | Yes | database url |
| user | Yes | name of the database user |
| token | Yes | token to authenticate to the database ([Intructions about how to create one](https://v2.docs.influxdata.com/v2.0/security/tokens/create-token/)) |
| organization | Yes | database organization name |
| bucket | Yes | database bucket name |

### query

The `Query` thing defines a native query that provides several channels that you can bind to items.

#### Query parameters

The query items support the following parameters:

| Parameter | Required | Default | Description |
|--------------|----------|----------|-----------------------------------------------------------------------|
| query | true | | Query string in native syntax |
| interval | false | 0 | Interval in seconds in which the query is automatically executed |
| hasParameters| false | false | True if the query has parameters, false otherwise |
| timeout | false | 0 | Query execution timeout in seconds |
| scalarResult | false | true | If query always returns a single value or not |
| scalarColumn | false | | In case of multiple columns, it indicates which to use for scalarResult|

These are described further in the following subsections.

##### query

The query the items represents in the native language of your database:

- Flux for `influxdb2`

#### hasParameters

If `hasParameters=true` you can use parameters in the query string that can be dynamically set with the `setQueryParameters` action.

For InfluxDB use the `${paramName}` syntax for each parameter, and keep in mind that the values from that parameters must be from a trusted source as current
parameter substitution is subject to query injection attacks.

#### timeout

A time-out in seconds to wait for the query result, if it's exceeded, the result will be discarded and the addon will do its best to cancel the query.
Currently it's ignored and it will be implemented in a future version.

#### scalarResult

If `true` the query is expected to return a single scalar value that will be available to `result` channels as string, number, boolean,...
If the query can return several rows and/or several columns per row then it needs to be set to `false` and the result can be retrieved in `resultString`
channel as JSON or using the `getLastQueryResult` action.

#### scalarColumn

In case `scalarResult` is `true` and the select returns multiple columns you can use that parameter to choose which column to use to extract the result.

## Channels

Query items offer the following channels to be able to query / bind them to items:

| Channel Type ID | Item Type | Description |
|-----------------|-----------|------------------------------------------------------------------------------------------------------------------------------------|
| execute | Switch | Send `ON` to execute the query manually. It also indicates if query is currently running (`ON`) or not running (`OFF`) |
| resultString | String | Result of last executed query as a String |
| resultNumber | Number | Result of last executed query as a Number, query must have `scalarResult=true` |
| resultDateTime | DateTime | Result of last executed query as a DateTime, query must have `scalarResult=true` |
| resultContact | Contact | Result of last executed query as Contact, query must have `scalarResult=true` |
| resultSwitch | Switch | Result of last executed query as Switch, query must have `scalarResult=true` |
| parameters | String | Contains parameters of last executed query as JSON|
| correct | Switch | `ON` if the last executed query completed successfully, `OFF` if the query failed.|

All the channels, except `execute`, are updated when the query execution finishes, and while there is a query in execution they have the values from
last previous executed query.

The `resultString` channel is the only valid one if `scalarResult=false`, and in that case it contains the query result serialized to JSON in that format:

{
correct : true,
data : [
{
column1 : value,
column2 : value
},
{ ... }, //row2
{ ... } //row3
]
}

### Channel Triggers

#### calculateParameters

Triggers when there's a need to calculate parameters before query execution.
When a query has `hasParameters=true` it fires the `calculateParameters` channel trigger and pauses the execution until `setQueryParameters` action is call in
that query.

In the case a query has parameters, it's expected that there is a rule that catches the `calculateParameters` trigger, calculate the parameters with the corresponding logic and then calls the `setQueryParameters` action, after that the query will be executed.

## Actions

### For DatabaseBridge

#### executeQuery

It allows executing a query synchronously from a script/rule without defining it in a Thing.

To execute the action you need to pass the following parameters:

- String query: The query to execute
- Map<String,Object>: Query parameters (empty map if not needed)
- int timeout: Query timeout in seconds

And it returns an `ActionQueryResult` that has the following properties:

- correct (boolean) : True if the query was executed correctly, false otherwise
- data (List<Map<String,Object>>): A list where each element is a row that is stored in a map with (columnName,value) entries
- isScalarResult: It returns if the result is scalar one (only one row with one column)
- resultAsScalar: It returns the result as a scalar if possible, if not returns null


Example (using Jython script):

from core.log import logging, LOG_PREFIX
log = logging.getLogger("{}.action_example".format(LOG_PREFIX))
map = {"time" : "-2h"}
influxdb = actions.get("dbquery","dbquery:influxdb2:sampleQuery") //Get bridge thing
result = influxdb.executeQuery("from(bucket: \"default\") |> range(start:-2h) |> filter(fn: (r) => r[\"_measurement\"] == \"go_memstats_frees_total\") |> filter(fn: (r) => r[\"_field\"] == \"counter\") |> mean()",{},5)
log.info("execute query result is "+str(result.data))


Use this action with care, because as the query is executed synchronously, it is not good to execute long-running queries that can block script execution.

### For Queries

#### setQueryParameters

It's used for queries with parameters to set them.
To execute the action you need to pass the parameters as a Map.

Example (using Jython script):

params = {"time" : "-2h"}
dbquery = actions.get("dbquery","dbquery:query:queryWithParams") //Get query thing
dbquery.setQueryParameters(params)

#### getLastQueryResult

It can be used in scripts to get the last query result.
It doesn't have any parameters and returns an `ActionQueryResult` as defined in `executeQuery` action.

Example (using Jython script):

dbquery = actions.get("dbquery","dbquery:query:queryWithParams") //Get query thing
result = dbquery.getLastQueryResult()


## Examples

### The Simplest case

Define a InfluxDB2 database thing and a query with an interval execution.
That executes the query every 15 seconds and punts the result in `myItem`.

# Bridge Thing definition
Bridge dbquery:influxdb2:mydatabase "InfluxDB2 Bridge" [ bucket="default", user="admin", url="http://localhost:8086", organization="openhab", token="*******" ]

# Query Thing definition
Thing dbquery:query:myquery "My Query" [ interval=15, hasParameters=false, scalarResult=true, timeout=0, query="from(bucket: \"default\") |> range(start:-1h) |> filter(fn: (r) => r[\"_measurement\"] == \"go_memstats_frees_total\") |> filter(fn: (r) => r[\"_field\"] == \"counter\") |> mean()", scalarColumn="_value" ]

# Item definition
Number myItem "QueryResult" {channel="dbquery:query:myquery:resultNumber"}

### A query with parameters

Using the previous example you change the `range(start:-1h)` for `range(start:${time})`

Create a rule that is fired

- **When** `calculateParameters` is triggered in `myquery`
- **Then** executes the following script action (in that example Jython):

map = {"time" : "-2h"}
dbquery = actions.get("dbquery","dbquery:query:myquery")
dbquery.setQueryParameters(map)
107 changes: 107 additions & 0 deletions bundles/org.openhab.binding.dbquery/pom.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://maven.apache.org/POM/4.0.0"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

<modelVersion>4.0.0</modelVersion>

<parent>
<groupId>org.openhab.addons.bundles</groupId>
<artifactId>org.openhab.addons.reactor.bundles</artifactId>
<version>3.2.0-SNAPSHOT</version>
</parent>

<artifactId>org.openhab.binding.dbquery</artifactId>

<name>openHAB Add-ons :: Bundles :: DBQuery Binding</name>

<properties>
<bnd.importpackage>
!javax.annotation;!android.*,!com.android.*,!com.google.appengine.*,!dalvik.system,!kotlin.*,!kotlinx.*,!org.conscrypt,!sun.security.ssl,!org.apache.harmony.*,!org.apache.http.*,!rx.*,!org.msgpack.*
</bnd.importpackage>
</properties>

<dependencies>
<!-- influxdb-client-java -->
<dependency>
<groupId>com.influxdb</groupId>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you consider adding the dependencies via Karaf (feature.xml), so that the dependencies need to be downloaded only once if the influxdb persistence is also installed? (To make this work, the same adjustment needs to be done to the persistence addon)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My intent, if it's possible, is to maintain independence between the persistence plugins.
To be able to evolve differently if needed and use different versions or implementations in the future if it's desired.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the versions differ, both versions will be used. But if the versions are identical, the dependency will be reused. You won't introduce a dependency between the two.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right.
Ok, I will try then but will need some support and guidance on that.
Is there any recent PR or documentation that I can take as a reference on how to do it?

Also, I will try to do all corrections in a row when you finish the review.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could take a look here:

<bundle dependency="true">mvn:org.jsoup/jsoup/1.8.3</bundle>

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@fwolter: It's only adding that line and leaving the pom the same way?
Or it's needed to create the bundle too?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You don't need to create any bundle. But you need to change the scope in the pom.xml from compile to provided.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it's possible I prefer to leave this as it is for this first version.
Because I prefer to don't put more changes including modification of persistence addon.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK

<artifactId>influxdb-client-java</artifactId>
<version>1.6.0</version>
</dependency>
<dependency>
<artifactId>influxdb-client-core</artifactId>
<groupId>com.influxdb</groupId>
<version>1.6.0</version>
</dependency>
<dependency>
<artifactId>converter-gson</artifactId>
<groupId>com.squareup.retrofit2</groupId>
<version>2.5.0</version>
</dependency>
<dependency>
<artifactId>converter-scalars</artifactId>
<groupId>com.squareup.retrofit2</groupId>
<version>2.5.0</version>
</dependency>
<dependency> <!-- also used for querydb library -->
<artifactId>gson</artifactId>
<groupId>com.google.code.gson</groupId>
<version>2.8.5</version>
</dependency>
<dependency>
<artifactId>gson-fire</artifactId>
<groupId>io.gsonfire</groupId>
<version>1.8.0</version>
</dependency>
<dependency>
<artifactId>okio</artifactId>
<groupId>com.squareup.okio</groupId>
<version>1.17.3</version>
</dependency>
<dependency>
<artifactId>commons-csv</artifactId>
<groupId>org.apache.commons</groupId>
<version>1.6</version>
</dependency>
Skinah marked this conversation as resolved.
Show resolved Hide resolved
<dependency>
<artifactId>json</artifactId>
<groupId>org.json</groupId>
<version>20180813</version>
</dependency>
<dependency>
<artifactId>okhttp</artifactId>
<groupId>com.squareup.okhttp3</groupId>
<version>3.14.4</version>
</dependency>
<dependency>
<artifactId>retrofit</artifactId>
<groupId>com.squareup.retrofit2</groupId>
<version>2.6.2</version>
</dependency>
<dependency>
<artifactId>jsr305</artifactId>
<groupId>com.google.code.findbugs</groupId>
<version>3.0.2</version>
</dependency>
<dependency>
<artifactId>logging-interceptor</artifactId>
<groupId>com.squareup.okhttp3</groupId>
<version>3.14.4</version>
</dependency>
<dependency>
<artifactId>rxjava</artifactId>
<groupId>io.reactivex.rxjava2</groupId>
<version>2.2.17</version>
</dependency>
<dependency>
<artifactId>reactive-streams</artifactId>
<groupId>org.reactivestreams</groupId>
<version>1.0.3</version>
</dependency>
<dependency>
<artifactId>swagger-annotations</artifactId>
<groupId>io.swagger</groupId>
<version>1.5.22</version>
</dependency>
<!-- end influxdb-client-java -->
</dependencies>
</project>
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
<?xml version="1.0" encoding="UTF-8"?>
<features name="org.openhab.binding.dbquery-${project.version}" xmlns="http://karaf.apache.org/xmlns/features/v1.4.0">
<repository>mvn:org.openhab.core.features.karaf/org.openhab.core.features.karaf.openhab-core/${ohc.version}/xml/features</repository>

<feature name="openhab-binding-dbquery" description="DBQuery Binding" version="${project.version}">
<feature>openhab-runtime-base</feature>
<bundle start-level="80">mvn:org.openhab.addons.bundles/org.openhab.binding.dbquery/${project.version}</bundle>
</feature>
</features>
Loading