-
-
Notifications
You must be signed in to change notification settings - Fork 3.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[dbquery] Initial contribution #8780
Merged
Merged
Changes from all commits
Commits
Show all changes
49 commits
Select commit
Hold shift + click to select a range
32ce395
Initial commit
lujop 93acd40
Implement reconnect attempts
lujop c60896e
Minor documentation changes and fix trigger channel name
lujop 7974e3a
Fix NPE bug initializing ThingActions
lujop ac330ae
Implement query actions and another fixes
lujop a275713
Update parameters and correct channel
lujop 98a55ad
Fix formatting and forgot part on previous commit
lujop 5874268
Improve documentation
lujop 4e75253
Add javadoc comment and license to all classes
lujop 99f4dbf
Code cleanup
lujop e7ee6b2
Untrack unused i18n file
lujop f219a4a
Fix log level for query actions trace information
lujop 41e7827
Add dbquery addon to bundles pom
lujop 21c0995
Temporary remove mqtt bindings that make travis build to fail
lujop b4ac7b5
Fix formatting issue
lujop 06802a8
Revert "Temporary remove mqtt bindings that make travis build to fail"
lujop 2a6fec8
Code clean up from static analysis
lujop ba2ae9f
Merge branch 'main' into dbquery_initial_submission
Skinah c7efbcf
Update code to be compatible with 3.1.0
lujop 6f0c114
Merge branch 'main' into dbquery_initial_submission
lujop 4afe2af
Requested PR changes
lujop c5ec59e
Update bundles/org.openhab.binding.dbquery/src/main/java/org/openhab/…
lujop a05e81a
Update bundles/org.openhab.binding.dbquery/README.md
lujop 318e4e7
Update bundles/org.openhab.binding.dbquery/README.md
lujop 89a4ac0
Update bundles/org.openhab.binding.dbquery/README.md
lujop dae8934
Update bundles/org.openhab.binding.dbquery/README.md
lujop 2fb00ab
Update bundles/org.openhab.binding.dbquery/README.md
lujop 143ac55
Update bundles/org.openhab.binding.dbquery/README.md
lujop fdf75a0
Update bundles/org.openhab.binding.dbquery/README.md
lujop 8513271
Update bundles/org.openhab.binding.dbquery/README.md
lujop c0abd28
Update bundles/org.openhab.binding.dbquery/README.md
lujop 729b1a2
Update bundles/org.openhab.binding.dbquery/README.md
lujop ef8d6d0
Update bundles/org.openhab.binding.dbquery/README.md
lujop 701583f
Update bundles/org.openhab.binding.dbquery/README.md
lujop 24d9a18
Update bundles/org.openhab.binding.dbquery/README.md
lujop 0ea5efe
Update bundles/org.openhab.binding.dbquery/src/main/java/org/openhab/…
lujop 2a696b0
Apply suggestions from code review
lujop e900721
Suggestions from code review
lujop 255e561
Merge remote-tracking branch 'openhab/main' into dbquery_initial_subm…
lujop 7a9ec0b
Update parent version
lujop 0abdf2f
Update bundles/org.openhab.binding.dbquery/src/main/resources/OH-INF/…
lujop dd2ec87
Update bundles/org.openhab.binding.dbquery/src/main/resources/OH-INF/…
lujop e5cc8c1
Update bundles/org.openhab.binding.dbquery/src/main/resources/OH-INF/…
lujop 2f58d26
Update bundles/org.openhab.binding.dbquery/src/main/resources/OH-INF/…
lujop e1453e0
Update bundles/org.openhab.binding.dbquery/src/main/resources/OH-INF/…
lujop b175337
Changes asked in PR review
lujop 0895e03
Update bundles/org.openhab.binding.dbquery/README.md
lujop cf1155a
README documentation imporovements
lujop fc2be94
Fix format issue
lujop File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
Validating CODEOWNERS rules …
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,210 @@ | ||
# DBQuery Binding | ||
|
||
This binding allows creating items from the result of native database queries. | ||
It currently only supports InfluxDB 2.X. | ||
|
||
You can use the addon in any situation where you want to create an item from a native query. | ||
The source of the query can be any supported database, and doesn't need to be the one you use as the persistence service in openHAB. | ||
Some use cases can be: | ||
|
||
- Integrate a device that stores its data in a database | ||
- Query derived data from you openHAB persistence, for example with Influx2 tasks you can process your data to create a new one | ||
- Bypass limitations of current openHAB persistence queries | ||
|
||
## Supported Things | ||
|
||
There are two types of supported things: `influxdb2` and a `query`. | ||
For each different database you want to connect to, you must define a `Bridge` thing for that database. | ||
Then each `Bridge` can define as many `Query` things that you want to execute. | ||
|
||
## Thing Configuration | ||
|
||
### Bridges | ||
|
||
#### influxdb2 | ||
|
||
Defines a connection to an Influx2 database and allows creating queries on it. | ||
|
||
| Parameter | Required | Description | | ||
|--------------|----------|----------------------------------------- | | ||
| url | Yes | database url | | ||
| user | Yes | name of the database user | | ||
| token | Yes | token to authenticate to the database ([Intructions about how to create one](https://v2.docs.influxdata.com/v2.0/security/tokens/create-token/)) | | ||
| organization | Yes | database organization name | | ||
| bucket | Yes | database bucket name | | ||
|
||
### query | ||
|
||
The `Query` thing defines a native query that provides several channels that you can bind to items. | ||
|
||
#### Query parameters | ||
|
||
The query items support the following parameters: | ||
|
||
| Parameter | Required | Default | Description | | ||
|--------------|----------|----------|-----------------------------------------------------------------------| | ||
| query | true | | Query string in native syntax | | ||
| interval | false | 0 | Interval in seconds in which the query is automatically executed | | ||
| hasParameters| false | false | True if the query has parameters, false otherwise | | ||
| timeout | false | 0 | Query execution timeout in seconds | | ||
| scalarResult | false | true | If query always returns a single value or not | | ||
| scalarColumn | false | | In case of multiple columns, it indicates which to use for scalarResult| | ||
|
||
These are described further in the following subsections. | ||
|
||
##### query | ||
|
||
The query the items represents in the native language of your database: | ||
|
||
- Flux for `influxdb2` | ||
|
||
#### hasParameters | ||
|
||
If `hasParameters=true` you can use parameters in the query string that can be dynamically set with the `setQueryParameters` action. | ||
|
||
For InfluxDB use the `${paramName}` syntax for each parameter, and keep in mind that the values from that parameters must be from a trusted source as current | ||
parameter substitution is subject to query injection attacks. | ||
|
||
#### timeout | ||
|
||
A time-out in seconds to wait for the query result, if it's exceeded, the result will be discarded and the addon will do its best to cancel the query. | ||
Currently it's ignored and it will be implemented in a future version. | ||
|
||
#### scalarResult | ||
|
||
If `true` the query is expected to return a single scalar value that will be available to `result` channels as string, number, boolean,... | ||
If the query can return several rows and/or several columns per row then it needs to be set to `false` and the result can be retrieved in `resultString` | ||
channel as JSON or using the `getLastQueryResult` action. | ||
|
||
#### scalarColumn | ||
|
||
In case `scalarResult` is `true` and the select returns multiple columns you can use that parameter to choose which column to use to extract the result. | ||
|
||
## Channels | ||
|
||
Query items offer the following channels to be able to query / bind them to items: | ||
|
||
| Channel Type ID | Item Type | Description | | ||
|-----------------|-----------|------------------------------------------------------------------------------------------------------------------------------------| | ||
| execute | Switch | Send `ON` to execute the query manually. It also indicates if query is currently running (`ON`) or not running (`OFF`) | | ||
| resultString | String | Result of last executed query as a String | | ||
| resultNumber | Number | Result of last executed query as a Number, query must have `scalarResult=true` | | ||
| resultDateTime | DateTime | Result of last executed query as a DateTime, query must have `scalarResult=true` | | ||
| resultContact | Contact | Result of last executed query as Contact, query must have `scalarResult=true` | | ||
| resultSwitch | Switch | Result of last executed query as Switch, query must have `scalarResult=true` | | ||
| parameters | String | Contains parameters of last executed query as JSON| | ||
| correct | Switch | `ON` if the last executed query completed successfully, `OFF` if the query failed.| | ||
|
||
All the channels, except `execute`, are updated when the query execution finishes, and while there is a query in execution they have the values from | ||
last previous executed query. | ||
|
||
The `resultString` channel is the only valid one if `scalarResult=false`, and in that case it contains the query result serialized to JSON in that format: | ||
|
||
{ | ||
correct : true, | ||
data : [ | ||
{ | ||
column1 : value, | ||
column2 : value | ||
}, | ||
{ ... }, //row2 | ||
{ ... } //row3 | ||
] | ||
} | ||
|
||
### Channel Triggers | ||
|
||
#### calculateParameters | ||
|
||
Triggers when there's a need to calculate parameters before query execution. | ||
When a query has `hasParameters=true` it fires the `calculateParameters` channel trigger and pauses the execution until `setQueryParameters` action is call in | ||
that query. | ||
|
||
In the case a query has parameters, it's expected that there is a rule that catches the `calculateParameters` trigger, calculate the parameters with the corresponding logic and then calls the `setQueryParameters` action, after that the query will be executed. | ||
|
||
## Actions | ||
|
||
### For DatabaseBridge | ||
|
||
#### executeQuery | ||
|
||
It allows executing a query synchronously from a script/rule without defining it in a Thing. | ||
|
||
To execute the action you need to pass the following parameters: | ||
|
||
- String query: The query to execute | ||
- Map<String,Object>: Query parameters (empty map if not needed) | ||
- int timeout: Query timeout in seconds | ||
|
||
And it returns an `ActionQueryResult` that has the following properties: | ||
|
||
- correct (boolean) : True if the query was executed correctly, false otherwise | ||
- data (List<Map<String,Object>>): A list where each element is a row that is stored in a map with (columnName,value) entries | ||
- isScalarResult: It returns if the result is scalar one (only one row with one column) | ||
- resultAsScalar: It returns the result as a scalar if possible, if not returns null | ||
|
||
|
||
Example (using Jython script): | ||
|
||
from core.log import logging, LOG_PREFIX | ||
log = logging.getLogger("{}.action_example".format(LOG_PREFIX)) | ||
map = {"time" : "-2h"} | ||
influxdb = actions.get("dbquery","dbquery:influxdb2:sampleQuery") //Get bridge thing | ||
result = influxdb.executeQuery("from(bucket: \"default\") |> range(start:-2h) |> filter(fn: (r) => r[\"_measurement\"] == \"go_memstats_frees_total\") |> filter(fn: (r) => r[\"_field\"] == \"counter\") |> mean()",{},5) | ||
log.info("execute query result is "+str(result.data)) | ||
|
||
|
||
Use this action with care, because as the query is executed synchronously, it is not good to execute long-running queries that can block script execution. | ||
|
||
### For Queries | ||
|
||
#### setQueryParameters | ||
|
||
It's used for queries with parameters to set them. | ||
To execute the action you need to pass the parameters as a Map. | ||
|
||
Example (using Jython script): | ||
|
||
params = {"time" : "-2h"} | ||
dbquery = actions.get("dbquery","dbquery:query:queryWithParams") //Get query thing | ||
dbquery.setQueryParameters(params) | ||
|
||
#### getLastQueryResult | ||
|
||
It can be used in scripts to get the last query result. | ||
It doesn't have any parameters and returns an `ActionQueryResult` as defined in `executeQuery` action. | ||
|
||
Example (using Jython script): | ||
|
||
dbquery = actions.get("dbquery","dbquery:query:queryWithParams") //Get query thing | ||
result = dbquery.getLastQueryResult() | ||
|
||
|
||
## Examples | ||
|
||
### The Simplest case | ||
|
||
Define a InfluxDB2 database thing and a query with an interval execution. | ||
That executes the query every 15 seconds and punts the result in `myItem`. | ||
|
||
# Bridge Thing definition | ||
Bridge dbquery:influxdb2:mydatabase "InfluxDB2 Bridge" [ bucket="default", user="admin", url="http://localhost:8086", organization="openhab", token="*******" ] | ||
|
||
# Query Thing definition | ||
Thing dbquery:query:myquery "My Query" [ interval=15, hasParameters=false, scalarResult=true, timeout=0, query="from(bucket: \"default\") |> range(start:-1h) |> filter(fn: (r) => r[\"_measurement\"] == \"go_memstats_frees_total\") |> filter(fn: (r) => r[\"_field\"] == \"counter\") |> mean()", scalarColumn="_value" ] | ||
|
||
# Item definition | ||
Number myItem "QueryResult" {channel="dbquery:query:myquery:resultNumber"} | ||
|
||
### A query with parameters | ||
|
||
Using the previous example you change the `range(start:-1h)` for `range(start:${time})` | ||
|
||
Create a rule that is fired | ||
|
||
- **When** `calculateParameters` is triggered in `myquery` | ||
- **Then** executes the following script action (in that example Jython): | ||
|
||
map = {"time" : "-2h"} | ||
dbquery = actions.get("dbquery","dbquery:query:myquery") | ||
dbquery.setQueryParameters(map) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,107 @@ | ||
<?xml version="1.0" encoding="UTF-8"?> | ||
<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://maven.apache.org/POM/4.0.0" | ||
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> | ||
|
||
<modelVersion>4.0.0</modelVersion> | ||
|
||
<parent> | ||
<groupId>org.openhab.addons.bundles</groupId> | ||
<artifactId>org.openhab.addons.reactor.bundles</artifactId> | ||
<version>3.2.0-SNAPSHOT</version> | ||
</parent> | ||
|
||
<artifactId>org.openhab.binding.dbquery</artifactId> | ||
|
||
<name>openHAB Add-ons :: Bundles :: DBQuery Binding</name> | ||
|
||
<properties> | ||
<bnd.importpackage> | ||
!javax.annotation;!android.*,!com.android.*,!com.google.appengine.*,!dalvik.system,!kotlin.*,!kotlinx.*,!org.conscrypt,!sun.security.ssl,!org.apache.harmony.*,!org.apache.http.*,!rx.*,!org.msgpack.* | ||
</bnd.importpackage> | ||
</properties> | ||
|
||
<dependencies> | ||
<!-- influxdb-client-java --> | ||
<dependency> | ||
<groupId>com.influxdb</groupId> | ||
<artifactId>influxdb-client-java</artifactId> | ||
<version>1.6.0</version> | ||
</dependency> | ||
<dependency> | ||
<artifactId>influxdb-client-core</artifactId> | ||
<groupId>com.influxdb</groupId> | ||
<version>1.6.0</version> | ||
</dependency> | ||
<dependency> | ||
<artifactId>converter-gson</artifactId> | ||
<groupId>com.squareup.retrofit2</groupId> | ||
<version>2.5.0</version> | ||
</dependency> | ||
<dependency> | ||
<artifactId>converter-scalars</artifactId> | ||
<groupId>com.squareup.retrofit2</groupId> | ||
<version>2.5.0</version> | ||
</dependency> | ||
<dependency> <!-- also used for querydb library --> | ||
<artifactId>gson</artifactId> | ||
<groupId>com.google.code.gson</groupId> | ||
<version>2.8.5</version> | ||
</dependency> | ||
<dependency> | ||
<artifactId>gson-fire</artifactId> | ||
<groupId>io.gsonfire</groupId> | ||
<version>1.8.0</version> | ||
</dependency> | ||
<dependency> | ||
<artifactId>okio</artifactId> | ||
<groupId>com.squareup.okio</groupId> | ||
<version>1.17.3</version> | ||
</dependency> | ||
<dependency> | ||
<artifactId>commons-csv</artifactId> | ||
<groupId>org.apache.commons</groupId> | ||
<version>1.6</version> | ||
</dependency> | ||
Skinah marked this conversation as resolved.
Show resolved
Hide resolved
|
||
<dependency> | ||
<artifactId>json</artifactId> | ||
<groupId>org.json</groupId> | ||
<version>20180813</version> | ||
</dependency> | ||
<dependency> | ||
<artifactId>okhttp</artifactId> | ||
<groupId>com.squareup.okhttp3</groupId> | ||
<version>3.14.4</version> | ||
</dependency> | ||
<dependency> | ||
<artifactId>retrofit</artifactId> | ||
<groupId>com.squareup.retrofit2</groupId> | ||
<version>2.6.2</version> | ||
</dependency> | ||
<dependency> | ||
<artifactId>jsr305</artifactId> | ||
<groupId>com.google.code.findbugs</groupId> | ||
<version>3.0.2</version> | ||
</dependency> | ||
<dependency> | ||
<artifactId>logging-interceptor</artifactId> | ||
<groupId>com.squareup.okhttp3</groupId> | ||
<version>3.14.4</version> | ||
</dependency> | ||
<dependency> | ||
<artifactId>rxjava</artifactId> | ||
<groupId>io.reactivex.rxjava2</groupId> | ||
<version>2.2.17</version> | ||
</dependency> | ||
<dependency> | ||
<artifactId>reactive-streams</artifactId> | ||
<groupId>org.reactivestreams</groupId> | ||
<version>1.0.3</version> | ||
</dependency> | ||
<dependency> | ||
<artifactId>swagger-annotations</artifactId> | ||
<groupId>io.swagger</groupId> | ||
<version>1.5.22</version> | ||
</dependency> | ||
<!-- end influxdb-client-java --> | ||
</dependencies> | ||
</project> |
9 changes: 9 additions & 0 deletions
9
bundles/org.openhab.binding.dbquery/src/main/feature/feature.xml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,9 @@ | ||
<?xml version="1.0" encoding="UTF-8"?> | ||
<features name="org.openhab.binding.dbquery-${project.version}" xmlns="http://karaf.apache.org/xmlns/features/v1.4.0"> | ||
<repository>mvn:org.openhab.core.features.karaf/org.openhab.core.features.karaf.openhab-core/${ohc.version}/xml/features</repository> | ||
|
||
<feature name="openhab-binding-dbquery" description="DBQuery Binding" version="${project.version}"> | ||
<feature>openhab-runtime-base</feature> | ||
<bundle start-level="80">mvn:org.openhab.addons.bundles/org.openhab.binding.dbquery/${project.version}</bundle> | ||
</feature> | ||
</features> |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Did you consider adding the dependencies via Karaf (feature.xml), so that the dependencies need to be downloaded only once if the influxdb persistence is also installed? (To make this work, the same adjustment needs to be done to the persistence addon)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My intent, if it's possible, is to maintain independence between the persistence plugins.
To be able to evolve differently if needed and use different versions or implementations in the future if it's desired.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If the versions differ, both versions will be used. But if the versions are identical, the dependency will be reused. You won't introduce a dependency between the two.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You're right.
Ok, I will try then but will need some support and guidance on that.
Is there any recent PR or documentation that I can take as a reference on how to do it?
Also, I will try to do all corrections in a row when you finish the review.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You could take a look here:
openhab-addons/bundles/org.openhab.binding.ahawastecollection/src/main/feature/feature.xml
Line 7 in d36e1f1
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@fwolter: It's only adding that line and leaving the pom the same way?
Or it's needed to create the bundle too?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You don't need to create any bundle. But you need to change the scope in the pom.xml from compile to provided.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If it's possible I prefer to leave this as it is for this first version.
Because I prefer to don't put more changes including modification of persistence addon.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK