Return table from plugin #22

tobiaslindulf · 2017-12-19T07:02:35Z

Support for returning table from plugin.
Updated protocol with TableDescription.
Python Examples updated with examples of returning a table.
Prepared for release of version 1.1.0.

Status

[ ] Under development
[x ] Waiting for code review
[x ] Waiting for merge

Information

[ ] Contains breaking changes
[x ] Contains new API(s)
[x ] Contains documentation
[x ] Contains test
[x ] Contains examples

To-do list

[ ] [Fix this thing]
[ ] [Fix another thing]
[ ] ...

LinneaStovring-Nielsen · 2017-12-19T07:33:27Z

docs/writing_a_plugin.md

@@ -10,9 +31,9 @@ There are three possible RPC methods that Qlik can call.

 But what is the difference between a _script function_ and a _plugin function_?  

-For script functions you write any script directly in the Qlik client, and pass it as a parameter to the SSE plugin in one of the many pre-defined script functions. The plugin functions, on the other hand, are defined and implemented on the plugin side. The user calls the function directly by name and inserts the data as parameters in the Qlik client. The two options are described in detail below.
+For script functions you write any script directly in the Qlik client, and pass it as a parameter to the SSE plugin in one of the many pre-defined script functions. The plugin functions, on the other hand, are defined and implemented on the plugin side. The user calls the function directly by name and inserts the data as parameters in the Qlik client. The two options are described in details below.


I think the previous text was correct (but don't trust my English too much). I would prefer "The two options are described in detail below" without the added "s".

My mistake, I'll fix that :)

LinneaStovring-Nielsen · 2017-12-19T09:38:57Z

examples/python/ColumnOperations/ExtensionService_column.py

+            # Iterating over rows
+            for row in request_rows.rows:
+                # Retrieve numerical value of parameter and append to the params variable
+                # Length of param is 1 since one column is received, the [0] collects the first value in the list


This comment is a bit confusing, is 'the params variable' the result? And 'Length of param is 1 since one column is received' kind of implies that the function only expects one column but the function returns two columns...

In earlier examples/versions of the example, 'row' has been called 'param'. Updating the comment must have been missed in the change. I can change that as well.

Okej :). The code is included in one .md document as well, you might want to change that one too.

Should it be length of row is 2 then?

Thanks for pointing that out, I'll change it there as well. I plan to change the documentation for the python examples to avoid this issue with duplicated information. We'll see when I have the time to do that.

Yes in this case, 'row' will be of length 2.

josefinestal · 2017-12-19T11:33:47Z

For your information, the commits will be squashed later.

LinneaStovring-Nielsen · 2017-12-19T11:44:02Z

examples/python/FullScriptSupport_Pandas/ScriptEval_scriptPandas.py

+    def send_table_description(table, context):
+        """
+        # TableDescription is only handled in Qlik if sent from a 'Load ... Extension ...' script.
+        # If tableDescription is set when evaluating an expression the header will be neglected


ignored instead of neglected maybe?

Good catch, I'll fix it!

LinneaStovring-Nielsen · 2017-12-19T12:18:18Z

README.md

+
+| __Latest Product Version__ | __SSE Supported__ |
+| ----- | ----- |
+| Qlik Sense February 2018 | v1.1.0 |


It would be nice to quickly be able to find which new features are supported in each version. Mostly to make it clearer that Table Load is supported in v1.1.0 and that it is not supported in v1.0.0.

This information will be included in the release notes to v1.1.0

Aha, that should be fine then.

michellefthomas · 2017-12-20T19:16:23Z

docs/limitations.md

- If less rows, or lower cardinality, is sent back from the plugin, Qlik will add null values to match the number of rows sent from Qlik to the plugin originally. Note that the mapping may not work as intended if less data is sent back.
- If higher-order data is sent back, either too many rows or more than one column, Qlik will neglect the additional data and only take the first _n_ rows in the first column, assuming _n_ is the number of rows sent to the plugin. Note that the mapping may not work as intended if more data is sent back.
+#### Expressions using SSE must persist the cardinality
+If using SSE in a chart expression or in the Qlik load script (the `LOAD ... EXTENSION ...` statement excluded), you should preserve the cardinality and return a single column. In the case of aggregations, the response column should contain a single value (one row). In the case of tensor functions, the response column should contain the same number of rows as the request.


When you use SSE in a chart expression or in the Qlik load script (the LOAD ... EXTENSION ... statement excluded), you must preserve the cardinality and return a single column. In the case of aggregations, the response column must contain a single value (one row). In the case of tensor functions, the response column must contain the same number of rows as the request.

They actually do not have to, therefore we exchanged the word must to should, since it still works in some way even if they are doing it wrong. So I think we stick to the word should.

michellefthomas · 2017-12-20T19:31:41Z

examples/python/ColumnOperations/README.md

        }
 ```

 ### `SumOfRows` function
-The first function is a tensor function adding two columns row wise. We iterate over the `BundledRows` and extract the numerical values, which we then add togehter and transform into the desired form.
+The `SumOfRows` function is a tensor function summing two columns row wise. We iterate over the `BundledRows` and extract the numerical values, which we then add togehter and transform into the desired form.


row wise --> row-wise
togehter --> together

michellefthomas · 2017-12-20T19:32:19Z

examples/python/ColumnOperations/README.md

@@ -205,13 +208,59 @@ The second function is an aggregation and sums the values in a column. We iterat
        # Yield the row data constructed
        yield SSE.BundledRows(rows=[SSE.Row(duals=duals)])
 ```
+### `MaxOfColumns_2` function
+The `MaxOfColumns_2` function computes the maximum in each of two columns and returns the maximum values in two columns, therefore making it appropriate to be used from the Qlik load script. As you can see the function also set the TableDescription header before sending the result.


As you can see, the function also sets the TableDescription header before sending the result.

michellefthomas · 2017-12-20T19:33:56Z

examples/python/ColumnOperations/README.md


 The aggregating script function is called with the expression `Column.ScriptAggr('sum(args[0])', A)`, where `'sum(args[0])'` is the script and `A` is the data field. The script returns a single value when evaluated. The second script call is `Column.ScriptEval('args[0]+args[1]',A,B)` with the script `'args[0]+args[1]'` adding the two parameters `A` and `B` row-wise. The result is an array with five values, each a sum of the  corresponding values in `A` and `B`.

-The user defined function calls are straightforward as implemented. The aggregating function `SumOfColumn` is called with `Column.SumOfColumn(A)` and the tensor function `SumOfRows` is called with `Column.SumOfRows(A,B)`.
+The user defined function calls are straightforward to call since they become integrated in the script syntax.


The user-defined functions are straightforward to call, since they become integrated in the script syntax.

michellefthomas · 2017-12-20T19:40:24Z

docs/writing_a_plugin.md


-It is possible to make an SSE call from the load script or from a chart expression. The syntax of the call is the same regardless of where you choose to use it.
+Returning a table from the SSE plugin can be done using the `LOAD ... EXTENSION ...` statement in the Qlik Load script. Read more about the syntax for table load using SSE in the [Qlik Sense help](http://help.qlik.com/en-US/sense/February2018/Subsystems/Hub/Content/Scripting/ScriptRegularStatements/Load.htm) .  


You can return a table from the SSE plugin using the LOAD ... EXTENSION ... statement in the Qlik Load script.

michellefthomas · 2017-12-20T19:50:41Z

examples/python/FullScriptSupport_Pandas/README.md

+If the parameter is of type _Dual_ the plugin will create two additional columns in the `q` data frame, with the string and numerical representation. The column names will have the base as the parameter name but will end with '_str' and '_num' respectively. For example, a parameter called `Bar` with datatype _Dual_ will result in three columns in `q`: `Bar`, `Bar_str` and `Bar_num`. `Bar` will contain strings and numerics, `Bar_str` will contain only strings and `Bar_num` only numerics.
+
+### TableDescription
+In the load script, when using the `Load ... Extension ...` syntax you can create the `TableDescription` message within the script. This can be useful if you for instance want to name, set tags, or change datatype of the fields you are sending back to Qlik. Read more about what metadata can be included in the `TableDescription` in in the [SSE_Protocol.md](../../../docs/SSE_Protocol.md).


This can be useful if, for example, you want to name, set tags for, or change the datatype of the fields you are sending back to Qlik.

michellefthomas · 2017-12-20T19:52:14Z

examples/python/FullScriptSupport_Pandas/README.md

+Note that if a `TableDescription` is sent, the number of fields in the message must match the number of fields of data sent back to Qlik.
+
+### Result
+With the change to using the `exec` method to evaluate the script, there are some changes regarding what's possible to write in the script. See the Python documentation of `exec` [here](https://docs.python.org/3/library/functions.html#exec). One of them being that the `exec` method does not return anything. We must therefore set the result to a specific variable which we have chosen to call `qResult`. If nothing was set to the variable no data will be returned to Qlik. Note that `qResult` is not required to be a pandas data frame.  


One change is that the exec method does not return anything. We must therefore set the result to a specific variable, which we have chosen to call qResult. If nothing was set to the variable, no data will be returned to Qlik. Note that qResult is not required to be a Pandas data frame.

michellefthomas · 2017-12-20T19:53:56Z

examples/python/FullScriptSupport_Pandas/README.md

+
+
+## Qlik documents
+An example document is given for Qlik Sense (SSE_Full_Script_Support_pandas.qvf) and it's the same as the original Full Script Support example but with modified scripts to work with the pandas implementation and the use of `exec`.


We provide an example Qlik Sense document (SSE_Full_Script_Support_pandas.qvf). It's the same as the original Full Script Support example, but with modified scripts to work with the Pandas implementation and the use of exec.

michellefthomas · 2017-12-20T19:54:41Z

examples/python/ColumnOperations/README.md


 ## Qlik documents
-An example document is given for Qlik Sense (SSE_Column_Operations.qvf) and QlikView (SSE_Column_Operations.qvw). There are are two sheets in this example. One contains script calls and the other contains user defined function calls. We demonstrate a tensor function, which sums two columns row-wise, and an aggregating function, which sums all rows in a column returning a single value. The data loaded in the Data Load Editor are two fields, *A* and *B*, each of which contain five numeric values.
+An example document is given for Qlik Sense (SSE_Column_Operations.qvf) and QlikView (SSE_Column_Operations.qvw). 


We provide example documents for Qlik Sense (SSE_Column_Operations.qvf) and QlikView (SSE_Column_Operations.qvw).

michellefthomas · 2017-12-20T19:55:32Z

examples/python/HelloWorld/README.md

 ## Qlik documents
-An example document is given for Qlik Sense (SSE_Hello_World.qvf) and QlikView (SSE_Hello_World.qvw). The example consists of three sheets, one with script function calls and two with user defined function calls. The function calls on each of the first two sheets demonstrate the same functionality. We use a table for the tensor call and a KPI object for the aggregation call. On the third sheet we demonstrate enabling and disabling the cache for a specific function. A field called __HelloWorldData__ consisting of two rows of strings is loaded into the Qlik engine.
+An example document is given for Qlik Sense (SSE_Hello_World.qvf) and QlikView (SSE_Hello_World.qvw). The example consists of four sheets, one with script function calls, two with user defined function calls and one with the result of table load. The function calls on each of the first two sheets demonstrate the same functionality. We use a table for the tensor call and a KPI object for the aggregation call. On the third sheet we demonstrate enabling and disabling the cache for a specific function. A field called __HelloWorldData__ consisting of two rows of strings is loaded into the Qlik engine.


We provide example documents for Qlik Sense (SSE_Hello_World.qvf) and QlikView (SSE_Hello_World.qvw).

josefinestal · 2017-12-28T11:09:05Z

Please note that the history of the branch is rewritten due to my cleanup of all commits. If you plan to make any changes, make sure your local repository is updated with the latest changes. Thanks!

tobiaslindulf

I have reviewed all changes and it looks good.
I have also run all the examples and it works good.
Waiting for the last change before approving.

tobiaslindulf

Looks very good. Approved.

Update proto-file and its generated files with TableDescription and FieldDescription.

tobiaslindulf requested review from Bjornwa, josefinestal, kjartan-qlik, qlik-aoh, joesiltbergqlik and LinneaStovring-Nielsen December 19, 2017 07:02

LinneaStovring-Nielsen reviewed Dec 19, 2017

View reviewed changes

LinneaStovring-Nielsen approved these changes Dec 19, 2017

View reviewed changes

michellefthomas reviewed Dec 20, 2017

View reviewed changes

josefinestal force-pushed the table-support branch 5 times, most recently from c064516 to f27f0e3 Compare December 28, 2017 10:47

tobiaslindulf commented Jan 12, 2018

View reviewed changes

tobiaslindulf commented Jan 14, 2018

View reviewed changes

Bjornwa and others added 4 commits January 16, 2018 11:19

Add support for TableDescription

fed442a

Update proto-file and its generated files with TableDescription and FieldDescription.

Update helloworld python example with table load

d013ba2

Fixed typo in a log line in the CSharp example.

7387d56

Update Column operation example with table load

587969e

josefinestal force-pushed the table-support branch from 9315309 to 7c8c241 Compare January 16, 2018 10:24

josefinestal added 4 commits January 16, 2018 12:45

Update general documentation

f76b62f

Update FullScriptSupport example with table load

26d229e

Add a Full Script Support example using Pandas

4c9a5af

Remove duplicated ServerSideExtension_pb2.py file in test folder

18954b3

Add Changelog

c04ef9b

josefinestal force-pushed the table-support branch from 7c8c241 to c04ef9b Compare January 16, 2018 11:47

josefinestal merged commit 92b91ec into master Jan 16, 2018

josefinestal deleted the table-support branch January 16, 2018 12:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Return table from plugin #22

Return table from plugin #22

tobiaslindulf commented Dec 19, 2017

LinneaStovring-Nielsen Dec 19, 2017

josefinestal Dec 19, 2017

LinneaStovring-Nielsen Dec 19, 2017

josefinestal Dec 19, 2017

LinneaStovring-Nielsen Dec 19, 2017

josefinestal Dec 19, 2017

josefinestal commented Dec 19, 2017 •

edited

Loading

LinneaStovring-Nielsen Dec 19, 2017

josefinestal Dec 19, 2017

LinneaStovring-Nielsen Dec 19, 2017

josefinestal Dec 19, 2017

LinneaStovring-Nielsen Dec 19, 2017

michellefthomas Dec 20, 2017

tobiaslindulf Dec 20, 2017

michellefthomas Dec 20, 2017

michellefthomas Dec 20, 2017

michellefthomas Dec 20, 2017

michellefthomas Dec 20, 2017

michellefthomas Dec 20, 2017

michellefthomas Dec 20, 2017

michellefthomas Dec 20, 2017

michellefthomas Dec 20, 2017

michellefthomas Dec 20, 2017

josefinestal commented Dec 28, 2017

tobiaslindulf left a comment

tobiaslindulf left a comment


		It is possible to make an SSE call from the load script or from a chart expression. The syntax of the call is the same regardless of where you choose to use it.
		Returning a table from the SSE plugin can be done using the `LOAD ... EXTENSION ...` statement in the Qlik Load script. Read more about the syntax for table load using SSE in the [Qlik Sense help](http://help.qlik.com/en-US/sense/February2018/Subsystems/Hub/Content/Scripting/ScriptRegularStatements/Load.htm) .



		## Qlik documents
		An example document is given for Qlik Sense (SSE_Full_Script_Support_pandas.qvf) and it's the same as the original Full Script Support example but with modified scripts to work with the pandas implementation and the use of `exec`.

Return table from plugin #22

Return table from plugin #22

Conversation

tobiaslindulf commented Dec 19, 2017

Status

Information

To-do list

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

josefinestal commented Dec 19, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

josefinestal commented Dec 28, 2017

tobiaslindulf left a comment

Choose a reason for hiding this comment

tobiaslindulf left a comment

Choose a reason for hiding this comment

josefinestal commented Dec 19, 2017 •

edited

Loading