visualization doesn't work in Kibana 7.9.x

[mmguero]

I don't know if you're looking forward at all to the Kibana 7.9.x and beyond, but the kbn_sankey_vis plugin won't work as relies on stuff in the legacy plugin format that was removed.

I'm really not a very good Kibana plugin developer yet, but I have done a little bit of research trying to figure out why. I'll refer you to this thread:

Debugging missing plugins after updating from 7.6.2 to 7.9.2

The advice I got there was to either put in a shim/compatibility to temporarily fix it for 7.9.2, or to migrate to the "new plugin framework."

I'm not really knowledgeable enough at this point to do either, but I wanted to log it here to see if it was on your radar and at least have somewhere we could track it.

Thanks.

[ch-bas]

Hi @mmguero,
Thanks for your suggestion and interest in our plug-in. We are currently thinking about a total code refactoring for the whole plug-in in order to work with the new Kibana platform. We will keep you updated about the new outcome soon.

[mmguero]

Very excited about this, thanks for the update.

[bfgiordano]

That would be great to get the plugin available moving forward. Appreciate it!

…

On Mon, Nov 2, 2020 at 3:35 PM SG ***@***.***> wrote: Very excited about this, thanks for the update. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#15 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADIYKVPSBUNPGSLNDNAFKNDSN4JYPANCNFSM4TH32T6A> .

[hakanyildiz]

Hi,

Thx for great plugin. However, It does not work in Kibana 7.10.* . Is there will be any update?

[ch-bas]

Hi @hakanyildiz, so far the update for the new Kibana platform is planed for next year. We will keep you updated for the new releases. Thanks!

[ch-bas]

Hi @mmguero @bfgiordano @hakanyildiz , a new pre-release was published today, it is still under testing but a final version should be ready in the upcoming days.

[bfgiordano]

Awesome - thank you for continuing to move this plugin forward. Barbara

…

Sent from my iPhone

On Feb 19, 2021, at 11:00 AM, bassem chagra ***@***.***> wrote:  Hi @mmguero @bfgiordano @hakanyildiz , a new pre-release was published today, it is still under testing but a final version should be ready in the upcoming days. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[mmguero]

Thank you! You really made my day with this. In my opinion this is one of the essential third party visualizations, and I've looked at a lot of them. I will test out this prerelease today.

[mmguero]

I tried downloading, building and installing the plugin as of commit 8ac7a8b this morning, and ran into the following issue:

The plugin installed okay, but when loading Kibana I see this in the browser:

Elastic did not load properly. Check the server output for more information.

Looking at my NGINX logs:

nginx-proxy_1    | 172.22.26.1 - user [19/Feb/2021:21:03:01 +0000] "GET /kibana/app/visualize HTTP/1.1" 200 17244 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:85.0) Gecko/20100101 Firefox/85.0"
nginx-proxy_1    | 172.22.26.1 - user [19/Feb/2021:21:03:01 +0000] "GET /kibana/bootstrap.js HTTP/1.1" 200 2354 "https://localhost/kibana/app/visualize" "Mozilla/5.0 (X11; Linux x86_64; rv:85.0) Gecko/20100101 Firefox/85.0"
nginx-proxy_1    | 172.22.26.1 - user [19/Feb/2021:21:03:01 +0000] "GET /kibana/36136/bundles/plugin/kbnSankeyVis/kbnSankeyVis.plugin.js HTTP/1.1" 404 60 "https://localhost/kibana/app/visualize" "Mozilla/5.0 (X11; Linux x86_64; rv:85.0) Gecko/20100101 Firefox/85.0"

I don't see a kbnSankeyVis.plugin.js anywhere in my Kibana container, and I'm not sure what the 36136 thing would be about either. Perhaps the installations for building and installing a release .zip are different in the new version of Kibana? My build process looks like this, which more or less lines up with the instructions at the bottom of the README:

curl -sSL -o /tmp/kibana-sankey.zip "https://codeload.github.com/uniberg/kbn_sankey_vis/zip/feature/update_7.10.1" && \
cd /tmp && \
  echo "Installing Sankey visualization..." && \
  unzip /tmp/kibana-sankey.zip && \
  mkdir ./kibana &&\
  mv ./kbn_sankey_vis-* ./kibana/sankey_vis && \
  cd ./kibana/sankey_vis && \
  npm install --production && \
  cd /tmp && \
  zip -r sankey_vis.zip kibana --exclude ./kibana/sankey_vis/.git\* && \
  cd /usr/share/kibana/plugins && \
  /usr/share/kibana/bin/kibana-plugin install file:///tmp/sankey_vis.zip --allow-root && \
  rm -rf /tmp/kibana /tmp/*sankey*

I don't seem to get errors during the install of the plugin:

Installing Sankey visualization...
Archive:  /tmp/kibana-sankey.zip
8ac7a8bffc018716b94747ecb328c771ff42b63a
   creating: kbn_sankey_vis-feature-update_7.10.1/
 extracting: kbn_sankey_vis-feature-update_7.10.1/.eslintignore  
  inflating: kbn_sankey_vis-feature-update_7.10.1/.eslintrc.yml  
...
  inflating: kbn_sankey_vis-feature-update_7.10.1/test/filterTest.js  
kbn-sankey-vis@7.10.2 /tmp/kibana/sankey_vis
+-- d3-plugins-sankey@1.2.1  (git+https://github.com/uniberg/d3-plugins-sankey.git#c1ec50f3d3e88ec5cc5b2a9ab8e7987ce38ea6cd)
| `-- d3@3.5.17 
`-- json-stable-stringify@1.0.1 
  `-- jsonify@0.0.0 

  adding: kibana/ (stored 0%)
  adding: kibana/sankey_vis/ (stored 0%)
...
  adding: kibana/sankey_vis/node_modules/json-stable-stringify/test/to-json.js (deflated 63%)
Attempting to transfer from file:///tmp/sankey_vis.zip
Transferring 677223 bytes....................
Transfer complete
Retrieving metadata from plugin archive
Extracting plugin archive
Extraction complete
Plugin installation complete

If this isn't helpful, or not applicable while you're still in pre-release, then please ignore this comment. I'm fine waiting until your done. Just wanted to help as much as possible.

[ch-bas]

Hi @mmguero , thanks for reporting that!
I updated the readme file to include the new instructions for building the plugin.
Could you follow them and see if your problems will be solved?
Thanks!

[mmguero]

Here's what I'm running into with the updated build release instructions:

[root@000bdbb111ba tmp]# git clone --single-branch --recursive --depth=1 --branch=feature/update_7.10.1 https://github.com/uniberg/kbn_sankey_vis.git sankey_vis
Cloning into 'sankey_vis'...
remote: Enumerating objects: 42, done.
remote: Counting objects: 100% (42/42), done.
remote: Compressing objects: 100% (36/36), done.
remote: Total 42 (delta 4), reused 29 (delta 4), pack-reused 0
Unpacking objects: 100% (42/42), done.
[root@000bdbb111ba tmp]# cd sankey_vis/
[root@000bdbb111ba sankey_vis]# yarn install
yarn install v1.22.5
info No lockfile found.
[1/4] Resolving packages...
warning mocha > mkdirp@0.5.1: Legacy versions of mkdirp are no longer supported. Please update to mkdirp 1.x. (Note that the API surface has changed to use Promises in 1.x.)
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
success Saved lockfile.
Done in 9.15s.
[root@000bdbb111ba sankey_vis]# yarn build --kibana-version 7.10.2
yarn run v1.22.5
$ yarn plugin-helpers build --kibana-version 7.10.2
$ node ../../scripts/plugin_helpers build --kibana-version 7.10.2
internal/modules/cjs/loader.js:638
    throw err;
    ^

Error: Cannot find module '/scripts/plugin_helpers'
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:636:15)
    at Function.Module._load (internal/modules/cjs/loader.js:562:25)
    at Function.Module.runMain (internal/modules/cjs/loader.js:831:12)
    at startup (internal/bootstrap/node.js:283:19)
    at bootstrapNodeJSCore (internal/bootstrap/node.js:623:3)
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

[root@000bdbb111ba sankey_vis]# node --version
v10.23.3
[root@000bdbb111ba sankey_vis]# npm --version
6.14.11
[root@000bdbb111ba sankey_vis]# yarn --version
1.22.5

The important parts of my build environment (in a Docker image) can be seen as such:

RUN yum install -y epel-release && \
      yum update -y && \
      curl --silent --location https://rpm.nodesource.com/setup_10.x | bash - && \
      curl --silent --location https://dl.yarnpkg.com/rpm/yarn.repo | tee /etc/yum.repos.d/yarn.repo && \
      rpm --import https://dl.yarnpkg.com/rpm/pubkey.gpg

[mmguero]

I'm wondering if maybe it matters where I am in the filesystem (e.g., somewhere under the /usr/share/kibana source tree?) where I clone the repo?

[mmguero]

What I'm really wondering is if my Docker image is no longer sufficient to build the plugin from scratch. I'm going to research more in the Kibana contributing guide and see what I might be missing.

[ch-bas]

Running 'yarn build' will invoke using some dependencies from Kibana, so unless you have cloned the Kibana repository in the docker container and installed the dependencies(yarn kbn bootstrap), you will not be able to use the build process like you are doing right now.
I suggest to use the docker image provided by the Elastic team in docker hub and simply install the plugin using the normal way( kibana-plugin install zip file).

[mmguero]

I understand now, I did some more reading about the plugin development environment for Kibana 7.10+ and have adjusted my build process. I'm getting it built now and will report back with the results on how the plugin works.

[mmguero]

I got it installed correctly and it's mostly working. I have noticed a few issues:

1. If I run the visualization over a time frame with no data present I get a warning dialog with "Minimum two sub aggs needed."
2. If I have run the visualization as part of a dashboard and correctly see data in the graph, then if I do something to invalidate the canvas (for example, resize my browser window) I get "No results found." Hard-refreshing the page will then cause it to render correctly.

[mmguero]

One more I found:

When trying to use the "Group other values in separate bucket" visualization option, I get this search error:

sankeyProvider/<@https://localhost/kibana/36136/bundles/plugin/kbnSankeyVis/kbnSankeyVis.plugin.js:26:218644
sankeyProvider@https://localhost/kibana/36136/bundles/plugin/kbnSankeyVis/kbnSankeyVis.plugin.js:26:218418
_callee$@https://localhost/kibana/36136/bundles/plugin/visualizations/visualizations.plugin.js:6:215955
tryCatch@https://localhost/kibana/36136/bundles/plugin/opendistroQueryWorkbenchKibana/opendistroQueryWorkbenchKibana.plugin.js:1:32004
invoke@https://localhost/kibana/36136/bundles/plugin/opendistroQueryWorkbenchKibana/opendistroQueryWorkbenchKibana.plugin.js:1:35976
defineIteratorMethods/</prototype[method]@https://localhost/kibana/36136/bundles/plugin/opendistroQueryWorkbenchKibana/opendistroQueryWorkbenchKibana.plugin.js:1:33129
visualization_function_asyncGeneratorStep@https://localhost/kibana/36136/bundles/plugin/visualizations/visualizations.plugin.js:6:212591
_next@https://localhost/kibana/36136/bundles/plugin/visualizations/visualizations.plugin.js:6:212965

[ch-bas]

Thanks for reporting! I will try to take a look soon.

[ch-bas]

@mmguero , the mentioned issues were fixed in the new pre-release.

[mmguero]

Thanks! I tested it and the issues mentioned in this comment #15 (comment) are fixed. I still get an error when I try "Group other values in separate bucket" or "Show missing values":

bucketHelper(...) is undefined

sankeyProvider/<@https://localhost/kibana/36136/bundles/plugin/kbnSankeyVis/kbnSankeyVis.plugin.js:26:224456
sankeyProvider@https://localhost/kibana/36136/bundles/plugin/kbnSankeyVis/kbnSankeyVis.plugin.js:26:224230
_callee$@https://localhost/kibana/36136/bundles/plugin/visualizations/visualizations.plugin.js:6:215955
tryCatch@https://localhost/kibana/36136/bundles/plugin/opendistroQueryWorkbenchKibana/opendistroQueryWorkbenchKibana.plugin.js:1:32004
invoke@https://localhost/kibana/36136/bundles/plugin/opendistroQueryWorkbenchKibana/opendistroQueryWorkbenchKibana.plugin.js:1:35976
defineIteratorMethods/</prototype[method]@https://localhost/kibana/36136/bundles/plugin/opendistroQueryWorkbenchKibana/opendistroQueryWorkbenchKibana.plugin.js:1:33129
visualization_function_asyncGeneratorStep@https://localhost/kibana/36136/bundles/plugin/visualizations/visualizations.plugin.js:6:212591
_next@https://localhost/kibana/36136/bundles/plugin/visualizations/visualizations.plugin.js:6:212965

[ch-bas]

HI @mmguero , could you elaborate how to reproduce the bug, and in which Kibana's version is this happening ?
I was testing with Kibana 7.10.2, with 4 layers defined in the Sankey visualisation but couldn't see that problem.
Thanks in advance!

[mmguero]

Sure. I'm using Kibana 7.10.2 as well. Here's my entire build process for the plugin.

All I'm doing is setting up a sankey visualization with 3 layers, going into the visualization editor and choosing any of the buckets, and enabling "group other values in separate bucket" or "show missing values." I have attached a video file here, does this match what you are doing? And you're not getting that error?

out.mp4

[ch-bas]

@mmguero , the new pre-release should fix the issue you mentioned, thanks again for explaining how to reproduce it!

[mmguero]

Hmmm, I'm still getting the same error. Looking at your comment with the commit and the contents of the source, I'm not sure you fixed the same thing I was talking about (although there certainly may have been an issue with custom labels as well?).

The issue I'm seeing is not with "Custom label", but is with "Group other values in separate bucket" and "Show missing values."

"Group other values in separate bucket," if enabled, will do just that. Suppose you have "Size" set to 5. If "Group other values in separate bucket" is disabled you will only have 5 values displayed, even if you have 100 unique values. However, if you have "group other values in separate bucket" enabled, you'd have 6 values displayed: your top 5, then another bucket that says "Other" (or whatever you call it) that contains the sum total of the other 95 values.

Similarly, "Show missing values" will create an extra bucket for any documents that do not contain a value for the field chosen.

Even with your latest update, I am getting the bucketHelper error with either "Group other values in separate bucket" and "Show missing values" enabled. Is that working for you?

[ch-bas]

Sorry for this inconvenient behaviour of the plugin.
I tested now the new changes with Kibana's version 7.10.2 and I couldn't reproduce it.
Please feel free to report any new issue and I will try to take a look as soon as I find some free time, your contribution is much appreciated :) .

[mmguero]

I apparently hadn't grabbed the commits you made on the day you commented. My apologize. The missing/other buckets function is working now. Thanks so much!