Migrate from deprecated Kibana dashboard import API to general Saved Objects import API #20672
Comments
botelastic
bot
added
the
needs_team
botelastic
bot
removed
the
needs_team
|
Related / subset of #14196 |
andresrc
added
Team:Integrations
|
Pinging @elastic/integrations (Team:Integrations) |
|
@jen-huang We'd like to be able to remove this API from master in the near future. Do you know if Fleet is still this API from Kibana and/or if any Beats modules are still using this API on master/8.0? |
botelastic
bot
added
the
needs_team
botelastic
bot
removed
the
needs_team
|
@joshdover Fleet is not using this API, we use the saved object API for dashboards. Looking at the Beats code today it doesn't appear that there have been any changes in this area so that API is still being used on master/8.0. cc @ph @masci - not sure whether Agent or Integrations team would handle such a change? |
|
@ph @masci @jen-huang The cc @kobelb |
|
Pinging @elastic/agent (Team:Agent) |
|
I understand removing this API from Kibana will likely break your CI. We are tentatively planning to tackle this in the first few weeks of September. Would that be enough lead time for this group to make the changes needed to Beats? |
|
I have updated the issue description to describe the changes that need to happen. We looking at squeezing this into 7.15, but if it's not possible, we might need to ask for some additional delay before removing the current API. |
How does dashboard management work today with the legacy API?ImportDashboards should be placed under the Dashboards are loaded when the Beat starts or the user calls ExportDashboards can be exported with Possible requirements for management with new Saved object API
What we should support?Essential supportFrom 7.15, we only export and import using the Saved Objects API. Old assets can be removed from the repository, only converted JSONs are required. We would also remove the old API from libbeat completely. We do not support loading assets from and to Kibana 7.14 or older. Limitation 1: Users can only import assets they have exported using Beats 7.15 and the assets we ship with Beats. External module developers have to export their dashboards from Kibana 7.14+ using Complete supportFrom 7.15, we export and import using the Saved Objects API. Old assets are kept around so when loading to Kibana 7.14 or older, those assets can be loaded. When exporting dashboards from Kibana 7.14 or older, dashboards are retrieved in legacy format. External module developers can keep their dashboards in legacy format and load them into Kibana 7.14 or older. They can convert their dashboards using Conversion supportThis only impacts external module developers. If we include this support, we have to provide a way for external module developers to convert their dashboards automatically. We could provide a script or convert it in place, whatever. IMHO this is unnecessary. Users can write a script to load legacy assets and export them in the new format. |
|
++ on essential support. The export command is targeted on Beats developer and not the "usual" user. It might be considered a breaking change but I think this is one we should make. Kibana changed the API and we need to adjust. On the compatibility we always wanted to enforce that KB / ES >= Beats but never fully did. @kvch What you mean by "new index templates"? |
|
I would also go for just essential support as well, thanks! |
|
@ruflin By new index template, I mean the index templates that are in the new format. But AFAIK the only difference between the old and the new format is just a new line character. |
## What does this PR do?
This PR adopts the Saved Objects API for exporting and importing dashboards and index templates. We have agreed to provide essential support, so there are a few limitations:
* You can only import assets they have exported using Beats 7.15 and the assets we ship with Beats.
* You can only export new assets, not legacy format.
## Why is it important?
We have to move forward to let Kibana remove the old API.
## How to test this PR locally
Try to setup the dashboards of a Beat:
```sh
./filebeat setup --dashboards
```
Export dashboards:
```sh
./filebeat export dashboard --id {my-id}
```
and:
```sh
go run dev-tools/cmd/dashboards/export_dashboards.go -dashboard {my-id}
```
## Notes to the reviewer
The changeset is huge, but most of it comes from converting the dashboards. I suggest you use the file filter and disable json, ndjson and yml files. The changes are moving json to ndjson format and changing the extensions in the module.yml files from json to ndjson.
Please also test it manually with both the setup command and `export_dashboards.go`.
## Related issues
Closes #20672
This PR adopts the Saved Objects API for exporting and importing dashboards and index templates. We have agreed to provide essential support, so there are a few limitations:
* You can only import assets they have exported using Beats 7.15 and the assets we ship with Beats.
* You can only export new assets, not legacy format.
We have to move forward to let Kibana remove the old API.
Try to setup the dashboards of a Beat:
```sh
./filebeat setup --dashboards
```
Export dashboards:
```sh
./filebeat export dashboard --id {my-id}
```
and:
```sh
go run dev-tools/cmd/dashboards/export_dashboards.go -dashboard {my-id}
```
The changeset is huge, but most of it comes from converting the dashboards. I suggest you use the file filter and disable json, ndjson and yml files. The changes are moving json to ndjson format and changing the extensions in the module.yml files from json to ndjson.
Please also test it manually with both the setup command and `export_dashboards.go`.
Closes #20672
…#27398) This PR adopts the Saved Objects API for exporting and importing dashboards and index templates. We have agreed to provide essential support, so there are a few limitations: * You can only import assets they have exported using Beats 7.15 and the assets we ship with Beats. * You can only export new assets, not legacy format. We have to move forward to let Kibana remove the old API. Try to setup the dashboards of a Beat: ```sh ./filebeat setup --dashboards ``` Export dashboards: ```sh ./filebeat export dashboard --id {my-id} ``` and: ```sh go run dev-tools/cmd/dashboards/export_dashboards.go -dashboard {my-id} ``` The changeset is huge, but most of it comes from converting the dashboards. I suggest you use the file filter and disable json, ndjson and yml files. The changes are moving json to ndjson format and changing the extensions in the module.yml files from json to ndjson. Please also test it manually with both the setup command and `export_dashboards.go`. Closes #20672 Co-authored-by: Noémi Ványi <kvch@users.noreply.github.com>
|
This was validated as part of 7.l5 Agent / Beats testing, just fyi. |
In Kibana 7.0, the
/api/kibana/dashboards/importendpoint was deprecated and replaced with the new general Saved Objects import API.Today Beats still uses the legacy API here:
beats/libbeat/dashboards/kibana_loader.go
Line 36 in 0b12bb4
The legacy API will be removed in Kibana 8.0 and all Beats from 7.last+ will need to be using the new API. If you have any questions, feel free to reach out to @elastic/kibana-platform for assistance.
In order to implement this change we need to:
Regarding (2) a similar conversion is currently performed as part of the code that creates packages from modules so we could use some of that code, removing the transformations that are needed as part of the transition from modules to packages.
For (3) we would need to change the current migration to just performing the transformations in the saved object format.
TODO
The text was updated successfully, but these errors were encountered: