Filebeat import dashboards returns internal server error #28773

liza-mae · 2021-10-28T17:35:26Z

For confirmed bugs, please report:

Version: main
Operating System: Linux
Steps to Reproduce:
./filebeat setup

10:54:30 Exiting: Failed to import dashboard: Failed to load directory /home/vagrant/filebeat-8.0.0-SNAPSHOT-linux-x86_64/kibana/7:
10:54:30   error loading /home/vagrant/filebeat-8.0.0-SNAPSHOT-linux-x86_64/kibana/7/dashboard/Filebeat-cyberarkpas-audit.json: error dashboard asset: returned 500 to import file: An internal server error occurred.: <nil>. Response: {"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred."}
10:54:30

The text was updated successfully, but these errors were encountered:

elasticmachine · 2021-10-28T17:35:54Z

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

elasticmachine · 2021-10-28T17:35:55Z

Pinging @elastic/integrations (Team:Integrations)

liza-mae · 2021-11-02T17:16:53Z

I see these error in kibana log while filebeat setup is running:


[2021-11-02T17:15:46.984+00:00][ERROR][savedobjects-service] SyntaxError: Unexpected token o in JSON at position 1
    at JSON.parse (<anonymous>)
    at setEmsTmsDefaultModes (/home/vagrant/kibana-8.0.0-beta1/x-pack/plugins/maps/common/migrations/set_ems_tms_default_modes.js:28:26)
    at Object.8.0.0 (/home/vagrant/kibana-8.0.0-beta1/x-pack/plugins/maps/server/embeddable_migrations.js:35:72)
    at migrateFn (/home/vagrant/kibana-8.0.0-beta1/src/plugins/embeddable/common/lib/migrate.js:24:49)
    at migrations.<computed> (/home/vagrant/kibana-8.0.0-beta1/src/plugins/embeddable/common/lib/get_all_migrations.js:34:42)
    at /home/vagrant/kibana-8.0.0-beta1/src/plugins/dashboard/server/saved_objects/dashboard_migrations.js:189:29
    at Array.forEach (<anonymous>)
    at /home/vagrant/kibana-8.0.0-beta1/src/plugins/dashboard/server/saved_objects/dashboard_migrations.js:183:10
    at tryTransformDoc (/home/vagrant/kibana-8.0.0-beta1/src/core/server/saved_objects/migrations/core/document_migrator.js:597:22)
    at migrateProp (/home/vagrant/kibana-8.0.0-beta1/src/core/server/saved_objects/migrations/core/document_migrator.js:679:22)
    at applyMigrations (/home/vagrant/kibana-8.0.0-beta1/src/core/server/saved_objects/migrations/core/document_migrator.js:397:20)
    at DocumentMigrator.transformAndValidate [as transformDoc] (/home/vagrant/kibana-8.0.0-beta1/src/core/server/saved_objects/migrations/core/document_migrator.js:337:22)
    at DocumentMigrator.migrate (/home/vagrant/kibana-8.0.0-beta1/src/core/server/saved_objects/migrations/core/document_migrator.js:130:16)
    at KibanaMigrator.migrateDocument (/home/vagrant/kibana-8.0.0-beta1/src/core/server/saved_objects/migrations/kibana/kibana_migrator.js:204:34)
    at /home/vagrant/kibana-8.0.0-beta1/src/core/server/saved_objects/service/lib/repository.js:411:74
    at Array.map (<anonymous>)
[2021-11-02T17:15:46.987+00:00][ERROR][http] Error: Migration function for version 8.0.0 threw an error
    at tryTransformDoc (/home/vagrant/kibana-8.0.0-beta1/src/core/server/saved_objects/migrations/core/document_migrator.js:610:13)
    at migrateProp (/home/vagrant/kibana-8.0.0-beta1/src/core/server/saved_objects/migrations/core/document_migrator.js:679:22)
    at applyMigrations (/home/vagrant/kibana-8.0.0-beta1/src/core/server/saved_objects/migrations/core/document_migrator.js:397:20)
    at DocumentMigrator.transformAndValidate [as transformDoc] (/home/vagrant/kibana-8.0.0-beta1/src/core/server/saved_objects/migrations/core/document_migrator.js:337:22)
    at DocumentMigrator.migrate (/home/vagrant/kibana-8.0.0-beta1/src/core/server/saved_objects/migrations/core/document_migrator.js:130:16)
    at KibanaMigrator.migrateDocument (/home/vagrant/kibana-8.0.0-beta1/src/core/server/saved_objects/migrations/kibana/kibana_migrator.js:204:34)
    at /home/vagrant/kibana-8.0.0-beta1/src/core/server/saved_objects/service/lib/repository.js:411:74
    at Array.map (<anonymous>)
    at SavedObjectsRepository.bulkCreate (/home/vagrant/kibana-8.0.0-beta1/src/core/server/saved_objects/service/lib/repository.js:351:49)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at SavedObjectsClient.bulkCreate (/home/vagrant/kibana-8.0.0-beta1/src/core/server/saved_objects/service/saved_objects_client.js:54:12)
    at EncryptedSavedObjectsClientWrapper.bulkCreate (/home/vagrant/kibana-8.0.0-beta1/x-pack/plugins/encrypted_saved_objects/server/saved_objects/encrypted_saved_objects_client_wrapper.js:69:63)
    at SecureSavedObjectsClientWrapper.bulkCreate (/home/vagrant/kibana-8.0.0-beta1/x-pack/plugins/security/server/saved_objects/secure_saved_objects_client_wrapper.js:154:22)
    at SpacesSavedObjectsClient.bulkCreate (/home/vagrant/kibana-8.0.0-beta1/x-pack/plugins/spaces/server/saved_objects/spaces_saved_objects_client.js:82:12)
    at createSavedObjects (/home/vagrant/kibana-8.0.0-beta1/src/core/server/saved_objects/import/lib/create_saved_objects.js:104:32)
Caused by:
SyntaxError: Unexpected token o in JSON at position 1
    at JSON.parse (<anonymous>)
    at setEmsTmsDefaultModes (/home/vagrant/kibana-8.0.0-beta1/x-pack/plugins/maps/common/migrations/set_ems_tms_default_modes.js:28:26)
    at Object.8.0.0 (/home/vagrant/kibana-8.0.0-beta1/x-pack/plugins/maps/server/embeddable_migrations.js:35:72)
    at migrateFn (/home/vagrant/kibana-8.0.0-beta1/src/plugins/embeddable/common/lib/migrate.js:24:49)
    at migrations.<computed> (/home/vagrant/kibana-8.0.0-beta1/src/plugins/embeddable/common/lib/get_all_migrations.js:34:42)
    at /home/vagrant/kibana-8.0.0-beta1/src/plugins/dashboard/server/saved_objects/dashboard_migrations.js:189:29
    at Array.forEach (<anonymous>)
    at /home/vagrant/kibana-8.0.0-beta1/src/plugins/dashboard/server/saved_objects/dashboard_migrations.js:183:10
    at tryTransformDoc (/home/vagrant/kibana-8.0.0-beta1/src/core/server/saved_objects/migrations/core/document_migrator.js:597:22)
    at migrateProp (/home/vagrant/kibana-8.0.0-beta1/src/core/server/saved_objects/migrations/core/document_migrator.js:679:22)
    at applyMigrations (/home/vagrant/kibana-8.0.0-beta1/src/core/server/saved_objects/migrations/core/document_migrator.js:397:20)
    at DocumentMigrator.transformAndValidate [as transformDoc] (/home/vagrant/kibana-8.0.0-beta1/src/core/server/saved_objects/migrations/core/document_migrator.js:337:22)
    at DocumentMigrator.migrate (/home/vagrant/kibana-8.0.0-beta1/src/core/server/saved_objects/migrations/core/document_migrator.js:130:16)
    at KibanaMigrator.migrateDocument (/home/vagrant/kibana-8.0.0-beta1/src/core/server/saved_objects/migrations/kibana/kibana_migrator.js:204:34)
    at /home/vagrant/kibana-8.0.0-beta1/src/core/server/saved_objects/service/lib/repository.js:411:74
    at Array.map (<anonymous>)

kvch · 2021-11-02T17:18:13Z

I see this error in Kibana when trying to load the specific artifact as above:

[2021-11-02T17:14:58.911+00:00][ERROR][savedobjects-service] SyntaxError: Unexpected token o in JSON at position 1
    at JSON.parse (<anonymous>)
    at setEmsTmsDefaultModes (/usr/share/kibana/x-pack/plugins/maps/common/migrations/set_ems_tms_default_modes.js:28:26)
    at Object.8.0.0 (/usr/share/kibana/x-pack/plugins/maps/server/embeddable_migrations.js:35:72)
    at migrateFn (/usr/share/kibana/src/plugins/embeddable/common/lib/migrate.js:24:49)
    at migrations.<computed> (/usr/share/kibana/src/plugins/embeddable/common/lib/get_all_migrations.js:34:42)
    at /usr/share/kibana/src/plugins/dashboard/server/saved_objects/dashboard_migrations.js:189:29
    at Array.forEach (<anonymous>)
    at /usr/share/kibana/src/plugins/dashboard/server/saved_objects/dashboard_migrations.js:183:10
    at tryTransformDoc (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:597:22)
    at migrateProp (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:679:22)
    at applyMigrations (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:397:20)
    at DocumentMigrator.transformAndValidate [as transformDoc] (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:337:22)
    at DocumentMigrator.migrate (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:130:16)
    at KibanaMigrator.migrateDocument (/usr/share/kibana/src/core/server/saved_objects/migrations/kibana/kibana_migrator.js:204:34)
    at /usr/share/kibana/src/core/server/saved_objects/service/lib/repository.js:411:74
    at Array.map (<anonymous>)
[2021-11-02T17:14:58.914+00:00][ERROR][http] Error: Migration function for version 8.0.0 threw an error
    at tryTransformDoc (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:610:13)
    at migrateProp (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:679:22)
    at applyMigrations (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:397:20)
    at DocumentMigrator.transformAndValidate [as transformDoc] (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:337:22)
    at DocumentMigrator.migrate (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:130:16)
    at KibanaMigrator.migrateDocument (/usr/share/kibana/src/core/server/saved_objects/migrations/kibana/kibana_migrator.js:204:34)
    at /usr/share/kibana/src/core/server/saved_objects/service/lib/repository.js:411:74
    at Array.map (<anonymous>)
    at SavedObjectsRepository.bulkCreate (/usr/share/kibana/src/core/server/saved_objects/service/lib/repository.js:351:49)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at SavedObjectsClient.bulkCreate (/usr/share/kibana/src/core/server/saved_objects/service/saved_objects_client.js:54:12)
    at EncryptedSavedObjectsClientWrapper.bulkCreate (/usr/share/kibana/x-pack/plugins/encrypted_saved_objects/server/saved_objects/encrypted_saved_objects_client_wrapper.js:69:63)
    at SpacesSavedObjectsClient.bulkCreate (/usr/share/kibana/x-pack/plugins/spaces/server/saved_objects/spaces_saved_objects_client.js:82:12)
    at createSavedObjects (/usr/share/kibana/src/core/server/saved_objects/import/lib/create_saved_objects.js:104:32)
    at importSavedObjectsFromStream (/usr/share/kibana/src/core/server/saved_objects/import/import_saved_objects.js:89:36)
Caused by:
SyntaxError: Unexpected token o in JSON at position 1
    at JSON.parse (<anonymous>)
    at setEmsTmsDefaultModes (/usr/share/kibana/x-pack/plugins/maps/common/migrations/set_ems_tms_default_modes.js:28:26)
    at Object.8.0.0 (/usr/share/kibana/x-pack/plugins/maps/server/embeddable_migrations.js:35:72)
    at migrateFn (/usr/share/kibana/src/plugins/embeddable/common/lib/migrate.js:24:49)
    at migrations.<computed> (/usr/share/kibana/src/plugins/embeddable/common/lib/get_all_migrations.js:34:42)
    at /usr/share/kibana/src/plugins/dashboard/server/saved_objects/dashboard_migrations.js:189:29
    at Array.forEach (<anonymous>)
    at /usr/share/kibana/src/plugins/dashboard/server/saved_objects/dashboard_migrations.js:183:10
    at tryTransformDoc (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:597:22)
    at migrateProp (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:679:22)
    at applyMigrations (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:397:20)
    at DocumentMigrator.transformAndValidate [as transformDoc] (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:337:22)
    at DocumentMigrator.migrate (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:130:16)
    at KibanaMigrator.migrateDocument (/usr/share/kibana/src/core/server/saved_objects/migrations/kibana/kibana_migrator.js:204:34)
    at /usr/share/kibana/src/core/server/saved_objects/service/lib/repository.js:411:74
    at Array.map (<anonymous>)

This errors does not show up with earlier snapshots of Kibana and we haven't touched dashboard loading and the dashboards at all.

liza-mae · 2021-11-02T17:21:40Z

We believe this is a kibana issue so going to transfer issue there to be looked at.

elasticmachine · 2021-11-02T17:23:36Z

Pinging @elastic/kibana-core (Team:Core)

elasticmachine · 2021-11-02T17:51:12Z

Pinging @elastic/kibana-gis (Team:Geo)

liza-mae · 2021-11-02T19:36:51Z

@kvch from maps slack channel:
Filebeats have old tile-maps that are now being migrated to a different SO

liza-mae · 2021-11-02T20:39:54Z

Spoke to @nreese he says this should work, so he is looking to put a PR for it. Thanks!

nreese · 2021-11-02T21:11:05Z

The exception is getting thrown in a map saved object migration which was merged on Oct 27th.

The problem is not with the migration but the shape of the maps saved object provided by filebeat. attributes.layerListJSON is supposed to contain a string, not an array. Filebeat is providing an array.

https://github.com/elastic/beats/blob/master/x-pack/filebeat/module/cyberarkpas/_meta/kibana/7/dashboard/Filebeat-cyberarkpas-audit.json#L975

It looks like layerListJSON was converted from a string to JSON in #27901. @kvch I don't think this is going to work layerListJSON must be provided as a string.

If I remove the migration, the map panel is broken in the dashboard with the same problem

nreese · 2021-11-02T21:20:24Z

Closing since issue is with filebeat dashboard. Filebeat dashboard needs to pass maps saved objects with layerListJSON as a string and not an array. Regression caused by #27901 in beats

liza-mae · 2021-11-02T21:21:23Z

I will transfer it back to beats repo

kvch · 2021-11-03T07:05:22Z

I am cross posting if from Slack. We save dashboards in this JSON format to make reviewing easier for module developers. The dashboard object goes through several transformation before it is loaded to Kibana. LayerListJSON is transformed to string in this code: https://github.com/elastic/beats/blob/master/libbeat/dashboards/modify_json.go#L389-L418 since #28530

I still not believe that it is a Beats issue. Prevously, if a dashboard was not formatted correctly, Kibana responded with exact error messages where the formatting has to be changed. In this case we do not receive any errors on Beats side.

pgayvallet · 2021-11-03T09:01:21Z

I still not believe that it is a Beats issue

I will have to agree with @nreese on this one. The error from #28773 (comment) seems rather explicit, the migration tries to parse attributes.layerListJSON and fails because of invalid JSON.

[2021-11-02T17:14:58.914+00:00][ERROR][http] Error: Migration function for version 8.0.0 threw an error

Caused by:
SyntaxError: Unexpected token o in JSON at position 1
    at JSON.parse (<anonymous>)
    at setEmsTmsDefaultModes (/usr/share/kibana/x-pack/plugins/maps/common/migrations/set_ems_tms_default_modes.js:28:26)

Prevously, if a dashboard was not formatted correctly, Kibana responded with exact error messages where the formatting has to be changed

Hum, I'm surprised, as we (unfortunately) never had such kind of advanced validation for imports, even with the legacy dashboard import that was using create instead of import. Do you have an example of such exact error message?

rudolf · 2021-11-03T09:13:26Z

Is it possible to get the NDJSON payload that beats sends to the Kibana saved objects import API? We can use that to decide if the payload is well formatted or not.

I would also add that just because an import worked in a previous version doesn't mean that the payload was correct, it could be that the map was imported but that the maps app was unable to use the layer metadata (just speculating because I'm not familiar with the internals of the maps app).

kvch · 2021-11-03T09:14:05Z

I still not believe that it is a Beats issue

I will have to agree with @nreese on this one. The error from #28773 (comment) seems rather explicit, the migration tries to parse attributes.layerListJSON and fails because of invalid JSON.
[2021-11-02T17:14:58.914+00:00][ERROR][http] Error: Migration function for version 8.0.0 threw an error
Caused by:
SyntaxError: Unexpected token o in JSON at position 1
    at JSON.parse (<anonymous>)
    at setEmsTmsDefaultModes (/usr/share/kibana/x-pack/plugins/maps/common/migrations/set_ems_tms_default_modes.js:28:26)

Where is this error referring to the attribute?

Prevously, if a dashboard was not formatted correctly, Kibana responded with exact error messages where the formatting has to be changed

Hum, I'm surprised, as we (unfortunately) never had such kind of advanced validation for imports, even with the legacy dashboard import that was using create instead of import. Do you have an example of such exact error message?

The error can be found here: #27988
It was reported a few weeks ago that layerListJSON should be a string and I fixed it in #28530

kvch · 2021-11-03T09:16:03Z

@rudolf I will get more detailed logs for you in a few hours. Thanks!

pgayvallet · 2021-11-03T09:20:07Z

Where is this error referring to the attribute?

In the source code associated to the setEmsTmsDefaultModes failing migration:

https://github.com/elastic/kibana/blob/3b3c4b2f4a8fc120b2f9fc35c8945f59042079d0/x-pack/plugins/maps/common/migrations/set_ems_tms_default_modes.ts#L16-L25

kvch · 2021-11-03T09:26:37Z

These are the logs I get: https://gist.github.com/kvch/dcef6cc6be4c0e8b262d5769f9a0df50
I will look into it once more when I get back. Thanks.

kvch · 2021-11-03T11:43:25Z

Ok, that was the wrong output.

But I have checked it locally, and it seems it's indeed a Beats issue. Thank you for pointing me in the right direction.

liza-mae · 2021-11-03T17:59:39Z

Thanks @kvch for #28787 PR, should this be back-ported to 8.0 branch?

andresrc · 2021-11-03T18:31:42Z

Backport to 8.0: #28796

kvch · 2021-11-05T15:15:38Z

The backport is merged.

liza-mae transferred this issue from elastic/beats Nov 2, 2021

nreese closed this as completed Nov 2, 2021

liza-mae reopened this Nov 2, 2021

liza-mae transferred this issue from elastic/kibana Nov 2, 2021

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Nov 2, 2021

liza-mae added Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team Team:Integrations Label for the Integrations team bug regression and removed needs_team Indicates that the issue/PR needs a Team:* label labels Nov 2, 2021

kvch mentioned this issue Nov 3, 2021

Convert required fields to string in embeddedConfig recursively #28787

Merged

1 task

nreese mentioned this issue Nov 3, 2021

[Maps] do not fail migration when JSON.parse fails elastic/kibana#117342

Merged

kvch closed this as completed in #28787 Nov 3, 2021

liza-mae reopened this Nov 3, 2021

kvch closed this as completed Nov 5, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Filebeat import dashboards returns internal server error #28773

Filebeat import dashboards returns internal server error #28773

liza-mae commented Oct 28, 2021

elasticmachine commented Oct 28, 2021

elasticmachine commented Oct 28, 2021

liza-mae commented Nov 2, 2021

kvch commented Nov 2, 2021 •

edited

Loading

liza-mae commented Nov 2, 2021

elasticmachine commented Nov 2, 2021

elasticmachine commented Nov 2, 2021

liza-mae commented Nov 2, 2021

liza-mae commented Nov 2, 2021

nreese commented Nov 2, 2021 •

edited

Loading

nreese commented Nov 2, 2021 •

edited

Loading

liza-mae commented Nov 2, 2021

kvch commented Nov 3, 2021

pgayvallet commented Nov 3, 2021 •

edited

Loading

rudolf commented Nov 3, 2021

kvch commented Nov 3, 2021

kvch commented Nov 3, 2021

pgayvallet commented Nov 3, 2021

kvch commented Nov 3, 2021

kvch commented Nov 3, 2021

liza-mae commented Nov 3, 2021

andresrc commented Nov 3, 2021

kvch commented Nov 5, 2021

Filebeat import dashboards returns internal server error #28773

Filebeat import dashboards returns internal server error #28773

Comments

liza-mae commented Oct 28, 2021

elasticmachine commented Oct 28, 2021

elasticmachine commented Oct 28, 2021

liza-mae commented Nov 2, 2021

kvch commented Nov 2, 2021 • edited Loading

liza-mae commented Nov 2, 2021

elasticmachine commented Nov 2, 2021

elasticmachine commented Nov 2, 2021

liza-mae commented Nov 2, 2021

liza-mae commented Nov 2, 2021

nreese commented Nov 2, 2021 • edited Loading

nreese commented Nov 2, 2021 • edited Loading

liza-mae commented Nov 2, 2021

kvch commented Nov 3, 2021

pgayvallet commented Nov 3, 2021 • edited Loading

rudolf commented Nov 3, 2021

kvch commented Nov 3, 2021

kvch commented Nov 3, 2021

pgayvallet commented Nov 3, 2021

kvch commented Nov 3, 2021

kvch commented Nov 3, 2021

liza-mae commented Nov 3, 2021

andresrc commented Nov 3, 2021

kvch commented Nov 5, 2021

kvch commented Nov 2, 2021 •

edited

Loading

nreese commented Nov 2, 2021 •

edited

Loading

nreese commented Nov 2, 2021 •

edited

Loading

pgayvallet commented Nov 3, 2021 •

edited

Loading