-
Notifications
You must be signed in to change notification settings - Fork 8.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Corrupt Saved Object documents can cause a migration every time Kibana starts up #65612
Comments
Pinging @elastic/kibana-platform (Team:Platform) |
This bug can also lead to data loss if multiple Kibana nodes are using the same Elasticsearch index. If one of the nodes are restarted while the other nodes accept writes, the restarted node will create a new |
Once we complete #75780 Kibana upgrade migrations will fail if there are corrupt saved object documents encountered. Users will have to delete the corrupt document to proceed. This prevents hard to debug migration problems where we're unable to transform the corrupt document but still try persist it leading which might lead to a mappings exception. |
When starting up, Kibana scans the
.kibana
alias/index for documents that require a migration. For each document being migrated, we do a basic validation that checks if the type and namespace in the id matches the type and namespace fields. If these don't match, we don't apply the migration and just write the document as-is. Because the document's migrationVersion wasn't bumped, the next time Kibana starts up it will again see outdated documents and start a new migration.kibana/src/core/server/saved_objects/migrations/core/migrate_raw_docs.ts
Line 41 in ea7c78d
Because the Saved Objects API ensures that written documents are always valid, this will only affect users which write saved object documents directly into Elasticsearch instead of using the Kibana API's.
As a first step we can add logging to highlight this case to users.
The text was updated successfully, but these errors were encountered: