[meta] Better handling of the most common migration issues

[pgayvallet]

Our most common migration failure causes could highly benefit from better identification and documentation.

This is a meta issue to list and track the process of each individual failures.

For each failure, we want to:

In any case:

• Be able to identify it, and to assign a unique error code to it
• Add online documentation describing how to fix, or work around, the failure
  • it can either be one page per failure or one page listing all the failures, TBD
• Surface the error code, and the link to the documentation, in the failure's log

When the failure's cause can be predetermined:

• fail-fast during the migration (e.g as it was done in [SO migrations] exit early if cluster routing allocation is disabled #126612)
• surface the problem in Upgrade Assistant

Note: ideally, these changes would be backported to the 7.17 branch to ease the migration experience from 7.last to 8+

Individual issues:

• migrations fail with "Not enough active copies to meet shard count of [ALL] (have 1, needed 2)" #127136
• migrations fail with "this action would add [2] shards, but this cluster currently has [X]/[Y] maximum normal shards open" #128578
• migrations fail with "Timeout waiting for the status of the [.kibana_{VERSION}_reindex_temp] index to become 'yellow'" #128585
• Improve user-experience when encountering unknown or corrupted SO documents during a migration  #129018

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[TinaHeiligers]

assign a unique error code to it

Do we want to build on what we already have for the actions/es_errors or implement something entirely new?
For example, we have a rather nicely-readable error format for the saved objects service.

As for the code ranges, assuming the errors would fall in the 5xx, we could add a generic CODE_SAVED_OBJECTS_MIGRATIONS_ERROR as a 51x and decorate that.

OTOH, we'll probably run out of integers as we start reporting individual elasticsearch API errors.

@pgayvallet @rudolf If there's already a documented strategy for the error codes, please let me know!

[pgayvallet]

I don't think we should be using the SO service errors for SO migrations. SO errors are errors that are meant to be surfaced to the client via http transport, and therefor are directly bound to a specific http error code and structure.

For SO migration failures, the only thing we need ihmo is a unique error identifier that will be surfaced in the logs and can be used to search, or link to, the error in our documentation.

I was naively thinking of something like

ERROR_SO_MIGRATION_FAILURE_XXX, with XXX being either a number or an alpha-num value

e.g

ERROR_SO_MIGRATION_FAILURE_001

or

ERROR_SO_MIGRATION_FAILURE_ROUTING_ALLOCATION_DISABLED

The ERROR_SO_MIGRATION_FAILURE prefix could easily be changed too if someone comes with a better idea

[Bamieh]

@pgayvallet Just started looking into this. Historically ES avoided using error codes and I think it makes sense for Kibana to adopt their philosophy as well. I understand error codes might be easier to document however searching specific error strings on google is a lot easier and gives more relevant results. It also becomes another thing we need to maintain on the long run that we can avoid completely.

Update: Reflecting our discussion during the team sync:
The intention is to use an "error type" rather than an error code which should make it easier to search additionally we'll be linking to our docs to help people resolve these errors so using these error types should make it easier for users to troubleshoot and fix the migration errors.

[rudolf]

Closing as all the individual issues have been addressed