[Ingest Manager] Handle Legacy ES client errors #76865

jfsiii · 2020-09-07T11:14:56Z

Summary

Use an HTTP status code if ES client error provides one. extended Non-deterministic behavior while installing a package #75862 (comment) to all 4xx-5xx errors
Format Error message as described in Non-deterministic behavior while installing a package #75862 (comment) & agreed to Non-deterministic behavior while installing a package #75862 (comment)

example Request/Response:

> curl --user elastic:changeme -X POST http://localhost:5601/api/ingest_manager/epm/packages/aws-0.2.7 -H 'kbn-xsrf: xyz'
{
  "statusCode":400,
  "error":"Bad Request",
  "message":"[parse_exception] Failed to parse content to map from ES at /_ingest/pipeline/logs-aws.cloudtrail-0.2.7: {\"error\":{\"root_cause\":[{\"type\":\"parse_exception\",\"reason\":\"Failed to parse content to map\"}],\"type\":\"parse_exception\",\"reason\":\"Failed to parse content to map\",\"caused_by\":{\"type\":\"json_parse_exception\",\"reason\":\"Duplicate field 'ListGroupsForUser'\\n at [Source: (byte[])\\\"---\\ndescription: Pipeline for AWS CloudTrail Logs\\nprocessors:\\n  - set:\\n      field: event.ingested\\n      value: '{{_ingest.timestamp}}'\\n  - rename:\\n      field: \\\"message\\\"\\n      target_field: \\\"event.original\\\"\\n  - json:\\n      field: \\\"event.original\\\"\\n      target_field: \\\"json\\\"\\n  - date:\\n      field: \\\"json.eventTime\\\"\\n      target_field: \\\"@timestamp\\\"\\n      ignore_failure: true\\n      formats:\\n        - ISO8601\\n  - rename:\\n      field: \\\"json.eventVersion\\\"\\n      target_field: \\\"aws.cloudtrail.event_versi\\\"[truncated 16425 bytes]; line: 489, column: 26]\"}},\"status\":400}"
}

example Kibana Logs:

[parse_exception] Failed to parse content to map

Used `test.each` to generate tests for each 4xx - 5xx error. Call each error 3 different ways.

  defaultIngestErrorHandler
    use the HTTP error status code provided by LegacyESErrors
      ✓ 400 - with path & response (12ms)
      ✓ 401 - with path & response (5ms)
      ✓ 402 - with path & response (5ms)
      ✓ 403 - with path & response (6ms)
      ✓ 404 - with path & response (5ms)
      ✓ 405 - with path & response (17ms)
      ✓ 406 - with path & response (2ms)
      ✓ 407 - with path & response (3ms)
      ✓ 408 - with path & response (6ms)
      ✓ 409 - with path & response (5ms)
      ✓ 410 - with path & response (1ms)
      ✓ 411 - with path & response (1ms)
      ✓ 412 - with path & response (1ms)
      ✓ 413 - with path & response (1ms)
      ✓ 414 - with path & response (1ms)
      ✓ 415 - with path & response (1ms)
      ✓ 416 - with path & response (1ms)
      ✓ 417 - with path & response (9ms)
      ✓ 418 - with path & response (1ms)
      ✓ 421 - with path & response (1ms)
      ✓ 426 - with path & response (1ms)
      ✓ 429 - with path & response (1ms)
      ✓ 450 - with path & response (1ms)
      ✓ 494 - with path & response (1ms)
      ✓ 497 - with path & response (1ms)
      ✓ 499 - with path & response (3ms)
      ✓ 500 - with path & response (2ms)
      ✓ 501 - with path & response (1ms)
      ✓ 502 - with path & response (2ms)
      ✓ 503 - with path & response (1ms)
      ✓ 504 - with path & response (1ms)
      ✓ 505 - with path & response (8ms)
      ✓ 506 - with path & response (2ms)
      ✓ 510 - with path & response (1ms)
      ✓ 400 - with other metadata (1ms)
      ✓ 401 - with other metadata (1ms)
      ✓ 402 - with other metadata (1ms)
      ✓ 403 - with other metadata (1ms)
      ✓ 404 - with other metadata (2ms)
      ✓ 405 - with other metadata (1ms)
      ✓ 406 - with other metadata (2ms)
      ✓ 407 - with other metadata (1ms)
      ✓ 408 - with other metadata (1ms)
      ✓ 409 - with other metadata (1ms)
      ✓ 410 - with other metadata (10ms)
      ✓ 411 - with other metadata (2ms)
      ✓ 412 - with other metadata (1ms)
      ✓ 413 - with other metadata (1ms)
      ✓ 414 - with other metadata (1ms)
      ✓ 415 - with other metadata (1ms)
      ✓ 416 - with other metadata (7ms)
      ✓ 417 - with other metadata (2ms)
      ✓ 418 - with other metadata (1ms)
      ✓ 421 - with other metadata (1ms)
      ✓ 426 - with other metadata (1ms)
      ✓ 429 - with other metadata (1ms)
      ✓ 450 - with other metadata (1ms)
      ✓ 494 - with other metadata (11ms)
      ✓ 497 - with other metadata (1ms)
      ✓ 499 - with other metadata (1ms)
      ✓ 500 - with other metadata (1ms)
      ✓ 501 - with other metadata (1ms)
      ✓ 502 - with other metadata (1ms)
      ✓ 503 - with other metadata (1ms)
      ✓ 504 - with other metadata (2ms)
      ✓ 505 - with other metadata (2ms)
      ✓ 506 - with other metadata (1ms)
      ✓ 510 - with other metadata (1ms)
      ✓ 400 - without metadata (1ms)
      ✓ 401 - without metadata (1ms)
      ✓ 402 - without metadata (10ms)
      ✓ 403 - without metadata (1ms)
      ✓ 404 - without metadata (1ms)
      ✓ 405 - without metadata (1ms)
      ✓ 406 - without metadata (1ms)
      ✓ 407 - without metadata (1ms)
      ✓ 408 - without metadata (1ms)
      ✓ 409 - without metadata (1ms)
      ✓ 410 - without metadata (1ms)
      ✓ 411 - without metadata (1ms)
      ✓ 412 - without metadata (1ms)
      ✓ 413 - without metadata (1ms)
      ✓ 414 - without metadata (1ms)
      ✓ 415 - without metadata (12ms)
      ✓ 416 - without metadata (1ms)
      ✓ 417 - without metadata (2ms)
      ✓ 418 - without metadata (1ms)
      ✓ 421 - without metadata (1ms)
      ✓ 426 - without metadata (2ms)
      ✓ 429 - without metadata (2ms)
      ✓ 450 - without metadata (3ms)
      ✓ 494 - without metadata (2ms)
      ✓ 497 - without metadata (2ms)
      ✓ 499 - without metadata (1ms)
      ✓ 500 - without metadata (1ms)
      ✓ 501 - without metadata (2ms)
      ✓ 502 - without metadata (1ms)
      ✓ 503 - without metadata (10ms)
      ✓ 504 - without metadata (2ms)
      ✓ 505 - without metadata (1ms)
      ✓ 506 - without metadata (2ms)
      ✓ 510 - without metadata (1ms)

Checklist

Unit or functional tests were updated or added to match the most common scenarios

Manual testing

Manual testing

Checkout the branch from elastic/package-storage#370

git clone https://github.com/elastic/package-storage.git
cd package-storage/
git switch -C update-aws-0.2.7-1598281986 origin/update-aws-0.2.7-1598281986

start the stack using the registry from elastic/package-storage#370

cd testing/environments/
docker-compose -f snapshot.yml pull
docker-compose -f snapshot.yml -f local.yml up --force-recreate

Try to install the broken package

> curl --user elastic:changeme -X POST http://localhost:5601/api/ingest_manager/epm/packages/aws-0.2.7 -H 'kbn-xsrf: xyz'
{"statusCode":500,"error":"Internal Server Error","message":"Bad Request"}

observe the same error as #75862

start only local registry with the broken package

# CTRL-C the stack (shell where you ran `docker-compose`)
cd ../.. # back to package-storage root
docker build .
docker run -p 8080:8080 id_from_prior_step

start the stack from this PR, pointing at the local registry from prior step

yarn start --no-base-path --xpack.ingestManager.registryUrl=http://localhost:8080
yarn es snapshot --license=trial  -E xpack.security.authc.api_key.enabled=true

Try to install the broken package

curl --user elastic:changeme -X POST http://localhost:5601/api/ingest_manager/epm/packages/aws-0.2.7 -H 'kbn-xsrf: xyz'
{
  "statusCode": 400,
  "error": "Bad Request",
  "message": "[parse_exception] Failed to parse content to map response from /_ingest/pipeline/logs-aws.cloudtrail-0.2.7: {\"error\":{\"root_cause\":[{\"type\":\"parse_exception\",\"reason\":\"Failed to parse content to map\"}],\"type\":\"parse_exception\",\"reason\":\"Failed to parse content to map\",\"caused_by\":{\"type\":\"json_parse_exception\",\"reason\":\"Duplicate field 'ListGroupsForUser'\\n at [Source: (byte[])\\\"---\\ndescription: Pipeline for AWS CloudTrail Logs\\nprocessors:\\n  - set:\\n      field: event.ingested\\n      value: '{{_ingest.timestamp}}'\\n  - rename:\\n      field: \\\"message\\\"\\n      target_field: \\\"event.original\\\"\\n  - json:\\n      field: \\\"event.original\\\"\\n      target_field: \\\"json\\\"\\n  - date:\\n      field: \\\"json.eventTime\\\"\\n      target_field: \\\"@timestamp\\\"\\n      ignore_failure: true\\n      formats:\\n        - ISO8601\\n  - rename:\\n      field: \\\"json.eventVersion\\\"\\n      target_field: \\\"aws.cloudtrail.event_versi\\\"[truncated 16425 bytes]; line: 489, column: 26]\"}},\"status\":400}"
}

observe new error format

@timestamp

``` > curl --user elastic:changeme -X POST http://localhost:5601/api/ingest_manager/epm/packages/aws-0.2.7 -H 'kbn-xsrf: xyz' {"statusCode":400,"error":"Bad Request","message":"[parse_exception] Failed to parse content to map from ES at /_ingest/pipeline/logs-aws.cloudtrail-0.2.7: {\"error\":{\"root_cause\":[{\"type\":\"parse_exception\",\"reason\":\"Failed to parse content to map\"}],\"type\":\"parse_exception\",\"reason\":\"Failed to parse content to map\",\"caused_by\":{\"type\":\"json_parse_exception\",\"reason\":\"Duplicate field 'ListGroupsForUser'\\n at [Source: (byte[])\\\"---\\ndescription: Pipeline for AWS CloudTrail Logs\\nprocessors:\\n - set:\\n field: event.ingested\\n value: '{{_ingest.timestamp}}'\\n - rename:\\n field: \\\"message\\\"\\n target_field: \\\"event.original\\\"\\n - json:\\n field: \\\"event.original\\\"\\n target_field: \\\"json\\\"\\n - date:\\n field: \\\"json.eventTime\\\"\\n target_field: \\\"@timestamp\\\"\\n ignore_failure: true\\n formats:\\n - ISO8601\\n - rename:\\n field: \\\"json.eventVersion\\\"\\n target_field: \\\"aws.cloudtrail.event_versi\\\"[truncated 16425 bytes]; line: 489, column: 26]\"}},\"status\":400}"} ```

x-pack/plugins/ingest_manager/server/errors/handlers.ts

``` use the HTTP error status code provided by LegacyESErrors ✓ 400 - with meta (21ms) ✓ 401 - with meta (8ms) ✓ 402 - with meta (18ms) ✓ 403 - with meta (7ms) ✓ 404 - with meta (5ms) ✓ 405 - with meta (4ms) ✓ 406 - with meta (4ms) ✓ 407 - with meta (11ms) ✓ 408 - with meta (14ms) ✓ 409 - with meta (7ms) ✓ 410 - with meta (4ms) ✓ 411 - with meta (4ms) ✓ 412 - with meta (4ms) ✓ 413 - with meta (4ms) ✓ 414 - with meta (4ms) ✓ 415 - with meta (4ms) ✓ 416 - with meta (7ms) ✓ 417 - with meta (5ms) ✓ 418 - with meta (5ms) ✓ 421 - with meta (4ms) ✓ 426 - with meta (4ms) ✓ 429 - with meta (13ms) ✓ 450 - with meta (4ms) ✓ 494 - with meta (5ms) ✓ 497 - with meta (5ms) ✓ 499 - with meta (6ms) ✓ 500 - with meta (7ms) ✓ 501 - with meta (6ms) ✓ 502 - with meta (6ms) ✓ 503 - with meta (7ms) ✓ 504 - with meta (5ms) ✓ 505 - with meta (5ms) ✓ 506 - with meta (6ms) ✓ 510 - with meta (6ms) ✓ 400 - without meta (5ms) ✓ 401 - without meta (12ms) ✓ 402 - without meta (5ms) ✓ 403 - without meta (7ms) ✓ 404 - without meta (7ms) ✓ 405 - without meta (6ms) ✓ 406 - without meta (6ms) ✓ 407 - without meta (5ms) ✓ 408 - without meta (6ms) ✓ 409 - without meta (5ms) ✓ 410 - without meta (6ms) ✓ 411 - without meta (5ms) ✓ 412 - without meta (6ms) ✓ 413 - without meta (5ms) ✓ 414 - without meta (11ms) ✓ 415 - without meta (6ms) ✓ 416 - without meta (7ms) ✓ 417 - without meta (5ms) ✓ 418 - without meta (3ms) ✓ 421 - without meta (4ms) ✓ 426 - without meta (77ms) ✓ 429 - without meta (13ms) ✓ 450 - without meta (3ms) ✓ 494 - without meta (2ms) ✓ 497 - without meta (2ms) ✓ 499 - without meta (2ms) ✓ 500 - without meta (2ms) ✓ 501 - without meta (8ms) ✓ 502 - without meta (2ms) ✓ 503 - without meta (2ms) ✓ 504 - without meta (3ms) ✓ 505 - without meta (2ms) ✓ 506 - without meta (2ms) ✓ 510 - without meta (2ms) ```

elasticmachine · 2020-09-08T19:34:09Z

Pinging @elastic/ingest-management (Team:Ingest Management)

jfsiii · 2020-09-08T20:46:11Z

@elasticmachine merge upstream

jfsiii · 2020-09-08T21:24:03Z

@elasticmachine merge upstream

``` use the HTTP error status code provided by LegacyESErrors ✓ 400 - with path & response (11ms) ✓ 401 - with path & response (7ms) ✓ 402 - with path & response (5ms) ✓ 403 - with path & response (7ms) ✓ 404 - with path & response (7ms) ✓ 405 - with path & response (20ms) ✓ 406 - with path & response (2ms) ✓ 407 - with path & response (3ms) ✓ 408 - with path & response (6ms) ✓ 409 - with path & response (8ms) ✓ 410 - with path & response (1ms) ✓ 411 - with path & response (1ms) ✓ 412 - with path & response (1ms) ✓ 413 - with path & response (1ms) ✓ 414 - with path & response (1ms) ✓ 415 - with path & response (1ms) ✓ 416 - with path & response (2ms) ✓ 417 - with path & response (13ms) ✓ 418 - with path & response (1ms) ✓ 421 - with path & response (1ms) ✓ 426 - with path & response (1ms) ✓ 429 - with path & response (2ms) ✓ 450 - with path & response (1ms) ✓ 494 - with path & response (1ms) ✓ 497 - with path & response (1ms) ✓ 499 - with path & response (1ms) ✓ 500 - with path & response (2ms) ✓ 501 - with path & response (1ms) ✓ 502 - with path & response (1ms) ✓ 503 - with path & response (1ms) ✓ 504 - with path & response (1ms) ✓ 505 - with path & response (10ms) ✓ 506 - with path & response (1ms) ✓ 510 - with path & response (2ms) ✓ 400 - with other metadata (1ms) ✓ 401 - with other metadata (1ms) ✓ 402 - with other metadata (1ms) ✓ 403 - with other metadata (1ms) ✓ 404 - with other metadata (1ms) ✓ 405 - with other metadata (1ms) ✓ 406 - with other metadata (1ms) ✓ 407 - with other metadata (1ms) ✓ 408 - with other metadata (2ms) ✓ 409 - with other metadata (1ms) ✓ 410 - with other metadata (10ms) ✓ 411 - with other metadata (1ms) ✓ 412 - with other metadata (1ms) ✓ 413 - with other metadata (1ms) ✓ 414 - with other metadata (1ms) ✓ 415 - with other metadata (1ms) ✓ 416 - with other metadata (10ms) ✓ 417 - with other metadata (1ms) ✓ 418 - with other metadata (1ms) ✓ 421 - with other metadata (1ms) ✓ 426 - with other metadata (1ms) ✓ 429 - with other metadata (1ms) ✓ 450 - with other metadata (1ms) ✓ 494 - with other metadata (11ms) ✓ 497 - with other metadata (2ms) ✓ 499 - with other metadata (1ms) ✓ 500 - with other metadata (1ms) ✓ 501 - with other metadata (1ms) ✓ 502 - with other metadata (1ms) ✓ 503 - with other metadata (1ms) ✓ 504 - with other metadata (2ms) ✓ 505 - with other metadata (1ms) ✓ 506 - with other metadata (1ms) ✓ 510 - with other metadata (1ms) ✓ 400 - without metadata (1ms) ✓ 401 - without metadata (1ms) ✓ 402 - without metadata (13ms) ✓ 403 - without metadata (1ms) ✓ 404 - without metadata (1ms) ✓ 405 - without metadata (2ms) ✓ 406 - without metadata (1ms) ✓ 407 - without metadata (1ms) ✓ 408 - without metadata (1ms) ✓ 409 - without metadata (1ms) ✓ 410 - without metadata (1ms) ✓ 411 - without metadata (1ms) ✓ 412 - without metadata (1ms) ✓ 413 - without metadata (1ms) ✓ 414 - without metadata (1ms) ✓ 415 - without metadata (2ms) ✓ 416 - without metadata (10ms) ✓ 417 - without metadata (2ms) ✓ 418 - without metadata (1ms) ✓ 421 - without metadata (1ms) ✓ 426 - without metadata (1ms) ✓ 429 - without metadata (1ms) ✓ 450 - without metadata (1ms) ✓ 494 - without metadata (1ms) ✓ 497 - without metadata (2ms) ✓ 499 - without metadata (1ms) ✓ 500 - without metadata (1ms) ✓ 501 - without metadata (1ms) ✓ 502 - without metadata (1ms) ✓ 503 - without metadata (11ms) ✓ 504 - without metadata (1ms) ✓ 505 - without metadata (1ms) ✓ 506 - without metadata (3ms) ✓ 510 - without metadata (1ms) ```

…bana into 75862-surface-legacy-es-errors

skh · 2020-09-09T10:41:36Z

x-pack/plugins/ingest_manager/server/errors/handlers.ts

+    // there was a problem communicating with ES (e.g. via `callCluster`)
+    // only log the message
+    logger.error(error.message);
+


Why not log the more elaborate message that is constructed below, and passed to response.customError()?

No real reason not to. I considered it at one point. Happy to update it.

I think it would be helpful to have the same level of detail in both the log and the returned message.

Totally. Just pushed fdaf588

skh · 2020-09-09T10:43:18Z

Have you thought about adding an integration test for this in x-pack/test/ingest_manager_api_integration with an actual broken package?

jfsiii · 2020-09-09T15:31:09Z

Have you thought about adding an integration test for this in x-pack/test/ingest_manager_api_integration with an actual broken package?

@skh yes, but I was hoping to get this improvement in for us & our users then do that in a separate PR.

skh

Do you have a broken package or instructions how to trigger the error case(s) locally? I'd be happy to do that as well.

Otherwise, code looks good 👍

jfsiii · 2020-09-09T18:08:21Z

@skh Thanks for the testing offer & reminder. I added a "Manual testing" section to the description

jfsiii · 2020-09-09T19:28:56Z

@elasticmachine merge upstream

kibanamachine · 2020-09-09T21:12:02Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 00a87dc

Build metrics

distributable file count

id	value	diff	baseline
default	45478	+1	45477

History

💚 Build #73429 succeeded fdaf588
💚 Build #73286 succeeded e05362c
💚 Build #73246 succeeded 1bba375
💚 Build #73214 succeeded 154f82b
💚 Build #72909 succeeded 8e26047

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

skh · 2020-09-10T09:25:56Z

Thank you for the testing instructions! The branch origin/update-aws-0.2.7-1598281986 seems to be no longer available on package-storage, so here's what I ended up doing instead:

check out any branch that has packages (production, staging or snapshot), I used production
in any package, find a pipeline in YAML format and delete the first line that contains only ---. This makes it valid YAML that happens to be not acceptable to ES, so it passes the build step of the registry, but triggers an error when the package is installed. I used apache-0.1.4/dataset/access/elasticsearch/ingest_pipeline/default.yml to do so.

Installing apache-0.1.4 then resulted in this response:

{"statusCode":500,"error":"Internal Server Error","message":"[not_x_content_exception] Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes response from /_ingest/pipeline/logs-apache.access-0.1.4: {\"error\":{\"root_cause\":[{\"type\":\"not_x_content_exception\",\"reason\":\"Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes\"}],\"type\":\"not_x_content_exception\",\"reason\":\"Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes\"},\"status\":500}"}

and this log entry:

server    log   [11:24:00.539] [error][ingestManager][plugins] [not_x_content_exception] Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes response from /_ingest/pipeline/logs-apache.access-0.1.4: {"error":{"root_cause":[{"type":"not_x_content_exception","reason":"Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes"}],"type":"not_x_content_exception","reason":"Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes"},"status":500}

jfsiii · 2020-09-10T10:48:49Z

Oh, woops. I just noticed elastic/package-storage#370 (comment)

@kaiyan-sheng kaiyan-sheng deleted the update-aws-0.2.7-1598281986 branch 15 days ago

I guess I still had it checked out

Nice to have more examples. Especially one so straightforward. The … response from /_ingest/pipeline/… message in log/response is from this PR so I'm calling this confirmed :)

…rok/new-patterns-component-use-array * 'master' of github.com:elastic/kibana: (39 commits) [APM] Always load esarchives from common (elastic#77139) [Ingest Manager] Handle Legacy ES client errors (elastic#76865) [Docs] URL Drilldown (elastic#76529) [bugfix] Replace panel flyout opens 2 flyouts (elastic#76931) clean up test (elastic#76887) [Enterprise Search] Update shared API request handler (elastic#77112) [Maps] convert ESAggSource to TS (elastic#76999) Add plugin status API - take 2 (elastic#76732) Adds lens as a readable saved object for read-only dashboard users (elastic#77067) Skip checking for the reserved realm (elastic#76687) Reporting/diagnostics (elastic#74314) Reporting/Test: unskip non-screenshot tests (elastic#77088) Move metrics to setup and add cgroup metrics (elastic#76730) [Enterprise Search] Add Overview landing page/plugin (elastic#76734) First pass. Change TS type. Update OpenAPI (elastic#76434) [CI] Balance xpack ci groups a bit (elastic#77068) [Security_solution][Detections] Refactor signal ancestry to allow multiple parents (elastic#76531) [Maps] convert MetricsEditor to TS (elastic#76727) IndexMigrator: fix non blocking migration wrapper promise rejection (elastic#77018) [Enterprise Search] Update config data endpoint to v2 (elastic#76970) ... # Conflicts: # src/plugins/es_ui_shared/static/forms/hook_form_lib/components/use_array.ts

@timestamp

## Summary closes #75862 1. Use an HTTP status code if ES client error provides one. extended #75862 (comment) to all 4xx-5xx errors 1. Format Error message as described in #75862 (comment) & agreed to #75862 (comment) ### example Request/Response: ``` > curl --user elastic:changeme -X POST http://localhost:5601/api/ingest_manager/epm/packages/aws-0.2.7 -H 'kbn-xsrf: xyz' { "statusCode":400, "error":"Bad Request", "message":"[parse_exception] Failed to parse content to map from ES at /_ingest/pipeline/logs-aws.cloudtrail-0.2.7: {\"error\":{\"root_cause\":[{\"type\":\"parse_exception\",\"reason\":\"Failed to parse content to map\"}],\"type\":\"parse_exception\",\"reason\":\"Failed to parse content to map\",\"caused_by\":{\"type\":\"json_parse_exception\",\"reason\":\"Duplicate field 'ListGroupsForUser'\\n at [Source: (byte[])\\\"---\\ndescription: Pipeline for AWS CloudTrail Logs\\nprocessors:\\n - set:\\n field: event.ingested\\n value: '{{_ingest.timestamp}}'\\n - rename:\\n field: \\\"message\\\"\\n target_field: \\\"event.original\\\"\\n - json:\\n field: \\\"event.original\\\"\\n target_field: \\\"json\\\"\\n - date:\\n field: \\\"json.eventTime\\\"\\n target_field: \\\"@timestamp\\\"\\n ignore_failure: true\\n formats:\\n - ISO8601\\n - rename:\\n field: \\\"json.eventVersion\\\"\\n target_field: \\\"aws.cloudtrail.event_versi\\\"[truncated 16425 bytes]; line: 489, column: 26]\"}},\"status\":400}" } ``` ### example Kibana Logs: `[parse_exception] Failed to parse content to map` <details><summary>Used `test.each` to generate tests for each 4xx - 5xx error. Call each error 3 different ways.</summary> ``` defaultIngestErrorHandler use the HTTP error status code provided by LegacyESErrors ✓ 400 - with path & response (12ms) ✓ 401 - with path & response (5ms) ✓ 402 - with path & response (5ms) ✓ 403 - with path & response (6ms) ✓ 404 - with path & response (5ms) ✓ 405 - with path & response (17ms) ✓ 406 - with path & response (2ms) ✓ 407 - with path & response (3ms) ✓ 408 - with path & response (6ms) ✓ 409 - with path & response (5ms) ✓ 410 - with path & response (1ms) ✓ 411 - with path & response (1ms) ✓ 412 - with path & response (1ms) ✓ 413 - with path & response (1ms) ✓ 414 - with path & response (1ms) ✓ 415 - with path & response (1ms) ✓ 416 - with path & response (1ms) ✓ 417 - with path & response (9ms) ✓ 418 - with path & response (1ms) ✓ 421 - with path & response (1ms) ✓ 426 - with path & response (1ms) ✓ 429 - with path & response (1ms) ✓ 450 - with path & response (1ms) ✓ 494 - with path & response (1ms) ✓ 497 - with path & response (1ms) ✓ 499 - with path & response (3ms) ✓ 500 - with path & response (2ms) ✓ 501 - with path & response (1ms) ✓ 502 - with path & response (2ms) ✓ 503 - with path & response (1ms) ✓ 504 - with path & response (1ms) ✓ 505 - with path & response (8ms) ✓ 506 - with path & response (2ms) ✓ 510 - with path & response (1ms) ✓ 400 - with other metadata (1ms) ✓ 401 - with other metadata (1ms) ✓ 402 - with other metadata (1ms) ✓ 403 - with other metadata (1ms) ✓ 404 - with other metadata (2ms) ✓ 405 - with other metadata (1ms) ✓ 406 - with other metadata (2ms) ✓ 407 - with other metadata (1ms) ✓ 408 - with other metadata (1ms) ✓ 409 - with other metadata (1ms) ✓ 410 - with other metadata (10ms) ✓ 411 - with other metadata (2ms) ✓ 412 - with other metadata (1ms) ✓ 413 - with other metadata (1ms) ✓ 414 - with other metadata (1ms) ✓ 415 - with other metadata (1ms) ✓ 416 - with other metadata (7ms) ✓ 417 - with other metadata (2ms) ✓ 418 - with other metadata (1ms) ✓ 421 - with other metadata (1ms) ✓ 426 - with other metadata (1ms) ✓ 429 - with other metadata (1ms) ✓ 450 - with other metadata (1ms) ✓ 494 - with other metadata (11ms) ✓ 497 - with other metadata (1ms) ✓ 499 - with other metadata (1ms) ✓ 500 - with other metadata (1ms) ✓ 501 - with other metadata (1ms) ✓ 502 - with other metadata (1ms) ✓ 503 - with other metadata (1ms) ✓ 504 - with other metadata (2ms) ✓ 505 - with other metadata (2ms) ✓ 506 - with other metadata (1ms) ✓ 510 - with other metadata (1ms) ✓ 400 - without metadata (1ms) ✓ 401 - without metadata (1ms) ✓ 402 - without metadata (10ms) ✓ 403 - without metadata (1ms) ✓ 404 - without metadata (1ms) ✓ 405 - without metadata (1ms) ✓ 406 - without metadata (1ms) ✓ 407 - without metadata (1ms) ✓ 408 - without metadata (1ms) ✓ 409 - without metadata (1ms) ✓ 410 - without metadata (1ms) ✓ 411 - without metadata (1ms) ✓ 412 - without metadata (1ms) ✓ 413 - without metadata (1ms) ✓ 414 - without metadata (1ms) ✓ 415 - without metadata (12ms) ✓ 416 - without metadata (1ms) ✓ 417 - without metadata (2ms) ✓ 418 - without metadata (1ms) ✓ 421 - without metadata (1ms) ✓ 426 - without metadata (2ms) ✓ 429 - without metadata (2ms) ✓ 450 - without metadata (3ms) ✓ 494 - without metadata (2ms) ✓ 497 - without metadata (2ms) ✓ 499 - without metadata (1ms) ✓ 500 - without metadata (1ms) ✓ 501 - without metadata (2ms) ✓ 502 - without metadata (1ms) ✓ 503 - without metadata (10ms) ✓ 504 - without metadata (2ms) ✓ 505 - without metadata (1ms) ✓ 506 - without metadata (2ms) ✓ 510 - without metadata (1ms) ``` </details> ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios ### Manual testing <details><summary><strong>Manual testing</strong></summary> #### Checkout the branch from elastic/package-storage#370 ``` git clone https://github.com/elastic/package-storage.git cd package-storage/ git switch -C update-aws-0.2.7-1598281986 origin/update-aws-0.2.7-1598281986 ``` #### start the stack using the registry from elastic/package-storage#370 ``` cd testing/environments/ docker-compose -f snapshot.yml pull docker-compose -f snapshot.yml -f local.yml up --force-recreate ``` #### Try to install the broken package ``` > curl --user elastic:changeme -X POST http://localhost:5601/api/ingest_manager/epm/packages/aws-0.2.7 -H 'kbn-xsrf: xyz' {"statusCode":500,"error":"Internal Server Error","message":"Bad Request"} ``` _observe the same error as #75862_ #### start only local registry with the broken package ``` # CTRL-C the stack (shell where you ran `docker-compose`) cd ../.. # back to package-storage root docker build . docker run -p 8080:8080 id_from_prior_step ``` #### start the stack from this PR, pointing at the local registry from prior step ``` yarn start --no-base-path --xpack.ingestManager.registryUrl=http://localhost:8080 yarn es snapshot --license=trial -E xpack.security.authc.api_key.enabled=true ``` #### Try to install the broken package ``` curl --user elastic:changeme -X POST http://localhost:5601/api/ingest_manager/epm/packages/aws-0.2.7 -H 'kbn-xsrf: xyz' { "statusCode": 400, "error": "Bad Request", "message": "[parse_exception] Failed to parse content to map response from /_ingest/pipeline/logs-aws.cloudtrail-0.2.7: {\"error\":{\"root_cause\":[{\"type\":\"parse_exception\",\"reason\":\"Failed to parse content to map\"}],\"type\":\"parse_exception\",\"reason\":\"Failed to parse content to map\",\"caused_by\":{\"type\":\"json_parse_exception\",\"reason\":\"Duplicate field 'ListGroupsForUser'\\n at [Source: (byte[])\\\"---\\ndescription: Pipeline for AWS CloudTrail Logs\\nprocessors:\\n - set:\\n field: event.ingested\\n value: '{{_ingest.timestamp}}'\\n - rename:\\n field: \\\"message\\\"\\n target_field: \\\"event.original\\\"\\n - json:\\n field: \\\"event.original\\\"\\n target_field: \\\"json\\\"\\n - date:\\n field: \\\"json.eventTime\\\"\\n target_field: \\\"@timestamp\\\"\\n ignore_failure: true\\n formats:\\n - ISO8601\\n - rename:\\n field: \\\"json.eventVersion\\\"\\n target_field: \\\"aws.cloudtrail.event_versi\\\"[truncated 16425 bytes]; line: 489, column: 26]\"}},\"status\":400}" } ``` _observe new error format_ </details>

@timestamp

* [Ingest Manager] Handle Legacy ES client errors (#76865) ## Summary closes #75862 1. Use an HTTP status code if ES client error provides one. extended #75862 (comment) to all 4xx-5xx errors 1. Format Error message as described in #75862 (comment) & agreed to #75862 (comment) ### example Request/Response: ``` > curl --user elastic:changeme -X POST http://localhost:5601/api/ingest_manager/epm/packages/aws-0.2.7 -H 'kbn-xsrf: xyz' { "statusCode":400, "error":"Bad Request", "message":"[parse_exception] Failed to parse content to map from ES at /_ingest/pipeline/logs-aws.cloudtrail-0.2.7: {\"error\":{\"root_cause\":[{\"type\":\"parse_exception\",\"reason\":\"Failed to parse content to map\"}],\"type\":\"parse_exception\",\"reason\":\"Failed to parse content to map\",\"caused_by\":{\"type\":\"json_parse_exception\",\"reason\":\"Duplicate field 'ListGroupsForUser'\\n at [Source: (byte[])\\\"---\\ndescription: Pipeline for AWS CloudTrail Logs\\nprocessors:\\n - set:\\n field: event.ingested\\n value: '{{_ingest.timestamp}}'\\n - rename:\\n field: \\\"message\\\"\\n target_field: \\\"event.original\\\"\\n - json:\\n field: \\\"event.original\\\"\\n target_field: \\\"json\\\"\\n - date:\\n field: \\\"json.eventTime\\\"\\n target_field: \\\"@timestamp\\\"\\n ignore_failure: true\\n formats:\\n - ISO8601\\n - rename:\\n field: \\\"json.eventVersion\\\"\\n target_field: \\\"aws.cloudtrail.event_versi\\\"[truncated 16425 bytes]; line: 489, column: 26]\"}},\"status\":400}" } ``` ### example Kibana Logs: `[parse_exception] Failed to parse content to map` <details><summary>Used `test.each` to generate tests for each 4xx - 5xx error. Call each error 3 different ways.</summary> ``` defaultIngestErrorHandler use the HTTP error status code provided by LegacyESErrors ✓ 400 - with path & response (12ms) ✓ 401 - with path & response (5ms) ✓ 402 - with path & response (5ms) ✓ 403 - with path & response (6ms) ✓ 404 - with path & response (5ms) ✓ 405 - with path & response (17ms) ✓ 406 - with path & response (2ms) ✓ 407 - with path & response (3ms) ✓ 408 - with path & response (6ms) ✓ 409 - with path & response (5ms) ✓ 410 - with path & response (1ms) ✓ 411 - with path & response (1ms) ✓ 412 - with path & response (1ms) ✓ 413 - with path & response (1ms) ✓ 414 - with path & response (1ms) ✓ 415 - with path & response (1ms) ✓ 416 - with path & response (1ms) ✓ 417 - with path & response (9ms) ✓ 418 - with path & response (1ms) ✓ 421 - with path & response (1ms) ✓ 426 - with path & response (1ms) ✓ 429 - with path & response (1ms) ✓ 450 - with path & response (1ms) ✓ 494 - with path & response (1ms) ✓ 497 - with path & response (1ms) ✓ 499 - with path & response (3ms) ✓ 500 - with path & response (2ms) ✓ 501 - with path & response (1ms) ✓ 502 - with path & response (2ms) ✓ 503 - with path & response (1ms) ✓ 504 - with path & response (1ms) ✓ 505 - with path & response (8ms) ✓ 506 - with path & response (2ms) ✓ 510 - with path & response (1ms) ✓ 400 - with other metadata (1ms) ✓ 401 - with other metadata (1ms) ✓ 402 - with other metadata (1ms) ✓ 403 - with other metadata (1ms) ✓ 404 - with other metadata (2ms) ✓ 405 - with other metadata (1ms) ✓ 406 - with other metadata (2ms) ✓ 407 - with other metadata (1ms) ✓ 408 - with other metadata (1ms) ✓ 409 - with other metadata (1ms) ✓ 410 - with other metadata (10ms) ✓ 411 - with other metadata (2ms) ✓ 412 - with other metadata (1ms) ✓ 413 - with other metadata (1ms) ✓ 414 - with other metadata (1ms) ✓ 415 - with other metadata (1ms) ✓ 416 - with other metadata (7ms) ✓ 417 - with other metadata (2ms) ✓ 418 - with other metadata (1ms) ✓ 421 - with other metadata (1ms) ✓ 426 - with other metadata (1ms) ✓ 429 - with other metadata (1ms) ✓ 450 - with other metadata (1ms) ✓ 494 - with other metadata (11ms) ✓ 497 - with other metadata (1ms) ✓ 499 - with other metadata (1ms) ✓ 500 - with other metadata (1ms) ✓ 501 - with other metadata (1ms) ✓ 502 - with other metadata (1ms) ✓ 503 - with other metadata (1ms) ✓ 504 - with other metadata (2ms) ✓ 505 - with other metadata (2ms) ✓ 506 - with other metadata (1ms) ✓ 510 - with other metadata (1ms) ✓ 400 - without metadata (1ms) ✓ 401 - without metadata (1ms) ✓ 402 - without metadata (10ms) ✓ 403 - without metadata (1ms) ✓ 404 - without metadata (1ms) ✓ 405 - without metadata (1ms) ✓ 406 - without metadata (1ms) ✓ 407 - without metadata (1ms) ✓ 408 - without metadata (1ms) ✓ 409 - without metadata (1ms) ✓ 410 - without metadata (1ms) ✓ 411 - without metadata (1ms) ✓ 412 - without metadata (1ms) ✓ 413 - without metadata (1ms) ✓ 414 - without metadata (1ms) ✓ 415 - without metadata (12ms) ✓ 416 - without metadata (1ms) ✓ 417 - without metadata (2ms) ✓ 418 - without metadata (1ms) ✓ 421 - without metadata (1ms) ✓ 426 - without metadata (2ms) ✓ 429 - without metadata (2ms) ✓ 450 - without metadata (3ms) ✓ 494 - without metadata (2ms) ✓ 497 - without metadata (2ms) ✓ 499 - without metadata (1ms) ✓ 500 - without metadata (1ms) ✓ 501 - without metadata (2ms) ✓ 502 - without metadata (1ms) ✓ 503 - without metadata (10ms) ✓ 504 - without metadata (2ms) ✓ 505 - without metadata (1ms) ✓ 506 - without metadata (2ms) ✓ 510 - without metadata (1ms) ``` </details> ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios ### Manual testing <details><summary><strong>Manual testing</strong></summary> #### Checkout the branch from elastic/package-storage#370 ``` git clone https://github.com/elastic/package-storage.git cd package-storage/ git switch -C update-aws-0.2.7-1598281986 origin/update-aws-0.2.7-1598281986 ``` #### start the stack using the registry from elastic/package-storage#370 ``` cd testing/environments/ docker-compose -f snapshot.yml pull docker-compose -f snapshot.yml -f local.yml up --force-recreate ``` #### Try to install the broken package ``` > curl --user elastic:changeme -X POST http://localhost:5601/api/ingest_manager/epm/packages/aws-0.2.7 -H 'kbn-xsrf: xyz' {"statusCode":500,"error":"Internal Server Error","message":"Bad Request"} ``` _observe the same error as #75862_ #### start only local registry with the broken package ``` # CTRL-C the stack (shell where you ran `docker-compose`) cd ../.. # back to package-storage root docker build . docker run -p 8080:8080 id_from_prior_step ``` #### start the stack from this PR, pointing at the local registry from prior step ``` yarn start --no-base-path --xpack.ingestManager.registryUrl=http://localhost:8080 yarn es snapshot --license=trial -E xpack.security.authc.api_key.enabled=true ``` #### Try to install the broken package ``` curl --user elastic:changeme -X POST http://localhost:5601/api/ingest_manager/epm/packages/aws-0.2.7 -H 'kbn-xsrf: xyz' { "statusCode": 400, "error": "Bad Request", "message": "[parse_exception] Failed to parse content to map response from /_ingest/pipeline/logs-aws.cloudtrail-0.2.7: {\"error\":{\"root_cause\":[{\"type\":\"parse_exception\",\"reason\":\"Failed to parse content to map\"}],\"type\":\"parse_exception\",\"reason\":\"Failed to parse content to map\",\"caused_by\":{\"type\":\"json_parse_exception\",\"reason\":\"Duplicate field 'ListGroupsForUser'\\n at [Source: (byte[])\\\"---\\ndescription: Pipeline for AWS CloudTrail Logs\\nprocessors:\\n - set:\\n field: event.ingested\\n value: '{{_ingest.timestamp}}'\\n - rename:\\n field: \\\"message\\\"\\n target_field: \\\"event.original\\\"\\n - json:\\n field: \\\"event.original\\\"\\n target_field: \\\"json\\\"\\n - date:\\n field: \\\"json.eventTime\\\"\\n target_field: \\\"@timestamp\\\"\\n ignore_failure: true\\n formats:\\n - ISO8601\\n - rename:\\n field: \\\"json.eventVersion\\\"\\n target_field: \\\"aws.cloudtrail.event_versi\\\"[truncated 16425 bytes]; line: 489, column: 26]\"}},\"status\":400}" } ``` _observe new error format_ </details> * Remove TS 4 feature b/c 7.9 is on TS 3.9.5 Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

ghost · 2020-09-22T10:55:39Z

Hi @skh

We have gone through the ticket and observed it requires DEV_Validation to test the ticket .

Please let us know if anything is missing from our end.

EricDavisX · 2020-09-22T20:29:12Z

Thanks @rahulgupta-qasource I believe you are commenting because the issue is found / cited as part of the 7.9.2 test query. we are ok with the testing done here by Engineering team, thank you for the review.

John Schulz added 2 commits September 5, 2020 16:44

Use code from legacy repo for type guard

8e26047

jfsiii self-assigned this Sep 7, 2020

jfsiii added the Team:Fleet Team label for Observability Data Collection Fleet team label Sep 7, 2020

jfsiii mentioned this pull request Sep 7, 2020

Non-deterministic behavior while installing a package #75862

Closed

jfsiii added the release_note:skip Skip the PR/issue when compiling release notes label Sep 7, 2020

jfsiii commented Sep 7, 2020

View reviewed changes

x-pack/plugins/ingest_manager/server/errors/handlers.ts Outdated Show resolved Hide resolved

jfsiii marked this pull request as ready for review September 8, 2020 19:34

jfsiii requested a review from a team September 8, 2020 19:34

jfsiii added v7.10.0 v7.9.2 v8.0.0 labels Sep 8, 2020

elasticmachine and others added 4 commits September 8, 2020 15:24

Merge branch 'master' into 75862-surface-legacy-es-errors

1bba375

Add types to test function

9a06d7b

Merge branch '75862-surface-legacy-es-errors' of github.com:jfsiii/ki…

e05362c

…bana into 75862-surface-legacy-es-errors

skh reviewed Sep 9, 2020

View reviewed changes

jfsiii requested a review from skh September 9, 2020 15:26

Use same message for logs & HTTP response

fdaf588

skh approved these changes Sep 9, 2020

View reviewed changes

Merge branch 'master' into 75862-surface-legacy-es-errors

00a87dc

jfsiii merged commit 524f30a into elastic:master Sep 10, 2020

This was referenced Sep 10, 2020

[7.x] [Ingest Manager] Handle Legacy ES client errors (#76865) #77213

Merged

[7.9] [Ingest Manager] Handle Legacy ES client errors (#76865) #77215

Merged

jfsiii mentioned this pull request Oct 19, 2020

[Fleet] Handle ES failures on template installation #70586

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Ingest Manager] Handle Legacy ES client errors #76865

[Ingest Manager] Handle Legacy ES client errors #76865

jfsiii commented Sep 7, 2020 •

edited

Loading

elasticmachine commented Sep 8, 2020

jfsiii commented Sep 8, 2020

jfsiii commented Sep 8, 2020

skh Sep 9, 2020

jfsiii Sep 9, 2020

skh Sep 9, 2020

jfsiii Sep 9, 2020

skh Sep 9, 2020

skh commented Sep 9, 2020

jfsiii commented Sep 9, 2020

skh left a comment

jfsiii commented Sep 9, 2020

jfsiii commented Sep 9, 2020

kibanamachine commented Sep 9, 2020

skh commented Sep 10, 2020

jfsiii commented Sep 10, 2020

ghost commented Sep 22, 2020

EricDavisX commented Sep 22, 2020

[Ingest Manager] Handle Legacy ES client errors #76865

[Ingest Manager] Handle Legacy ES client errors #76865

Conversation

jfsiii commented Sep 7, 2020 • edited Loading

Summary

example Request/Response:

example Kibana Logs:

Checklist

Manual testing

Checkout the branch from elastic/package-storage#370

start the stack using the registry from elastic/package-storage#370

Try to install the broken package

start only local registry with the broken package

start the stack from this PR, pointing at the local registry from prior step

Try to install the broken package

elasticmachine commented Sep 8, 2020

jfsiii commented Sep 8, 2020

jfsiii commented Sep 8, 2020

skh Sep 9, 2020

Choose a reason for hiding this comment

jfsiii Sep 9, 2020

Choose a reason for hiding this comment

skh Sep 9, 2020

Choose a reason for hiding this comment

jfsiii Sep 9, 2020

Choose a reason for hiding this comment

skh Sep 9, 2020

Choose a reason for hiding this comment

skh commented Sep 9, 2020

jfsiii commented Sep 9, 2020

skh left a comment

Choose a reason for hiding this comment

jfsiii commented Sep 9, 2020

jfsiii commented Sep 9, 2020

kibanamachine commented Sep 9, 2020

💚 Build Succeeded

Build metrics

distributable file count

History

skh commented Sep 10, 2020

jfsiii commented Sep 10, 2020

ghost commented Sep 22, 2020

EricDavisX commented Sep 22, 2020

jfsiii commented Sep 7, 2020 •

edited

Loading