[EPM] Add mapping field types to index template generation v2 #60266
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
neptunian
reviewed
Mar 17, 2020
x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/template/template.ts
Outdated
Show resolved
Hide resolved
* move expand stage to expandFields * fix expandFields * add deduplication stage dedupFields
skh
force-pushed
the
55865-index-template-types-2
branch
from
2e1d7cb
to
36fdca0
Compare
This was referenced Mar 18, 2020
neptunian
added
v7.7.0
v8.0.0
release_note:skip
|
Pinging @elastic/ingest-management (Feature:EPM) |
|
@neptunian thank you for your comments on this PR and #59894 ! I've created a bunch of new issues for most of it, see the initial description of this PR. Changes to this PR:
This is ready for another look now. |
neptunian
approved these changes
Mar 18, 2020
💚 Build SucceededHistory
To update your PR or re-run it, just comment with: |
jloleysens
added a commit
that referenced
this pull request
Mar 18, 2020
…nless * 'app/painless' of github.com:elastic/kibana: (64 commits) Fix filter scope in bool query (#60488) change index pattern id to be the same as index pattern title (#60436) [Endpoint] resolver v1 events (#59233) Branding fixes for dashboard, loader and space selector (#60073) skip flaky suite (#60535) [SIEM][Detection Engine] Fixes bug with timeline templates not working Fixed errors which are happening if switch between alert types (#60453) [EPM] Add mapping field types to index template generation v2 (#60266) [NP] Cutover ensureDefaultIndexPattern to kibana_utils (#59895) Closes #60265. Adds Beta badge to service map (#60482) [Visualize] Duplicated query filters in es request (#60106) [ML] Disable functional transform tests Fixes to service map single node banner (#60072) [Uptime] replace fetch with kibana http (#59881) Upgrade @types/node to match Node.js runtime (#60368) [License Management] NP migration (#60250) Fix create alert button from not showing in alerts list (#60444) [SIEM][Case] Update connector through flyout (#60307) add data-test-subj where possible on SO management table (#60226) Enforce `required` presence for value/key validation of `recordOf` and `mapOf`. (#60406) ...
skh
added a commit
that referenced
this pull request
Mar 19, 2020
#60533) * Add properties needed for index templates to Field * Add data type handling to template generation * Adjust tests * Update fields test snapshots * Remove duplicate fields from test file * Add test cases * Enhance processFields * move expand stage to expandFields * fix expandFields * add deduplication stage dedupFields * Use processField() to preprocess fields * Remove alias fields with invalid path * Remove obsolete code. * Fix documentation. * Add unit tests for getField() * Don't fail on invalid input for now. * Validate array fields. * Guard against invalid input.
jloleysens
added a commit
to jloleysens/kibana
that referenced
this pull request
Mar 19, 2020
* upstream/app/painless: (66 commits) Another i18n issue Fix i18n Fix filter scope in bool query (elastic#60488) change index pattern id to be the same as index pattern title (elastic#60436) [Endpoint] resolver v1 events (elastic#59233) Branding fixes for dashboard, loader and space selector (elastic#60073) skip flaky suite (elastic#60535) [SIEM][Detection Engine] Fixes bug with timeline templates not working Fixed errors which are happening if switch between alert types (elastic#60453) [EPM] Add mapping field types to index template generation v2 (elastic#60266) [NP] Cutover ensureDefaultIndexPattern to kibana_utils (elastic#59895) Closes elastic#60265. Adds Beta badge to service map (elastic#60482) [Visualize] Duplicated query filters in es request (elastic#60106) [ML] Disable functional transform tests Fixes to service map single node banner (elastic#60072) [Uptime] replace fetch with kibana http (elastic#59881) Upgrade @types/node to match Node.js runtime (elastic#60368) [License Management] NP migration (elastic#60250) Fix create alert button from not showing in alerts list (elastic#60444) [SIEM][Case] Update connector through flyout (elastic#60307) ...
jen-huang
added
the
Team:Fleet
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Mostly implements #55865
flattens all fields (instead of expanding all fields, as beats does) after discussion with @ruflinpathpointing to a non-existing fieldobjectdata typefields/field.tsandkibana/index_pattern/install.tswhich I kept because it still might divergeHow to test this?
GET _cat/templates, thenGET /_template/$TEMPLATE_NAME). The relevant templates arelogs-$PACKAGE.$DATASETandmetrics-$PACKAGE.$DATASET, so e.g.metrics-system.processserver/services/epm/elasticsearch/template/__snapshots__/template.test.ts.snap, especially the snapshot for thesystem.ymltestserver/services/epm/fields/__snapshots__/field.test.ts.snapmight also be interesting, as it shows the intermediate data structure afterprocessFields(), but before the transformation into themappingsstructure.Follow-up issues:
[EPM] Implement correct handling of array field type in mappings [EPM] Implement correct handling of array field type in mappings #60498fields.ymlprocessing for index template mapping generation [EPM] Unit-test fields.yml processing for index template mapping generation #60503