Iterate package

[jalvz]

Motivation/summary

This does a few things:

• Splits metrics data stream into app metrics, internal and profiles. This is following the indexing strategy proposal.
• Adds the dataset property to the manifest files so the right templates are created.
• Sort and dedup fields (within each yml file). Fields are now flattened.
• Generates data stream pipelines out of our definitions.json file

Missing from this PR / WIP: Pipelines are tied to data streams, we need to copy the ones that are used for other event types to the right data stream folder. Right now all are under traces. This shouldn't block the review.

Checklist

• I have signed the Contributor License Agreement.
• I have updated CHANGELOG.asciidoc

I have considered changes for:

• documentation
• logging (add log lines, choose appropriate log selector, etc.)
• metrics and monitoring (create issue for Kibana team to add metrics to visualizations, e.g. Kibana#44001)
• automated tests (add tests for the code changes, all unit tests pass locally)
• telemetry
• Elasticsearch Service (https://cloud.elastic.co)
• Elastic Cloud Enterprise (https://www.elastic.co/products/ece)
• Elastic Cloud on Kubernetes (https://www.elastic.co/elastic-cloud-kubernetes)

How to test these changes

In short: run the package, install it, and check that all templates and pipelines installed are correct.

Related issues

Part of #4004

[apmmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Build Cause: Pull request #4473 updated

• Start Time: 2020-12-02T13:35:15.167+0000

• Duration: 50 min 16 sec

Test stats 🧪

Test	Results
Failed	0
Passed	4623
Skipped	141
Total	4764

Steps errors

Expand to view the steps failures

Compress

• Took 0 min 0 sec . View more details on here
• Description: tar --exclude=coverage-files.tgz -czf coverage-files.tgz coverage

Compress

• Took 0 min 0 sec . View more details on here
• Description: tar --exclude=system-tests-linux-files.tgz -czf system-tests-linux-files.tgz system-tests

Test Sync

• Took 3 min 15 sec . View more details on here
• Description: ./.ci/scripts/sync.sh

[codecov-io]

Codecov Report

Merging #4473 (7bed6bd) into master (49d55cc) will decrease coverage by 0.02%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##           master    #4473      +/-   ##
==========================================
- Coverage   76.07%   76.04%   -0.03%     
==========================================
  Files         160      160              
  Lines        9775     9781       +6     
==========================================
+ Hits         7436     7438       +2     
- Misses       2339     2343       +4     

Impacted Files	Coverage Δ
beater/beater.go	56.52% <0.00%> (-1.69%)	⬇️
processor/otel/consumer.go	93.95% <0.00%> (+0.44%)	⬆️

[axw]

Due to issues getting Elastic Agent + APM Server to run with 7.11.0 BC, and given that this isn't user facing yet, I'm testing with 8.0.0-SNAPSHOT.

• Installed package apm 0.1.0-dev.3
• Ran a program that sent a transaction, span, error, Go runtime metrics, custom metric
• Checked data streams:

• traces-apm-default contains the transaction and span
• logs-apm.error-default contains the error
• metrics-apm-default contains contains the Go runtime metrics and custom metric
• metrics-apm.internal-default contains breakdown metrics

That all seems good. I found a couple of issues.

The Data Streams list in the Fleet app does not list the "traces" data stream:

The "labels.foo" dynamic templates are missing from mappings:

Looking more closely at the package, I see that the fields.yml definitions for labels doesn't have all the bits from our overrides:

apm-server/apmpackage/apm/0.1.0/data_stream/traces/fields/ecs.yml

Lines 130 to 133 in 6d06e68

	- name: labels
	type: object
	description: |
	A flat mapping of user-defined labels with string, boolean or number values.

apm-server/model/transaction/_meta/fields.yml

Lines 131 to 141 in 6d06e68

	- name: labels
	type: object
	object_type_params:
	- object_type: keyword
	- object_type: boolean
	- object_type: scaled_float
	scaling_factor: 1000000
	dynamic: true
	overwrite: true
	description: >
	A flat mapping of user-defined labels with string, boolean or number values.

[axw]

Opened #4610 for the dynamic templates issue.

[axw]

The data stream list issue is due to this:

https://github.com/elastic/kibana/blob/94b02d97ed12c7d93081f88768def083b439473a/x-pack/plugins/fleet/server/routes/data_streams/handlers.ts#L12

I'll create a PR to fix that.

[axw]

For want of a better label, marking this "regression" given the labels issue.