Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

enhancement(external docs): Add Cue docs for component-specific metrics #4820

Merged
merged 20 commits into from
Nov 19, 2020

Conversation

lucperkins
Copy link
Contributor

@lucperkins lucperkins commented Oct 31, 2020

Addresses #4810 by adding Cue configs for the internal_metrics source. This PR also restructures the metrics into a single metrics.cue file and adds a telemetry.metrics block to all components to which that applies.

@lucperkins lucperkins force-pushed the internal-metrics-docs branch 2 times, most recently from 2e6a49d to b5430b1 Compare October 31, 2020 01:32
@lucperkins lucperkins marked this pull request as draft November 1, 2020 03:44
@binarylogic
Copy link
Contributor

Seems like a good start, but we're missing quite a few. Ex: memory_used. I was thinking about a script that would pattern match and find metrics in the internal_events folder. This way we could run it and compare it against our cue data.

Finally, I think we'll want to note which metrics are available based on which components. So, for example, the lua transform should document the telemetry it exposes. This makes me think that we might be better off documeting all of this in each component .cue file and then referencing that data in the internal_metrics.cue file. Does that make sense?

@lucperkins
Copy link
Contributor Author

@binarylogic Sorry, thought I had marked this PR as a draft. It's indeed not there yet.

@lucperkins lucperkins force-pushed the internal-metrics-docs branch 3 times, most recently from 16fd43e to f8c47a2 Compare November 1, 2020 23:02
@lucperkins
Copy link
Contributor Author

@binarylogic I thought about it, and I think the way to go here is to define all of the potential metrics "lower" in the config tree as a kind of grab bag, potentially split into groups for convenience, that each component can draw from. That will enable us to have per-component granularity while also using the good parts of Cue to full effect.

@lucperkins lucperkins force-pushed the internal-metrics-docs branch 8 times, most recently from 41acffc to 42c2f08 Compare November 8, 2020 00:35
@lucperkins lucperkins marked this pull request as ready for review November 8, 2020 00:49
@lucperkins lucperkins force-pushed the internal-metrics-docs branch 4 times, most recently from 599c325 to 462f6ea Compare November 18, 2020 20:32
Luc Perkins added 19 commits November 19, 2020 11:12
Signed-off-by: Luc Perkins <luc@timber.io>
Signed-off-by: Luc Perkins <luc@timber.io>
Signed-off-by: Luc Perkins <luc@timber.io>
Signed-off-by: Luc Perkins <luc@timber.io>
Signed-off-by: Luc Perkins <luc@timber.io>
Signed-off-by: Luc Perkins <luc@timber.io>
Signed-off-by: Luc Perkins <luc@timber.io>
Signed-off-by: Luc Perkins <luc@timber.io>
Signed-off-by: Luc Perkins <luc@timber.io>
Signed-off-by: Luc Perkins <luc@timber.io>
Signed-off-by: Luc Perkins <luc@timber.io>
Signed-off-by: Luc Perkins <luc@timber.io>
Signed-off-by: Luc Perkins <luc@timber.io>
Signed-off-by: Luc Perkins <luc@timber.io>
Signed-off-by: Luc Perkins <luc@timber.io>
Signed-off-by: Luc Perkins <luc@timber.io>
Signed-off-by: Luc Perkins <luc@timber.io>
Signed-off-by: Luc Perkins <luc@timber.io>
Signed-off-by: Luc Perkins <luc@timber.io>
Signed-off-by: Luc Perkins <luc@timber.io>
Copy link
Contributor

@binarylogic binarylogic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work!

@lucperkins lucperkins merged commit a076b6e into master Nov 19, 2020
@lucperkins lucperkins deleted the internal-metrics-docs branch November 19, 2020 23:15
casserni pushed a commit that referenced this pull request Nov 20, 2020
…cs (#4820)

* Begin transferring metrics

Signed-off-by: Luc Perkins <luc@timber.io>

* Metrics docs for sinks

Signed-off-by: Luc Perkins <luc@timber.io>

* Add source metrics and refactor metrics.cue

Signed-off-by: Luc Perkins <luc@timber.io>

* Add metrics for transforms

Signed-off-by: Luc Perkins <luc@timber.io>

* Fill in remaining TODO descriptions

Signed-off-by: Luc Perkins <luc@timber.io>

* Add internal_metrics source

Signed-off-by: Luc Perkins <luc@timber.io>

* Fix service/aggregator discrepancy

Signed-off-by: Luc Perkins <luc@timber.io>

* Move Apache, MongoDB, and host metrics out of metrics.cue

Signed-off-by: Luc Perkins <luc@timber.io>

* Move Docker metrics out of metrics.cue

Signed-off-by: Luc Perkins <luc@timber.io>

* Move k8s metrics out of metrics.cue

Signed-off-by: Luc Perkins <luc@timber.io>

* Migrate internal metrics out of metrics.cue

Signed-off-by: Luc Perkins <luc@timber.io>

* Remove metrics.cue file

Signed-off-by: Luc Perkins <luc@timber.io>

* Apply explicit default namespace to all metrics

Signed-off-by: Luc Perkins <luc@timber.io>

* Refactor Splunk common metrics into separate file

Signed-off-by: Luc Perkins <luc@timber.io>

* Address feedback on metric descriptions

Signed-off-by: Luc Perkins <luc@timber.io>

* Add default namespace to AWS ECS metrics

Signed-off-by: Luc Perkins <luc@timber.io>

* Add _total suffix to Windows metrics

Signed-off-by: Luc Perkins <luc@timber.io>

* Add new options for the error_type tag

Signed-off-by: Luc Perkins <luc@timber.io>

* Add AWS S3+SQS metrics to docs

Signed-off-by: Luc Perkins <luc@timber.io>

* Refactor all vector namespace metrics into internal_metrics.cue

Signed-off-by: Luc Perkins <luc@timber.io>
Signed-off-by: casserni <nicholascassera@gmail.com>
mengesb pushed a commit to jacobbraaten/vector that referenced this pull request Dec 9, 2020
…cs (vectordotdev#4820)

* Begin transferring metrics

Signed-off-by: Luc Perkins <luc@timber.io>

* Metrics docs for sinks

Signed-off-by: Luc Perkins <luc@timber.io>

* Add source metrics and refactor metrics.cue

Signed-off-by: Luc Perkins <luc@timber.io>

* Add metrics for transforms

Signed-off-by: Luc Perkins <luc@timber.io>

* Fill in remaining TODO descriptions

Signed-off-by: Luc Perkins <luc@timber.io>

* Add internal_metrics source

Signed-off-by: Luc Perkins <luc@timber.io>

* Fix service/aggregator discrepancy

Signed-off-by: Luc Perkins <luc@timber.io>

* Move Apache, MongoDB, and host metrics out of metrics.cue

Signed-off-by: Luc Perkins <luc@timber.io>

* Move Docker metrics out of metrics.cue

Signed-off-by: Luc Perkins <luc@timber.io>

* Move k8s metrics out of metrics.cue

Signed-off-by: Luc Perkins <luc@timber.io>

* Migrate internal metrics out of metrics.cue

Signed-off-by: Luc Perkins <luc@timber.io>

* Remove metrics.cue file

Signed-off-by: Luc Perkins <luc@timber.io>

* Apply explicit default namespace to all metrics

Signed-off-by: Luc Perkins <luc@timber.io>

* Refactor Splunk common metrics into separate file

Signed-off-by: Luc Perkins <luc@timber.io>

* Address feedback on metric descriptions

Signed-off-by: Luc Perkins <luc@timber.io>

* Add default namespace to AWS ECS metrics

Signed-off-by: Luc Perkins <luc@timber.io>

* Add _total suffix to Windows metrics

Signed-off-by: Luc Perkins <luc@timber.io>

* Add new options for the error_type tag

Signed-off-by: Luc Perkins <luc@timber.io>

* Add AWS S3+SQS metrics to docs

Signed-off-by: Luc Perkins <luc@timber.io>

* Refactor all vector namespace metrics into internal_metrics.cue

Signed-off-by: Luc Perkins <luc@timber.io>
Signed-off-by: Brian Menges <brian.menges@anaplan.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants