Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix(file source, kubernetes_logs source, file sink): make file internal metric tag opt-in #19145

Merged
merged 3 commits into from
Nov 14, 2023
Merged

fix(file source, kubernetes_logs source, file sink): make file internal metric tag opt-in #19145

merged 3 commits into from
Nov 14, 2023

Conversation

dsmith3197
Copy link
Contributor

Similar to #19083, we have decided to make this opt-in immediately due to the impact it is having on users and downstream systems.

@dsmith3197 dsmith3197 requested review from a team as code owners November 14, 2023 13:20
@github-actions GitHub Actions
Copy link

Your preview site for the vector.dev will be ready in a few minutes, please allow time for it to build.

Heres your preview link:
vector.dev preview

@github-actions GitHub Actions
Copy link

Your preview site for the Rust Doc will be ready in a few minutes, please allow time for it to build.

Heres your preview link:
Rust Doc preview

@github-actions GitHub Actions
Copy link

Your preview site for the VRL Playground will be ready in a few minutes, please allow time for it to build.

Heres your preview link:
VRL Playground preview

@github-actions github-actions bot added the domain: external docs Anything related to Vector's external, public documentation label Nov 14, 2023
@datadog-vectordotdev
Copy link

datadog-vectordotdev bot commented Nov 14, 2023 •
edited
Loading

Datadog Report

Branch report: dougsmith/file-tag-opt-in
Commit report: b80ecca

✅ vector: 0 Failed, 0 New Flaky, 2004 Passed, 0 Skipped, 1m 21.4s Wall Time

@dsmith3197 dsmith3197 requested a review from a team November 14, 2023 14:13
@github-actions github-actions bot added domain: sources Anything related to the Vector's sources domain: sinks Anything related to the Vector's sinks labels Nov 14, 2023
@dsmith3197 dsmith3197 added this pull request to the merge queue Nov 14, 2023
@github-actions GitHub Actions
Copy link

Regression Detector Results

Run ID: 9299e927-ae50-4a93-acf7-09a18631a5c5
Baseline: 0eb5396
Comparison: ec088ba
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine quickly if vector performance is changed and to what degree by a pull request.

Because a target's optimization goal performance in each experiment will vary somewhat each time it is run, we can only estimate mean differences in optimization goal relative to the baseline target. We express these differences as a percentage change relative to the baseline target, denoted "Δ mean %". These estimates are made to a precision that balances accuracy and cost control. We represent this precision as a 90.00% confidence interval denoted "Δ mean % CI": there is a 90.00% chance that the true value of "Δ mean %" is in that interval.

We decide whether a change in performance is a "regression" -- a change worth investigating further -- if both of the following two criteria are true:

  1. The estimated |Δ mean %| ≥ 5.00%. This criterion intends to answer the question "Does the estimated change in mean optimization goal performance have a meaningful impact on your customers?". We assume that when |Δ mean %| < 5.00%, the impact on your customers is not meaningful. We also assume that a performance change in optimization goal is worth investigating whether it is an increase or decrease, so long as the magnitude of the change is sufficiently large.

  2. Zero is not in the 90.00% confidence interval "Δ mean % CI" about "Δ mean %". This statement is equivalent to saying that there is at least a 90.00% chance that the mean difference in optimization goal is not zero. This criterion intends to answer the question, "Is there a statistically significant difference in mean optimization goal performance?". It also means there is no more than a 10.00% chance this criterion reports a statistically significant difference when the true difference in mean optimization goal is zero -- a "false positive". We assume you are willing to accept a 10.00% chance of inaccurately detecting a change in performance when no true difference exists.

The table below, if present, lists those experiments that have experienced a statistically significant change in mean optimization goal performance between baseline and comparison SHAs with 90.00% confidence OR have been detected as newly erratic. Negative values of "Δ mean %" mean that baseline is faster, whereas positive values of "Δ mean %" mean that comparison is faster. Results that do not exhibit more than a ±5.00% change in their mean optimization goal are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting change is observed.

No interesting changes in experiment optimization goals with confidence ≥ 90.00% and |Δ mean %| ≥ 5.00%.

Fine details of change detection per experiment.
experiment goal Δ mean % Δ mean % CI confidence
socket_to_socket_blackhole ingress throughput +0.94 [+0.89, +0.99] 100.00%
syslog_splunk_hec_logs ingress throughput +0.40 [+0.33, +0.46] 100.00%
syslog_loki ingress throughput +0.38 [+0.34, +0.42] 100.00%
datadog_agent_remap_datadog_logs_acks ingress throughput +0.28 [+0.21, +0.35] 100.00%
http_to_http_acks ingress throughput +0.26 [-1.05, +1.56] 25.37%
datadog_agent_remap_datadog_logs ingress throughput +0.25 [+0.15, +0.34] 100.00%
http_text_to_http_json ingress throughput +0.19 [+0.08, +0.31] 99.39%
splunk_hec_route_s3 ingress throughput +0.13 [-0.38, +0.64] 32.25%
http_to_http_noack ingress throughput +0.12 [+0.02, +0.23] 94.87%
http_to_http_json ingress throughput +0.05 [-0.03, +0.12] 68.02%
splunk_hec_to_splunk_hec_logs_acks ingress throughput +0.00 [-0.14, +0.14] 0.92%
http_elasticsearch ingress throughput -0.00 [-0.06, +0.06] 1.22%
splunk_hec_indexer_ack_blackhole ingress throughput -0.00 [-0.14, +0.14] 2.02%
splunk_hec_to_splunk_hec_logs_noack ingress throughput -0.05 [-0.16, +0.07] 48.65%
otlp_grpc_to_blackhole ingress throughput -0.06 [-0.16, +0.05] 61.12%
enterprise_http_to_http ingress throughput -0.08 [-0.16, -0.01] 92.33%
fluent_elasticsearch ingress throughput -0.09 [-0.54, +0.36] 25.33%
syslog_log2metric_splunk_hec_metrics ingress throughput -0.12 [-0.25, +0.02] 83.54%
syslog_regex_logs2metric_ddmetrics ingress throughput -0.25 [-0.35, -0.16] 100.00%
datadog_agent_remap_blackhole_acks ingress throughput -0.30 [-0.39, -0.22] 100.00%
syslog_log2metric_humio_metrics ingress throughput -0.34 [-0.43, -0.24] 100.00%
otlp_http_to_blackhole ingress throughput -0.34 [-0.49, -0.19] 99.97%
http_to_s3 ingress throughput -0.35 [-0.63, -0.07] 96.06%
syslog_humio_logs ingress throughput -0.38 [-0.47, -0.29] 100.00%
datadog_agent_remap_blackhole ingress throughput -0.41 [-0.50, -0.33] 100.00%
file_to_blackhole egress throughput -1.69 [-4.11, +0.74] 74.69%

@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Nov 14, 2023
@dsmith3197 dsmith3197 added this pull request to the merge queue Nov 14, 2023
@github-actions GitHub Actions
Copy link

Regression Detector Results

Run ID: 5e7870df-91b4-4c8b-bb0a-c6aab42f7d0a
Baseline: 20de822
Comparison: 9155659
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine quickly if vector performance is changed and to what degree by a pull request.

Because a target's optimization goal performance in each experiment will vary somewhat each time it is run, we can only estimate mean differences in optimization goal relative to the baseline target. We express these differences as a percentage change relative to the baseline target, denoted "Δ mean %". These estimates are made to a precision that balances accuracy and cost control. We represent this precision as a 90.00% confidence interval denoted "Δ mean % CI": there is a 90.00% chance that the true value of "Δ mean %" is in that interval.

We decide whether a change in performance is a "regression" -- a change worth investigating further -- if both of the following two criteria are true:

  1. The estimated |Δ mean %| ≥ 5.00%. This criterion intends to answer the question "Does the estimated change in mean optimization goal performance have a meaningful impact on your customers?". We assume that when |Δ mean %| < 5.00%, the impact on your customers is not meaningful. We also assume that a performance change in optimization goal is worth investigating whether it is an increase or decrease, so long as the magnitude of the change is sufficiently large.

  2. Zero is not in the 90.00% confidence interval "Δ mean % CI" about "Δ mean %". This statement is equivalent to saying that there is at least a 90.00% chance that the mean difference in optimization goal is not zero. This criterion intends to answer the question, "Is there a statistically significant difference in mean optimization goal performance?". It also means there is no more than a 10.00% chance this criterion reports a statistically significant difference when the true difference in mean optimization goal is zero -- a "false positive". We assume you are willing to accept a 10.00% chance of inaccurately detecting a change in performance when no true difference exists.

The table below, if present, lists those experiments that have experienced a statistically significant change in mean optimization goal performance between baseline and comparison SHAs with 90.00% confidence OR have been detected as newly erratic. Negative values of "Δ mean %" mean that baseline is faster, whereas positive values of "Δ mean %" mean that comparison is faster. Results that do not exhibit more than a ±5.00% change in their mean optimization goal are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting change is observed.

No interesting changes in experiment optimization goals with confidence ≥ 90.00% and |Δ mean %| ≥ 5.00%.

Fine details of change detection per experiment.
experiment goal Δ mean % Δ mean % CI confidence
syslog_log2metric_humio_metrics ingress throughput +3.39 [+3.29, +3.50] 100.00%
syslog_humio_logs ingress throughput +1.31 [+1.22, +1.40] 100.00%
datadog_agent_remap_datadog_logs_acks ingress throughput +0.80 [+0.72, +0.87] 100.00%
syslog_loki ingress throughput +0.50 [+0.45, +0.55] 100.00%
syslog_regex_logs2metric_ddmetrics ingress throughput +0.46 [+0.37, +0.54] 100.00%
otlp_http_to_blackhole ingress throughput +0.41 [+0.25, +0.57] 100.00%
otlp_grpc_to_blackhole ingress throughput +0.40 [+0.30, +0.50] 100.00%
http_elasticsearch ingress throughput +0.31 [+0.25, +0.36] 100.00%
syslog_log2metric_splunk_hec_metrics ingress throughput +0.16 [+0.02, +0.29] 94.87%
http_to_http_noack ingress throughput +0.13 [+0.03, +0.22] 97.24%
datadog_agent_remap_blackhole_acks ingress throughput +0.07 [-0.02, +0.16] 78.79%
http_to_http_json ingress throughput +0.04 [-0.04, +0.11] 60.75%
http_to_s3 ingress throughput +0.01 [-0.26, +0.28] 4.82%
splunk_hec_to_splunk_hec_logs_acks ingress throughput +0.00 [-0.14, +0.15] 0.78%
splunk_hec_indexer_ack_blackhole ingress throughput -0.00 [-0.14, +0.14] 1.71%
splunk_hec_to_splunk_hec_logs_noack ingress throughput -0.06 [-0.18, +0.05] 63.92%
enterprise_http_to_http ingress throughput -0.09 [-0.17, -0.02] 96.02%
datadog_agent_remap_datadog_logs ingress throughput -0.30 [-0.39, -0.20] 100.00%
fluent_elasticsearch ingress throughput -0.30 [-0.75, +0.15] 72.86%
syslog_splunk_hec_logs ingress throughput -0.37 [-0.41, -0.32] 100.00%
socket_to_socket_blackhole ingress throughput -0.45 [-0.53, -0.37] 100.00%
http_text_to_http_json ingress throughput -0.69 [-0.80, -0.58] 100.00%
splunk_hec_route_s3 ingress throughput -0.70 [-1.21, -0.20] 97.90%
file_to_blackhole egress throughput -0.71 [-3.20, +1.78] 36.10%
http_to_http_acks ingress throughput -0.82 [-2.11, +0.47] 70.51%
datadog_agent_remap_blackhole ingress throughput -1.07 [-1.16, -0.98] 100.00%

@github-actions GitHub Actions
Copy link

Regression Detector Results

Run ID: d88d76a7-c237-44a7-8694-33a3833182bc
Baseline: c668def
Comparison: 981fb8c
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine quickly if vector performance is changed and to what degree by a pull request.

Because a target's optimization goal performance in each experiment will vary somewhat each time it is run, we can only estimate mean differences in optimization goal relative to the baseline target. We express these differences as a percentage change relative to the baseline target, denoted "Δ mean %". These estimates are made to a precision that balances accuracy and cost control. We represent this precision as a 90.00% confidence interval denoted "Δ mean % CI": there is a 90.00% chance that the true value of "Δ mean %" is in that interval.

We decide whether a change in performance is a "regression" -- a change worth investigating further -- if both of the following two criteria are true:

  1. The estimated |Δ mean %| ≥ 5.00%. This criterion intends to answer the question "Does the estimated change in mean optimization goal performance have a meaningful impact on your customers?". We assume that when |Δ mean %| < 5.00%, the impact on your customers is not meaningful. We also assume that a performance change in optimization goal is worth investigating whether it is an increase or decrease, so long as the magnitude of the change is sufficiently large.

  2. Zero is not in the 90.00% confidence interval "Δ mean % CI" about "Δ mean %". This statement is equivalent to saying that there is at least a 90.00% chance that the mean difference in optimization goal is not zero. This criterion intends to answer the question, "Is there a statistically significant difference in mean optimization goal performance?". It also means there is no more than a 10.00% chance this criterion reports a statistically significant difference when the true difference in mean optimization goal is zero -- a "false positive". We assume you are willing to accept a 10.00% chance of inaccurately detecting a change in performance when no true difference exists.

The table below, if present, lists those experiments that have experienced a statistically significant change in mean optimization goal performance between baseline and comparison SHAs with 90.00% confidence OR have been detected as newly erratic. Negative values of "Δ mean %" mean that baseline is faster, whereas positive values of "Δ mean %" mean that comparison is faster. Results that do not exhibit more than a ±5.00% change in their mean optimization goal are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting change is observed.

No interesting changes in experiment optimization goals with confidence ≥ 90.00% and |Δ mean %| ≥ 5.00%.

Fine details of change detection per experiment.
experiment goal Δ mean % Δ mean % CI confidence
otlp_http_to_blackhole ingress throughput +2.63 [+2.47, +2.79] 100.00%
socket_to_socket_blackhole ingress throughput +2.04 [+1.99, +2.09] 100.00%
http_text_to_http_json ingress throughput +1.71 [+1.58, +1.85] 100.00%
syslog_regex_logs2metric_ddmetrics ingress throughput +1.68 [+1.60, +1.77] 100.00%
http_elasticsearch ingress throughput +1.39 [+1.33, +1.45] 100.00%
syslog_loki ingress throughput +1.23 [+1.18, +1.27] 100.00%
syslog_humio_logs ingress throughput +0.93 [+0.86, +1.00] 100.00%
syslog_log2metric_splunk_hec_metrics ingress throughput +0.90 [+0.77, +1.03] 100.00%
file_to_blackhole egress throughput +0.59 [-1.89, +3.08] 30.64%
otlp_grpc_to_blackhole ingress throughput +0.21 [+0.11, +0.31] 99.95%
http_to_s3 ingress throughput +0.21 [-0.07, +0.48] 78.25%
http_to_http_noack ingress throughput +0.10 [+0.02, +0.18] 96.04%
splunk_hec_route_s3 ingress throughput +0.03 [-0.48, +0.54] 7.83%
http_to_http_json ingress throughput +0.02 [-0.05, +0.09] 30.72%
splunk_hec_indexer_ack_blackhole ingress throughput +0.00 [-0.14, +0.14] 2.01%
splunk_hec_to_splunk_hec_logs_acks ingress throughput -0.00 [-0.14, +0.14] 1.40%
splunk_hec_to_splunk_hec_logs_noack ingress throughput -0.02 [-0.13, +0.10] 18.78%
syslog_splunk_hec_logs ingress throughput -0.08 [-0.13, -0.03] 98.88%
enterprise_http_to_http ingress throughput -0.12 [-0.22, -0.02] 94.48%
datadog_agent_remap_blackhole_acks ingress throughput -0.17 [-0.26, -0.07] 99.61%
http_to_http_acks ingress throughput -0.45 [-1.76, +0.85] 43.06%
fluent_elasticsearch ingress throughput -0.49 [-0.95, -0.04] 92.59%
syslog_log2metric_humio_metrics ingress throughput -1.48 [-1.58, -1.38] 100.00%
datadog_agent_remap_datadog_logs ingress throughput -1.72 [-1.82, -1.63] 100.00%
datadog_agent_remap_datadog_logs_acks ingress throughput -1.74 [-1.81, -1.66] 100.00%
datadog_agent_remap_blackhole ingress throughput -1.88 [-1.97, -1.80] 100.00%

Merged via the queue into master with commit 981fb8c Nov 14, 2023
41 checks passed
@dsmith3197 dsmith3197 deleted the dougsmith/file-tag-opt-in branch November 14, 2023 20:52
neuronull pushed a commit that referenced this pull request Nov 14, 2023
@dsmith3197 @neuronull
fix(file source, kubernetes_logs source, file sink): make file intern…
16b9bf6 
…al metric tag opt-in (#19145)

* fix(file source, kubernetes_logs source, file sink): make file internal metric tag opt-in

* update cue

* fix tests
pront pushed a commit to dygfloyd/vector that referenced this pull request Nov 15, 2023
@dsmith3197 @pront
fix(file source, kubernetes_logs source, file sink): make file intern…
9b37792 
…al metric tag opt-in (vectordotdev#19145)

* fix(file source, kubernetes_logs source, file sink): make file internal metric tag opt-in

* update cue

* fix tests
pront pushed a commit to dygfloyd/vector that referenced this pull request Nov 15, 2023
@dsmith3197 @pront
fix(file source, kubernetes_logs source, file sink): make file intern…
fa19972 
…al metric tag opt-in (vectordotdev#19145)

* fix(file source, kubernetes_logs source, file sink): make file internal metric tag opt-in

* update cue

* fix tests
pront added a commit to dygfloyd/vector that referenced this pull request Nov 15, 2023
@pront
fix test config
3b931e6 
fix yaml test config and also format the code

fix wrong usage log.get, now all tests are passing

Ran cargo vdev fmt

Fix reference in to_metrics_metadata in log_to_metric.rs

Convert type errors to single error with type.

Added test for multiple metrics but it is failing.

Additional attempts to get the multiple_metadata_metrics working.

fix(playground): fix playground vrl version and link (vectordotdev#19119)

* fix(playground): fix playground vrl version and link

* Ran cargo vdev fmt

* remove redundant line

* replace clone() with as_ref() whenever possible

* Ran cargo vdev fmt

chore(website): Fix commenting step on workflow (vectordotdev#19134)

* fix: extra env vars into comment step

* fix: add sleep to allow branch to connect

* fix: update where branch name and sanitized branch name are used

Remove multiple metrics test.

chore(ci): Bump bufbuild/buf-setup-action from 1.27.2 to 1.28.0 (vectordotdev#19137)

Bumps [bufbuild/buf-setup-action](https://github.com/bufbuild/buf-setup-action) from 1.27.2 to 1.28.0.
- [Release notes](https://github.com/bufbuild/buf-setup-action/releases)
- [Commits](bufbuild/buf-setup-action@v1.27.2...v1.28.0)

---
updated-dependencies:
- dependency-name: bufbuild/buf-setup-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

chore(deps): Bump the clap group with 1 update (vectordotdev#19127)

Bumps the clap group with 1 update: [clap](https://github.com/clap-rs/clap).

- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md)
- [Commits](clap-rs/clap@v4.4.7...v4.4.8)

---
updated-dependencies:
- dependency-name: clap
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: clap
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

chore(releasing): Add known issue for Datadog Metrics sink in v0.34.0 (vectordotdev#19122)

* chore(releasing): Add known issue for Datadog Metrics sink in v0.34.0

Ref: vectordotdev#19110

Signed-off-by: Jesse Szwedko <jesse.szwedko@datadoghq.com>

* Update website/cue/reference/releases/0.34.0.cue

Co-authored-by: Brett Blue <84536271+brett0000FF@users.noreply.github.com>

---------

Signed-off-by: Jesse Szwedko <jesse.szwedko@datadoghq.com>
Co-authored-by: Brett Blue <84536271+brett0000FF@users.noreply.github.com>

chore(deps): Bump proptest from 1.3.1 to 1.4.0 (vectordotdev#19131)

Bumps [proptest](https://github.com/proptest-rs/proptest) from 1.3.1 to 1.4.0.
- [Release notes](https://github.com/proptest-rs/proptest/releases)
- [Changelog](https://github.com/proptest-rs/proptest/blob/master/CHANGELOG.md)
- [Commits](proptest-rs/proptest@v1.3.1...v1.4.0)

---
updated-dependencies:
- dependency-name: proptest
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

chore(deps): Bump env_logger from 0.10.0 to 0.10.1 (vectordotdev#19130)

Bumps [env_logger](https://github.com/rust-cli/env_logger) from 0.10.0 to 0.10.1.
- [Release notes](https://github.com/rust-cli/env_logger/releases)
- [Changelog](https://github.com/rust-cli/env_logger/blob/main/CHANGELOG.md)
- [Commits](rust-cli/env_logger@v0.10.0...v0.10.1)

---
updated-dependencies:
- dependency-name: env_logger
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

chore(docs): Add alpha to traces and beta to metrics in descriptions (vectordotdev#19139)

add alpha to traces and beta to metrics in descriptions

Update README.md (vectordotdev#19142)

chore(deps): Bump hdrhistogram from 7.5.2 to 7.5.3 (vectordotdev#19129)

Bumps [hdrhistogram](https://github.com/HdrHistogram/HdrHistogram_rust) from 7.5.2 to 7.5.3.
- [Release notes](https://github.com/HdrHistogram/HdrHistogram_rust/releases)
- [Changelog](https://github.com/HdrHistogram/HdrHistogram_rust/blob/main/CHANGELOG.md)
- [Commits](HdrHistogram/HdrHistogram_rust@v7.5.2...v7.5.3)

---
updated-dependencies:
- dependency-name: hdrhistogram
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

enhancement(file sink, aws_s3 sink, gcp_cloud_storage): configurable filename timezone (vectordotdev#18506)

* add TzOffset as File Sink configuration

* integrate TzOffset into File Sink

* apply tz offset to all log event in render_timestamp.

* added TzOffset tests

* adding chrono-tz for parsing timezones

* rename tz_offset to path_tz. timezones are safer than offsets

* update tz_offset references to path_tz

* cargo fmt

* remove unnecessary commented out code. fmt and generate-component-docs

* clippy suggestions and remove TryFrom<&str> - serde handles converting to String

* rename Template config option `path_tz` to `timezone`

* move `path_tz.rs` to `src/config`

preparing for applying the same to `aws_s3` sink for filename timezone

* update doc configuration description for path_tz

* fix wrong method name

* AWS and GCS filename timezone support

  * remove custom tz config
  * use VRL's timezone config
  * pass around SinkContext

* use TzOffset to pass down to request builders. VRL's TimeZone can't be hash derived

* make key_prefix timezone aware and use Option `or` syntax

* move tz to offset conversion codes to sink util

* remove empty line

* update timezone docs in vector-config

* get timezone and convert to offset in one go in FileSink

* just pass the sinkconfig directly

* vector_common to vector_lib

* configurable_component is in vector_lib now

* lookup to vector_lib

* fix aws s3 integration test. pass the context to build_processor in tests

* formatting

* add sinkcontext to FileSink in file tests

* key_prefix is expected to be a template. no need for into

Update documentation

Updated documentation to include example

enhancement(networking, sinks): add full jitter to retry backoff policy (vectordotdev#19106)

* enhancement(networking, sinks): add full jitter to retry backoff policy

* fmt

* fix tests

* add test

* fix

* force ci

fix(file source, kubernetes_logs source, file sink): make file internal metric tag opt-in (vectordotdev#19145)

* fix(file source, kubernetes_logs source, file sink): make file internal metric tag opt-in

* update cue

* fix tests

fix(datadog_metrics sink): evaluate series v1 env var at runtime (vectordotdev#19148)

fix(datadog_metrics sink): evaluate v1 env var at runtime

chore(website): WEB-4247 | Update references from s3 to setup.vector.dev (vectordotdev#19149)

feat: update references from s3 to setup.vector.dev

fix(ARC, networking): improve request settings (vectordotdev#19101)

* fix(ARC, networking): improve request settings

* fix spelling

* change defaults

* refactor

* self-review

* clippy

* update default overrides

* fmt nit

* add upgrade guide entry

Update src/transforms/log_to_metric.rs

Co-authored-by: Pavlos Rontidis <pavlos.rontidis@gmail.com>

Tweak and generate docs

Tweak and generate docs

Update website/cue/reference/components/transforms/base/log_to_metric.cue

Co-authored-by: Heston Hoffman <hestonhoffman@gmail.com>

Update website/cue/reference/components/transforms/base/log_to_metric.cue

Co-authored-by: Heston Hoffman <hestonhoffman@gmail.com>

Update website/cue/reference/components/transforms/base/log_to_metric.cue

Co-authored-by: Heston Hoffman <hestonhoffman@gmail.com>

Update src/transforms/log_to_metric.rs

Co-authored-by: Heston Hoffman <hestonhoffman@gmail.com>

Update src/transforms/log_to_metric.rs

Co-authored-by: Heston Hoffman <hestonhoffman@gmail.com>

Update src/transforms/log_to_metric.rs

Co-authored-by: Heston Hoffman <hestonhoffman@gmail.com>

Fix styling in example for all_metrics.

Minor tweaks to formatting for documentation

Fix minor issues with formatting that was causing tests to fail.
neuronull pushed a commit that referenced this pull request Nov 16, 2023
@dsmith3197 @neuronull
fix(file source, kubernetes_logs source, file sink): make file intern…
8b68c2b 
…al metric tag opt-in (#19145)

* fix(file source, kubernetes_logs source, file sink): make file internal metric tag opt-in

* update cue

* fix tests
jszwedko pushed a commit that referenced this pull request Nov 16, 2023
@dsmith3197 @jszwedko
fix(file source, kubernetes_logs source, file sink): make file intern…
be744f4 
…al metric tag opt-in (#19145)

* fix(file source, kubernetes_logs source, file sink): make file internal metric tag opt-in

* update cue

* fix tests
dsmith3197 added a commit that referenced this pull request Jan 8, 2024
@dsmith3197
fix(file source, kubernetes_logs source, file sink): make file intern…
c4cdb5d 
…al metric tag opt-in (#19145)

* fix(file source, kubernetes_logs source, file sink): make file internal metric tag opt-in

* update cue

* fix tests
AndrooTheChen pushed a commit to discord/vector that referenced this pull request Sep 23, 2024
@dsmith3197 @AndrooTheChen
fix(file source, kubernetes_logs source, file sink): make file intern…
5f48cae 
…al metric tag opt-in (vectordotdev#19145)

* fix(file source, kubernetes_logs source, file sink): make file internal metric tag opt-in

* update cue

* fix tests
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@maycmlee maycmlee maycmlee approved these changes

@jszwedko jszwedko jszwedko approved these changes

Assignees
No one assigned
Labels
domain: external docs Anything related to Vector's external, public documentation domain: sinks Anything related to the Vector's sinks domain: sources Anything related to the Vector's sources
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@dsmith3197 @jszwedko @maycmlee