Feature/upstream update #16
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Bumps [tokio-util](https://github.com/tokio-rs/tokio) from 0.6.5 to 0.6.6. - [Release notes](https://github.com/tokio-rs/tokio/releases) - [Commits](tokio-rs/tokio@tokio-util-0.6.5...tokio-util-0.6.6) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [actions/cache](https://github.com/actions/cache) from v2.1.4 to v2.1.5. - [Release notes](https://github.com/actions/cache/releases) - [Commits](actions/cache@v2.1.4...1a9e213) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* fix(datadog service): Document region parameter Closes vectordotdev#7079 This was added to the shared Datadog documentation in vectordotdev#4174 but never added specifically to the sinks. Signed-off-by: Jesse Szwedko <jesse@szwedko.me> * Add missing cue config for region Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
…ev#7047) * Delete socket Signed-off-by: ktf <krunotf@gmail.com> * Add tests Signed-off-by: ktf <krunotf@gmail.com> * Remove spaces Signed-off-by: ktf <krunotf@gmail.com>
Bumps [reqwest](https://github.com/seanmonstar/reqwest) from 0.11.2 to 0.11.3. - [Release notes](https://github.com/seanmonstar/reqwest/releases) - [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md) - [Commits](seanmonstar/reqwest@v0.11.2...v0.11.3) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…ectordotdev#7081) * fix bug:clickhouse sink incorrectly encodes arrays Signed-off-by: shumin <shumin1027@gmail.com> * Add test for clickhouse arrays Signed-off-by: Jesse Szwedko <jesse@szwedko.me> Co-authored-by: shumin <shumin@compubiq.com> Co-authored-by: Jesse Szwedko <jesse@szwedko.me>
Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
…vectordotdev#7111) * fix(docker_logs source): Docker logs missing when container has a tty Enum bollard::container::LogOutput is Console when docker started with a tty. LogOutput::Console was missing in the new_event of a stream value received https://docs.rs/bollard/0.10.1/bollard/container/enum.LogOutput.html Fixes vectordotdev#5903 Signed-off-by: Jean Prat <jean.prat@sinch.com>
This advisory seems to no longer be valid. Closes: vectordotdev#6223 Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
…dev#7122) Fixes vectordotdev#7121 Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
…#7127) Ensure `except_fields`, `only_fields`, and `timestamp_format` appear on all sinks supporting `encoding`. Previously they only appeared on sinks that had healthchecks. Fixes: vectordotdev#6949 Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
…tdev#7133) * chore(ci): Add advisory RUSTSEC-2021-0013 back to deny.toml I neglected to enable all features. The `wasm` dependencies still depend on an affected version of rust-cpuid. This reverts commit f30d4c6. Signed-off-by: Jesse Szwedko <jesse@szwedko.me> * Run cargo deny in CI if deny.toml changes Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
Bumps [async-trait](https://github.com/dtolnay/async-trait) from 0.1.48 to 0.1.49. - [Release notes](https://github.com/dtolnay/async-trait/releases) - [Commits](dtolnay/async-trait@0.1.48...0.1.49) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* fix(ci): Ensure all benchmark artaficts are included CRITERION_HOME having "s was causing it to put it in the wrong place (locally it created a directory called "). Also just upload `./target/criterion` since Github does the zipping for us. We did need to zip when previously downloading the artifact in the same workflow as this was not zipped yet and so we were hitting limits trying to individually pull a very large number of files. Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.4.0 to 1.5.0. - [Release notes](https://github.com/tokio-rs/tokio/releases) - [Commits](tokio-rs/tokio@tokio-1.4.0...tokio-1.5.0) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [notify](https://github.com/notify-rs/notify) from 4.0.15 to 4.0.16. - [Release notes](https://github.com/notify-rs/notify/releases) - [Changelog](https://github.com/notify-rs/notify/blob/v4.0.16/CHANGELOG.md) - [Commits](notify-rs/notify@v4.0.15...v4.0.16) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* fix(remap): Preserve type defs when assigning fields Previously, remap would overwrite the type def of `.` whenever a new field was assigned. That is: ``` .foo = 5 .bar = 6 ``` Would result in the compiler only having type info for `.bar`. This change causes it to merge the typedefs whenever a list of path segments appears. Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
…#7143) Bumps [pretty_assertions](https://github.com/colin-kiegel/rust-pretty-assertions) from 0.7.1 to 0.7.2. - [Release notes](https://github.com/colin-kiegel/rust-pretty-assertions/releases) - [Changelog](https://github.com/colin-kiegel/rust-pretty-assertions/blob/main/CHANGELOG.md) - [Commits](rust-pretty-assertions/rust-pretty-assertions@v0.7.1...v0.7.2) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
vectordotdev#7140) * fix(kubernetes_logs source): Always use file checkpoints if they exist The `kubernetes_logs` source exposes a `PathProvider` that breaks one of the `FileServer`s assumptions that all available files will be listed at Vector startup time. Instead, the files are only returned once the k8s metadata is available to the `kubernetes_logs` source. This caused the `FileServer` to ignore any checkpoints that existed for these files. As a short-term fix, we just always use the checkpoint, if available, for any new files that are seen. This fixes the case for the `kubernetes_logs` source where they are seen as "new" after start-up. vectordotdev#6564 exists to test this behavior, but it seems to pass even without this change, so that test will need to be updated. Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
…vectordotdev#7091) Fixes vectordotdev#7044 The docs and examples indicated that it should default to case sensitive; however, `contains`, `starts_with`, and `ends_with`, defaulted to case insensitive matching. Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
…ordotdev#7138) * fix(compression): Switch to MultiGzDecoder instead ef GzDecoder Fixes vectordotdev#7061 It appears that AWS's ALB logging gzip's multi-part files which we were only reading the first part of. I tested that `MultiGzDecoder` works on simple gzip files so I figured we should switch to it everywhere. It was already being used by the `file` source. Signed-off-by: Jesse Szwedko <jesse@szwedko.me> * Add test for multi-part zst files Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
* chore(performance): Fix remap benches I broke them in vectordotdev#7118 Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
I backed out a few changes: ``` Upgrading grok v~1.0.1 -> v1.2.0 Upgrading async-graphql-warp v=2.6.4 -> v2.8.2 Upgrading once_cell v1.3 -> v1.7.2 Upgrading async-graphql v=2.6.4 -> v2.8.2 ``` Due to incompatibilities in package dependencies. I backed out ``` Upgrading db-key v0.0.5 -> v0.1.0 ``` Because it requires code changes. Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
Bumps [rust_decimal](https://github.com/paupino/rust-decimal) from 1.10.3 to 1.11.0. - [Release notes](https://github.com/paupino/rust-decimal/releases) - [Commits](paupino/rust-decimal@1.10.3...1.11.0) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…tordotdev#7152) Fixes: vectordotdev#5716 Previously labels on docker containers were inserted in such a way that dots in their names would end up creating a nested structure in the log event due to the semantics of `LogEvent.insert`. For example: ```json { "container_created_at": "2021-04-16T18:53:19.946155600Z", "container_id": "d6bd69d4bc64bef20b4e992dcc23113741067e8268762694f92899504ae14319", "container_name": "docker_echo_1", "host": "COMP-C02DV25MML87", "image": "hashicorp/http-echo:latest", "label": { "com": { "docker": { "compose": { "config-hash": "e7e5ba19811180f27a7af36667652d0cd686599e6184cb023d9b71d791ff6a1e", "container-number": "1", "oneoff": "False", "project": { "config_files": "docker-compose.yml", "working_dir": "/private/tmp/docker" }, "service": "echo", "version": "1.27.4" } } } }, "message": "2021/04/16 19:14:10 localhost:5678 172.29.0.1:61824 \"GET / HTTP/1.1\" 200 6 \"curl/7.64.1\" 35.6µs", "source_type": "docker", "stream": "stdout", "timestamp": "2021-04-16T19:14:10.400790400Z" } ``` This change ensures that labels are inserted as-is as keys: ```json { "container_created_at": "2021-04-16T18:53:19.946155600Z", "container_id": "d6bd69d4bc64bef20b4e992dcc23113741067e8268762694f92899504ae14319", "container_name": "docker_echo_1", "host": "COMP-C02DV25MML87", "image": "hashicorp/http-echo:latest", "label": { "com.docker.compose.config-hash": "e7e5ba19811180f27a7af36667652d0cd686599e6184cb023d9b71d791ff6a1e", "com.docker.compose.container-number": "1", "com.docker.compose.oneoff": "False", "com.docker.compose.project": "docker", "com.docker.compose.project.config_files": "docker-compose.yml", "com.docker.compose.project.working_dir": "/private/tmp/docker", "com.docker.compose.service": "echo", "com.docker.compose.version": "1.27.4" }, "message": "2021/04/16 19:12:01 localhost:5678 172.29.0.1:61820 \"GET / HTTP/1.1\" 200 6 \"curl/7.64.1\" 18.1µs", "source_type": "docker", "stream": "stdout", "timestamp": "2021-04-16T19:12:01.622769500Z" } ``` Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
Bumps [gouth](https://github.com/mechiru/gouth) from 0.2.0 to 0.2.1. - [Release notes](https://github.com/mechiru/gouth/releases) - [Commits](mechiru/gouth@v0.2.0...v0.2.1) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [async-trait](https://github.com/dtolnay/async-trait) from 0.1.49 to 0.1.50. - [Release notes](https://github.com/dtolnay/async-trait/releases) - [Commits](dtolnay/async-trait@0.1.49...0.1.50) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Resolves CVE-2021-22876 CVE-2021-22890 Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
This commit introduces a new RFC for extracting the core of vector. Closes vectordotdev#7027. Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
…ordotdev#7487) This is the documented behavior but we were not honoring it. Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
This commit adds an explicit +Unpin to our buffer type definition, introducing `EventStream` type alias to tidy up some repitition. This is done to support pull request vectordotdev#7576 where this changed is needed but is a little outside the scope of that PR. The major change here is in `src/utilization` where I had to re-implement the select loop to use an explicit stream definition. The `stream!` macro used previously couldn't make the `+ Unpin` guarantee. It wasn't clear to me that it could be fixed upstream. Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
…7581) * fix(cli): include error details in `vector vrl` output Currently, the actual details of the errors are masked so that users just see things like: `parse error` and `json error` without additional detail about what the issue is. This change ensures we display the source of the error. I also fixed the help doc for `--input` that was confusing a user as they thought, understandably, that they could pass a pretty-printed JSON object as a file; however `vector vrl` expects one JSON object per line. We can improve this in the future to actually allow a prety printed object here, but this at least makes the help doc accurate. Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
…dev#7264) * Early compact Signed-off-by: ktf <krunotf@gmail.com> * Exact compaction Signed-off-by: ktf <krunotf@gmail.com> * Add comments Signed-off-by: ktf <krunotf@gmail.com> * More comments Signed-off-by: ktf <krunotf@gmail.com> * More comments Signed-off-by: ktf <krunotf@gmail.com> * Remove dashes Signed-off-by: ktf <krunotf@gmail.com> * Add timeout Signed-off-by: ktf <krunotf@gmail.com> * Clippy Signed-off-by: ktf <krunotf@gmail.com> * Move changes Signed-off-by: ktf <krunotf@gmail.com>
Add initial implementation of `redact` to VRL. Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
Bumps [rust_decimal](https://github.com/paupino/rust-decimal) from 1.14.0 to 1.14.1. - [Release notes](https://github.com/paupino/rust-decimal/releases) - [Commits](paupino/rust-decimal@1.14.0...1.14.1) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [semver](https://github.com/dtolnay/semver) from 0.11.0 to 1.0.0. - [Release notes](https://github.com/dtolnay/semver/releases) - [Commits](dtolnay/semver@0.11.0...1.0.0) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…otdev#7611) Bumps [docker/setup-qemu-action](https://github.com/docker/setup-qemu-action) from 1.1.0 to 1.2.0. - [Release notes](https://github.com/docker/setup-qemu-action/releases) - [Commits](docker/setup-qemu-action@v1.1.0...v1.2.0) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…otdev#7612) Bumps [docker/build-push-action](https://github.com/docker/build-push-action) from 2.4.0 to 2.5.0. - [Release notes](https://github.com/docker/build-push-action/releases) - [Commits](docker/build-push-action@v2.4.0...v2.5.0) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…lated changes (vectordotdev#7602) * chore(dev): remove 'built' dep and stop needless rebuilding on unrelated changes Signed-off-by: Toby Lawrence <toby@nuclearfurnace.com>
Bumps [actions/cache](https://github.com/actions/cache) from 2.1.5 to 2.1.6. - [Release notes](https://github.com/actions/cache/releases) - [Commits](actions/cache@v2.1.5...v2.1.6) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…vectordotdev#7620) Signed-off-by: Toby Lawrence <toby@nuclearfurnace.com>
Commit a07b55c didn't include all the changes to `Cargo.lock` caused to removal of `semver` crate. Signed-off-by: Bruce Guenter <bruce.guenter@datadoghq.com>
…v#7576) This commit moves the top-level benchmarks of buffers into the core buffers crate. This is not a simple code move. There are two fundamental changes being proposed here: introduction of quickcheck tests and intentionally spare benchmarks. ## QuickCheck The first major change in this PR are the newly introduce QuickCheck tests for buffers, both for in-memory and on-disk buffers. The test is simplistic: every `T` that is sent into the buffer should come back out in order unless the buffer was over-full, in which was the `WhenFull` condition applies. The model we compare the buffer against is a `VecDeque` with logic hung off it to mimic the [`buffers + num-senders`](https://docs.rs/futures/0.3.15/futures/channel/mpsc/fn.channel.html) quirk of futures' mpsc, our own shedding logic. The test loop is done without reference to any external runtime: we use the bare interface of `Sink` and Stream`. This avoids the non-determinism introduced by a runtime and has the happy side-benefit of making buffers more runtime agnostic. Notes are left in the model test indicating how we could expand the model. ## Intentionally Spare Benchmarks The benchmark code follow a similar tactic to the quickcheck tests: they do not use a runtime and instead use the bare `Sink` and `Stream` API. While criterion does have support for running async/await code their documentation notes that this setup will introduce overhead and noise, undesirable in this core serialization point of vector. The benchmarks are done with respect to the in-memory and on-disk buffer variants, crossed by a write-then-read and write-and-read benchmark variation. Every effort has been taken to drive down noise as much as possible on my development machine and the results are promising, though we'll see how things shake out in the CI system. Perhaps controversially the benchmark code _does not_ concern itself with the correctness of the buffer response, except in the most coarse sense. This is in line with the goal of driving down noise, reducing the amount of code running in the measurement loop, but runs a little counter to the project's previous strategy with regard to benchmarks. The intention here is for the quickcheck tests to tackle correctness issues and for the benchmark code to focus solely on measurement. The benchmark measurement loops are simplistic and fail-fast with regard to error conditions. Resolves vectordotdev#7458 Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
Signed-off-by: Luke Steensen <luke.steensen@gmail.com>
* Min size Signed-off-by: ktf <krunotf@gmail.com> * Raise to 4MB Signed-off-by: ktf <krunotf@gmail.com> * Add references Signed-off-by: ktf <krunotf@gmail.com>
…#7616 (vectordotdev#7617) Signed-off-by: Milen Georgiev <shogo@shogo.eu>
vectordotdev#7618) * Add `fn {Event,Metric}::with_batch_notifier` * Add support for acknowledgements to vector source Signed-off-by: Bruce Guenter <bruce.guenter@datadoghq.com>
…rdotdev#7603) We don't have a good replacement for this yet for all cases, so just remove the deprecation warnings for now. We plan to revisit schema management at some point which will likely involve dropping these.
…7536) * enhancement(aws_s3 sink): Add acknowledgements support Signed-off-by: Bruce Guenter <bruce.guenter@datadoghq.com>
Signed-off-by: Bruce Guenter <bruce.guenter@datadoghq.com>
…ordotdev#7535) As there are no integration tests for this sink, this is untested. Signed-off-by: Bruce Guenter <bruce.guenter@datadoghq.com>
… creation at runtime (vectordotdev#7601) * fix(aws_cloudwatch_logs sink): Fix healthcheck to allow for log group creation at runtime Signed-off-by: Spencer Gilbert <spencer.gilbert@gmail.com> * Fix clippy violation Signed-off-by: Spencer Gilbert <spencer.gilbert@gmail.com> * Both dynamic group names and create_missing_group short circuit healthcheck Signed-off-by: Spencer Gilbert <spencer.gilbert@gmail.com> * Always try to describe log group, don't error for dynamic or create on demand Signed-off-by: Spencer Gilbert <spencer.gilbert@gmail.com> * Fix event messages to pass linting Signed-off-by: Spencer Gilbert <spencer.gilbert@gmail.com> * Update healthcheck log messages Signed-off-by: Spencer Gilbert <spencer.gilbert@gmail.com>
* Batch deletes Signed-off-by: ktf <krunotf@gmail.com> * Use VecDeque Signed-off-by: ktf <krunotf@gmail.com> * Use take Signed-off-by: ktf <krunotf@gmail.com> * Return unread_size Signed-off-by: ktf <krunotf@gmail.com> * Unpin Signed-off-by: ktf <krunotf@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.