From 2b1c4a6c3702d2bce46edae8ff6a7071d884c857 Mon Sep 17 00:00:00 2001 From: Andrzej Stencel Date: Fri, 31 May 2024 14:43:22 +0200 Subject: [PATCH] [extension/filestorage] remove feature flag `extension.filestorage.replaceUnsafeCharacters` (#33236) **Description:** Removes the feature flag according to schedule. **Link to tracking Issue:** - https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/3148 **Testing:** No changes. **Documentation:** Replaced the description of the feature gate with the description of the feature. --- .../remove-file-storage-feature-gate.yaml | 27 ++++++++++++ extension/storage/filestorage/README.md | 41 +++++-------------- extension/storage/filestorage/extension.go | 16 +------- extension/storage/filestorage/go.mod | 2 - extension/storage/filestorage/go.sum | 4 -- 5 files changed, 39 insertions(+), 51 deletions(-) create mode 100644 .chloggen/remove-file-storage-feature-gate.yaml diff --git a/.chloggen/remove-file-storage-feature-gate.yaml b/.chloggen/remove-file-storage-feature-gate.yaml new file mode 100644 index 000000000000..dc3f3dd57472 --- /dev/null +++ b/.chloggen/remove-file-storage-feature-gate.yaml @@ -0,0 +1,27 @@ +# Use this changelog template to create an entry for release notes. + +# One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix' +change_type: breaking + +# The name of the component, or a single word describing the area of concern, (e.g. filelogreceiver) +component: extension/filestorage + +# A brief description of the change. Surround your text with quotes ("") if it needs to start with a backtick (`). +note: Replace path-unsafe characters in component names + +# Mandatory: One or more tracking issues related to the change. You can use the PR number here if no issue exists. +issues: [3148] + +# (Optional) One or more lines of additional information to render under the primary note. +# These lines will be padded with 2 spaces and then inserted directly into the document. +# Use pipe (|) for multiline entries. +subtext: The feature gate `extension.filestorage.replaceUnsafeCharacters` is now removed. + +# If your change doesn't affect end users or the exported elements of any package, +# you should instead start your pull request title with [chore] or use the "Skip Changelog" label. +# Optional: The change log or logs in which this entry should be included. +# e.g. '[user]' or '[user, api]' +# Include 'user' if the change is relevant to end users. +# Include 'api' if there is a change to a library API. +# Default: '[user]' +change_logs: [] diff --git a/extension/storage/filestorage/README.md b/extension/storage/filestorage/README.md index 36fb14346685..ee41b647fb88 100644 --- a/extension/storage/filestorage/README.md +++ b/extension/storage/filestorage/README.md @@ -100,44 +100,23 @@ exporters: nop: ``` -## Feature Gates +## Replacing unsafe characters in component names -See the [Collector feature gates](https://github.com/open-telemetry/opentelemetry-collector/blob/main/featuregate/README.md#collector-feature-gates) for an overview of feature gates in the collector. +The extension uses the type and name of the component using the extension to create a file where the component's data is stored. +For example, if a Filelog receiver named `filelog/logs` uses the extension, its data is stored in a file named `receiver_filelog_logs`. -### `extension.filestorage.replaceUnsafeCharacters` +Sometimes the component name contains characters that either have special meaning in paths - like `/` - or are problematic or even forbidden in file names (depending on the host operating system), like `?` or `|`. +To prevent surprising or erroneous behavior, some characters in the component names are replaced before creating the file name to store data by the extension. -When enabled, characters that are not safe in file names are replaced in component name using the extension before creating the file name to store data by the extension. +For example, for a Filelog receiver named `filelog/logs/container`, the component name `logs/container` is sanitized into `logs~007Econtainer` and the data is stored in a file named `receiver_filelog_logs~007Econtainer`. -For example, for a Filelog receiver named `filelog/logs/json`, the data is stored: - -- in path `receiver_filelog_logs/json` with the feature flag disabled (note that this is a file named `json` inside directory named `receiver_filelog_logs`). -- in file `receiver_filelog_logs~007Ejson` with the feature flag enabled. - -This replacement is done to prevent surprising behavior or errors in the File Storage extension. - -The feature replaces all usafe characters with a tilde `~` and the character's [Unicode number][unicode_chars] in hex. +Every unsafe character is replaced with a tilde `~` and the character's [Unicode number][unicode_chars] in hex. The only safe characters are: uppercase and lowercase ASCII letters `A-Z` and `a-z`, digits `0-9`, dot `.`, hyphen `-`, underscore `_`. - -Changing the state of this feature gate may change the path to the file that the extension is writing component's data to. This may lead to loss of the data stored in the original path. - -Before enabling this feature gate, ideally make sure that all component names that use the File Storage extension have names that only contain safe characters. -In case you want to keep using unsafe characters in your component names, you may want to rename the files used for storage before enabling this feature gate. -For example, `mv ./receiver_filelog_logs/json ./receiver_filelog_logs~007Ejson`. - -For more details, see the following issues: - -- [File storage extension - invalid file name characters must be encoded #3148](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/3148) -- [[filestorage] receiver name sanitization #20731](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/20731) - -The schedule for this feature gate is: - -- Introduced in v0.87.0 (October 2023) as `alpha` - disabled by default. -- Moved to `beta` in v0.92.0 (January 2024) - enabled by default. -- Moved to `stable` in v0.99.0 (April 2024) - cannot be disabled. -- Removed in v0.102.0 (three releases after `stable`). +The tilde `~` character is also replaced even though it is a safe character, to make sure that the sanitized component name never overlaps with a component name that does not require sanitization. [unicode_chars]: https://en.wikipedia.org/wiki/List_of_Unicode_characters + ## Troubleshooting _Currently, the File Storage extension uses [bbolt](https://github.com/etcd-io/bbolt) to store and read data on disk. The @@ -165,4 +144,4 @@ default file_input.knownFiles2 {"Fingerprint":{"first_bytes":"MzEwNzkKMjE5Cg=="},"Offset":10,"FileAttributes":{"log.file.name":"1.log"},"HeaderFinalized":false,"FlushState":{"LastDataChange":"2024-03-20T18:16:18.164331-07:00","LastDataLength":0}} {"Fingerprint":{"first_bytes":"MjQ0MDMK"},"Offset":6,"FileAttributes":{"log.file.name":"2.log"},"HeaderFinalized":false,"FlushState":{"LastDataChange":"2024-03-20T18:16:39.96429-07:00","LastDataLength":0}} -``` \ No newline at end of file +``` diff --git a/extension/storage/filestorage/extension.go b/extension/storage/filestorage/extension.go index fdbf533158c3..f9d0467accbf 100644 --- a/extension/storage/filestorage/extension.go +++ b/extension/storage/filestorage/extension.go @@ -14,19 +14,9 @@ import ( "go.opentelemetry.io/collector/component" "go.opentelemetry.io/collector/extension" "go.opentelemetry.io/collector/extension/experimental/storage" - "go.opentelemetry.io/collector/featuregate" "go.uber.org/zap" ) -var replaceUnsafeCharactersFeatureGate = featuregate.GlobalRegistry().MustRegister( - "extension.filestorage.replaceUnsafeCharacters", - featuregate.StageStable, - featuregate.WithRegisterDescription("When enabled, characters that are not safe in file paths are replaced in component name using the extension. For example, the data for component `filelog/logs/json` will be stored in file `receiver_filelog_logs~007Ejson` and not in `receiver_filelog_logs/json`."), - featuregate.WithRegisterReferenceURL("https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/3148"), - featuregate.WithRegisterFromVersion("v0.87.0"), - featuregate.WithRegisterToVersion("v0.102.0"), -) - type localFileStorage struct { cfg *Config logger *zap.Logger @@ -66,9 +56,7 @@ func (lfs *localFileStorage) GetClient(_ context.Context, kind component.Kind, e rawName = fmt.Sprintf("%s_%s_%s_%s", kindString(kind), ent.Type(), ent.Name(), name) } - if replaceUnsafeCharactersFeatureGate.IsEnabled() { - rawName = sanitize(rawName) - } + rawName = sanitize(rawName) absoluteName := filepath.Join(lfs.cfg.Directory, rawName) client, err := newClient(lfs.logger, absoluteName, lfs.cfg.Timeout, lfs.cfg.Compaction, !lfs.cfg.FSync) @@ -110,7 +98,7 @@ func sanitize(name string) string { // https://en.wikipedia.org/wiki/List_of_Unicode_characters // For example, the slash is replaced with "~002F", and the tilde itself is replaced with "~007E". // We perform replacement on the tilde even though it is a safe character to make sure that the sanitized component name - // never overlaps with a component name that does not reqire sanitization. + // never overlaps with a component name that does not require sanitization. var sanitized strings.Builder for _, character := range name { if isSafe(character) { diff --git a/extension/storage/filestorage/go.mod b/extension/storage/filestorage/go.mod index 7fc1f898d2eb..71194a4427d0 100644 --- a/extension/storage/filestorage/go.mod +++ b/extension/storage/filestorage/go.mod @@ -8,7 +8,6 @@ require ( go.opentelemetry.io/collector/component v0.101.1-0.20240529223953-eaab76e46d38 go.opentelemetry.io/collector/confmap v0.101.1-0.20240529223953-eaab76e46d38 go.opentelemetry.io/collector/extension v0.101.1-0.20240529223953-eaab76e46d38 - go.opentelemetry.io/collector/featuregate v1.8.1-0.20240529223953-eaab76e46d38 go.opentelemetry.io/otel/metric v1.27.0 go.opentelemetry.io/otel/trace v1.27.0 go.uber.org/goleak v1.3.0 @@ -25,7 +24,6 @@ require ( github.com/go-logr/stdr v1.2.2 // indirect github.com/gogo/protobuf v1.3.2 // indirect github.com/google/uuid v1.6.0 // indirect - github.com/hashicorp/go-version v1.7.0 // indirect github.com/knadh/koanf/maps v0.1.1 // indirect github.com/knadh/koanf/providers/confmap v0.1.0 // indirect github.com/knadh/koanf/v2 v2.1.1 // indirect diff --git a/extension/storage/filestorage/go.sum b/extension/storage/filestorage/go.sum index ffedab5254f4..df107c560a29 100644 --- a/extension/storage/filestorage/go.sum +++ b/extension/storage/filestorage/go.sum @@ -17,8 +17,6 @@ github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI= github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY= github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0= github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= -github.com/hashicorp/go-version v1.7.0 h1:5tqGy27NaOTB8yJKUZELlFAS/LTKJkrmONwQKeRZfjY= -github.com/hashicorp/go-version v1.7.0/go.mod h1:fltr4n8CU8Ke44wwGCBoEymUuxUHl09ZGVZPK5anwXA= github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8= github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck= github.com/knadh/koanf/maps v0.1.1 h1:G5TjmUh2D7G2YWf5SQQqSiHRJEjaicvU0KpypqB3NIs= @@ -61,8 +59,6 @@ go.opentelemetry.io/collector/confmap v0.101.1-0.20240529223953-eaab76e46d38 h1: go.opentelemetry.io/collector/confmap v0.101.1-0.20240529223953-eaab76e46d38/go.mod h1:KgpS7UxH5rkd69CzAzlY2I1heH8Z7eNCZlHmwQBMxNg= go.opentelemetry.io/collector/extension v0.101.1-0.20240529223953-eaab76e46d38 h1:SMVLOe6rqzJRkc7yygpIJakvRr8H9ZA1LZr1KqIb6dM= go.opentelemetry.io/collector/extension v0.101.1-0.20240529223953-eaab76e46d38/go.mod h1:dCC2JXO2XLrjnRENVd+b8kiaM+jJaiUXowGuupa0ouo= -go.opentelemetry.io/collector/featuregate v1.8.1-0.20240529223953-eaab76e46d38 h1:4tVKrXF8oux8ATbESgPuIV57T85GhjdQHdjSIkFqlIQ= -go.opentelemetry.io/collector/featuregate v1.8.1-0.20240529223953-eaab76e46d38/go.mod h1:PsOINaGgTiFc+Tzu2K/X2jP+Ngmlp7YKGV1XrnBkH7U= go.opentelemetry.io/collector/pdata v1.8.1-0.20240529223953-eaab76e46d38 h1:WOatQWY4MmVW4N4J3exbCb0RinS3B5pqKhcRjNWSNk8= go.opentelemetry.io/collector/pdata v1.8.1-0.20240529223953-eaab76e46d38/go.mod h1:vk7LrfpyVpGZrRWcpjyy0DDZzL3SZiYMQxfap25551w= go.opentelemetry.io/otel v1.27.0 h1:9BZoF3yMK/O1AafMiQTVu0YDj5Ea4hPhxCs7sGva+cg=