Merge pull request #922 from gravwell/dev

Merge dev -> master for v5.4.5 release

_static/versions.json

@@ -1,10 +1,14 @@
 [
   {
-    "name": "v5.4.4 (latest)",
-    "version": "v5.4.4",
+    "name": "v5.4.5 (latest)",
+    "version": "v5.4.5",
     "url": "https://docs.gravwell.io/",
     "preferred": true
   },
+  {
+    "version": "v5.4.4",
+    "url": "https://docs.gravwell.io/v5.4.4/"
+  },
   {
     "version": "v5.4.3",
     "url": "https://docs.gravwell.io/v5.4.3/"

changelog/5.4.5.md

@@ -0,0 +1,32 @@
+# Changelog for version 5.4.5
+
+## Released 09 February 2024
+
+## Gravwell
+
+### Additions
+
+* Added the ability to share write access with a group for Query Library items.
+
+### Bug Fixes
+
+* Fixed an issue with permissions checking when de-referencing a Query Library item in a Scheduled Search.
+* Fixed an issue with dereferencing a Query Library item in a Scheduled Search when there are multiple versions installed via kits (e.g. one kit installed globally by an admin and the same installed/edited by a user).
+* Fixed an issue with Scheduled Searches prompting for unsaved changes when no change was made.
+* Fixed an issue with emitting entries outside of the end of a time window when using the `dump` module . 
+* Fixed an issue where installing with an expired license would not properly display the validation error.
+* Fixed an issue where the ingesters list would re-sort. 
+* Fixed an issue where the explore entry in the Query Studio details pane would sometimes not show. 
+* Fixed an issue where the backend would respond twice to explore requests on the websocket in Query Studio. 
+* Fixed an issue where the browser would hang while navigating away from Query Library. 
+
+## Ingesters
+
+### Additions
+
+* Added a new `Trim` flag to Windows File Follower.
+* Added more logging to the S3 ingester.
+
+### Bug Fixes
+
+* Fixed an issue where Windows File Follower would not respect `Regex-Delimiter` configuration.  

changelog/list.md

@@ -7,7 +7,7 @@
 maxdepth: 1
 caption: Current Release
 ---
-5.4.4 <5.4.4>
+5.4.5 <5.4.5>
 ```
 
 ## Previous Versions
@@ -18,6 +18,7 @@ maxdepth: 1
 caption: Previous Releases
 ---
 
+5.4.4 <5.4.4>
 5.4.3 <5.4.3>
 5.4.2 <5.4.2>
 5.4.1 <5.4.1>

conf.py

@@ -21,7 +21,7 @@
 project = "Gravwell"
 copyright = f"Gravwell, Inc. {date.today().year}"
 author = "Gravwell, Inc."
-release = "v5.4.4"
+release = "v5.4.5"
 
 # -- General configuration ---------------------------------------------------
 # https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration

configuration/environment-variables.md

@@ -6,7 +6,7 @@ The indexer, webserver, and ingester components support configuring some paramet
 GRAVWELL_INGEST_SECRET=MyIngestSecret /opt/gravwell/bin/gravwell_federator
 ```
 
-### Loading Values From Files
+## Loading Values From Files
 
 If "_FILE" is added to the end of the environment variable name, Gravwell assumes the variable contains the path to a file which in turn contains the desired data. This is particularly useful in combination with [Docker's "secrets" feature](https://docs.docker.com/engine/swarm/secrets/).
 

ingesters/file_follow.md

@@ -129,6 +129,7 @@ Each follower specifies at minimum a base directory and a filename filtering pat
 | Timestamp-Format-String   | string |          |               | Format string used for parsing timestamps. |
 | Preprocessor              | string(s) |       |               | List of preprocessors to use when processing entries after extraction. |
 | Attach-Filename           | bool  |           | false         | Attach complete filepath to each entry. |
+| Trim                      | bool  |           | false         | Trim leading and trailing space characters from each entry. |
 
 
 ### Base-Directory
@@ -339,3 +340,29 @@ The format defined using these options will be inserted at the top of the list o
 ### Attach-Filename
 
 Each follower has the option to attach the complete file path of the source file to each entry at the time of ingest using the `Attach-Filename=true` parameter to the Follower configuration block.  Setting `Attach-Filename=true` to the follower will attach a value named `file` to each entry which is available at query time in Gravwell.  Be aware that attaching long file paths to entries will have an impact on storage; compression will reduce that impact but it will not eliminate it.
+
+### Trim
+
+Each follower has the option to trim leading and trailing spaces from entries prior to ingest; this option can be useful when your data sources have leading or trailing spaces, tabs, newlines, and other whitespace characters in the ASCII character set.  Trim does not trim whitespace characters in the UTF-8 (or any other non-ASCII encoding) character sets.
+
+The Trim flag can be useful when ingesting entries that have unusual record breaks.  For example, consider the following set of log entries:
+
+```
+starting action baking:
+   * pouring milk
+   * adding cheese
+   * mixing
+   * inserting into oven
+   * setting temperature
+   * remove from oven
+   * set on cooling rack
+starting action consumption:
+   * fending off children
+   * hiding in closet
+   * denying existence of fresh-baked cookies
+starting action cleanup:
+   * washing cookie sheet
+   * replace cookie sheet
+```
+
+You may wish to have each multiline action be a single entry; this can be accomplished with "Regex-Delimiter=`\n\S`" which will break records on a newline that does not have spaces afterwords.  However, a `Regex-Delimiter` that uses newlines will also capture newlines in the entries; adding the `Trim=true` config option will remove the leading and trailing newlines (and any other space characters).  Trim will not remove any interior whitespace characters.

ingesters/win_file_follow.md

@@ -14,7 +14,7 @@ Download the Gravwell Windows File Follower installer:
 
 | Ingester Name | Installer    | More Info |
 | :------------ | :----------- | :-------- |
-| Windows File Follower | <a data-bs-custom-class="hash-popover" href="https://update.gravwell.io/archive/5.4.4/installers/gravwell_file_follow_5.4.4.msi">Download <i class="fa-solid fa-download"></i></a>&nbsp;&nbsp;&nbsp;<a data-bs-custom-class="hash-popover" href="javascript:void(0);" data-bs-toggle="popover" data-bs-placement="bottom" data-bs-html="true" data-bs-content='<code class="docutils literal notranslate"><span class="pre">d94a05ddbe247728a8a12c8884413bba3e11cb6ae2a5c5fe3a2b2cc5ad8fa7e1</span></code>'>(SHA256)</a> | [Documentation](/ingesters/win_file_follow) |
+| Windows File Follower | <a data-bs-custom-class="hash-popover" href="https://update.gravwell.io/archive/5.4.5/installers/gravwell_file_follow_5.4.5.1.msi">Download <i class="fa-solid fa-download"></i></a>&nbsp;&nbsp;&nbsp;<a data-bs-custom-class="hash-popover" href="javascript:void(0);" data-bs-toggle="popover" data-bs-placement="bottom" data-bs-html="true" data-bs-content='<code class="docutils literal notranslate"><span class="pre">97a38d6a24f1aaabd57d5184f29ac8703a8749211c466cb5b5bd75ac0e1278f5</span></code>'>(SHA256)</a> | [Documentation](/ingesters/win_file_follow) |
 
 The Gravwell Windows file follower is installed using a signed MSI package.  Gravwell signs both the Windows executable and MSI installer with our private key pairs, but depending on download volumes, you may see a warning about the MSI being untrusted.  This is due to the way Microsoft "weighs" files.   Basically, as they see more people download and install a given package, it becomes more trustworthy.  Don't worry though, we have a well audited build pipeline and we sign every package.
 

ingesters/winevent.md

@@ -49,7 +49,7 @@ Download the Gravwell Windows Events installer:
 
 | Ingester Name | Installer    | More Info |
 | :------------ | :----------- | :-------- |
-| Windows Events | <a data-bs-custom-class="hash-popover" href="https://update.gravwell.io/archive/5.4.4/installers/gravwell_win_events_5.4.4.msi">Download <i class="fa-solid fa-download"></i></a>&nbsp;&nbsp;&nbsp;<a data-bs-custom-class="hash-popover" href="javascript:void(0);" data-bs-toggle="popover" data-bs-placement="bottom" data-bs-html="true" data-bs-content='<code class="docutils literal notranslate"><span class="pre">ba875403dc51b1c659feece9e1a953c05beebf0eda625a5ce9a7e41692fa8a56</span></code>'>(SHA256)</a> | [Documentation](/ingesters/winevent) |
+| Windows Events | <a data-bs-custom-class="hash-popover" href="https://update.gravwell.io/archive/5.4.5/installers/gravwell_win_events_5.4.5.1.msi">Download <i class="fa-solid fa-download"></i></a>&nbsp;&nbsp;&nbsp;<a data-bs-custom-class="hash-popover" href="javascript:void(0);" data-bs-toggle="popover" data-bs-placement="bottom" data-bs-html="true" data-bs-content='<code class="docutils literal notranslate"><span class="pre">80e11a65821b831aeb72429af6be536d282f8e018e647fca4cd547b54aadea91</span></code>'>(SHA256)</a> | [Documentation](/ingesters/winevent) |
 
 Run the .msi installation wizard to install the Gravwell events service.  On first installation the installation wizard will prompt to configure the indexer endpoint and ingest secret.  Subsequent installations and/or upgrades will identify a resident configuration file and will not prompt.