Merge pull request #1314 from gravwell/merge/next-minor_to_next-major

chore: Merge next-minor into next-major

_static/versions.json

@@ -1,10 +1,14 @@
 [
   {
-    "name": "v5.6.7 (latest)",
-    "version": "v5.6.7",
+    "name": "v5.6.8 (latest)",
+    "version": "v5.6.8",
     "url": "/",
     "preferred": true
   },
+  {
+    "version": "v5.6.7",
+    "url": "/v5.6.7/"
+  },
   {
     "version": "v5.6.6",
     "url": "/v5.6.6/"

changelog/5.6.8.md

@@ -0,0 +1,16 @@
+# Changelog for version 5.6.8
+
+## Released 16 December 2024
+
+## Gravwell
+
+### Additions
+* Added hotkeys to support auto-closing pairs for `"`, `(`, `[`, and `{` in the query editor.
+* Added strict transport security header when running in TLS mode.
+* Added support for start/end constraints in inner queries when using compound queries.
+
+### Bug Fixes
+
+* Fixed an issue where JavaScript returning `undefined` could improperly halt execution of a Flow.
+* Fixed an issue with macro expansion with invalid macros. 
+* Improved timestamp processing to truncate subsecond precision when using start/end constraints. 

changelog/list.md

@@ -7,7 +7,7 @@
 maxdepth: 1
 caption: Current Release
 ---
-5.6.7 <5.6.7>
+5.6.8 <5.6.8>
 ```
 
 ## Previous Versions
@@ -18,6 +18,7 @@ maxdepth: 1
 caption: Previous Releases
 ---
 
+5.6.7 <5.6.7>
 5.6.6 <5.6.6>
 5.6.5 <5.6.5>
 5.6.4 <5.6.4>

conf.py

@@ -21,7 +21,7 @@
 project = "Gravwell"
 copyright = f"Gravwell, Inc. {date.today().year}"
 author = "Gravwell, Inc."
-release = "v5.6.7"
+release = "v5.6.8"
 
 # Default to localhost:8000, so the version switcher looks OK on livehtml
 version_list_url = os.environ.get(

configuration/accelerators.md

@@ -249,6 +249,11 @@ Note that the tag `zeekconn` can be matched against both accelerators, however t
 	Tags=foo*
 ```
 
+(intrinsic-acceleration-target)=
+## Acceleration with Intrinsic Enumerated Values
+
+When acceleration is enabled, [intrinsic enumerated values](#attach-target) will always be accelerated with the fulltext engine. This enables queries using the [intrinsic](/search/intrinsic/intrinsic) module to be accelerated. No specific configuration is required for acceleration with intrinsic EVs other than having acceleration enabled.
+
 ## Fulltext
 
 The fulltext accelerator is designed to index words within text logs and is considered the most flexible acceleration option.  Many of the other search modules support invoking the fulltext accelerator when executing queries.  However, the primary search module for engaging with the fulltext accelerator is the [grep](/search/grep/grep) module with the `-w` flag.  Much like the Unix grep utility, `grep -w` specifies that the provided filter is expected to a word, rather than a subset of bytes.  Running a search with `words foo bar baz` will look for the words foo, bar, and baz and engage the fulltext accelerator.
@@ -686,7 +691,6 @@ The results show why fulltext may often be worth the storage and ingest penalty:
 | fulltextindex   | 2.99s      | 12.49X  |
 | fulltextbloom   | 3.40s      | 12.49X  |
 
-
 #### Query AX modules
 
 The AX definition file for all four tags is below, see the [AX](/configuration/autoextractors) documentation for more information:

configuration/parameters.md

@@ -902,7 +902,7 @@ Description:	Sets the storage location for data replicated from other Gravwell i
 ### **Max-Replicated-Data-GB**
 Default Value:  
 Example:		`Max-Replicated-Data-GB=100`  
-Description:	Sets, in gigabytes, the maximum amount of replicated data to store. When this is exceeded, the indexer will begin walking the replicated data to clean up; it will first remove any shards which have been deleted on the original indexer, then it will begin deleting the oldest shards. Once the storage size is below the limit, deletion will stop.
+Description:	Sets, in gigabytes, the maximum amount of replicated data to store. When this is exceeded, the indexer will begin walking the replicated data to clean up; it will first remove any shards which have been deleted on the original indexer, then cold shards, then by oldest date. Once the storage size is below the limit, deletion will stop.
 
 ### **Replication-Secret-Override**
 Default Value:  

configuration/replication.md

@@ -11,7 +11,7 @@ The replication system is logically separated into "Clients" and "Peers", with e
 
 Replication connections are encrypted by default and require that indexers have functioning X509 certificates.  If the certificates are not signed by a valid certificate authority (CA) then `Insecure-Skip-TLS-Verify=true` must be added to the Replication configuration section.
 
-Replication storage nodes (nodes which receive replicated data) are allotted a specific amount of storage and will not delete data until that storage is exhausted.  If a remote client node deletes data as part of normal ageout, the data shard is marked as deleted and prioritized for deletion when the replication node hits its storage limit.  The replication system prioritizes deleted shards first, cold shards second, and oldest shards last.  All replicated data is compressed; if a cold storage location is provided it is usually recommended that the replication storage location have the same storage capacity as the cold and hot storage combined.
+Replication storage nodes (nodes which receive replicated data) are allotted a specific amount of storage and will not delete data unless the `Max-Replicated-Data-GB` parameter is set. Even with `Max-Replicated-Data-GB` set, the replication system will not delete replicated shards until the storage limit has been reached.  If a remote client node deletes data as part of normal ageout, the data shard is marked as deleted and prioritized for deletion when the replication node hits its storage limit.  The replication system prioritizes deleted shards first, cold shards second, and oldest shards last.  All replicated data is compressed; if a cold storage location is provided it is usually recommended that the replication storage location have the same storage capacity as the cold and hot storage combined.
 
 ```{note}
 By default, the replication engine uses port 9406.

gui/queries/queries.md

@@ -43,6 +43,10 @@ The queries stored in the query library are also available through the right-han
 (timeframe_selector)=
 ## Selecting a Timeframe
 
+```{note}
+Timeframes are always aligned to one second boundaries. Sub-second timeframes will be automatically rounded down to the second.
+```
+
 By default, queries run over the last hour of data. This is easily changed by clicking on the calendar icon or timeframe above the query and selecting a timeframe from the dropdown:
 
 ![](timeframe-icon.png)

ingesters/ingesters.md

@@ -326,6 +326,7 @@ Log-Source-Override=DEAD:BEEF::FEED:FEBE
 Log-Source-Override=::1
 ```
 
+(attach-target)=
 ### Attach
 
 All ingesters support the `Attach` global configuration stanza, which allows [intrinsic enumerated values](intrinsic_enumerated_values) to be attached to entries during ingest. Intrinsic enumerated values can later be accessed with the [intrinsic](/search/intrinsic/intrinsic) search module.

ingesters/win_file_follow.md

@@ -14,7 +14,7 @@ Download the Gravwell Windows File Follower installer:
 
 | Ingester Name | Installer    | More Info |
 | :------------ | :----------- | :-------- |
-| Windows File Follower | <a data-bs-custom-class="hash-popover" href="https://update.gravwell.io/archive/5.6.7/installers/gravwell_file_follow_5.6.7.1.msi">Download <i class="fa-solid fa-download"></i></a>&nbsp;&nbsp;&nbsp;<a data-bs-custom-class="hash-popover" href="javascript:void(0);" data-bs-toggle="popover" data-bs-placement="bottom" data-bs-html="true" data-bs-content='<code class="docutils literal notranslate"><span class="pre">ef27478ee05cd92c682386d35cc6d471c662114d2635893904a3fdf7df0efb1c</span></code>'>(SHA256)</a> | [Documentation](/ingesters/win_file_follow) |
+| Windows File Follower | <a data-bs-custom-class="hash-popover" href="https://update.gravwell.io/archive/5.6.8/installers/gravwell_file_follow_5.6.8.1.msi">Download <i class="fa-solid fa-download"></i></a>&nbsp;&nbsp;&nbsp;<a data-bs-custom-class="hash-popover" href="javascript:void(0);" data-bs-toggle="popover" data-bs-placement="bottom" data-bs-html="true" data-bs-content='<code class="docutils literal notranslate"><span class="pre">392c1400f0d4fae3410fea6fcc510c9fe641b73445d19bb4b0726faa523ee7b9</span></code>'>(SHA256)</a> | [Documentation](/ingesters/win_file_follow) |
 
 The Gravwell Windows file follower is installed using a signed MSI package.  Gravwell signs both the Windows executable and MSI installer with our private key pairs, but depending on download volumes, you may see a warning about the MSI being untrusted.  This is due to the way Microsoft "weighs" files.   Basically, as they see more people download and install a given package, it becomes more trustworthy.  Don't worry though, we have a well audited build pipeline and we sign every package.
 

ingesters/winevent.md

@@ -49,7 +49,7 @@ Download the Gravwell Windows Events installer:
 
 | Ingester Name | Installer    | More Info |
 | :------------ | :----------- | :-------- |
-| Windows Events | <a data-bs-custom-class="hash-popover" href="https://update.gravwell.io/archive/5.6.7/installers/gravwell_win_events_5.6.7.1.msi">Download <i class="fa-solid fa-download"></i></a>&nbsp;&nbsp;&nbsp;<a data-bs-custom-class="hash-popover" href="javascript:void(0);" data-bs-toggle="popover" data-bs-placement="bottom" data-bs-html="true" data-bs-content='<code class="docutils literal notranslate"><span class="pre">14242932f36dbc4a726db4156b5ebed2cdaf5e01c5e49239e79d3344bd4b4a73</span></code>'>(SHA256)</a> | [Documentation](/ingesters/winevent) |
+| Windows Events | <a data-bs-custom-class="hash-popover" href="https://update.gravwell.io/archive/5.6.8/installers/gravwell_win_events_5.6.8.1.msi">Download <i class="fa-solid fa-download"></i></a>&nbsp;&nbsp;&nbsp;<a data-bs-custom-class="hash-popover" href="javascript:void(0);" data-bs-toggle="popover" data-bs-placement="bottom" data-bs-html="true" data-bs-content='<code class="docutils literal notranslate"><span class="pre">2b664f995fb404929f01e1e5b1b525b3f0d0a05a0e51c339aba7f90d9968a60e</span></code>'>(SHA256)</a> | [Documentation](/ingesters/winevent) |
 
 Run the .msi installation wizard to install the Gravwell events service.  On first installation the installation wizard will prompt to configure the indexer endpoint and ingest secret.  Subsequent installations and/or upgrades will identify a resident configuration file and will not prompt.