Merge pull request #1205 from gravwell/merge/main_to_next-patch

chore: Merge main into next-patch

3rdparty-frontend-licenses.txt

@@ -2892,32 +2892,6 @@ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 SOFTWARE.
 
 
-has
-MIT
-Copyright (c) 2013 Thiago de Arruda
-
-Permission is hereby granted, free of charge, to any person
-obtaining a copy of this software and associated documentation
-files (the "Software"), to deal in the Software without
-restriction, including without limitation the rights to use,
-copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the
-Software is furnished to do so, subject to the following
-conditions:
-
-The above copyright notice and this permission notice shall be
-included in all copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
-EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
-OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
-NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
-HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
-WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
-FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
-OTHER DEALINGS IN THE SOFTWARE.
-
-
 has-bigints
 MIT
 MIT License
@@ -3540,32 +3514,6 @@ SOFTWARE.
 
 
 
-is-typed-array
-MIT
-The MIT License (MIT)
-
-Copyright (c) 2015 Jordan Harband
-
-Permission is hereby granted, free of charge, to any person obtaining a copy
-of this software and associated documentation files (the "Software"), to deal
-in the Software without restriction, including without limitation the rights
-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the Software is
-furnished to do so, subject to the following conditions:
-
-The above copyright notice and this permission notice shall be included in all
-copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-SOFTWARE.
-
-
-
 is-weakmap
 MIT
 MIT License
@@ -4325,6 +4273,31 @@ IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
 CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
 
 
+possible-typed-array-names
+MIT
+MIT License
+
+Copyright (c) 2024 Jordan Harband
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+
+
 regenerator-runtime
 MIT
 

_static/versions.json

@@ -1,10 +1,14 @@
 [
   {
-    "name": "v5.5.7 (latest)",
-    "version": "v5.5.7",
+    "name": "v5.6.0 (latest)",
+    "version": "v5.6.0",
     "url": "/",
     "preferred": true
   },
+  {
+    "version": "v5.5.7",
+    "url": "/v5.5.7/"
+  },
   {
     "version": "v5.5.6",
     "url": "/v5.5.6/"

architecture/architecture.md

@@ -8,6 +8,7 @@ maxdepth: 1
 caption: Architecture
 hidden: true
 ---
+Filesystems </configuration/filesystems>
 Networking Considerations for Gravwell </configuration/networking>
 Gravwell Clusters </distributed/cluster>
 Distributed Gravwell Webserver </distributed/frontend>
@@ -100,9 +101,9 @@ The core ingest mechanic requires only three data items: a byte array, timestamp
 
 ## Compatibility 
 
-### Minimum Software Requirements
+### Minimum Requirements
 
-Gravwell runs on most common Linux distributions which support [SystemD](https://en.wikipedia.org/wiki/Systemd). A minimum Linux kernel version of 3.2 and a 64bit X86 architecture are required.
+Gravwell runs on most common Linux distributions which support [SystemD](https://en.wikipedia.org/wiki/Systemd). A minimum Linux kernel version of 3.2, 2GB of RAM, and a 64bit X86 architecture is required.
 
 ### Version Locking
 

cbac/cbac.md

@@ -4,6 +4,7 @@ Capability Based Access Control (CBAC) is a feature access system that enables u
 
 CBAC is based around a deny-all default policy. Capabilities and tag access must be granted to each user (or group a user belongs to) in order to access those features. Admin users are not restricted by CBAC and always have full system access.
 
+(enabling-cbac)=
 ## Enabling CBAC
 
 CBAC is enabled by adding the following clause to the global section of the webserver's `gravwell.conf` and restarting the webserver:

changelog/5.6.0.md

@@ -0,0 +1,54 @@
+# Changelog for version 5.6.0
+
+## Released 15 October 2024
+
+## Gravwell
+
+### Additions
+
+* Added the Free and CE Advanced license tiers.
+* Added the ability to download installed Kits.
+* Added the [Attach flow node](/flows/nodes/attach).
+* Added support for single and double quotes in Data field extractions in winlog.
+* Added the ability to share results from scheduled searches and alerts globally or with multiple groups.
+* Added `-maxtracked` and `-maxsize` flags to the `fuse` module. 
+* Added maps to persistent variables in the `eval` module.
+* Added acceleration hints to the `intrinsic` module.
+* Added src acceleration hints to the `eval` module.
+* Added additional error handling to searches.
+* Added support for an ERROR state on the Persistent Searches page.
+
+### Bug Fixes
+
+* Improved Renderer Storage Limit notifications.
+* Improved recovery for searches resulting in errors. 
+* Improved search agent detection of searches which hit an error during a query.
+* Improved sharing options for the Persistent Searches pages.
+* Improved ageout to prevent hot aging to cold when cold data storage is over its threshold.
+* Improved overview chart colors to better reflect the search status for default, warn, and error.
+* Fixed an edge case on the Scheduled Search API to improve compliance with OpenAPI spec.
+* Fixed an issue where overview stats could be incomplete when the Renderer Storage Limit was reached due to partial results returned.
+* Fixed an issue where SSO logins would fail when a token cookie gets too big (e.g. when the groups list is long).
+* Fixed an issue where a validation error could be shown on a Dispatcher owned by another user when changing an Alert schema.
+* Fixed an issue where a duplicate warning would be incorrectly shown when saving your first query.
+* Fixed an issue where uploading an invalid Flow would not display an error message.
+* Fixed an issue where a custom label added to a Flow node could be reset by changing focus.
+* Fixed an issue where a configuration Macro name would not be saved on Kit download.
+* Fixed an issue where Scripts were not properly displayed in the Kit Content List when deploying.
+* Fixed an issue where the cursor would jump to the end when trying to add characters to the beginning or middle of a Macro name.
+* Fixed an issue where the Last Run time would not be updated without refreshing for Scheduled Searches and Scripts.
+* Fixed an issue where the `Scheduled` value for Flows was incorrectly populated with the executed time instead of the scheduled time.
+* Fixed an issue where the text renderer did not show intrinsic EVs without using the `intrinsic` module.
+* Fixed an issue where acceleration was not working with the `src` module.
+* Fixed an issue where `lookup` module could not read a CSV smaller than 8 bytes.
+* Fixed an issue with resource name resolution for queries run as admin.
+* Fixed an issue where a timeframe lock would be lost after two consecutive launches in Query Studio.
+* Fixed an issue where enabling live search would cause the 'Fetching data...' message to be displayed until the next update.
+* Fixed permissions in shell installers to ensure all files are owned by gravwell:gravwell instead of root.
+* Sorted EVs in the Query Studio Fields tab to prevent them from rearranging.
+
+## Ingester Changes
+
+### Bug Fixes
+
+* Fixed a bug in the syslog ingester preprocessor that would crash given certain malformed input.

changelog/list.md

@@ -7,7 +7,7 @@
 maxdepth: 1
 caption: Current Release
 ---
-5.5.7 <5.5.7>
+5.6.0 <5.6.0>
 ```
 
 ## Previous Versions
@@ -18,6 +18,7 @@ maxdepth: 1
 caption: Previous Releases
 ---
 
+5.5.7 <5.5.7>
 5.5.6 <5.5.6>
 5.5.5 <5.5.5>
 5.5.4 <5.5.4>

conf.py

@@ -21,7 +21,7 @@
 project = "Gravwell"
 copyright = f"Gravwell, Inc. {date.today().year}"
 author = "Gravwell, Inc."
-release = "v5.5.7"
+release = "v5.6.0"
 
 # Default to localhost:8000, so the version switcher looks OK on livehtml
 version_list_url = os.environ.get(
@@ -56,6 +56,7 @@
     ".DS_Store",
     "env",
     "_vendor",
+    "_tools",
 ]
 
 language = "en"

configuration/configuration.md

@@ -69,7 +69,7 @@ By default, Gravwell does not generate TLS certificates. For instructions on set
 (configuration_indexer)=
 ## Indexer Configuration
 
-Indexers are the storage centers of Gravwell and are responsible for storing, retrieving, and processing data.  Indexers perform the first heavy lifting when executing a query, first finding the data then pushing it into the search pipeline.  The search pipeline will perform as much work as possible in parallel on the indexers for efficiency.  Indexers benefit from high-speed low-latency storage and as much RAM as possible.  Gravwell can take advantage of file system caches, which means that as you are running multiple queries over the same data it won’t even have to go to the disks.  We have seen Gravwell operate at over 5GB/s per node on well-cached data.  The more memory, the more data can be cached.  When searching over large pools that exceed the memory capacity of even the largest machines, high speed RAID arrays can help increase throughput.
+Indexers are the storage centers of Gravwell and are responsible for storing, retrieving, and processing data.  Indexers perform the first heavy lifting when executing a query, first finding the data then pushing it into the search pipeline.  The search pipeline will perform as much work as possible in parallel on the indexers for efficiency.  Indexers benefit from high-speed low-latency storage and as much RAM as possible.  Gravwell can take advantage of filesystem caches, which means that as you are running multiple queries over the same data it won’t even have to go to the disks.  We have seen Gravwell operate at over 5GB/s per node on well-cached data.  The more memory, the more data can be cached.  When searching over large pools that exceed the memory capacity of even the largest machines, high speed RAID arrays can help increase throughput.
 
 We recommend indexers have at least 32GB of memory with 8 CPU cores.  If possible, Gravwell also recommends a very high speed NVME solid state disk that can act as a hot well, holding just a few days of of the most recent data and aging out to the slower spinning disk pools.  The hot well enables very fast access to the most recent data, while enabling Gravwell to organize and consolidate older data so that he can be searched as efficiently as possible.
 
@@ -103,6 +103,14 @@ Tag-to-well mappings are defined in the `/opt/gravwell/etc/gravwell.conf` config
 
 The well named "raw" is thus used to store data tagged "pcap" and "video", which we could reasonably assume will consume a significant amount of storage.
 
+
+(well_storage)=
+#### Well Storage
+
+Gravwell Indexers require seekable POSIX compliant filesystems for hot and cold storage volumes.  Picking the right filesystem for your well storage can open up opportunities for optimizations and fault tolerance above and beyond what Gravwell offers in the default configuration.
+
+See the section on [Filesystems](/configuration/filesystems) for more details on supported filesystems and filesystem options.
+
 #### Tag Restrictions and Gotchas
 
 Tag names can only contain alpha numeric values; dashes, underscores, special characters, etc are not allowed in tag names.  Tags should be simple names like "syslog" or "apache" that are easy to type and reflect the type of data in use.

configuration/filesystems.md

@@ -0,0 +1,55 @@
+# Gravwell Indexer Supported Filesystems
+
+Gravwell Indexers require robust, seekable, and POSIX complaint filesystems in order to function properly.  The Gravwell system makes extensive use of memory mapping, madvise calls, and filesystem specific optimizations to maximize compression ratios and query throughput.  Picking a good filesystem for your deployment is critical to ensuring a manageable and fast Gravwell system.
+
+## Supported Filesystems
+
+Gravwell officially supports the following Linux filesystems.
+
+| Filesystem | Minimum Kernel Version | Supports Transparent Compression |
+|:-----------|:-----------------------|:--------------------------------:|
+| EXT4       | 3.2                    |                                  |
+| XFS        | 3.2                    |                                  |
+| BTRFS      | 5.0                    | ✅                               |
+| ZFS        | N/A                    | ✅                               |
+| NFSv4      | N/A                    |                                  |
+
+
+
+
+### Ext4
+
+The Ext4 filesystem is well supported and an excellent default choice as a backing filesystem.  Ext4 supports volume sizes up to 1EiB and up to 4 Billion files, Gravwell extensively tests on Ext4.
+
+### XFS
+
+The XFS filesystem is extremely fast, well tested, and praised by kernel developers.  XFS supports a wide array of configuration options to optimize the filesystem for a specific storage device topology.
+
+### BTRFS
+
+The BTRFS filesystem has been a core part of the Linux kernel for over a decade, but due to some rocky starts and conservative warnings about stability early on in its life cycle it gets a bad rap.  Gravwell extensively tests the BTRFS filesystem in a transparent compression topology and has found it to be exceedingly fast, memory efficient, and well supported.  While BTRFS is supported all the way back to Linux Kernel 3.2, 5.X and newer kernels contain a highly optimized and stable code base.  Gravwell recommends BTRFS with ZSTD compression for a hot store when transparent compression is enabled and users want the best performance.
+
+### ZFS
+
+The ZFS filesystem has long been praised as **THE** next generation filesystem. It has a stable, well-maintained code base with robust documentation.  However, ZFS is in a bit of a strange situation in the Linux kernel in that many distributions do not natively support it and the kernel maintainers believe it has an incompatible license.  ZFS also employs its own caching strategy that is not well blended with the Linux page cache; this means you need to manually tune the ZFS ARC cache and be aware that the ARC cache competes for memory with the Gravwell processes.  When memory gets tight, ZFS will not free memory in the same way that BTRFS may.  That being said, the additional configuration options available in ZFS make it a good choice for cold storage when compression ratios are of the utmost importance.
+
+Gravwell recommends ZFS when transparent compression is desired for a cold storage tier and compression efficiency is more important than raw speed.  Setting the block size to 1MB and the compression system to zstd-10 can yield impressive compression ratios that still perform well.  ZFS however is significantly slower than BTRFS when using transparent compression and a fast storage device.  ZFS also does not support the ability to disable copy-on-write and compression on a per file basis, so ZFS will attempt to compress and fragment highly orthogonal data structures like well indexes.
+
+### NFSv4
+
+Some customers desire storage arrays to be fully remote with dedicated storage appliances doing the dirty work of data management.  Gravwell tentatively supports NFSv4 with a few caveats.  The filesystem must be configured with all supporting daemons and mount options such that file permissions can be properly mapped to the NFS volume.  While it is possible to disable user/group management on NFS entirely, this is not recommended.
+
+Gravwell Indexers also maintain long-lived file handles with very high I/O requirements. NFS, being a network filesystem, suffers from network interruptions, which can cause process hangs, unexpected performance drops, and increased complexity of management.  Gravwell only tests on NFSv4 and generally does not recommend it.
+
+
+## Unsupported Filesystems
+
+Gravwell requires full, robust POSIX compatibility. The following filesystems are not supported at all.  Gravwell may still function, but we make no guarantees about performance, reliability, or correctness.
+
+* FAT32
+* VFAT
+* NTFS
+* SMB/CIFS
+* FUSE mounts
+
+Other POSIX compliant filesystems like EXT2, EXT3, and ReiserFS are not tested.  Cluster filesystems such as GlusterFS, LusterFS, and CephFS are fully POSIX compliant and customers have reported good results, however Gravwell has not done extensive testing and does not officially support them.

configuration/parameters.md

@@ -312,6 +312,7 @@ Default Value:
 Example:        `Web-Listen-Address=10.0.0.1`  
 Description:        The Web-Listen-Address parameter specifies the address the webserver should bind to and serve from.  By default the parameter is empty, meaning the webserver binds to all interfaces and addresses.
 
+(remote-indexers-conig)=
 ### **Remote-Indexers**
 Applies to:        Webserver  
 Default Value:        `net:10.0.0.1:9404`  

configuration/replication.md

@@ -1,3 +1,4 @@
+(data-replication)=
 # Data Replication
 
 Replication is included with all Gravwell Cluster Edition licenses, allowing for fault-tolerant high availability deployments.  The Gravwell replication engine transparently manages data replication across distributed indexers with automatic failover, load balanced data distribution, and compression.  Gravwell also provides fine tuned control over exactly which wells are included in replication and how the data is distributed across peers.  Customers can rapidly deploy a Gravwell cluster with uniform data distribution, or design a replication scheme that can tolerate entire data center failures using region-aware peer selection.  The online failover system also allows continued access to data even when some indexers are offline.
@@ -113,7 +114,7 @@ Replication is controlled by the "Replication" configuration group in the gravwe
 | Connect-Wait-Timeout | Connect-Wait-Timeout=30 | Specifies the number of seconds an Indexer should wait when attempting to connect to replication peers during startup. |
 | Disable-Server | Disable-Server=true | Disable the indexer replication server, it will only act as a client.  This is important when using offline replication. | 
 | Disable-Compression | Disable-Compression=true | Disable compression on the storage for the replicated data. |
-| Enable-Transparent-Compression | Enable-Transparent-Compression=true | Enable transparent compression on using the host file system for replicated data. |
+| Enable-Transparent-Compression | Enable-Transparent-Compression=true | Enable transparent compression on using the host filesystem for replicated data. |
 | Enable-Transport-Compression | Enable-Transparent-Compression=true | Enable transport compression when transmitting data to replication peer.  Defaults to `true`. |
 
 ## Disabling Replication Per Well