Merge pull request #1118 from gravwell/dev

chore: Merge dev to main

_static/versions.json

@@ -1,10 +1,14 @@
 [
   {
-    "name": "v5.5.3 (latest)",
-    "version": "v5.5.3",
+    "name": "v5.5.4 (latest)",
+    "version": "v5.5.4",
     "url": "https://docs.gravwell.io/",
     "preferred": true
   },
+  {
+    "version": "v5.5.3",
+    "url": "https://docs.gravwell.io/v5.5.3/"
+  },
   {
     "version": "v5.5.2",
     "url": "https://docs.gravwell.io/v5.5.2/"

admin/limits.md

@@ -1,11 +1,29 @@
 # Controlling System Resource Usage
 
-This section describes how you can tune the Gravwell system's consumption of system resources.
+This page describes the limits Gravwell places on user's consumption of system resources. These limits are important because incautious users can otherwise make the system unusable for others, by filling up the disk, causing an out-of-memory (OOM) situation, or just by making everyone else's queries extremely slow.
+
+If you've been linked to this page from within the Gravwell UI, one of your queries likely hit one of the limits described here. Besides descriptions of the limit and why it's important, we also offer suggestions for how to avoid hitting these limits.
 
 ## Webserver Renderer Storage
 
-Render modules such as chart and table store their results on-disk on the webserver. Depending on the search, the results may consume significant amounts of disk space. To prevent users from filling up the disk with over-large queries, use the `Render-Store-Limit` parameter in the webserver's gravwell.conf file. Setting `Render-Store-Limit=64`, for instance, would set a limit of 64 MB of on-disk storage per query.
+Render modules such as chart and table store their results on-disk on the webserver. Depending on the search, the results may consume significant amounts of disk space. For instance, the query `tag=*` tells Gravwell to fetch every entry in the system from the specified timeframe; in a large production environment, this could be terabytes or even petabytes of data! Therefore, the webserver will stop storing data after a certain threshold is reached, to avoid filling up the disk.
+
+If you're seeing messages about render storage limits when running your searches, consider the following options:
+
+* If you're just trying to get an idea of what raw entries look like, consider using the Preview timeframe. Running `tag=foo` across the Preview timeframe tells Gravwell to fetch a small number of the most recent entries, just enough to get an idea of the data.
+* If you want to see a few entries from a particular timeframe (and thus can't use Preview), try the limit module: rather than running `tag=foo`, try `tag=foo limit 100`.
+* Often, users want to see how *much* data exists in a timeframe, so they run `tag=foo` and look at the overview chart. It's much more efficient to instead run `tag=foo chart` (by default, the `chart` renderer will chart the entry count).
+  - Running `tag=foo ax | chart`, if an autoextractor exists for the tag, will also populate the Fields tab of the search results with useful stats about the most common values in each field of the entries.
+
+### Controlling Renderer Storage
+
+To prevent users from filling up the disk with over-large queries, use the `Render-Store-Limit` parameter in the webserver's gravwell.conf file. Setting `Render-Store-Limit=64`, for instance, would set a limit of 64 MB of on-disk storage per query. The default is 1024MB; this gives users a great deal of space to work with while hopefully preventing free space issues on modern disks.
+
+## Gravwell Resource Size
+
+User-created [resources](/resources/resources) can take up a lot of space on disk, on both the webserver and the indexers. In addition, one of the most common uses of resources is to provide lookup tables; in order to use a resource as a lookup table, the *entire* resource must be loaded into memory, meaning that running many simultaneous queries with extremely large lookup tables can put the system at risk of an out-of-memory state. The `Resource-Max-Size` parameter in gravwell.conf specifies a limit, in bytes, for resource size. Thus, setting `Resource-Max-Size=20971520` will limit resources to no more than 20 megabytes.
 
-## Limiting Gravwell Resource Size
+If you're seeing error messages indicating that your resource exceeds the maximum size, consider the following options:
 
-User-created [resources](/resources/resources) can take up a lot of space on disk, on both the webserver and the indexers. The `Resource-Max-Size` parameter in gravwell.conf specifies a limit, in bytes, for resource size. Thus, setting `Resource-Max-Size=20971520` will limit resources to no more than 20 megabytes.
+* Consider if you really need all the columns in your lookup table, especially if you're generating the table from a query. If you only need to match MAC addresses to hostnames in your queries, don't say `tag=inventory ax | unique mac | table -save inv_lookup`, because that will build a table with columns for *every* field on the data; instead, extract only what you need: `tag=inventory ax mac hostname | unique mac | table -save inv_lookup`.
+* If you're building resources to do lookups on IP addresses or subnets, check if the specialized formats used by the [ipexist](/search/ipexist/ipexist) or [iplookup](/search/iplookup/iplookup) modules might be more suitable to your needs. These special types of lookup tables can be more space-efficient for some needs.

changelog/5.5.4.md

@@ -0,0 +1,9 @@
+# Changelog for version 5.5.4
+
+## Released 20 September 2024
+
+## Gravwell
+
+### Bug Fixes
+
+* Fixed an issue where the stats module would incorrectly calculate the maximum tracked elements.

changelog/list.md

@@ -7,7 +7,7 @@
 maxdepth: 1
 caption: Current Release
 ---
-5.5.3 <5.5.3>
+5.5.4 <5.5.4>
 ```
 
 ## Previous Versions
@@ -18,6 +18,7 @@ maxdepth: 1
 caption: Previous Releases
 ---
 
+5.5.3 <5.5.3>
 5.5.2 <5.5.2>
 5.5.1 <5.5.1>
 5.5.0 <5.5.0>

conf.py

@@ -21,7 +21,7 @@
 project = "Gravwell"
 copyright = f"Gravwell, Inc. {date.today().year}"
 author = "Gravwell, Inc."
-release = "v5.5.3"
+release = "v5.5.4"
 
 # -- General configuration ---------------------------------------------------
 # https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration

configuration/certificates.md

@@ -10,6 +10,22 @@ The Gravwell administrator has three options for certificates:
 * Install a properly-signed TLS certificate. This is the ideal configuration, but typically requires the Gravwell instance to have a publicly-accessible hostname.
 * Install a self-signed certificate. This makes sense when you want to encrypt traffic to Gravwell but for one reason or another cannot get a properly signed certificate.
 
+## Hostnames in TLS Certificates
+
+When connecting to a TLS-secured service, the client will check that the server's certificate is valid for the *hostname* the client is trying to access. If you have a valid certificate for `gravwell.example.org`, but you decide to access your Gravwell server directly by IP, e.g. `https://10.0.0.2/`, your web browser will most likely warn you that the certificate presented is invalid. Components of a Gravwell cluster may also communicate with each other over TLS-secured connections, and unless they are specifically configured to skip certificate validation these services will *fail* if the hostname they are given doesn't match the names on the certificate.
+
+When provisioning your TLS certificates, make sure the Common Name (CN) and Subject Alternative Names (SANs) cover *all* hostnames or IP addresses you will ever use to connect to the system. If you have multiple DNS entries for the same system -- for example, if your webserver is accessible from both the Internet (`gravwell.example.org`) and within your own LAN namespace (`gravwell.internal.lan`) -- be sure all those names are in the certificate!
+
+Here's a brief list of some of the ways Gravwell components communicate via TLS connections; make sure your certificates are appropriate for any of these situations you intend to deploy:
+
+- Ingesters talk to indexers (`Encrypted-Backend-Target` in the [ingester config file](/ingesters/ingesters))
+- Indexers talk to other indexers via [replication](/configuration/replication)
+- Indexers talk to the [cloud archive service](/configuration/archive)
+- Webservers in a cluster talk to the [datastore](/distributed/frontend)
+- Webservers in a cluster talk to other webservers (`External-Addr`, see [docs](/distributed/frontend))
+
+Generally, you will see entries in the relevant component's log files if there is a TLS problem; if you see logs about invalid certificates, but you believe that your certificate should be correct, double-check the hostnames you've set in the config files against the hostnames present in the certificate's CN and SAN fields.
+
 ## Allowed TLS Ciphers
 
 A number of TLS ciphers are considered cryptographically insecure, so Gravwell only supports the following TLS ciphers:

ingesters/win_file_follow.md

@@ -14,7 +14,7 @@ Download the Gravwell Windows File Follower installer:
 
 | Ingester Name | Installer    | More Info |
 | :------------ | :----------- | :-------- |
-| Windows File Follower | <a data-bs-custom-class="hash-popover" href="https://update.gravwell.io/archive/5.5.3/installers/gravwell_file_follow_5.5.3.1.msi">Download <i class="fa-solid fa-download"></i></a>&nbsp;&nbsp;&nbsp;<a data-bs-custom-class="hash-popover" href="javascript:void(0);" data-bs-toggle="popover" data-bs-placement="bottom" data-bs-html="true" data-bs-content='<code class="docutils literal notranslate"><span class="pre">d146269af0f368a01df2219f0e407fe1bf9bff1248c2fa98cc01e764199e585c</span></code>'>(SHA256)</a> | [Documentation](/ingesters/win_file_follow) |
+| Windows File Follower | <a data-bs-custom-class="hash-popover" href="https://update.gravwell.io/archive/5.5.4/installers/gravwell_file_follow_5.5.4.1.msi">Download <i class="fa-solid fa-download"></i></a>&nbsp;&nbsp;&nbsp;<a data-bs-custom-class="hash-popover" href="javascript:void(0);" data-bs-toggle="popover" data-bs-placement="bottom" data-bs-html="true" data-bs-content='<code class="docutils literal notranslate"><span class="pre">b60af513bd894622ab76a388cb12f0ad05a29831238579be50eddc5771ce6e26</span></code>'>(SHA256)</a> | [Documentation](/ingesters/win_file_follow) |
 
 The Gravwell Windows file follower is installed using a signed MSI package.  Gravwell signs both the Windows executable and MSI installer with our private key pairs, but depending on download volumes, you may see a warning about the MSI being untrusted.  This is due to the way Microsoft "weighs" files.   Basically, as they see more people download and install a given package, it becomes more trustworthy.  Don't worry though, we have a well audited build pipeline and we sign every package.
 

ingesters/winevent.md

@@ -49,7 +49,7 @@ Download the Gravwell Windows Events installer:
 
 | Ingester Name | Installer    | More Info |
 | :------------ | :----------- | :-------- |
-| Windows Events | <a data-bs-custom-class="hash-popover" href="https://update.gravwell.io/archive/5.5.3/installers/gravwell_win_events_5.5.3.1.msi">Download <i class="fa-solid fa-download"></i></a>&nbsp;&nbsp;&nbsp;<a data-bs-custom-class="hash-popover" href="javascript:void(0);" data-bs-toggle="popover" data-bs-placement="bottom" data-bs-html="true" data-bs-content='<code class="docutils literal notranslate"><span class="pre">60b7d966426939399f632d3a70fabb5b1957cafef240012a2884405d1fd3a08c</span></code>'>(SHA256)</a> | [Documentation](/ingesters/winevent) |
+| Windows Events | <a data-bs-custom-class="hash-popover" href="https://update.gravwell.io/archive/5.5.4/installers/gravwell_win_events_5.5.4.1.msi">Download <i class="fa-solid fa-download"></i></a>&nbsp;&nbsp;&nbsp;<a data-bs-custom-class="hash-popover" href="javascript:void(0);" data-bs-toggle="popover" data-bs-placement="bottom" data-bs-html="true" data-bs-content='<code class="docutils literal notranslate"><span class="pre">dbccf1914a85b6c1ff5d76a76c7e977971e363d64ded9c8c056718faa3058759</span></code>'>(SHA256)</a> | [Documentation](/ingesters/winevent) |
 
 Run the .msi installation wizard to install the Gravwell events service.  On first installation the installation wizard will prompt to configure the indexer endpoint and ingest secret.  Subsequent installations and/or upgrades will identify a resident configuration file and will not prompt.