diff --git a/_static/versions.json b/_static/versions.json index 6394a1df..43641f5b 100644 --- a/_static/versions.json +++ b/_static/versions.json @@ -1,10 +1,14 @@ [ { - "name": "v5.4.10 (latest)", - "version": "v5.4.10", + "name": "v5.4.11 (latest)", + "version": "v5.4.11", "url": "https://docs.gravwell.io/", "preferred": true }, + { + "version": "v5.4.10", + "url": "https://docs.gravwell.io/v5.4.10/" + }, { "version": "v5.4.9", "url": "https://docs.gravwell.io/v5.4.9/" diff --git a/changelog/5.4.11.md b/changelog/5.4.11.md new file mode 100644 index 00000000..3186e42c --- /dev/null +++ b/changelog/5.4.11.md @@ -0,0 +1,34 @@ +# Changelog for version 5.4.11 + +## Released 30 July 2024 + +## Gravwell + +```{note} +This release contains high priority bug fixes for ingesters and search modules. +Gravwell highly recommends that all users upgrade to 5.4.11 or above. +``` + +### Additions + +* Added debug interfaces to most components to get stack traces and CPU/Memory profiles using the SIGUSR1 signal. +* Added additional log on query completion that shows local disk usage which helps identify excessively expensive queries. +* Added `-maxtracked` and `-maxsize` flags to the `first`, `last`, and `unique` modules to prevent unexpected resource exhaustion. +* Improved internal logging around shard management. + +### Bug Fixes + +* Fixed an issue where an indexer could drop a file lock when repairing many shards. +* Fixed an issue where multiple overlapping AX modules targeting multiple tags could provide incorrect acceleration hints. +* Fixed an issue where the kits interface could sometimes show resources from other kits. + +## Ingester Changes + +### Additions + +* Improved health check API on HTTP ingester to indicate if the ingester would block on a request. +* Improved resource utilization on the Federator and Indexer when servicing many clients. + +### Bug Fixes + +* Fixed an issue in the ingest library that can cause resource starvation when under extremely high load with many concurrent data flows. diff --git a/changelog/list.md b/changelog/list.md index 6da67d18..1f264506 100644 --- a/changelog/list.md +++ b/changelog/list.md @@ -7,7 +7,7 @@ maxdepth: 1 caption: Current Release --- -5.4.10 <5.4.10> +5.4.11 <5.4.11> ``` ## Previous Versions @@ -18,6 +18,7 @@ maxdepth: 1 caption: Previous Releases --- +5.4.10 <5.4.10> 5.4.9 <5.4.9> 5.4.8 <5.4.8> 5.4.7 <5.4.7> diff --git a/conf.py b/conf.py index b4e64f02..19b78fbc 100644 --- a/conf.py +++ b/conf.py @@ -21,7 +21,7 @@ project = "Gravwell" copyright = f"Gravwell, Inc. {date.today().year}" author = "Gravwell, Inc." -release = "v5.4.10" +release = "v5.4.11" # -- General configuration --------------------------------------------------- # https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration diff --git a/ingesters/win_file_follow.md b/ingesters/win_file_follow.md index 9112b641..3c7ee210 100644 --- a/ingesters/win_file_follow.md +++ b/ingesters/win_file_follow.md @@ -14,7 +14,7 @@ Download the Gravwell Windows File Follower installer: | Ingester Name | Installer | More Info | | :------------ | :----------- | :-------- | -| Windows File Follower | Download    (SHA256) | [Documentation](/ingesters/win_file_follow) | +| Windows File Follower | Download    (SHA256) | [Documentation](/ingesters/win_file_follow) | The Gravwell Windows file follower is installed using a signed MSI package. Gravwell signs both the Windows executable and MSI installer with our private key pairs, but depending on download volumes, you may see a warning about the MSI being untrusted. This is due to the way Microsoft "weighs" files. Basically, as they see more people download and install a given package, it becomes more trustworthy. Don't worry though, we have a well audited build pipeline and we sign every package. diff --git a/ingesters/winevent.md b/ingesters/winevent.md index fa8817f7..9c6f7862 100644 --- a/ingesters/winevent.md +++ b/ingesters/winevent.md @@ -49,7 +49,7 @@ Download the Gravwell Windows Events installer: | Ingester Name | Installer | More Info | | :------------ | :----------- | :-------- | -| Windows Events | Download    (SHA256) | [Documentation](/ingesters/winevent) | +| Windows Events | Download    (SHA256) | [Documentation](/ingesters/winevent) | Run the .msi installation wizard to install the Gravwell events service. On first installation the installation wizard will prompt to configure the indexer endpoint and ingest secret. Subsequent installations and/or upgrades will identify a resident configuration file and will not prompt. diff --git a/quickstart/downloads.md b/quickstart/downloads.md index 4753ce75..d406541a 100644 --- a/quickstart/downloads.md +++ b/quickstart/downloads.md @@ -4,14 +4,14 @@ | Ingester Name | Installer | More Info | | :------------ | :----------- | :-------- | -| Windows Events | Download    (SHA256) | [Documentation](/ingesters/winevent) | -| Windows File Follower | Download    (SHA256) | [Documentation](/ingesters/win_file_follow) | +| Windows Events | Download    (SHA256) | [Documentation](/ingesters/winevent) | +| Windows File Follower | Download    (SHA256) | [Documentation](/ingesters/win_file_follow) | ## macOS Ingesters | Ingester Name | Installer | More Info | | :------------ | :----------- | :-------- | -| File Follower | Download    (SHA256) | [Documentation](/ingesters/file_follow) | +| File Follower | Download    (SHA256) | [Documentation](/ingesters/file_follow) | ## Other Installers @@ -23,7 +23,7 @@ The Debian and RHEL repositories are more easily maintained than these standalon The Gravwell core installer contains the indexer and webserver frontend. You'll need a license; either get a Community Edition free license, or contact info@gravwell.io for commercial options. -Download Gravwell Core Installer Download    (SHA256) +Download Gravwell Core Installer Download    (SHA256) ### Ingesters @@ -32,24 +32,24 @@ The core suite of ingesters are available for download as installable packages. #### Current Ingester Releases | Ingester Name | Installer | More Info | | :------------ | :----------- | :-------- | -| Amazon Kinesis | Download    (SHA256) | [Documentation](/ingesters/kinesis)| -| Amazon S3 | Download    (SHA256) | [Documentation](/ingesters/s3)| -| Amazon SQS | Download    (SHA256) | [Documentation](/ingesters/sqs)| -| Apache Kafka | Download    (SHA256) | [Documentation](/ingesters/kafka)| -| Apache Kafka Federator | Download    (SHA256) | [Documentation](/ingesters/federators/kafkafederator)| -| Collectd Collector | Download    (SHA256) | [Documentation](/ingesters/collectd) | -| File Follower | Download    (SHA256) | [Documentation](/ingesters/file_follow) | -| Google PubSub | Download    (SHA256) | [Documentation](/ingesters/pubsub)| -| HTTP Ingester | Download    (SHA256) | [Documentation](/ingesters/http) | -| Ingest Federator | Download    (SHA256) | [Documentation](/ingesters/federators/federator) | -| IPMI Ingester | Download    (SHA256) | [Documentation](/ingesters/ipmi)| -| Microsoft Azure EventHub | Download    (SHA256) | [Documentation](/ingesters/eventhubs)| -| Microsoft Graph API | Download    (SHA256) | [Documentation](/ingesters/msg)| -| Netflow Capture | Download    (SHA256) | [Documentation](/ingesters/netflow) | -| Network Capture | Download    (SHA256) | [Documentation](/ingesters/pcap) | -| Office 365 Logs | Download    (SHA256) | [Documentation](/ingesters/o365)| -| Simple Relay | Download    (SHA256) | [Documentation](/ingesters/simple_relay)| -| SNMP Traps | Download    (SHA256) | [Documentation](/ingesters/snmp)| +| Amazon Kinesis | Download    (SHA256) | [Documentation](/ingesters/kinesis)| +| Amazon S3 | Download    (SHA256) | [Documentation](/ingesters/s3)| +| Amazon SQS | Download    (SHA256) | [Documentation](/ingesters/sqs)| +| Apache Kafka | Download    (SHA256) | [Documentation](/ingesters/kafka)| +| Apache Kafka Federator | Download    (SHA256) | [Documentation](/ingesters/federators/kafkafederator)| +| Collectd Collector | Download    (SHA256) | [Documentation](/ingesters/collectd) | +| File Follower | Download    (SHA256) | [Documentation](/ingesters/file_follow) | +| Google PubSub | Download    (SHA256) | [Documentation](/ingesters/pubsub)| +| HTTP Ingester | Download    (SHA256) | [Documentation](/ingesters/http) | +| Ingest Federator | Download    (SHA256) | [Documentation](/ingesters/federators/federator) | +| IPMI Ingester | Download    (SHA256) | [Documentation](/ingesters/ipmi)| +| Microsoft Azure EventHub | Download    (SHA256) | [Documentation](/ingesters/eventhubs)| +| Microsoft Graph API | Download    (SHA256) | [Documentation](/ingesters/msg)| +| Netflow Capture | Download    (SHA256) | [Documentation](/ingesters/netflow) | +| Network Capture | Download    (SHA256) | [Documentation](/ingesters/pcap) | +| Office 365 Logs | Download    (SHA256) | [Documentation](/ingesters/o365)| +| Simple Relay | Download    (SHA256) | [Documentation](/ingesters/simple_relay)| +| SNMP Traps | Download    (SHA256) | [Documentation](/ingesters/snmp)| ### Other downloads @@ -57,8 +57,8 @@ Some Gravwell components are distributed as optional additional installers, such | Component Name | Installer | More Info | | :------------- | :----------- | :-------- | -| Datastore | Download    (SHA256) | [Documentation](/distributed/frontend) | -| Cloud Archive Server | Download    (SHA256) | [Documentation](/configuration/archive) | -| Offline Replicator | Download    (SHA256) | [Documentation](/configuration/replication) | -| Load Balancer | Download    (SHA256) | [Documentation](/distributed/loadbalancer) | -| Gravwell Tools | Download    (SHA256) | [Documentation](/tools/tools)| +| Datastore | Download    (SHA256) | [Documentation](/distributed/frontend) | +| Cloud Archive Server | Download    (SHA256) | [Documentation](/configuration/archive) | +| Offline Replicator | Download    (SHA256) | [Documentation](/configuration/replication) | +| Load Balancer | Download    (SHA256) | [Documentation](/distributed/loadbalancer) | +| Gravwell Tools | Download    (SHA256) | [Documentation](/tools/tools)| diff --git a/quickstart/quickstart.md b/quickstart/quickstart.md index e8dea0d7..c388e431 100644 --- a/quickstart/quickstart.md +++ b/quickstart/quickstart.md @@ -19,7 +19,7 @@ This guide is suitable for Community Edition users as well as users with a paid You may find the [installation checklist](checklist) and the [glossary](/glossary/glossary) useful companions to this document. -If you are interested in a complete training package, please see the [complete training PDF](https://github.com/gravwell/training/releases/download/v5.4.10/gravwell_training_v5.4.10.pdf). The Gravwell training PDF is the complete training manual which is paired with labs and exercises. The exercises are built from the open source [Gravwell Training](https://github.com/gravwell/training) repository. +If you are interested in a complete training package, please see the [complete training PDF](https://github.com/gravwell/training/releases/download/v5.4.11/gravwell_training_v5.4.11.pdf). The Gravwell training PDF is the complete training manual which is paired with labs and exercises. The exercises are built from the open source [Gravwell Training](https://github.com/gravwell/training) repository. ```{note} Community Edition users will need to obtain their own license from [https://www.gravwell.io/download](https://www.gravwell.io/download) before beginning installation. Paid users should already have received a license file via email. diff --git a/search/firstlast/firstlast.md b/search/firstlast/firstlast.md index 422151d7..078e1c1a 100644 --- a/search/firstlast/firstlast.md +++ b/search/firstlast/firstlast.md @@ -8,6 +8,11 @@ The modules use the same syntax. Each optionally takes one or more enumerated va first [enumerated value]... ``` +## Supported Options + +* `-maxtracked `: sets the maximum number of unique keys to track per operation, e.g. `first -maxtracked 5000 DstIP`. This is used to help avoid memory exhaustion if there are millions of IPv6 addresses in the data. If the maxtracked value is exceeded, the search will terminate with an error suggesting you should increase the max value. Defaults to 100000000. Refer to the [stats module documentation](/search/stats/stats) for more information about maxtracked. +* `-maxsize `: sets the maximum amount of memory in megabytes to hold when tracking keys. + ## Examples To get just the first entry in a query by time, simply invoke the `first` module with no arguments: diff --git a/search/text/dataexplorer.png b/search/text/dataexplorer.png new file mode 100644 index 00000000..79d31e55 Binary files /dev/null and b/search/text/dataexplorer.png differ diff --git a/search/text/text.md b/search/text/text.md index 55754d73..70a2f77e 100644 --- a/search/text/text.md +++ b/search/text/text.md @@ -1,6 +1,9 @@ # Text The text renderer is designed to show human readable entries in a text format. Any non-printable characters will be converted to the ‘.’ character. Text also fully supports Unicode and can render non-ASCII characters. Text is the default renderer and is applied if no renderer is specified. -Text also has a default limit of approximately 1000 characters per entry, to prevent accidentally displaying multiple megabytes of raw data. To increase the maximum length of output add the `limit ` argument, where `n` is the number of characters to display. -Example: `text limit 4096` \ No newline at end of file +![](text.png) + +The text renderer is also the main interface to [Data Explorer](/gui/data-explorer/de). + +![](dataexplorer.png) diff --git a/search/text/text.png b/search/text/text.png new file mode 100644 index 00000000..220fe26b Binary files /dev/null and b/search/text/text.png differ diff --git a/search/unique/unique.md b/search/unique/unique.md index 8badd595..5a19cde8 100644 --- a/search/unique/unique.md +++ b/search/unique/unique.md @@ -6,6 +6,7 @@ The unique module eliminates duplicate entries in the query data. ## Supported Options * `-maxtracked`: sets the maximum number of unique keys to track per operation, e.g. `unique -maxtracked 5000 DstIP`. This is used to help avoid memory exhaustion if there are millions of IPv6 addresses in the data. If the maxtracked value is exceeded, the search will terminate with an error suggesting you should increase the max value. Defaults to 100000000. Refer to the [stats module documentation](/search/stats/stats) for more information about maxtracked. +* `-maxsize `: sets the maximum amount of memory in megabytes to hold when tracking keys. ## Usage