From 77cb2c15f2d6e180ec70c87878286c3bcc6bf4ee Mon Sep 17 00:00:00 2001
From: kristopher watts <kris.watts@gravwell.io>
Date: Fri, 20 Aug 2021 10:17:09 -0600
Subject: [PATCH 1/6] cleaning up the notifications APIs in anko a bit
---
scripting/scriptingsearch.md | 19 ++++++++++++++++---
1 file changed, 16 insertions(+), 3 deletions(-)
diff --git a/scripting/scriptingsearch.md b/scripting/scriptingsearch.md
index 5754cc8b..524aedda 100644
--- a/scripting/scriptingsearch.md
+++ b/scripting/scriptingsearch.md
@@ -202,11 +202,24 @@ The following functions are deprecated but still available, allowing emails to b
## Creating Notifications
-Scripts may create notifications targeted at the script owner. A notification consists of an integer ID, a string message, an optional HTTP link, and an expiration. If the expiration is in the past, or more than 24 hours in the future, Gravwell will instead set the expiration to be 12 hours.
+Scripts may create notifications targeted at the script owner. A notification consists of an integer ID, a string message, an optional HTTP link, and an expiration.
- addSelfTargetedNotification(7, "This is my notification", "https://gravwell.io", time.Now().Add(3 * time.Hour)
+* `addSelfTargetedNotification(uint32, string, string, time.Time) error`
-The notification ID uniquely identifies the notification. This allows the user to update existing notifications by calling the function again with the same notification ID, but it also allows the user to add multiple simultaneous notifications by specifying different IDs.
+If the expiration is in the past, or more than 24 hours in the future, Gravwell will instead set the expiration to be 12 hours. The notification ID uniquely identifies the notification. This allows the user to update existing notifications by calling the function again with the same notification ID, but it also allows the user to add multiple simultaneous notifications by specifying different IDs.
+
+### Example Notification Creation Script
+
+This script will create a notification that is targeted at the current user, it contains a link and expires 12 hours after the notification is created.
+
+```
+var time = import("time")
+MSG=`This is my notification`
+ID=0x7
+LINK="https://gravwell.io"
+EXPIRES=time.Now().Add(3*time.Hour)
+return addSelfTargetedNotification(ID, MSG, LINK, EXPIRES)
+```
## Creating and Ingesting Entries
From ea0edf8ea9f5512b70488395e8af3178e36f815a Mon Sep 17 00:00:00 2001
From: kristopher watts <kris.watts@gravwell.io>
Date: Tue, 31 Aug 2021 16:02:36 -0600
Subject: [PATCH 2/6] added nosort module documentation
---
search/complete-module-list.md | 1 +
search/nosort/nosort.md | 19 +++++++++++++++++++
search/processingmodules.md | 1 +
3 files changed, 21 insertions(+)
create mode 100644 search/nosort/nosort.md
diff --git a/search/complete-module-list.md b/search/complete-module-list.md
index 51ce4b28..4f4ada7d 100644
--- a/search/complete-module-list.md
+++ b/search/complete-module-list.md
@@ -45,6 +45,7 @@
* [min](math/math.md#Min)
* [namedfields](namedfields/namedfields.md)
* [netflow](netflow/netflow.md)
+* [nosort](nosort/nosort.md)
* [numbercard](gauge/gauge.md)
* [packet](packet/packet.md)
* [packetlayer](packetlayer/packetlayer.md)
diff --git a/search/nosort/nosort.md b/search/nosort/nosort.md
new file mode 100644
index 00000000..9c31ba8e
--- /dev/null
+++ b/search/nosort/nosort.md
@@ -0,0 +1,19 @@
+## Nosort
+
+By default, everything in the Gravwell search pipeline is temporally sorted (when appropriate). This means that if you execute the query `tag=gravwell` Gravwell will automatically insert a `sort by time desc` so that the data you see is strictly sorted.
+
+However, there may be times where the extra overhead from the sort may not be required or explicitly not wanted; this is where `nosort` comes into play. The `nosort` module does nothing but inform Gravwell that you explicitly do not want the data sorted at any stage, it basically turns off the `sort by time` injection.
+
+The `nosort` module is purely for query optimization and is never required, don't use it unless you really know what you are doing. The `nosort` module has a single optional flag `-asc` that tells Gravwell that you don not care about explicit time sorting but you would like the data read from oldest to newest (roughly). The `-asc` flag is useful on big aggregate queries because it means that we will likely pull data in the order it was ingested which means the disks are probably moving in a more or less linear pattern.
+
+### Examples
+
+`tag=syslog nosort`
+
+Show syslog entries in a rough ordering, do not strictly sort them by time.
+
+`tag=syslog nosort -asc`
+
+Show syslog entries in a rough ordering from oldest to newest, do not strictly sort them by time.
+
+Note: The `nosort` module collapses the pipeline, this means that if you are running on a cluster environment it may actually reduce the query performance if you place it in the wrong location.
diff --git a/search/processingmodules.md b/search/processingmodules.md
index 7493bb5f..685895dc 100644
--- a/search/processingmodules.md
+++ b/search/processingmodules.md
@@ -57,6 +57,7 @@ These can be used just like user-defined enumerated values, thus `table foo bar
* [max](math/math.md#Max)
* [mean](math/math.md#Mean)
* [min](math/math.md#Min)
+* [nosort](nosort/nosort.md)
* [packetlayer](packetlayer/packetlayer.md)
* [regex](regex/regex.md)
* [require](require/require.md)
From 1180dbcf64f7bf62b2cc42edf853c5aec2eef855 Mon Sep 17 00:00:00 2001
From: kristopher watts <kris.watts@gravwell.io>
Date: Tue, 31 Aug 2021 16:07:25 -0600
Subject: [PATCH 3/6] adding in IPv4 and IPv6 extractions to the wiki
---
search/slice/slice.md | 59 +++++++++++++++++++++++--------------------
1 file changed, 32 insertions(+), 27 deletions(-)
diff --git a/search/slice/slice.md b/search/slice/slice.md
index 865fb514..5b6cc189 100644
--- a/search/slice/slice.md
+++ b/search/slice/slice.md
@@ -49,6 +49,8 @@ An integral function of the slice module is casting the data to the appropriate
* float64be
* array
* string
+* IPv4
+* IPv6
### Inline filtering
@@ -71,31 +73,34 @@ The slice module supports inline filtering which allows for very fast processing
Type | == | != | ~ | !~ | < | <= | > | >=
----------|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:
-byte | X | X | | | X | X | X | X
-int16 | X | X | | | X | X | X | X
-int16le | X | X | | | X | X | X | X
-int16be | X | X | | | X | X | X | X
-uint16 | X | X | | | X | X | X | X
-uint16le | X | X | | | X | X | X | X
-uint16be | X | X | | | X | X | X | X
-int32 | X | X | | | X | X | X | X
-int32le | X | X | | | X | X | X | X
-int32be | X | X | | | X | X | X | X
-uint32 | X | X | | | X | X | X | X
-uint32le | X | X | | | X | X | X | X
-uint32be | X | X | | | X | X | X | X
-int64 | X | X | | | X | X | X | X
-int64le | X | X | | | X | X | X | X
-int64be | X | X | | | X | X | X | X
-uint64 | X | X | | | X | X | X | X
-uint64le | X | X | | | X | X | X | X
-uint64be | X | X | | | X | X | X | X
-float32 | X | X | | | X | X | X | X
-float32le| X | X | | | X | X | X | X
-float32be| X | X | | | X | X | X | X
-float64 | X | X | | | X | X | X | X
-float64le| X | X | | | X | X | X | X
-float64be| X | X | | | X | X | X | X
-array | X | X | X | X | | | |
-string | X | X | X | X | | | |
+byte | X | X | | | X | X | X | X
+int16 | X | X | | | X | X | X | X
+int16le | X | X | | | X | X | X | X
+int16be | X | X | | | X | X | X | X
+uint16 | X | X | | | X | X | X | X
+uint16le | X | X | | | X | X | X | X
+uint16be | X | X | | | X | X | X | X
+int32 | X | X | | | X | X | X | X
+int32le | X | X | | | X | X | X | X
+int32be | X | X | | | X | X | X | X
+uint32 | X | X | | | X | X | X | X
+uint32le | X | X | | | X | X | X | X
+uint32be | X | X | | | X | X | X | X
+int64 | X | X | | | X | X | X | X
+int64le | X | X | | | X | X | X | X
+int64be | X | X | | | X | X | X | X
+uint64 | X | X | | | X | X | X | X
+uint64le | X | X | | | X | X | X | X
+uint64be | X | X | | | X | X | X | X
+float32 | X | X | | | X | X | X | X
+float32le| X | X | | | X | X | X | X
+float32be| X | X | | | X | X | X | X
+float64 | X | X | | | X | X | X | X
+float64le| X | X | | | X | X | X | X
+float64be| X | X | | | X | X | X | X
+array | X | X | X | X | | | |
+string | X | X | X | X | | | |
+IPv4 | X | X | X | X | | | |
+IPv6 | X | X | X | X | | | |
+Note: The `IPv4` and `IPv6` operators expect 4 and 16 byte network encoded values, text encoding of IP addresses will not extract appropriately.
From af8defffca816d233811a7a2c52de407d6a6453b Mon Sep 17 00:00:00 2001
From: David Fritz <fritz@gravwell.io>
Date: Fri, 10 Sep 2021 09:21:46 -0600
Subject: [PATCH 4/6] move entropy to its own article
---
search/complete-module-list.md | 2 +-
search/entropy/entropy.md | 41 ++++++++++++++++++++++++++++++++++
search/math/math.md | 22 ------------------
search/processingmodules.md | 2 +-
4 files changed, 43 insertions(+), 24 deletions(-)
create mode 100644 search/entropy/entropy.md
diff --git a/search/complete-module-list.md b/search/complete-module-list.md
index 4f4ada7d..d66333cb 100644
--- a/search/complete-module-list.md
+++ b/search/complete-module-list.md
@@ -15,7 +15,7 @@
* [dns](dns/dns.md)
* [dump](dump/dump.md)
* [enrich](enrich/enrich.md)
-* [entropy](math/math.md#Entropy)
+* [entropy](entropy/entropy.md)
* [eval](eval/eval.md)
* [fdg](fdg/fdg.md)
* [fields](fields/fields.md)
diff --git a/search/entropy/entropy.md b/search/entropy/entropy.md
new file mode 100644
index 00000000..c7a36764
--- /dev/null
+++ b/search/entropy/entropy.md
@@ -0,0 +1,41 @@
+## Entropy
+
+The `entropy` module calculates the entropy of field values over time. Specifying `entropy` without any arguments will generate the entropy of all entries DATA fields across the search range. The `entropy` module supports temporal search mode allowing for charting of entropy over time. `entropy` can also operate on enumerated values and group by enumerated values. Output values are between 0 and 1.
+
+Syntax:
+
+```
+entropy [enumerated value] [by ...] [over <duration>]
+```
+
+The `entropy` module syntax allows for specifying an enumerated value to calculate entropy over. If not specified, `entropy` will calculate entropy over the entire DATA field. The module also supports specifying one or more arguments to group by, using the `by` keyword. For example, to calcaulte entropy on the enumerated value `foo`, grouped by `bar` and `baz`:
+
+```
+tag=gravwell entropy foo by bar baz
+```
+
+Queries can be temporally grouped over arbitrary time windows using the `over` keyword:
+
+```
+tag=gravwell entropy over 10m
+```
+
+All arguments are optional.
+
+### Supported Options
+
+`entropy` has no flags.
+
+### Examples
+
+This query calculates and charts the entropy of TCP packet payloads based on port:
+
+```
+tag=pcap packet tcp.Port tcp.Payload | entropy Payload by Port | chart entropy by Port
+```
+
+An example query which calculates the entropy of URLS by host and sorts the list based on highest entropy value:
+
+```
+tag=pcap packet tcp.Port==80 ipv4.IP !~ 10.0.0.0/8 tcp.Payload | grep -e Payload GET PUT HEAD POST | regex -e Payload "[A-Z]+\s(?P<url>\S+)\sHTTP\/" | entropy url by IP | table IP entropy
+```
diff --git a/search/math/math.md b/search/math/math.md
index dea2378c..1423ac45 100644
--- a/search/math/math.md
+++ b/search/math/math.md
@@ -100,25 +100,3 @@ tag=pcap packet tcp.DstPort tcp.DstIP | eval DstPort < 1024 | unique DstPort Dst
```
The search above will output every unique combination of IP + port, provided the port is less than 1024. This is a useful way to find servers on a network, for instance.
-
-## Entropy
-
-The entropy module calculates the Entropy of field values over time. Specifying `entropy` without any arguments will generate the entropy of all entries data sections across the search range. The entropy module supports temporal search mode allowing for charting of entropy over time. Entropy can also operate on enumerated values and group using multiple keys similar to other math modules. Entropy output values are between 0 and 1.
-
-An example query which calculates and charts the entropy of TCP packet payloads based on port:
-
-```
-tag=pcap packet tcp.Port tcp.Payload | entropy Payload by Port | chart entropy by Port
-```
-
-An example query which calculates the entropy of URLS by host and sorts the list based on highest entropy value:
-
-```
-tag=pcap packet tcp.Port==80 ipv4.IP !~ 10.0.0.0/8 tcp.Payload | grep -e Payload GET PUT HEAD POST | regex -e Payload "[A-Z]+\s(?P<url>\S+)\sHTTP\/" | entropy url by IP | table IP entropy
-```
-
-The entropy module can take the `-p` flag, which tells it to calculate entropy *per entry*, rather than over a window as normal. The following will calculate the entropy of each windows log entry and display the entropy plus the data:
-
-```
-tag=winlog entropy -p | table DATA entropy
-```
diff --git a/search/processingmodules.md b/search/processingmodules.md
index 685895dc..af239288 100644
--- a/search/processingmodules.md
+++ b/search/processingmodules.md
@@ -37,7 +37,7 @@ These can be used just like user-defined enumerated values, thus `table foo bar
* [diff](diff/diff.md)
* [dns](dns/dns.md)
* [enrich](enrich/enrich.md)
-* [entropy](math/math.md#Entropy)
+* [entropy](entropy/entropy.md)
* [eval](eval/eval.md)
* [first/last](firstlast/firstlast.md)
* [geoip](geoip/geoip.md)
From 20a15b5df86bbd9bfa6c2a67e442e672fda8ca49 Mon Sep 17 00:00:00 2001
From: kristopher watts <kris.watts@gravwell.io>
Date: Fri, 10 Sep 2021 09:52:16 -0600
Subject: [PATCH 5/6] adding changelog for 4.2.2
---
changelog/4.2.2.md | 43 +++++++++++++++++++++++++++++++++++++++++++
changelog/list.md | 3 ++-
2 files changed, 45 insertions(+), 1 deletion(-)
create mode 100644 changelog/4.2.2.md
diff --git a/changelog/4.2.2.md b/changelog/4.2.2.md
new file mode 100644
index 00000000..0ba382bf
--- /dev/null
+++ b/changelog/4.2.2.md
@@ -0,0 +1,43 @@
+# Changelog for version 4.2.2
+
+### Released Sept 10 2021
+
+## Frontend Changes
+* Various UI/UX cleanup.
+* Improved contrast of highlighting in Query Studio.
+* Improved table renderer on Query Studio.
+* Fixed issue where some investigative dashboards were not cleaning up queries quickly when completed.
+* Improved query cleanup logic on Query Studio.
+* Fixed issue where System Hardware stats stop updating when an indexer is down.
+* Fixed issue where history from Query Studio was logging expanded query and stripping formatting.
+* Improved time locking and zoom when user has custom timeframes defined.
+* Improved kit refresh after kit install.
+* Fixed issue where some point2point renderer images were not properly displaying lines.
+* Improved default info handling when adding new users.
+* Improved overview stats so that bucketing lines up on zoom.
+* Improved date detection when zooming on overview chart.
+
+## Backend Changes
+* Fixed issue where some large queries could cause windowing errors in overview chart.
+* Improved webserver compression on HTTP requests.
+* Reduced memory usage when distributing very large resources between webserver and indexers.
+* Fixed issue where the `limit` module was not generating accurate stats.
+* Added version handshake between datastore and webservers.
+* Added validation of backup file when generated by the CLI.
+* Fixed response code when templates are not found.
+* Added `-asc` flag to the [nosort](#!search/nosort/nosort.md) module.
+* Fixed issue where extremely large storage blocks could cause queries to stall in cluster mode.
+* Fixed issue where syslog was not dropping entries with missing `Message` fields when a filter was applied.
+* Improved default charting logic when in `chart` renderer when no operators are specified.
+* Fixed issue in `syslog` module where `]` characters were not escaped properly.
+* Improved efficiency of startup when using replication.
+* Added log in `gravwell` tag when a notification is fired.
+* Improved corrupted block recovery when using userland compression.
+* Fixed issue where large web uploads were not cleaning up temporary files when completed.
+* Fixed issue where internal logs were exceeding the 32 character limit for MsgID in RFC5424.
+* Fixed issue where table renderer was not reporting an error when it failed to save a resource.
+* Improved timeout on queries that cover very long time ranges.
+
+## Ingester, CLI, and Library Changes
+* Improved config file error logs to include line number containing the error.
+* Fixed shell installers to appropriately detect docker containers in Debian 11.
diff --git a/changelog/list.md b/changelog/list.md
index 12dac3e5..ef0c09ee 100644
--- a/changelog/list.md
+++ b/changelog/list.md
@@ -2,10 +2,11 @@
## Current Version
-* [4.2.1](4.2.1.md)
+[4.2.2](4.2.2.md)
## Previous Versions
+* [4.2.1](4.2.1.md)
* [4.2.0](4.2.0.md)
* [4.1.9](4.1.9.md)
* [4.1.8](4.1.8.md)
From 83c8e0ae361e71918f82aa9e4eb624fc9cce16c9 Mon Sep 17 00:00:00 2001
From: kristopher watts <kris.watts@gravwell.io>
Date: Fri, 10 Sep 2021 10:14:12 -0600
Subject: [PATCH 6/6] adding notes and running a gofmt
---
changelog/4.2.2.md | 4 ++++
serve.go | 4 +---
2 files changed, 5 insertions(+), 3 deletions(-)
diff --git a/changelog/4.2.2.md b/changelog/4.2.2.md
index 0ba382bf..5c125679 100644
--- a/changelog/4.2.2.md
+++ b/changelog/4.2.2.md
@@ -41,3 +41,7 @@
## Ingester, CLI, and Library Changes
* Improved config file error logs to include line number containing the error.
* Fixed shell installers to appropriately detect docker containers in Debian 11.
+
+## Notes
+
+Beginning with version 4.2.2 the webservers and the datastore are version locked. When upgrading you may see notifications and errors indicating that the version handshake failed until both the webservers and datastore have been upgraded. Webservers will continue to attempt to connect to the datastore until the version handshake succeeds.
diff --git a/serve.go b/serve.go
index 859b6e8d..8f4006a6 100644
--- a/serve.go
+++ b/serve.go
@@ -629,12 +629,10 @@ func isChangeLogRef(pg string) bool {
func isDirectSearchRef(pg string, terms []string) (ok bool) {
for _, term := range terms {
- pgSnippet := `search/`+term+`/`
+ pgSnippet := `search/` + term + `/`
if ok = strings.Contains(pg, pgSnippet); ok {
break
}
}
return
}
-
-