Merge pull request #278 from gravwell/dev

Dev->master

changelog/4.2.2.md

@@ -0,0 +1,47 @@
+# Changelog for version 4.2.2
+
+### Released Sept 10 2021
+
+## Frontend Changes
+* Various UI/UX cleanup.
+* Improved contrast of highlighting in Query Studio.
+* Improved table renderer on Query Studio.
+* Fixed issue where some investigative dashboards were not cleaning up queries quickly when completed.
+* Improved query cleanup logic on Query Studio.
+* Fixed issue where System Hardware stats stop updating when an indexer is down.
+* Fixed issue where history from Query Studio was logging expanded query and stripping formatting.
+* Improved time locking and zoom when user has custom timeframes defined.
+* Improved kit refresh after kit install.
+* Fixed issue where some point2point renderer images were not properly displaying lines.
+* Improved default info handling when adding new users.
+* Improved overview stats so that bucketing lines up on zoom.
+* Improved date detection when zooming on overview chart.
+
+## Backend Changes
+* Fixed issue where some large queries could cause windowing errors in overview chart.
+* Improved webserver compression on HTTP requests.
+* Reduced memory usage when distributing very large resources between webserver and indexers.
+* Fixed issue where the `limit` module was not generating accurate stats.
+* Added version handshake between datastore and webservers.
+* Added validation of backup file when generated by the CLI.
+* Fixed response code when templates are not found.
+* Added `-asc` flag to the [nosort](#!search/nosort/nosort.md) module.
+* Fixed issue where extremely large storage blocks could cause queries to stall in cluster mode.
+* Fixed issue where syslog was not dropping entries with missing `Message` fields when a filter was applied.
+* Improved default charting logic when in `chart` renderer when no operators are specified.
+* Fixed issue in `syslog` module where `]` characters were not escaped properly.
+* Improved efficiency of startup when using replication.
+* Added log in `gravwell` tag when a notification is fired.
+* Improved corrupted block recovery when using userland compression.
+* Fixed issue where large web uploads were not cleaning up temporary files when completed.
+* Fixed issue where internal logs were exceeding the 32 character limit for MsgID in RFC5424.
+* Fixed issue where table renderer was not reporting an error when it failed to save a resource.
+* Improved timeout on queries that cover very long time ranges.
+
+## Ingester, CLI, and Library Changes
+* Improved config file error logs to include line number containing the error.
+* Fixed shell installers to appropriately detect docker containers in Debian 11.
+
+## Notes
+
+Beginning with version 4.2.2 the webservers and the datastore are version locked.  When upgrading you may see notifications and errors indicating that the version handshake failed until both the webservers and datastore have been upgraded.  Webservers will continue to attempt to connect to the datastore until the version handshake succeeds.

changelog/list.md

@@ -2,10 +2,11 @@
 
 ## Current Version 
 
-* [4.2.1](4.2.1.md)
+[4.2.2](4.2.2.md)
 
 ## Previous Versions
 
+* [4.2.1](4.2.1.md)
 * [4.2.0](4.2.0.md)
 * [4.1.9](4.1.9.md)
 * [4.1.8](4.1.8.md)

scripting/scriptingsearch.md

@@ -202,11 +202,24 @@ The following functions are deprecated but still available, allowing emails to b
 
 ## Creating Notifications
 
-Scripts may create notifications targeted at the script owner. A notification consists of an integer ID, a string message, an optional HTTP link, and an expiration. If the expiration is in the past, or more than 24 hours in the future, Gravwell will instead set the expiration to be 12 hours.
+Scripts may create notifications targeted at the script owner. A notification consists of an integer ID, a string message, an optional HTTP link, and an expiration.
 
-	addSelfTargetedNotification(7, "This is my notification", "https://gravwell.io", time.Now().Add(3 * time.Hour)
+* `addSelfTargetedNotification(uint32, string, string, time.Time) error`
 
-The notification ID uniquely identifies the notification. This allows the user to update existing notifications by calling the function again with the same notification ID, but it also allows the user to add multiple simultaneous notifications by specifying different IDs.
+If the expiration is in the past, or more than 24 hours in the future, Gravwell will instead set the expiration to be 12 hours. The notification ID uniquely identifies the notification. This allows the user to update existing notifications by calling the function again with the same notification ID, but it also allows the user to add multiple simultaneous notifications by specifying different IDs.
+
+### Example Notification Creation Script
+
+This script will create a notification that is targeted at the current user, it contains a link and expires 12 hours after the notification is created.
+
+```
+var time = import("time")
+MSG=`This is my notification`
+ID=0x7
+LINK="https://gravwell.io"
+EXPIRES=time.Now().Add(3*time.Hour)
+return addSelfTargetedNotification(ID, MSG, LINK, EXPIRES)
+```
 
 ## Creating and Ingesting Entries
 

search/complete-module-list.md

@@ -15,7 +15,7 @@
 * [dns](dns/dns.md)
 * [dump](dump/dump.md)
 * [enrich](enrich/enrich.md)
-* [entropy](math/math.md#Entropy)
+* [entropy](entropy/entropy.md)
 * [eval](eval/eval.md)
 * [fdg](fdg/fdg.md)
 * [fields](fields/fields.md)
@@ -45,6 +45,7 @@
 * [min](math/math.md#Min)
 * [namedfields](namedfields/namedfields.md)
 * [netflow](netflow/netflow.md)
+* [nosort](nosort/nosort.md)
 * [numbercard](gauge/gauge.md)
 * [packet](packet/packet.md)
 * [packetlayer](packetlayer/packetlayer.md)

search/entropy/entropy.md

@@ -0,0 +1,41 @@
+## Entropy
+
+The `entropy` module calculates the entropy of field values over time. Specifying `entropy` without any arguments will generate the entropy of all entries DATA fields across the search range. The `entropy` module supports temporal search mode allowing for charting of entropy over time. `entropy` can also operate on enumerated values and group by enumerated values. Output values are between 0 and 1.
+
+Syntax: 
+
+```
+entropy [enumerated value] [by ...] [over <duration>]
+```
+
+The `entropy` module syntax allows for specifying an enumerated value to calculate entropy over. If not specified, `entropy` will calculate entropy over the entire DATA field. The module also supports specifying one or more arguments to group by, using the `by` keyword. For example, to calcaulte entropy on the enumerated value `foo`, grouped by `bar` and `baz`:
+
+```
+tag=gravwell entropy foo by bar baz
+```
+
+Queries can be temporally grouped over arbitrary time windows using the `over` keyword:
+
+```
+tag=gravwell entropy over 10m
+```
+
+All arguments are optional.
+
+### Supported Options
+
+`entropy` has no flags.
+
+### Examples
+
+This query calculates and charts the entropy of TCP packet payloads based on port:
+
+```
+tag=pcap packet tcp.Port tcp.Payload | entropy Payload by Port | chart entropy by Port
+```
+
+An example query which calculates the entropy of URLS by host and sorts the list based on highest entropy value:
+
+```
+tag=pcap packet tcp.Port==80 ipv4.IP !~ 10.0.0.0/8 tcp.Payload | grep -e Payload GET PUT HEAD POST | regex -e Payload "[A-Z]+\s(?P<url>\S+)\sHTTP\/" | entropy url by IP | table IP entropy
+```

search/math/math.md

@@ -100,25 +100,3 @@ tag=pcap packet tcp.DstPort tcp.DstIP | eval DstPort < 1024 | unique DstPort Dst
 ```
 
 The search above will output every unique combination of IP + port, provided the port is less than 1024. This is a useful way to find servers on a network, for instance.
-
-## Entropy
-
-The entropy module calculates the Entropy of field values over time.  Specifying `entropy` without any arguments will generate the entropy of all entries data sections across the search range.  The entropy module supports temporal search mode allowing for charting of entropy over time.  Entropy can also operate on enumerated values and group using multiple keys similar to other math modules.  Entropy output values are between 0 and 1.
-
-An example query which calculates and charts the entropy of TCP packet payloads based on port:
-
-```
-tag=pcap packet tcp.Port tcp.Payload | entropy Payload by Port | chart entropy by Port
-```
-
-An example query which calculates the entropy of URLS by host and sorts the list based on highest entropy value:
-
-```
-tag=pcap packet tcp.Port==80 ipv4.IP !~ 10.0.0.0/8 tcp.Payload | grep -e Payload GET PUT HEAD POST | regex -e Payload "[A-Z]+\s(?P<url>\S+)\sHTTP\/" | entropy url by IP | table IP entropy
-```
-
-The entropy module can take the `-p` flag, which tells it to calculate entropy *per entry*, rather than over a window as normal. The following will calculate the entropy of each windows log entry and display the entropy plus the data:
-
-```
-tag=winlog entropy -p | table DATA entropy
-```

search/nosort/nosort.md

@@ -0,0 +1,19 @@
+## Nosort
+
+By default, everything in the Gravwell search pipeline is temporally sorted (when appropriate).  This means that if you execute the query `tag=gravwell` Gravwell will automatically insert a `sort by time desc` so that the data you see is strictly sorted.
+
+However, there may be times where the extra overhead from the sort may not be required or explicitly not wanted; this is where `nosort` comes into play.  The `nosort` module does nothing but inform Gravwell that you explicitly do not want the data sorted at any stage, it basically turns off the `sort by time` injection.
+
+The `nosort` module is purely for query optimization and is never required, don't use it unless you really know what you are doing.  The `nosort` module has a single optional flag `-asc` that tells Gravwell that you don not care about explicit time sorting but you would like the data read from oldest to newest (roughly).  The `-asc` flag is useful on big aggregate queries because it means that we will likely pull data in the order it was ingested which means the disks are probably moving in a more or less linear pattern.
+
+### Examples
+
+`tag=syslog nosort`
+
+Show syslog entries in a rough ordering, do not strictly sort them by time.
+
+`tag=syslog nosort -asc`
+
+Show syslog entries in a rough ordering from oldest to newest, do not strictly sort them by time.
+
+Note: The `nosort` module collapses the pipeline, this means that if you are running on a cluster environment it may actually reduce the query performance if you place it in the wrong location.

search/processingmodules.md

@@ -37,7 +37,7 @@ These can be used just like user-defined enumerated values, thus `table foo bar
 * [diff](diff/diff.md)
 * [dns](dns/dns.md)
 * [enrich](enrich/enrich.md)
-* [entropy](math/math.md#Entropy)
+* [entropy](entropy/entropy.md)
 * [eval](eval/eval.md)
 * [first/last](firstlast/firstlast.md)
 * [geoip](geoip/geoip.md)
@@ -57,6 +57,7 @@ These can be used just like user-defined enumerated values, thus `table foo bar
 * [max](math/math.md#Max)
 * [mean](math/math.md#Mean)
 * [min](math/math.md#Min)
+* [nosort](nosort/nosort.md)
 * [packetlayer](packetlayer/packetlayer.md)
 * [regex](regex/regex.md)
 * [require](require/require.md)

search/slice/slice.md

@@ -49,6 +49,8 @@ An integral function of the slice module is casting the data to the appropriate
 * float64be
 * array
 * string
+* IPv4
+* IPv6
 
 ### Inline filtering
 
@@ -71,31 +73,34 @@ The slice module supports inline filtering which allows for very fast processing
 
 Type     | == | != | ~ | !~ | < | <= | > | >=
 ----------|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:
-byte     | X | X |  |  | X | X | X | X 
-int16    | X | X |  |  | X | X | X | X
-int16le  | X | X |  |  | X | X | X | X
-int16be  | X | X |  |  | X | X | X | X
-uint16   | X | X |  |  | X | X | X | X
-uint16le | X | X |  |  | X | X | X | X
-uint16be | X | X |  |  | X | X | X | X 
-int32    | X | X |  |  | X | X | X | X
-int32le  | X | X |  |  | X | X | X | X
-int32be  | X | X |  |  | X | X | X | X
-uint32   | X | X |  |  | X | X | X | X
-uint32le | X | X |  |  | X | X | X | X
-uint32be | X | X |  |  | X | X | X | X
-int64    | X | X |  |  | X | X | X | X
-int64le  | X | X |  |  | X | X | X | X
-int64be  | X | X |  |  | X | X | X | X
-uint64   | X | X |  |  | X | X | X | X
-uint64le | X | X |  |  | X | X | X | X
-uint64be | X | X |  |  | X | X | X | X
-float32  | X | X |  |  | X | X | X | X
-float32le| X | X |  |  | X | X | X | X
-float32be| X | X |  |  | X | X | X | X
-float64  | X | X |  |  | X | X | X | X
-float64le| X | X |  |  | X | X | X | X
-float64be| X | X |  |  | X | X | X | X
-array    | X | X | X | X |  |  |  |
-string   | X | X | X | X |  |  |  |
+byte     | X | X |   |   | X | X | X | X 
+int16    | X | X |   |   | X | X | X | X
+int16le  | X | X |   |   | X | X | X | X
+int16be  | X | X |   |   | X | X | X | X
+uint16   | X | X |   |   | X | X | X | X
+uint16le | X | X |   |   | X | X | X | X
+uint16be | X | X |   |   | X | X | X | X 
+int32    | X | X |   |   | X | X | X | X
+int32le  | X | X |   |   | X | X | X | X
+int32be  | X | X |   |   | X | X | X | X
+uint32   | X | X |   |   | X | X | X | X
+uint32le | X | X |   |   | X | X | X | X
+uint32be | X | X |   |   | X | X | X | X
+int64    | X | X |   |   | X | X | X | X
+int64le  | X | X |   |   | X | X | X | X
+int64be  | X | X |   |   | X | X | X | X
+uint64   | X | X |   |   | X | X | X | X
+uint64le | X | X |   |   | X | X | X | X
+uint64be | X | X |   |   | X | X | X | X
+float32  | X | X |   |   | X | X | X | X
+float32le| X | X |   |   | X | X | X | X
+float32be| X | X |   |   | X | X | X | X
+float64  | X | X |   |   | X | X | X | X
+float64le| X | X |   |   | X | X | X | X
+float64be| X | X |   |   | X | X | X | X
+array    | X | X | X | X |   |   |   |
+string   | X | X | X | X |   |   |   |
+IPv4     | X | X | X | X |   |   |   |
+IPv6     | X | X | X | X |   |   |   |
 
+Note: The `IPv4` and `IPv6` operators expect 4 and 16 byte network encoded values, text encoding of IP addresses will not extract appropriately.

serve.go

@@ -629,12 +629,10 @@ func isChangeLogRef(pg string) bool {
 
 func isDirectSearchRef(pg string, terms []string) (ok bool) {
 	for _, term := range terms {
-		pgSnippet := `search/`+term+`/`
+		pgSnippet := `search/` + term + `/`
 		if ok = strings.Contains(pg, pgSnippet); ok {
 			break
 		}
 	}
 	return
 }
-
-