Skip to content

Commit 7cf5baf

Browse files
andrewkrohandrewkroh
committedJun 3, 2020
Add Okta module documentation, config cleanup, _id field
This add documentation for the Okta module. It contains descriptions of the configuration options and general information about the module. I fixed an issue with the module where it was not setting the _id field for Elasticsearch events. I also did some cleanup to the pipeline.js (indentation, semi-colons, strict equality checks). The module's manifest was updated to not duplicate httpjson's default values. The module was accepting configuration as JSON strings for some parameters (http_headers, http_request_body, pagination, rate_limit, ssl) which is inconsistent with how other parts of Beats are configured so I removed this. Now these options expect regular YAML objects for values. None of these options are required to use the module so the impact to users should be minimal.
1 parent 1de8df2 commit 7cf5baf

File tree

9 files changed

+279
-97
lines changed

9 files changed

+279
-97
lines changed
 

‎CHANGELOG.next.asciidoc

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
4242
* iptables {pull}18756[18756]
4343
* Checkpoint {pull}18754[18754]
4444
- Preserve case of http.request.method. ECS prior to 1.6 specified normalizing to lowercase, which lost information. Affects filesets: apache/access, elasticsearch/audit, iis/access, iis/error, nginx/access, nginx/ingress_controller, aws/elb, suricata/eve, zeek/http. {issue}18154[18154] {pull}18359[18359]
45+
- Okta module now requires objects instead of JSON strings for the `http_headers`, `http_request_body`, `pagination`, `rate_limit`, and `ssl` variables. {pull}18953[18953]
4546

4647
*Heartbeat*
4748

@@ -163,6 +164,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
163164
- Fix `googlecloud.audit` pipeline to only take in fields that are explicitly defined by the dataset. {issue}18465[18465] {pull}18472[18472]
164165
- Fix `o365.audit` failing to ingest events when ip address is surrounded by square brackets. {issue}18587[18587] {pull}18591[18591]
165166
- Fix Kubernetes Watcher goroutine leaks when input config is invalid and `input.reload` is enabled. {issue}18629[18629] {pull}18630[18630]
167+
- Okta module now sets the Elasticsearch `_id` field to the Okta UUID value contained in each system log to minimize the possibility of duplicating events. {pull}18953[18953]
166168

167169
*Heartbeat*
168170

‎filebeat/docs/modules/okta.asciidoc

Lines changed: 93 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -12,15 +12,105 @@ This file is generated! See scripts/docs_collector.py
1212

1313
beta[]
1414

15-
This is a filebeat module for retrieving system logs from Okta (www.okta.com) via API.
15+
The Okta module collects events from the
16+
https://developer.okta.com/docs/reference/[Okta API]. Specifically this supports
17+
reading from the https://developer.okta.com/docs/reference/api/system-log/[Okta
18+
System Log API].
1619

17-
:has-dashboards!:
20+
:fileset_ex: system
21+
22+
include::../include/config-option-intro.asciidoc[]
23+
24+
[float]
25+
==== `system` fileset settings
26+
27+
The Okta System Log records system events related to your organization in order
28+
to provide an audit trail that can be used to understand platform activity and
29+
to diagnose problems. This module is implemented using the
30+
<<filebeat-input-httpjson,httpjson>> input and is configured to paginate through
31+
the logs while honoring any
32+
https://developer.okta.com/docs/reference/rate-limits/[rate-limiting] headers
33+
sent by Okta.
34+
35+
NOTE: This module does not persist the timestamp of the last read event in
36+
order to facilitate resuming on restart. This feature will be coming in a future
37+
version. When you restart the module will read events from the beginning of the
38+
log. To minimize duplicates documents the module uses the event's Okta UUID
39+
value as the Elasticsearch `_id`.
40+
41+
This is an example configuration for the module.
42+
43+
[source,yaml]
44+
----
45+
- module okta
46+
system:
47+
var.url: https://yourOktaDomain/api/v1/logs
48+
var.api_key: '00QCjAl4MlV-WPXM...0HmjFx-vbGua'
49+
----
50+
51+
[float]
52+
===== Configuration options
53+
54+
*`var.url`*::
55+
56+
Specifies the URL to the Okta System Log API. Required.
57+
+
58+
[source,yaml]
59+
----
60+
var.url: https://mycompany.okta.com/api/v1/logs
61+
----
62+
63+
*`var.api_key`*::
64+
65+
Specifies the Okta API token to use in requests to the API. Required.
66+
The token is used in an HTTP `Authorization` header with the `SSWS` scheme.
67+
See https://developer.okta.com/docs/guides/create-an-api-token/create-the-token/[
68+
Create an API token] for information on how to obtain a token.
69+
+
70+
[source,yaml]
71+
----
72+
var.api_key: '00QCjAl4MlV-WPXM...0HmjFx-vbGua'
73+
----
1874

19-
This module comes with a sample dashboard. For example:
75+
*`var.http_client_timeout`*::
76+
77+
Duration of the time limit on HTTP requests made by the module. Defaults to
78+
`60s`.
79+
80+
*`var.interval`*::
81+
82+
Duration between requests to the API. Defaults to `60s`.
83+
84+
*`var.keep_original_message`*::
85+
86+
Boolean flag indicating if the original JSON event string should be included in
87+
the `event.original` field. Defaults to `true`.
88+
89+
*`var.ssl`*::
90+
91+
Configuration options for SSL parameters like the certificate authority to use
92+
for HTTPS-based connections. If the `ssl` section is missing, the host CAs are
93+
used for HTTPS connections to Okta. See <<configuration-ssl>> for more
94+
information.
95+
+
96+
[source,yaml]
97+
----
98+
var.ssl:
99+
supported_protocols: [TLSv1.2]
100+
----
101+
102+
[float]
103+
=== Example dashboard
104+
105+
This module comes with a sample dashboard:
20106

21107
[role="screenshot"]
22108
image::./images/filebeat-okta-dashboard.png[]
23109

110+
:has-dashboards!:
111+
112+
:fileset_ex!:
113+
24114
:modulename!:
25115

26116

‎x-pack/filebeat/filebeat.reference.yml

Lines changed: 4 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -860,17 +860,10 @@ filebeat.modules:
860860
- module: okta
861861
system:
862862
enabled: true
863-
# API key to access Okta
864-
#var.api_key
865-
866-
# URL of the Okta REST API
867-
#var.url
868-
869-
# Disable SSL verification
870-
#var.ssl: |-
871-
# {
872-
# "verification_mode": "none"
873-
# }
863+
# You must configure the URL with your Okta domain and provide an
864+
# API token to access the logs API.
865+
#var.url: https://yourOktaDomain/api/v1/logs
866+
#var.api_key: 'yourApiTokenHere'
874867

875868
#------------------------------- Osquery Module -------------------------------
876869
- module: osquery

‎x-pack/filebeat/module/okta/_meta/config.yml

Lines changed: 4 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,7 @@
11
- module: okta
22
system:
33
enabled: true
4-
# API key to access Okta
5-
#var.api_key
6-
7-
# URL of the Okta REST API
8-
#var.url
9-
10-
# Disable SSL verification
11-
#var.ssl: |-
12-
# {
13-
# "verification_mode": "none"
14-
# }
4+
# You must configure the URL with your Okta domain and provide an
5+
# API token to access the logs API.
6+
#var.url: https://yourOktaDomain/api/v1/logs
7+
#var.api_key: 'yourApiTokenHere'

‎x-pack/filebeat/module/okta/_meta/docs.asciidoc

Lines changed: 93 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -7,13 +7,103 @@
77

88
beta[]
99

10-
This is a filebeat module for retrieving system logs from Okta (www.okta.com) via API.
10+
The Okta module collects events from the
11+
https://developer.okta.com/docs/reference/[Okta API]. Specifically this supports
12+
reading from the https://developer.okta.com/docs/reference/api/system-log/[Okta
13+
System Log API].
1114

12-
:has-dashboards!:
15+
:fileset_ex: system
16+
17+
include::../include/config-option-intro.asciidoc[]
18+
19+
[float]
20+
==== `system` fileset settings
21+
22+
The Okta System Log records system events related to your organization in order
23+
to provide an audit trail that can be used to understand platform activity and
24+
to diagnose problems. This module is implemented using the
25+
<<filebeat-input-httpjson,httpjson>> input and is configured to paginate through
26+
the logs while honoring any
27+
https://developer.okta.com/docs/reference/rate-limits/[rate-limiting] headers
28+
sent by Okta.
29+
30+
NOTE: This module does not persist the timestamp of the last read event in
31+
order to facilitate resuming on restart. This feature will be coming in a future
32+
version. When you restart the module will read events from the beginning of the
33+
log. To minimize duplicates documents the module uses the event's Okta UUID
34+
value as the Elasticsearch `_id`.
35+
36+
This is an example configuration for the module.
37+
38+
[source,yaml]
39+
----
40+
- module okta
41+
system:
42+
var.url: https://yourOktaDomain/api/v1/logs
43+
var.api_key: '00QCjAl4MlV-WPXM...0HmjFx-vbGua'
44+
----
45+
46+
[float]
47+
===== Configuration options
48+
49+
*`var.url`*::
50+
51+
Specifies the URL to the Okta System Log API. Required.
52+
+
53+
[source,yaml]
54+
----
55+
var.url: https://mycompany.okta.com/api/v1/logs
56+
----
57+
58+
*`var.api_key`*::
59+
60+
Specifies the Okta API token to use in requests to the API. Required.
61+
The token is used in an HTTP `Authorization` header with the `SSWS` scheme.
62+
See https://developer.okta.com/docs/guides/create-an-api-token/create-the-token/[
63+
Create an API token] for information on how to obtain a token.
64+
+
65+
[source,yaml]
66+
----
67+
var.api_key: '00QCjAl4MlV-WPXM...0HmjFx-vbGua'
68+
----
1369

14-
This module comes with a sample dashboard. For example:
70+
*`var.http_client_timeout`*::
71+
72+
Duration of the time limit on HTTP requests made by the module. Defaults to
73+
`60s`.
74+
75+
*`var.interval`*::
76+
77+
Duration between requests to the API. Defaults to `60s`.
78+
79+
*`var.keep_original_message`*::
80+
81+
Boolean flag indicating if the original JSON event string should be included in
82+
the `event.original` field. Defaults to `true`.
83+
84+
*`var.ssl`*::
85+
86+
Configuration options for SSL parameters like the certificate authority to use
87+
for HTTPS-based connections. If the `ssl` section is missing, the host CAs are
88+
used for HTTPS connections to Okta. See <<configuration-ssl>> for more
89+
information.
90+
+
91+
[source,yaml]
92+
----
93+
var.ssl:
94+
supported_protocols: [TLSv1.2]
95+
----
96+
97+
[float]
98+
=== Example dashboard
99+
100+
This module comes with a sample dashboard:
15101

16102
[role="screenshot"]
17103
image::./images/filebeat-okta-dashboard.png[]
18104

105+
:has-dashboards!:
106+
107+
:fileset_ex!:
108+
19109
:modulename!:

‎x-pack/filebeat/module/okta/system/config/input.yml

Lines changed: 35 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,48 @@
11
{{ if eq .input "httpjson" }}
22

33
type: httpjson
4+
5+
{{ if .api_key }}
46
api_key: {{ .api_key }}
5-
authentication_scheme: {{.authentication_scheme}}
7+
{{ end }}
8+
9+
authentication_scheme: {{ .authentication_scheme }}
10+
11+
{{ if .http_client_timeout }}
612
http_client_timeout: {{ .http_client_timeout }}
13+
{{ end }}
14+
15+
{{ if .http_method }}
716
http_method: {{ .http_method }}
8-
http_headers: {{ .http_headers }}
17+
{{ end }}
18+
19+
{{ if .http_headers }}
20+
http_headers: {{ .http_headers | tojson }}
21+
{{ end }}
22+
23+
{{ if .http_request_body }}
924
http_request_body: {{ .http_request_body }}
10-
no_http_body: {{ .no_http_body }}
25+
{{ end }}
26+
1127
interval: {{ .interval }}
28+
29+
{{ if .json_objects_array }}
1230
json_objects_array: {{ .json_objects_array }}
13-
pagination: {{ .pagination }}
14-
rate_limit: {{ .rate_limit }}
31+
{{ end }}
32+
33+
no_http_body: {{ .no_http_body }}
34+
35+
pagination: {{ .pagination | tojson }}
36+
37+
rate_limit: {{ .rate_limit | tojson }}
38+
39+
{{ if .ssl }}
40+
ssl: {{ .ssl | tojson }}
41+
{{ end }}
42+
43+
{{ if .url }}
1544
url: {{ .url }}
16-
ssl: {{ .ssl }}
45+
{{ end }}
1746

1847
{{ else if eq .input "file" }}
1948

0 commit comments

Comments
 (0)