Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cherry-pick #18511 to 7.7: Fix source.address not being set for nginx ingress_controller #18569

Merged
merged 5 commits into from
May 18, 2020
Merged

Cherry-pick #18511 to 7.7: Fix source.address not being set for nginx ingress_controller #18569

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
0ccffa0
Fix source.address not being set for nginx ingress_controller
ChrsMark May 15, 2020
9a337b2
Fix expected file
ChrsMark May 15, 2020
dafe67b
Update test file
ChrsMark May 15, 2020
32d7c59
fix test file
ChrsMark May 15, 2020
da70758
recreate test file
ChrsMark May 15, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.next.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Fixed typo in log message. {pull}17897[17897]
- Unescape file name from SQS message. {pull}18370[18370]
- Improve cisco asa and ftd pipelines' failure handler to avoid mapping temporary fields. {issue}18391[18391] {pull}18392[18392]
- Fix source.address not being set for nginx ingress_controller {pull}18511[18511]
- Fix `googlecloud.audit` pipeline to only take in fields that are explicitly defined by the dataset. {issue}18465[18465] {pull}18472[18472]
- Fix a rate limit related issue in httpjson input for Okta module. {issue}18530[18530] {pull}18534[18534]

Expand Down
2 changes: 1 addition & 1 deletion filebeat/module/nginx/ingress_controller/ingest/pipeline.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@
},
{
"script": {
"if": "ctx.nginx?.access?.remote_ip_list != null && ctx.nginx.ingress_controller.remote_ip_list.length > 0",
"if": "ctx.nginx?.ingress_controller?.remote_ip_list != null && ctx.nginx.ingress_controller.remote_ip_list.length > 0",
"lang": "painless",
"source": "boolean isPrivate(def dot, def ip) { try { StringTokenizer tok = new StringTokenizer(ip, dot); int firstByte = Integer.parseInt(tok.nextToken()); int secondByte = Integer.parseInt(tok.nextToken()); if (firstByte == 10) { return true; } if (firstByte == 192 && secondByte == 168) { return true; } if (firstByte == 172 && secondByte >= 16 && secondByte <= 31) { return true; } if (firstByte == 127) { return true; } return false; } catch (Exception e) { return false; } } try { ctx.source.address = null; if (ctx.nginx.ingress_controller.remote_ip_list == null) { return; } def found = false; for (def item : ctx.nginx.ingress_controller.remote_ip_list) { if (!isPrivate(params.dot, item)) { ctx.source.address = item; found = true; break; } } if (!found) { ctx.source.address = ctx.nginx.ingress_controller.remote_ip_list[0]; }} catch (Exception e) { ctx.source.address = null; }",
"params": {
Expand Down
140 changes: 114 additions & 26 deletions filebeat/module/nginx/ingress_controller/test/test.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,12 @@
"nginx.ingress_controller.upstream.response.length": 59,
"nginx.ingress_controller.upstream.response.status_code": 200,
"nginx.ingress_controller.upstream.response.time": 0.0,
"related.ip": [
"192.168.64.1"
],
"service.type": "nginx",
"source.address": "",
"source.address": "192.168.64.1",
"source.ip": "192.168.64.1",
"url.original": "/products",
"user.name": "-",
"user_agent.device.name": "Other",
Expand Down Expand Up @@ -60,8 +64,12 @@
"nginx.ingress_controller.upstream.response.length": 59,
"nginx.ingress_controller.upstream.response.status_code": 200,
"nginx.ingress_controller.upstream.response.time": 0.0,
"related.ip": [
"192.168.64.1"
],
"service.type": "nginx",
"source.address": "",
"source.address": "192.168.64.1",
"source.ip": "192.168.64.1",
"url.original": "/products/42",
"user.name": "-",
"user_agent.device.name": "Other",
Expand Down Expand Up @@ -95,8 +103,12 @@
"nginx.ingress_controller.upstream.response.length": 59,
"nginx.ingress_controller.upstream.response.status_code": 200,
"nginx.ingress_controller.upstream.response.time": 0.001,
"related.ip": [
"192.168.64.1"
],
"service.type": "nginx",
"source.address": "",
"source.address": "192.168.64.1",
"source.ip": "192.168.64.1",
"url.original": "/products/42",
"user.name": "-",
"user_agent.device.name": "Other",
Expand Down Expand Up @@ -130,8 +142,12 @@
"nginx.ingress_controller.upstream.response.length": 59,
"nginx.ingress_controller.upstream.response.status_code": 200,
"nginx.ingress_controller.upstream.response.time": 0.0,
"related.ip": [
"192.168.64.1"
],
"service.type": "nginx",
"source.address": "",
"source.address": "192.168.64.1",
"source.ip": "192.168.64.1",
"url.original": "/products/42",
"user.name": "-",
"user_agent.device.name": "Other",
Expand Down Expand Up @@ -160,13 +176,17 @@
],
"nginx.ingress_controller.upstream.alternative_name": "",
"nginx.ingress_controller.upstream.name": "",
"related.ip": [
"192.168.64.1"
],
"service.type": "nginx",
"source.address": "",
"url.original": "/products/42",
"user.name": "-",
"user_agent.device.name": "Other",
"user_agent.name": "Other",
"user_agent.original": "-"
"user_agent.original": "-",
"source.address": "192.168.64.1",
"source.ip": "192.168.64.1"
},
{
"@timestamp": "2020-02-07T11:50:09.000Z",
Expand All @@ -189,13 +209,17 @@
],
"nginx.ingress_controller.upstream.alternative_name": "",
"nginx.ingress_controller.upstream.name": "",
"related.ip": [
"192.168.64.1"
],
"service.type": "nginx",
"source.address": "",
"url.original": "/products/42",
"user.name": "-",
"user_agent.device.name": "Other",
"user_agent.name": "Other",
"user_agent.original": "-"
"user_agent.original": "-",
"source.address": "192.168.64.1",
"source.ip": "192.168.64.1",
"url.original": "/products/42"
},
{
"@timestamp": "2020-02-07T11:55:05.000Z",
Expand Down Expand Up @@ -223,8 +247,12 @@
"nginx.ingress_controller.upstream.response.length": 59,
"nginx.ingress_controller.upstream.response.status_code": 200,
"nginx.ingress_controller.upstream.response.time": 0.0,
"related.ip": [
"192.168.64.1"
],
"service.type": "nginx",
"source.address": "",
"source.address": "192.168.64.1",
"source.ip": "192.168.64.1",
"url.original": "/products/42",
"user.name": "-",
"user_agent.device.name": "Other",
Expand Down Expand Up @@ -258,8 +286,12 @@
"nginx.ingress_controller.upstream.response.length": 59,
"nginx.ingress_controller.upstream.response.status_code": 200,
"nginx.ingress_controller.upstream.response.time": 0.0,
"related.ip": [
"192.168.64.1"
],
"service.type": "nginx",
"source.address": "",
"source.address": "192.168.64.1",
"source.ip": "192.168.64.1",
"url.original": "/products/42",
"user.name": "-",
"user_agent.device.name": "Other",
Expand Down Expand Up @@ -296,8 +328,12 @@
"nginx.ingress_controller.upstream.response.length": 59,
"nginx.ingress_controller.upstream.response.status_code": 200,
"nginx.ingress_controller.upstream.response.time": 0.0,
"related.ip": [
"192.168.64.1"
],
"service.type": "nginx",
"source.address": "",
"source.address": "192.168.64.1",
"source.ip": "192.168.64.1",
"url.original": "/favicon.ico",
"user.name": "-",
"user_agent.device.name": "Other",
Expand Down Expand Up @@ -334,8 +370,12 @@
"nginx.ingress_controller.upstream.response.length": 61,
"nginx.ingress_controller.upstream.response.status_code": 200,
"nginx.ingress_controller.upstream.response.time": 0.001,
"related.ip": [
"192.168.64.1"
],
"service.type": "nginx",
"source.address": "",
"source.address": "192.168.64.1",
"source.ip": "192.168.64.1",
"url.original": "/v2",
"user.name": "-",
"user_agent.device.name": "Other",
Expand Down Expand Up @@ -372,8 +412,12 @@
"nginx.ingress_controller.upstream.response.length": 59,
"nginx.ingress_controller.upstream.response.status_code": 200,
"nginx.ingress_controller.upstream.response.time": 0.002,
"related.ip": [
"192.168.64.1"
],
"service.type": "nginx",
"source.address": "",
"source.address": "192.168.64.1",
"source.ip": "192.168.64.1",
"url.original": "/favicon.ico",
"user.name": "-",
"user_agent.device.name": "Other",
Expand Down Expand Up @@ -410,8 +454,12 @@
"nginx.ingress_controller.upstream.response.length": 59,
"nginx.ingress_controller.upstream.response.status_code": 200,
"nginx.ingress_controller.upstream.response.time": 0.001,
"related.ip": [
"192.168.64.1"
],
"service.type": "nginx",
"source.address": "",
"source.address": "192.168.64.1",
"source.ip": "192.168.64.1",
"url.original": "/products/42",
"user.name": "-",
"user_agent.device.name": "Other",
Expand Down Expand Up @@ -448,8 +496,12 @@
"nginx.ingress_controller.upstream.response.length": 59,
"nginx.ingress_controller.upstream.response.status_code": 200,
"nginx.ingress_controller.upstream.response.time": 0.001,
"related.ip": [
"192.168.64.1"
],
"service.type": "nginx",
"source.address": "",
"source.address": "192.168.64.1",
"source.ip": "192.168.64.1",
"url.original": "/favicon.ico",
"user.name": "-",
"user_agent.device.name": "Other",
Expand Down Expand Up @@ -486,8 +538,12 @@
"nginx.ingress_controller.upstream.response.length": 59,
"nginx.ingress_controller.upstream.response.status_code": 200,
"nginx.ingress_controller.upstream.response.time": 0.002,
"related.ip": [
"192.168.64.1"
],
"service.type": "nginx",
"source.address": "",
"source.address": "192.168.64.1",
"source.ip": "192.168.64.1",
"url.original": "/products/42",
"user.name": "-",
"user_agent.device.name": "Other",
Expand Down Expand Up @@ -524,8 +580,12 @@
"nginx.ingress_controller.upstream.response.length": 59,
"nginx.ingress_controller.upstream.response.status_code": 200,
"nginx.ingress_controller.upstream.response.time": 0.001,
"related.ip": [
"192.168.64.1"
],
"service.type": "nginx",
"source.address": "",
"source.address": "192.168.64.1",
"source.ip": "192.168.64.1",
"url.original": "/",
"user.name": "-",
"user_agent.device.name": "Other",
Expand Down Expand Up @@ -562,8 +622,12 @@
"nginx.ingress_controller.upstream.response.length": 59,
"nginx.ingress_controller.upstream.response.status_code": 200,
"nginx.ingress_controller.upstream.response.time": 0.002,
"related.ip": [
"192.168.64.1"
],
"service.type": "nginx",
"source.address": "",
"source.address": "192.168.64.1",
"source.ip": "192.168.64.1",
"url.original": "/favicon.ico",
"user.name": "-",
"user_agent.device.name": "Other",
Expand Down Expand Up @@ -600,8 +664,12 @@
"nginx.ingress_controller.upstream.response.length": 61,
"nginx.ingress_controller.upstream.response.status_code": 200,
"nginx.ingress_controller.upstream.response.time": 0.002,
"related.ip": [
"192.168.64.1"
],
"service.type": "nginx",
"source.address": "",
"source.address": "192.168.64.1",
"source.ip": "192.168.64.1",
"url.original": "/v2",
"user.name": "-",
"user_agent.device.name": "Other",
Expand Down Expand Up @@ -638,8 +706,12 @@
"nginx.ingress_controller.upstream.response.length": 59,
"nginx.ingress_controller.upstream.response.status_code": 200,
"nginx.ingress_controller.upstream.response.time": 0.0,
"related.ip": [
"192.168.64.1"
],
"service.type": "nginx",
"source.address": "",
"source.address": "192.168.64.1",
"source.ip": "192.168.64.1",
"url.original": "/favicon.ico",
"user.name": "-",
"user_agent.device.name": "Other",
Expand Down Expand Up @@ -676,8 +748,12 @@
"nginx.ingress_controller.upstream.response.length": 59,
"nginx.ingress_controller.upstream.response.status_code": 200,
"nginx.ingress_controller.upstream.response.time": 0.001,
"related.ip": [
"192.168.64.1"
],
"service.type": "nginx",
"source.address": "",
"source.address": "192.168.64.1",
"source.ip": "192.168.64.1",
"url.original": "/products/42?address=delhi+technological+university",
"user.name": "-",
"user_agent.device.name": "Other",
Expand Down Expand Up @@ -711,8 +787,12 @@
"nginx.ingress_controller.upstream.response.length": 61,
"nginx.ingress_controller.upstream.response.status_code": 200,
"nginx.ingress_controller.upstream.response.time": 0.001,
"related.ip": [
"192.168.64.1"
],
"service.type": "nginx",
"source.address": "",
"source.address": "192.168.64.1",
"source.ip": "192.168.64.1",
"url.original": "/v2",
"user.name": "-",
"user_agent.device.name": "Other",
Expand Down Expand Up @@ -749,8 +829,12 @@
"nginx.ingress_controller.upstream.response.length": 59,
"nginx.ingress_controller.upstream.response.status_code": 200,
"nginx.ingress_controller.upstream.response.time": 0.0,
"related.ip": [
"192.168.64.1"
],
"service.type": "nginx",
"source.address": "",
"source.address": "192.168.64.1",
"source.ip": "192.168.64.1",
"url.original": "/favicon.ico",
"user.name": "-",
"user_agent.device.name": "Other",
Expand Down Expand Up @@ -787,8 +871,12 @@
"nginx.ingress_controller.upstream.response.length": 61,
"nginx.ingress_controller.upstream.response.status_code": 200,
"nginx.ingress_controller.upstream.response.time": 0.0,
"related.ip": [
"192.168.64.1"
],
"service.type": "nginx",
"source.address": "",
"source.address": "192.168.64.1",
"source.ip": "192.168.64.1",
"url.original": "/v2/some",
"user.name": "-",
"user_agent.device.name": "Other",
Expand All @@ -799,4 +887,4 @@
"user_agent.os.version": "10.14",
"user_agent.version": "72.0."
}
]
]