From bf063b17c09c6718df5046b17a7e56f9934bafd4 Mon Sep 17 00:00:00 2001 From: Mike Bell Date: Wed, 28 Aug 2024 13:03:24 +0100 Subject: [PATCH 1/4] docs: intial commit of runbook --- .../investigating-blocked-ingress-spikes.html.md.erb | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 runbooks/source/investigating-blocked-ingress-spikes.html.md.erb diff --git a/runbooks/source/investigating-blocked-ingress-spikes.html.md.erb b/runbooks/source/investigating-blocked-ingress-spikes.html.md.erb new file mode 100644 index 00000000..6c31b485 --- /dev/null +++ b/runbooks/source/investigating-blocked-ingress-spikes.html.md.erb @@ -0,0 +1,8 @@ +--- +title: Investigating blocked ingress spikes +weight: 9999 +last_reviewed_on: 2024-08-28 +review_in: 6 months +--- + +# <%= current_page.data.title %> \ No newline at end of file From 9b54fcf9520ee3d20d8793de9030848d2043a299 Mon Sep 17 00:00:00 2001 From: Mike Bell Date: Wed, 28 Aug 2024 14:39:09 +0100 Subject: [PATCH 2/4] docs: first pass at investigating blocked ingress runbook --- ...igating-blocked-ingress-spikes.html.md.erb | 21 ++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/runbooks/source/investigating-blocked-ingress-spikes.html.md.erb b/runbooks/source/investigating-blocked-ingress-spikes.html.md.erb index 6c31b485..5a65a9b8 100644 --- a/runbooks/source/investigating-blocked-ingress-spikes.html.md.erb +++ b/runbooks/source/investigating-blocked-ingress-spikes.html.md.erb @@ -5,4 +5,23 @@ last_reviewed_on: 2024-08-28 review_in: 6 months --- -# <%= current_page.data.title %> \ No newline at end of file +# <%= current_page.data.title %> + +Things to look at while investigating a spike in blocked access: + +1. Is the spike isolation to that application? If there is an attack it could be either cluster wide or specifically targeted at a single app. +2. Is the ingress using modsec? +3. [Access denied with code 406 in the last 24 hours](https://kibana.cloud-platform.service.justice.gov.uk/_plugin/kibana/app/discover#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-1d,to:now))&_a=(columns:!(_source),filters:!(),index:'8a728bc0-00eb-11ec-9062-27aa363b66a2',interval:auto,query:(language:kuery,query:'%22Access%20denied%20with%20code%20406%22'),sort:!())) +3. Are there any suspect logs in the namespace? +4. Is there a wider impact on the platform? + * Has the cluster scaled up due to extra resource usage? + * Are there more 4xx/5xx errors than usual? + * Are we seeing ingress related alarms in #lower-priority-alarms +5. Note any suspicious IP addresses. + +## Communication + +It's important to clearly and efficiently communicate between the Cloud Platform team and user. It may be required to call an [incident](https://runbooks.cloud-platform.service.justice.gov.uk/incident-process.html). Where possible keep a record of findings as either part of the a Slack thread or Google document. + +Further issues can be raised in the Cloud Platform issue [tracker](https://github.com/ministryofjustice/cloud-platform/issues). + From 85ecce300dca601b7a21724125d2fad7491157e8 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" Date: Wed, 28 Aug 2024 13:40:20 +0000 Subject: [PATCH 3/4] Commit changes made by code formatters --- runbooks/source/investigating-blocked-ingress-spikes.html.md.erb | 1 - 1 file changed, 1 deletion(-) diff --git a/runbooks/source/investigating-blocked-ingress-spikes.html.md.erb b/runbooks/source/investigating-blocked-ingress-spikes.html.md.erb index 5a65a9b8..ca04af77 100644 --- a/runbooks/source/investigating-blocked-ingress-spikes.html.md.erb +++ b/runbooks/source/investigating-blocked-ingress-spikes.html.md.erb @@ -24,4 +24,3 @@ Things to look at while investigating a spike in blocked access: It's important to clearly and efficiently communicate between the Cloud Platform team and user. It may be required to call an [incident](https://runbooks.cloud-platform.service.justice.gov.uk/incident-process.html). Where possible keep a record of findings as either part of the a Slack thread or Google document. Further issues can be raised in the Cloud Platform issue [tracker](https://github.com/ministryofjustice/cloud-platform/issues). - From bd6ae815f295a7b7f2b77c51a98b9456b2cc73aa Mon Sep 17 00:00:00 2001 From: Mike Bell Date: Thu, 29 Aug 2024 09:15:41 +0100 Subject: [PATCH 4/4] docs: update based on feedback --- .../source/investigating-blocked-ingress-spikes.html.md.erb | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/runbooks/source/investigating-blocked-ingress-spikes.html.md.erb b/runbooks/source/investigating-blocked-ingress-spikes.html.md.erb index 5a65a9b8..15f2a866 100644 --- a/runbooks/source/investigating-blocked-ingress-spikes.html.md.erb +++ b/runbooks/source/investigating-blocked-ingress-spikes.html.md.erb @@ -11,13 +11,14 @@ Things to look at while investigating a spike in blocked access: 1. Is the spike isolation to that application? If there is an attack it could be either cluster wide or specifically targeted at a single app. 2. Is the ingress using modsec? -3. [Access denied with code 406 in the last 24 hours](https://kibana.cloud-platform.service.justice.gov.uk/_plugin/kibana/app/discover#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-1d,to:now))&_a=(columns:!(_source),filters:!(),index:'8a728bc0-00eb-11ec-9062-27aa363b66a2',interval:auto,query:(language:kuery,query:'%22Access%20denied%20with%20code%20406%22'),sort:!())) +3. [Access denied with code 406 in the last 24 hours](https://kibana.cloud-platform.service.justice.gov.uk/_plugin/kibana/app/discover#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-1d,to:now))&_a=(columns:!(_source),filters:!(),index:'8a728bc0-00eb-11ec-9062-27aa363b66a2',interval:auto,query:(language:kuery,query:'%22Access%20denied%20with%20code%20406%22'),sort:!())). Not every user uses the custom `406` status so this is not a catch all solution. 3. Are there any suspect logs in the namespace? 4. Is there a wider impact on the platform? * Has the cluster scaled up due to extra resource usage? * Are there more 4xx/5xx errors than usual? * Are we seeing ingress related alarms in #lower-priority-alarms 5. Note any suspicious IP addresses. +6. Has modsec been misconfigured? Further information can be found [here](https://www.netnea.com/cms/apache-tutorial-7_including-modsecurity-core-rules/) ## Communication @@ -25,3 +26,6 @@ It's important to clearly and efficiently communicate between the Cloud Platform Further issues can be raised in the Cloud Platform issue [tracker](https://github.com/ministryofjustice/cloud-platform/issues). +## Other Links + +* [Debugging 101](https://www.netnea.com/cms/apache-tutorial-7_including-modsecurity-core-rules/) \ No newline at end of file