Enhance troubleshoot page

content-repo/extra-docs/articles/advanced-networking-troubleshooting

@@ -0,0 +1,156 @@
+---
+title: Advanced Networking Troubleshooting Guide
+description: Advanced troubleshooting steps Newtowk issues.
+---
+
+Note that this part is only relevant for XSOAR on-rem.
+Examples of common errors indicating that there probably is a networking issue:
+
+* `[Errno -2] Name does not resolve`
+* `[Errno 110] Operation timed out`
+* `Failed to establish a new connection: [Errno -3] Try again`
+* `dial tcp: lookup ****: no such host`
+* `connect: operation timed out`
+* `connect: connection refused`
+* `ERR_CONNECTION_REFUSED`
+
+When troubleshooting networking issues, it is important to first understand what type of networking the integration or automation is using. Cortex XSOAR integrations and automations can be classified into two main types regarding their networking use:
+
+##### Host Based Networking
+
+Integrations/automations running within the server/engine will use the networking stack provided by the host machine of the server/engine. Such integrations/automations include native integrations (part of the server binary) such as the `RemoteAccess` integration and JavaScript integrations such as `VirusTotal` and `http`. Native integrations can be identified by the fact that they are shipped as part of the server and not associated with a Content Pack. JavaScript integrations/automations can be identified by checking the integration/automation settings to see that the *Language Type* is **JavaScript**. JavaScript integrations/automations run within the Cortex XSOAR server/engine process using a JavaScript virtual environment and therefore use the same network stack as the server/engine. The source IP addresses for these integrations/automations are the same as used by the server/engine.
+
+If the integration/automation is using HTTP-based communication, we recommend first testing locally using the `curl` utility to verify that it is possible to perform network communication with the HTTP endpoint. Run the `curl` command on the server or engine machine by logging in via SSH. Common `curl` command variants (`httpbin.org` is used as an example url):
+```bash
+# Run simple curl command with -v for verbose output:
+curl -v https://httpbin.org/status/200
+
+# Run with -k to trust any certificate in case you receive errors regarding certificates
+curl -vk https://httpbin.org/status/200
+
+# curl will use the machine env variables for proxy settings. If you wish to ignore the proxy settings run:
+curl -vk --noproxy "*" https://httpbin.org/status/200
+
+# Setting explicitly a proxy server to use by curl
+curl -x http://192.168.0.1:8080 https://httpbin.org/status/200
+
+# Passing an additional header as part of the curl request:
+curl -v -H 'Accept: application/json' https://httpbin.org/headers
+
+# In cases that the integration uses basic authentication, you can also easily test the credentials:
+curl -v --user myuser:mypass https://httpbin.org//basic-auth/myuser/mypass
+```
+
+More info about `curl` is available at [Everything curl](https://ec.haxx.se/).
+
+If you are not able to perform a basic `curl` request from the machine to the target HTTP endpoint, the issue is probably not a problem with the integration/automation but rather with the networking setup of the server/engine machine. Make sure to first resolve the networking issue so a basic `curl` command succeeds before continuing to test the integration/automation. Many times this resolves to a firewall, NAT or proxy issue.
+
+##### Docker Based Networking
+
+Docker Based integrations/automations are written in Python or Powershell. They can be identified by inspecting the integration/automation settings and under *Language type* will appear **Python** or **Powershell**. Docker creates its own networking, therefore the integrations/automations use a different networking stack from the Cortex XSOAR server/engine. The source IP addresses for these integrations/automations are different and provided according to the Docker networking configuration.
+
+As with [Host Based Networking](#host-based-networking), for integrations/automations that use HTTP endpoints we recommend testing with `curl` from within a Docker container as a first step. This can be done by logging in to the server/engine machine via SSH and running the following command:
+```bash
+docker run -it --rm demisto/netutils:1.0.0.6138 curl <curl parameters>
+```
+For example:
+```bash
+# Run simple curl command with -v for verbose output:
+docker run -it --rm demisto/netutils:1.0.0.6138 curl -v https://httpbin.org/status/200
+```
+For additional `curl` sample commands see the [Host Based Networking](#host-based-networking) section.
+
+**Note**: You may need to run `docker` with `sudo` or login with root if your user doesn't have sufficient permissions to execute the `docker` command.
+
+If running `curl` from within `docker` fails with networking errors, we recommend checking if the `curl` command succeeds or fails without `docker` by running the `curl` command directly on the host machine. If the `curl` command succeeds on the host machine and fails within Docker, you are probably experiencing a Docker networking issue due to how the Docker networking stack is configured. 
+
+We recommend that you use the Docker networking stack because it provides networking isolation. Try to resolve the [Docker networking issue](https://success.docker.com/article/troubleshooting-container-networking) and consult the [Docker networking docs](https://docs.docker.com/network/). 
+
+When running with Docker's networking stack continues to cause issues, there is an option to run Docker containers with host networking. In this mode, the container will share the host’s network stack and all interfaces from the host will be available to the container. The container’s hostname will match the hostname on the host system. You can test this mode by running a `curl` command via `docker` in the following form:
+```bash
+docker run -it --rm --network=host demisto/netutils:1.0.0.6138 curl -v https://httpbin.org/status/200
+```
+
+If running with `--network=host` succeeds, you can configure the server to use host networking for docker by adding the following advanced server configuration in Cortex XSOAR:
+
+Key | Value
+--- |  ----
+`python.pass.extra.keys` | `--network=host`
+
+It is also possible to configure only a specific docker image to use the host networking by stating `python.pass.extra.keys.<docker-image>` as the key. For example:
+
+Key | Value
+--- |  ----
+`python.pass.extra.keys.demisto/smbprotocol` | `--network=host`
+
+After you add the server configuration, run the `/reset_containers` command from the Cortex XSOAR CLI to reset all containers and to begin using the new configuration.
+
+**Notes:**
+
+* For multi-tenant deployments, you need to add this setting to each tenant.
+* When using engines, you need to add this setting to each engine.
+
+##### Read Timeout
+
+In case you encounter a *ReadTimeout* error, such as `ReadTimeout: HTTPSConnectionPool(host='www.google.com', port=443): Read timed out. (read timeout=10)`, it means that the server (or network) failed to deliver any data within 10 seconds. This might be due to a large response size.
+
+Starting from Base Content Pack version 1.17.6, we support controlling the read timeout value via server advanced configuration, as follows:
+* System wide
+
+  Key | Value
+  --- |  ----
+  `python.pass.extra.keys` | `--env=REQUESTS_TIMEOUT=<TIMEOUT>`
+
+* Per Integration
+
+  Key | Value
+  --- |  ----
+  `python.pass.extra.keys` | `--env=REQUESTS_TIMEOUT.<INTEGRATION-ID>=<TIMEOUT>`
+
+Examples:
+
+* Set the read timeout value to *120* seconds system wide, `--env=REQUESTS_TIMEOUT=120`
+* Set the read timeout value to *75* seconds for the Palo Alto Networks WildFire v2 integration, `--env=REQUESTS_TIMEOUT.WildFire-v2=75`
+
+**Note:** The `REQUESTS_TIMEOUT` settings only affects integrations which use the [BaseClient](https://xsoar.pan.dev/docs/integrations/code-conventions#client-class) class from CommonServerPython.
+
+##### TLS/SSL Troubleshooting
+
+Examples of common errors indicating that there is an issue with trusting a TLS/SSL networking connection:
+
+* `SSLCertVerificationError`
+* `SSL_CERTIFICATE_VERIFY_FAILED`
+* `SSL: CERTIFICATE_VERIFY_FAILED`
+* `SSLError: certificate verify failed`
+
+These errors are usually as a result of a server using an untrusted certificate or a proxy (might be transparent) that is doing TLS/SSL termination.
+
+**Notes**
+
+* Most integrations provide a configuration option of *Trust any certificate*, which will cause the integration to ignore TLS/SSL certificate validation errors. You can use this option to test the connection and verify that in fact the issue is certificate related.
+* To trust custom certificates in Cortex XSOAR server or engines, follow the following [instructions](https://docs.paloaltonetworks.com/cortex/cortex-xsoar/6-0/cortex-xsoar-admin/docker/configure-python-docker-integrations-to-trust-custom-certificates).
+
+##### CertificatesTroubleshoot Automation
+
+Use the [CertificatesTroubleshoot Automation](https://xsoar.pan.dev/docs/reference/scripts/certificates-troubleshoot) to retrieve and decode an endpoint certificate. Additionally, use it to retrieve, decode and validiate the custom certificates deployed in Docker containers. The automation is part of the [Troubleshoot Pack](https://xsoar.pan.dev/marketplace/details/Troubleshoot).
+
+**Common reasons for TLS/SSL issues and resolutions**
+
+* Endpoint certificate issues:
+  * Expiration date - The certificate has a start and end date which is not valid anymore.
+
+    * Identify: `Endpoint certificate` -> `General`-> `NotValidBefore/NotValidAfter`:
+
+    ![image-20201018155224381](../../../docs/doc_imgs/reference/certificate-verification-expire-date.png)
+
+    * Resolution: If the certificate expired, make sure to renew the certificate at the target endpoint.
+
+  * Common name / Alt name -  A certificate signed only for a specific URI, For example, if the certificate is signed for `test.com` and the integration is accessing the endpoint using `test1.com` the certification validation will fail.
+
+    * Identify: `Endpoint certificate` -> `Subject` -> `CommonName` and `certificate` -> `Extentions` -> `SubjectAlternativeName`:
+
+      ![image-20201018160939173](../../../docs/doc_imgs/reference/certificate-verification-altnames.png)
+
+      ![image-20201018160950403](../../../docs/doc_imgs/reference/certificate-verification-common-name.png)
+
+    * Resolution: If the URI isn't matching the URI endpoint (Regex), try to access the endpoint with one of the alt names/common names. If the endpoint isn't accessible via trusted names, sign the certificate with the correct common name or apply an additional alt name.