Skip to content

Latest commit

 

History

History
2758 lines (2397 loc) · 98.7 KB

cs_annotations.md

File metadata and controls

2758 lines (2397 loc) · 98.7 KB
copyright lastupdated
years
2014, 2018
2018-11-13

{:new_window: target="_blank"} {:shortdesc: .shortdesc} {:screen: .screen} {:pre: .pre} {:table: .aria-labeledby="caption"} {:codeblock: .codeblock} {:tip: .tip} {:note: .note} {:important: .important} {:deprecated: .deprecated} {:download: .download}

Customizing Ingress with annotations

{: #ingress_annotation}

To add capabilities to your Ingress application load balancer (ALB), you can specify annotations as metadata in an Ingress resource. {: shortdesc}

Before you use annotations, make sure you have properly set up your Ingress service configuration by following the steps in Exposing apps with Ingress. Once you have set up the Ingress ALB with a basic configuration, you can then expand its capabilities by adding annotations to the Ingress resource file. {: note}

General annotations
General annotations Name Description
External services proxy-external-service Add path definitions to external services, such as a service hosted in {{site.data.keyword.Bluemix_notm}}.
Location modifier location-modifier Modify the way the ALB matches the request URI against the app path.
Location snippets location-snippets Add a custom location block configuration for a service.
Private ALB routing ALB-ID Route incoming requests to your apps with a private ALB.
Rewrite paths rewrite-path Route incoming network traffic to a different path that your back-end app listens on.
Server snippets server-snippets Add a custom server block configuration.
TCP ports tcp-ports Access an app via a non-standard TCP port.

Connection annotations
Connection annotations Name Description
Custom connect-timeouts and read-timeouts proxy-connect-timeout, proxy-read-timeout Set the time that the ALB waits to connect to and read from the back-end app before the back-end app is considered unavailable.
Keepalive requests keepalive-requests Set the maximum number of requests that can be served through one keepalive connection.
Keepalive timeout keepalive-timeout Set the maximum time that a keepalive connection stays open on the server.
Proxy next upstream proxy-next-upstream-config Set when the ALB can pass a request to the next upstream server.
Session-affinity with cookies sticky-cookie-services Always route incoming network traffic to the same upstream server by using a sticky cookie.
Upstream failtimeout upstream-fail-timeout Set the amount of time during which the ALB can attempt to connect to the server before the server is considered unavailable.
Upstream keepalive upstream-keepalive Set the maximum number of idle keepalive connections for an upstream server.
Upstream maxfails upstream-max-fails Set the maximum number of unsuccessful attempts to communicate with the server before the server is considered unavailable.

HTTPS and TLS/SSL authentication annotations
HTTPS and TLS/SSL authentication annotations Name Description
{{site.data.keyword.appid_short}} Authentication appid-auth Use {{site.data.keyword.appid_full}} to authenticate with your app.
Custom HTTP and HTTPS ports custom-port Change the default ports for HTTP (port 80) and HTTPS (port 443) network traffic.
HTTP redirects to HTTPS redirect-to-https Redirect insecure HTTP requests on your domain to HTTPS.
HTTP Strict Transport Security (HSTS) hsts Set the browser to access the domain only by using HTTPS.
Mutual authentication mutual-auth Configure mutual authentication for the ALB.
SSL services support ssl-services Allow SSL services support to encrypt traffic to your upstream apps that require HTTPS.

Istio annotations
Istio annotations Name Description
Istio services istio-services Route traffic to Istio-managed services.

Proxy buffer annotations
Proxy buffer annotations Name Description
Client response data buffering proxy-buffering Disable the buffering of a client response on the ALB while sending the response to the client.
Proxy buffers proxy-buffers Set the number and size of the buffers that read a response for a single connection from the proxied server.
Proxy buffer size proxy-buffer-size Set the size of the buffer that reads the first part of the response that is received from the proxied server.
Proxy busy buffers size proxy-busy-buffers-size Set the size of proxy buffers that can be busy.

Request and response annotations
Request and response annotations Name Description
Add server port to host header add-host-port Add the server port to the host for routing requests.
Client request body size client-max-body-size Set the maximum size of the body that the client can send as part of a request.
Large client header buffers large-client-header-buffers Set the maximum number and size of buffers that read large client request headers.
Additional client request or response header proxy-add-headers, response-add-headers Add header information to a client request before forwarding the request to your back-end app or to a client response before sending the response to the client.
Client response header removal response-remove-headers Remove header information from a client response before forwarding the response to the client.

Service limit annotations
Service limit annotations Name Description
Global rate limits global-rate-limit Limit the request processing rate and number of connections per a defined key for all services.
Service rate limits service-rate-limit Limit the request processing rate and the number of connections per a defined key for specific services.

General annotations

{: #general}

External services (proxy-external-service)

{: #proxy-external-service}

Add path definitions to external services, such as services hosted in {{site.data.keyword.Bluemix_notm}}. {:shortdesc}

Description
Add path definitions to external services. Use this annotation only when your app operates on an external service instead of a back-end service. When you use this annotation to create an external service route, only `client-max-body-size`, `proxy-read-timeout`, `proxy-connect-timeout`, and `proxy-buffering` annotations are supported in conjunction. Any other annotations are not supported in conjunction with `proxy-external-service`.

You cannot specify multiple hosts for a single service and path.

Sample Ingress resource YAML

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: cafe-ingress
  annotations:
    ingress.bluemix.net/proxy-external-service: "path=<mypath> external-svc=https:<external_service> host=<mydomain>"
spec:
  tls:
  - hosts:
    - mydomain
    secretName: mysecret
  rules:
  - host: mydomain
    http:
      paths:
      - path: /
        backend:
          serviceName: myservice
          servicePort: 80
Understanding the annotation components
Idea icon Understanding the annotation components
path Replace <mypath> with the path that the external service listens on.
external-svc Replace <external_service> with the external service to be called. For example, https://<myservice>.<region>.mybluemix.net.
host Replace <mydomain> with the host domain for the external service.

Location modifier (location-modifier)

{: #location-modifier}

Modify the way the ALB matches the request URI against the app path. {:shortdesc}

Description
By default, ALBs process the paths that apps listen on as prefixes. When an ALB receives a request to an app, the ALB checks the Ingress resource for a path (as a prefix) that matches the beginning of the request URI. If a match is found, the request is forwarded to the IP address of the pod where the app is deployed.

The `location-modifier` annotation changes the way the ALB searches for matches by modifying the location block configuration. The location block determines how requests are handled for the app path.

To handle regular expression (regex) paths, this annotation is required.

Supported modifiers
Supported modifiers
Modifier Description
= The equal sign modifier causes the ALB to select exact matches only. When an exact match is found, the search stops and the matching path is selected.
For example, if you app listens on /tea, the ALB selects only exact /tea paths when matching a request to your app.
~ The tilde modifier causes the ALB to process paths as case-sensitive regex paths during matching.
For example, if you app listens on /coffee, the ALB can select /ab/coffee or /123/coffee paths when matching a request to your app even though the paths are not explicitly set for your app.
~\* The tilde followed by an asterisk modifier causes the ALB to process paths as case-insensitive regex paths during matching.
For example, if you app listens on /coffee, the ALB can select /ab/Coffee or /123/COFFEE paths when matching a request to your app even though the paths are not explicitly set for your app.
^~ The carat followed by a tilde modifier causes the ALB to select the best non-regex match instead of a regex path.
Sample Ingress resource YAML
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: myingress
annotations:
  ingress.bluemix.net/location-modifier: "modifier='<location_modifier>' serviceName=<myservice1>;modifier='<location_modifier>' serviceName=<myservice2>"
spec:
  tls:
  - hosts:
    - mydomain
    secretName: mysecret
  rules:
  - host: mydomain
    http:
      paths:
      - path: /
        backend:
          serviceName: <myservice>
          servicePort: 80
Understanding the annotation components
Idea icon Understanding the annotation components
modifier Replace <location_modifier> with the location modifier you want to use for the path. Supported modifiers are '=', '~', '~\*', and '^~'. You must surround the modifiers in single quotes.
serviceName Replace <myservice> with the name of the Kubernetes service you created for your app.

Location snippets (location-snippets)

{: #location-snippets}

Add a custom location block configuration for a service. {:shortdesc}

Description
A server block is an nginx directive that defines the configuration for the ALB virtual server. A location block is an nginx directive defined within the server block. Location blocks define how Ingress processes the request URI, or the part of the request that comes after the domain name or IP address and port.

When a server block receives a request, the location block matches the URI to a path and the request is forwarded to the IP address of the pod where the app is deployed. By using the location-snippets annotation, you can modify how the location block forwards requests to particular services.

To modify the server block as a whole instead, see the server-snippets annotation.
Sample Ingress resource YAML
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: myingress
annotations:
  ingress.bluemix.net/location-snippets: |
    serviceName=<myservice>
    # Example location snippet
    proxy_request_buffering off;
    rewrite_log on;
    proxy_set_header "x-additional-test-header" "location-snippet-header";
    <EOS>
spec:
tls:
- hosts:
  - mydomain
  secretName: mytlssecret
rules:
- host: mydomain
  http:
    paths:
    - path: /
      backend:
        serviceName: <myservice>
        servicePort: 8080
Understanding the annotation components
Idea icon Understanding the annotation components
serviceName Replace <myservice> with the name of the service that you created for your app.
Location snippet Provide the configuration snippet that you want to use for the specified service. This sample snippet configures the location block to turn off proxy request buffering, turn on log rewrites, and set additional headers when it forwards a request to the myservice service.

Private ALB routing (ALB-ID)

{: #alb-id}

Route incoming requests to your apps with a private ALB. {:shortdesc}

Description
Choose a private ALB to route incoming requests instead of the public ALB.
Sample Ingress resource YAML
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: myingress
annotations:
  ingress.bluemix.net/ALB-ID: "<private_ALB_ID>"
spec:
tls:
- hosts:
  - mydomain
  secretName: mytlssecret
rules:
- host: mydomain
  http:
    paths:
    - path: /
      backend:
        serviceName: myservice
        servicePort: 8080
Understanding the annotation components
Idea icon Understanding the annotation components
<private_ALB_ID> The ID for your private ALB. To find the private ALB ID, run ibmcloud ks albs --cluster <my_cluster>.

If you have a multizone cluster with more than one private ALB enabled, you can provide a list of ALB IDs separated by ;. For example: ingress.bluemix.net/ALB-ID: <private_ALB_ID_1>;<private_ALB_ID_2>;<private_ALB_ID_3>

Rewrite paths (rewrite-path)

{: #rewrite-path}

Route incoming network traffic on an ALB domain path to a different path that your back-end app listens on. {:shortdesc}

Description
Your Ingress ALB domain routes incoming network traffic on mykubecluster.us-south.containers.appdomain.cloud/beans to your app. Your app listens on /coffee, instead of /beans. To forward incoming network traffic to your app, add the rewrite annotation to your Ingress resource configuration file. The rewrite annotation ensures that incoming network traffic on /beans is forwarded to your app by using the /coffee path. When including multiple services, use only a semi-colon (;) to separate them.
Sample Ingress resource YAML
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: myingress
  annotations:
    ingress.bluemix.net/rewrite-path: "serviceName=<myservice1> rewrite=<target_path1>;serviceName=<myservice2> rewrite=<target_path2>"
spec:
  tls:
  - hosts:
    - mydomain
    secretName: mysecret
  rules:
  - host: mydomain
    http:
      paths:
      - path: /beans
        backend:
          serviceName: myservice1
          servicePort: 80
Understanding the annotation components
Idea icon Understanding the annotation components
serviceName Replace <myservice> with the name of the Kubernetes service that you created for your app.
rewrite Replace <target_path> with the path that your app listens on. Incoming network traffic on the ALB domain is forwarded to the Kubernetes service by using this path. Most apps do not listen on a specific path, but use the root path and a specific port. In the example above, the rewrite path was defined as /coffee.

Server snippets (server-snippets)

{: #server-snippets}

Add a custom server block configuration. {:shortdesc}

Description
A server block is an nginx directive that defines the configuration for the ALB virtual server. By using the server-snippets annotation, you can modify how the ALB handles requests by providing a custom configuration snippet.
Sample Ingress resource YAML
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: myingress
annotations:
  ingress.bluemix.net/server-snippets: |
    location = /health {
    return 200 'Healthy';
    add_header Content-Type text/plain;
    }
spec:
tls:
- hosts:
  - mydomain
  secretName: mytlssecret
rules:
- host: mydomain
  http:
    paths:
    - path: /
      backend:
        serviceName: <myservice>
        servicePort: 8080
Understanding the annotation components
Idea icon Understanding the annotation components
Server snippet Provide the configuration snippet that you want to use. This sample snippet specifies a location block to handle /health requests. The location block is configured to return a healthy response and add a header when it forwards a request.

TCP ports for appplication load balancers (tcp-ports)

{: #tcp-ports}

Access an app via a non-standard TCP port. {:shortdesc}

Description
Use this annotation for an app that is running a TCP streams workload.

The ALB operates in pass-through mode and forwards traffic to back-end apps. SSL termination is not supported in this case. The TLS connection is not terminated and passes through untouched.

Sample Ingress resource YAML
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: myingress
annotations:
  ingress.bluemix.net/tcp-ports: "serviceName=<myservice> ingressPort=<ingress_port> [servicePort=<service_port>]"
spec:
  tls:
  - hosts:
    - mydomain
    secretName: mysecret
  rules:
  - host: mydomain
    http:
      paths:
      - path: /
        backend:
          serviceName: <myservice>
          servicePort: 80
Understanding the annotation components
Idea icon Understanding the annotation components
serviceName Replace <myservice> with the name of the Kubernetes service to access over non-standard TCP port.
ingressPort Replace <ingress_port> with the TCP port on which you want to access your app.
servicePort This parameter is optional. When provided, the port is substituted to this value before traffic is sent to the back-end app. Otherwise, the port remains same as the Ingress port.
Usage
  1. Review open ports for your ALB. 
    kubectl get service -n kube-system
    Your CLI output looks similar to the following: 
    NAME                                             TYPE           CLUSTER-IP       EXTERNAL-IP    PORT(S)                      AGE
public-cr18e61e63c6e94b658596ca93d087eed9-alb1   LoadBalancer   10.xxx.xx.xxx    169.xx.xxx.xxx 80:30416/TCP,443:32668/TCP   109d
  2. Open the ALB config map. 
    kubectl edit configmap ibm-cloud-provider-ingress-cm -n kube-system
  3. Add the TCP ports to the config map. Replace <port> with the TCP ports that you want to open.

    By default, ports 80 and 443 are open. If you want to keep 80 and 443 open, you must also include them in addition to any other TCP ports you specify in the `public-ports` field. If you enabled a private ALB, you must also specify any ports you want to keep open in the `private-ports` field. For more information, see Opening ports in the Ingress ALB.

    apiVersion: v1
kind: ConfigMap
data:
 public-ports: 80;443;<port1>;<port2>
metadata:
 creationTimestamp: 2017-08-22T19:06:51Z
 name: ibm-cloud-provider-ingress-cm
 namespace: kube-system
 resourceVersion: "1320"
 selfLink: /api/v1/namespaces/kube-system/configmaps/ibm-cloud-provider-ingress-cm
 uid: <uid>
  4. Verify that your ALB is re-configured with the TCP ports. 
    kubectl get service -n kube-system
    Your CLI output looks similar to the following: 
    NAME                                             TYPE           CLUSTER-IP       EXTERNAL-IP    PORT(S)                      AGE
public-cr18e61e63c6e94b658596ca93d087eed9-alb1   LoadBalancer   10.xxx.xx.xxx  169.xx.xxx.xxx <port1>:30776/TCP,<port2>:30412/TCP   109d
  5. Configure your Ingress to access your app via a non-standard TCP port. Use the sample YAML file in this reference.
  6. Either create your ALB resource or update your existing ALB configuration. 
    kubectl apply -f myingress.yaml
  7. Curl the Ingress subdomain to access your app. Example: curl <domain>:<ingressPort>

Connection annotations

{: #connection}

Custom connect-timeouts and read-timeouts (proxy-connect-timeout, proxy-read-timeout)

{: #proxy-connect-timeout}

Set the time that the ALB waits to connect to and read from the back-end app before the back-end app is considered unavailable. {:shortdesc}

Description
When a client request is sent to the Ingress ALB, a connection to the back-end app is opened by the ALB. By default, the ALB waits 60 seconds to receive a reply from the back-end app. If the back-end app does not reply within 60 seconds, then the connection request is aborted and the back-end app is considered to be unavailable.



After the ALB is connected to the back-end app, response data is read from the back-end app by the ALB. During this read operation, the ALB waits a maximum of 60 seconds between two read operations to receive data from the back-end app. If the back-end app does not send data within 60 seconds, the connection to the back-end app is closed and the app is considered to be not available.

A 60 second connect-timeout and read-timeout is the default timeout on a proxy and usually should not be changed.

If the availability of your app is not steady or your app is slow to respond because of high workloads, you might want to increase the connect-timeout or read-timeout. Keep in mind that increasing the timeout impacts the performance of the ALB as the connection to the back-end app must stay open until the timeout is reached.

On the other hand, you can decrease the timeout to gain performance on the ALB. Ensure that your back-end app is able to handle requests within the specified timeout, even during higher workloads.

Sample Ingress resource YAML
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
 name: myingress
 annotations:
   ingress.bluemix.net/proxy-connect-timeout: "serviceName=<myservice> timeout=<connect_timeout>"
   ingress.bluemix.net/proxy-read-timeout: "serviceName=<myservice> timeout=<read_timeout>"
spec:
 tls:
 - hosts:
   - mydomain
   secretName: mytlssecret
 rules:
 - host: mydomain
   http:
     paths:
     - path: /
       backend:
         serviceName: myservice
         servicePort: 8080
Understanding the annotation components
Idea icon Understanding the annotation components
<connect_timeout> The number of seconds or minutes to wait to connect to the back-end app, for example 65s or 1m. A connect-timeout cannot exceed 75 seconds.
<read_timeout> The number of seconds or minutes to wait before the back-end app is read, for example 65s or 2m.

Keepalive requests (keepalive-requests)

{: #keepalive-requests}

Set the maximum number of requests that can be served through one keepalive connection. {:shortdesc}

Description
Sets the maximum number of requests that can be served through one keepalive connection.
Sample Ingress resource YAML
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: myingress
annotations:
  ingress.bluemix.net/keepalive-requests: "serviceName=<myservice> requests=<max_requests>"
spec:
tls:
- hosts:
  - mydomain
  secretName: mytlssecret
rules:
- host: mydomain
  http:
    paths:
    - path: /
      backend:
        serviceName: <myservice>
        servicePort: 8080
Understanding the annotation components
Idea icon Understanding the annotation components
serviceName Replace <myservice> with the name of the Kubernetes service that you created for your app. This parameter is optional. The configuration is applied to all of the services in the Ingress host unless a service is specified. If the parameter is provided, the keepalive requests are set for the given service. If the parameter is not provided, the keepalive requests are set at the server level of the nginx.conf for all the services that do not have the keepalive requests configured.
requests Replace <max_requests> with the maximum number of requests that can be served through one keepalive connection.

Keepalive timeout (keepalive-timeout)

{: #keepalive-timeout}

Set the maximum time that a keepalive connection stays open on the server side. {:shortdesc}

Description
Sets the maximum time that a keepalive connection stays open on the server.
Sample Ingress resource YAML
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
 name: myingress
 annotations:
   ingress.bluemix.net/keepalive-timeout: "serviceName=<myservice> timeout=<time>s"
spec:
 tls:
 - hosts:
   - mydomain
   secretName: mytlssecret
 rules:
 - host: mydomain
   http:
     paths:
     - path: /
       backend:
         serviceName: myservice
         servicePort: 8080
Understanding the annotation components
Idea icon Understanding the annotation components
serviceName Replace <myservice> with the name of the Kubernetes service that you created for your app. This parameter is optional. If the parameter is provided, the keepalive timeout is set for the given service. If the parameter is not provided, the keepalive timeout is set at the server level of the nginx.conf for all the services that do not have the keepalive timeout configured.
timeout Replace <time> with an amount of time in seconds. Example: timeout=20s. A zero value disables the keepalive client connections.

Proxy next upstream (proxy-next-upstream-config)

{: #proxy-next-upstream-config}

Set when the ALB can pass a request to the next upstream server. {:shortdesc}

Description
The Ingress ALB acts as a proxy between the client app and your app. Some app setups require multiple upstream servers that handle incoming client requests from the ALB. Sometimes the proxy server that the ALB uses cannot establish a connection with an upstream server that the app uses. The ALB can then try to establish a connection with the next upstream server to pass the request to it instead. You can use the `proxy-next-upstream-config` annotation to set in which cases, how long, and how many times the ALB can try to pass a request to the next upstream server.

Timeout is always configured when you use `proxy-next-upstream-config`, so don't add `timeout=true` to this annotation.

Sample Ingress resource YAML
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: myingress
  annotations:
    ingress.bluemix.net/proxy-next-upstream-config: "serviceName=<myservice1> retries=<tries> timeout=<time> error=true http_502=true; serviceName=<myservice2> http_403=true non_idempotent=true"
spec:
  tls:
  - hosts:
    - mydomain
    secretName: mysecret
  rules:
  - host: mydomain
    http:
      paths:
      - path: /
        backend:
          serviceName: myservice1
          servicePort: 80
Understanding the annotation components
Idea icon Understanding the annotation components
serviceName Replace <myservice> with the name of the Kubernetes service that you created for your app.
retries Replace <tries> with the maximum amount of times that the ALB tries to pass a request to the next upstream server. This number includes the original request. To turn off this limitation, use 0. If you do not specify a value, the default value 0 is used.
timeout Replace <time> with the maximum amount of time, in seconds, that the ALB tries to pass a request to the next upstream server. For example, to set a time of 30 seconds, enter 30s. To turn off this limitation, use 0. If you do not specify a value, the default value 0 is used.
error If set to true, the ALB passes a request to the next upstream server when an error occurred while establishing a connection with the first upstream server, passing a request to it, or reading the response header.
invalid_header If set to true, the ALB passes a request to the next upstream server when the first upstream server returns an empty or invalid response.
http_502 If set to true, the ALB passes a request to the next upstream server when the first upstream server returns a response with the code 502. You can designate the following HTTP response codes: 500, 502, 503, 504, 403, 404, 429.
non_idempotent If set to true, the ALB can pass requests with a non-idempotent method to the next upstream server. By default, the ALB does not pass these requests to the next upstream server.
off To prevent the ALB from passing requests to the next upstream server, set to true.

Session-affinity with cookies (sticky-cookie-services)

{: #sticky-cookie-services}

Use the sticky cookie annotation to add session affinity to your ALB and always route incoming network traffic to the same upstream server. {:shortdesc}

Description
For high availability, some app setups require you to deploy multiple upstream servers that handle incoming client requests. When a client connects to you back-end app, you can use session-affinity so that a client is served by the same upstream server for the duration of a session or for the time it takes to complete a task. You can configure your ALB to ensure session-affinity by always routing incoming network traffic to the same upstream server.



Every client that connects to your back-end app is assigned to one of the available upstream servers by the ALB. The ALB creates a session cookie that is stored in the client's app, which is included in the header information of every request between the ALB and the client. The information in the cookie ensures that all requests are handled by the same upstream server throughout the session.



When you include multiple services, use a semi-colon (;) to separate them.

Sample Ingress resource YAML
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: myingress
  annotations:
    ingress.bluemix.net/sticky-cookie-services: "serviceName=<myservice1> name=<cookie_name1> expires=<expiration_time1> path=<cookie_path1> hash=<hash_algorithm1>;serviceName=<myservice2> name=<cookie_name2> expires=<expiration_time2> path=<cookie_path2> hash=<hash_algorithm2>"
spec:
  tls:
  - hosts:
    - mydomain
    secretName: mytlssecret
  rules:
  - host: mydomain
    http:
      paths:
      - path: /service1_path
        backend:
          serviceName: <myservice1>
          servicePort: 8080
      - path: /service2_path
        backend:
          serviceName: <myservice2>
          servicePort: 80
Understanding the annotation components
Idea icon Understanding the annotation components
serviceName Replace <myservice> with the name of the Kubernetes service that you created for your app.
name Replace <cookie_name> with the name of a sticky cookie that is created during a session.
expires Replace <expiration_time> with the time in seconds (s), minutes (m), or hours (h) before the sticky cookie expires. This time is independent of the user activity. After the cookie is expired, the cookie is deleted by the client web browser and no longer sent to the ALB. For example, to set an expiration time of 1 second, 1 minute, or 1 hour, enter 1s, 1m, or 1h.
path Replace <cookie_path> with the path that is appended to the Ingress subdomain and that indicates for which domains and subdomains the cookie is sent to the ALB. For example, if your Ingress domain is www.myingress.com and you want to send the cookie in every client request, you must set path=/. If you want to send the cookie only for www.myingress.com/myapp and all its subdomains, then you must set path=/myapp.
hash Replace <hash_algorithm> with the hash algorithm that protects the information in the cookie. Only sha1 is supported. SHA1 creates a hash sum based on the information in the cookie and appends this hash sum to the cookie. The server can decrypt the information in the cookie and verify data integrity.

Upstream failtimeout (upstream-fail-timeout)

{: #upstream-fail-timeout}

Set the amount of time during which the ALB can attempt to connect to the server. {:shortdesc}

Description
Set the amount of time during which the ALB can attempt to connect to a server before the server is considered unavailable. For a server to be considered unavailable, the ALB must hit the maximum number of failed connection attempts set by the upstream-max-fails annotation within the set amount of time. This amount of time also determines how long the server is considered unavailable.
Sample Ingress resource YAML
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: myingress
  annotations:
    ingress.bluemix.net/upstream-fail-timeout: "serviceName=<myservice> fail-timeout=<fail_timeout>"
spec:
  tls:
  - hosts:
    - mydomain
    secretName: mytlssecret
  rules:
  - host: mydomain
    http:
      paths:
      - path: /
        backend:
          serviceName: myservice
          servicePort: 8080
Idea icon Understanding the annotation components
serviceName(Optional) Replace <myservice> with the name of the Kubernetes service that you created for your app.
fail-timeout Replace <fail_timeout> with the amount of time that the ALB can attempt to connect to a server before the server is considered unavailable. The default is 10s. Time must be in seconds.

Upstream keepalive (upstream-keepalive)

{: #upstream-keepalive}

Set the maximum number of idle keepalive connections for an upstream server. {:shortdesc}

Description
Set the maximum number of idle keepalive connections to the upstream server of a given service. The upstream server has 64 idle keepalive connections by default.
Sample Ingress resource YAML
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: myingress
  annotations:
    ingress.bluemix.net/upstream-keepalive: "serviceName=<myservice> keepalive=<max_connections>"
spec:
  tls:
  - hosts:
    - mydomain
    secretName: mytlssecret
  rules:
  - host: mydomain
    http:
      paths:
      - path: /
        backend:
          serviceName: myservice
          servicePort: 8080
Understanding the annotation components
Idea icon Understanding the annotation components
serviceName Replace <myservice> with the name of the Kubernetes service that you created for your app.
keepalive Replace <max_connections> with the maximum number of idle keepalive connections to the upstream server. The default is 64. A 0 value disables upstream keepalive connections for the given service.

Upstream maxfails (upstream-max-fails)

{: #upstream-max-fails}

Set the maximum number of unsuccessful attempts to communicate with the server. {:shortdesc}

Description
Set the maximum number of times the ALB can fail to connect to the server before the server is considered unavailable. For the server to be considered unavailable, the ALB must hit the maximum number within the duration of time set by the upstream-fail-timeout annotation. The duration of time that the server is considered unavailable is also set by the upstream-fail-timeout annotation.
Sample Ingress resource YAML
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: myingress
  annotations:
    ingress.bluemix.net/upstream-max-fails: "serviceName=<myservice> max-fails=<max_fails>"
spec:
  tls:
  - hosts:
    - mydomain
    secretName: mytlssecret
  rules:
  - host: mydomain
    http:
      paths:
      - path: /
        backend:
          serviceName: myservice
          servicePort: 8080
Idea icon Understanding the annotation components
serviceName(Optional) Replace <myservice> with the name of the Kubernetes service that you created for your app.
max-fails Replace <max_fails> with the maximum number of unsuccessful attempts the ALB can make to communicate with the server. The default is 1. A 0 value disables the annotation.

HTTPS and TLS/SSL authentication annotations

{: #https-auth}

{{site.data.keyword.appid_short_notm}} Authentication (appid-auth)

{: #appid-auth}

Use {{site.data.keyword.appid_full_notm}} to authenticate with your app. {:shortdesc}

Description
Authenticate web or API HTTP/HTTPS requests with {{site.data.keyword.appid_short_notm}}.

If you set the request type to web, a web request that contains an {{site.data.keyword.appid_short_notm}} access token is validated. If token validation fails, the web request is rejected. If the request does not contain an access token, then the request is redirected to the {{site.data.keyword.appid_short_notm}} login page. For {{site.data.keyword.appid_short_notm}} web authentication to work, cookies must be enabled in the user's browser.

If you set the request type to api, an API request that contains an {{site.data.keyword.appid_short_notm}} access token is validated. If the request does not contain an access token, a 401: Unauthorized error message is returned to the user.

For security reasons, {{site.data.keyword.appid_short_notm}} authentication only supports backends with TLS/SSL enabled.

Sample Ingress resource YAML
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: myingress
  annotations:
    ingress.bluemix.net/appid-auth: "bindSecret=<bind_secret> namespace=<namespace> requestType=<request_type> serviceName=<myservice>"
spec:
  tls:
  - hosts:
    - mydomain
    secretName: mytlssecret
  rules:
  - host: mydomain
    http:
      paths:
      - path: /
        backend:
          serviceName: myservice
          servicePort: 8080
Understanding the annotation components
Idea icon Understanding the annotation components
bindSecret Replace <bind_secret> with the Kubernetes secret which stores the bind secret for your {{site.data.keyword.appid_short_notm}} service instance.
namespace Replace <namespace> with the namespace of the bind secret. This field defaults to the `default` namespace.
requestType Replace <request_type> with the type of request you want to send to {{site.data.keyword.appid_short_notm}}. Accepted values are `web` or `api`. The default is `api`.
serviceName Replace <myservice> with the name of the Kubernetes service that you created for your app. This field is required. If a service name is not included, then the annotation is enabled for all services. If a service name is included, then the annotation is enabled only for that service. Separate multiple services with a comma (,).
Usage

Because the app uses {{site.data.keyword.appid_short_notm}} for authentication, you must provision an {{site.data.keyword.appid_short_notm}} instance, configure the instance with valid redirect URIs, and generate a bind secret by binding the instance to your cluster.

  1. Choose an existing or create a new {{site.data.keyword.appid_short_notm}} instance.

    • To use an existing instance, ensure that the service instance name doesn't contain spaces. To remove spaces, select the more options menu next to the name of your service instance and select Rename service.
    • To provision a new {{site.data.keyword.appid_short_notm}} instance:
      1. Replace the auto-filled Service name with your own unique name for the service instance. The service instance name can't contain spaces.
      2. Choose the same region that your cluster is deployed in.
      3. Click Create.

  2. Add redirect URLs for your app. A redirect URL is the callback endpoint of your app. To prevent phishing attacks, App ID validates the request URL against the whitelist of redirect URLs.

    1. In the {{site.data.keyword.appid_short_notm}} management console, navigate to Identity providers > Manage.
    2. Make sure that you have an Identity Provider selected. If no Identity Provider is selected, the user will not be authenticated but will be issued an access token for anonymous access to the app.
    3. In the Add web redirect URLs field, add redirect URLs for your app in the format http://<hostname>/<app_path>/appid_callback or https://<hostname>/<app_path>/appid_callback.
      • For example, an app that is registered with the IBM Ingress subdomain might look like https://mycluster.us-south.containers.appdomain.cloud/myapp1path/appid_callback.
      • An app that is registered with a custom domain might look like http://mydomain.net/myapp2path/appid_callback.

  3. Bind the {{site.data.keyword.appid_short_notm}} service instance to your cluster.

    ibmcloud ks cluster-service-bind <cluster_name_or_ID> <namespace> <service_instance_name>

    {: pre} When the service is successfully added to your cluster, a cluster secret is created that holds the credentials of your service instance. Example CLI output:

    ibmcloud ks cluster-service-bind mycluster mynamespace appid1
Binding service instance to namespace...
OK
Namespace:    mynamespace
Secret name:  binding-<service_instance_name>

    {: screen}

  4. Get the secret that was created in your cluster namespace.

    kubectl get secrets --namespace=<namespace>

    {: pre}

  5. Use the bind secret and the cluster namespace to add the appid-auth annotation to your Ingress resource.


Custom HTTP and HTTPS ports (custom-port)

{: #custom-port}

Change the default ports for HTTP (port 80) and HTTPS (port 443) network traffic. {:shortdesc}

Description
By default, the Ingress ALB is configured to listen for incoming HTTP network traffic on port 80 and for incoming HTTPS network traffic on port 443. You can change the default ports to add security to your ALB domain, or to enable only an HTTPS port.

To enable mutual authentication on a port, [configure the ALB to open the valid port](cs_ingress.html#opening_ingress_ports) and then specify that port in the [`mutual-auth` annotation](#mutual-auth). Do not use the `custom-port` annotation to specify a port for mutual authentication.

Sample Ingress resource YAML
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
 name: myingress
 annotations:
   ingress.bluemix.net/custom-port: "protocol=<protocol1> port=<port1>;protocol=<protocol2> port=<port2>"
spec:
 tls:
 - hosts:
   - mydomain
   secretName: mytlssecret
 rules:
 - host: mydomain
   http:
     paths:
     - path: /
       backend:
         serviceName: myservice
         servicePort: 8080
Understanding the annotation components
Idea icon Understanding the annotation components
<protocol> Enter http or https to change the default port for incoming HTTP or HTTPS network traffic.
<port> Enter the port number to use for incoming HTTP or HTTPS network traffic.

When a custom port is specified for either HTTP or HTTPS, the default ports are no longer valid for both HTTP and HTTPS. For example, to change the default port for HTTPS to 8443, but use the default port for HTTP, you must set custom ports for both: custom-port: "protocol=http port=80; protocol=https port=8443".
Usage
  1. Review open ports for your ALB. 
    kubectl get service -n kube-system
    Your CLI output looks similar to the following: 
    NAME                                             TYPE           CLUSTER-IP       EXTERNAL-IP    PORT(S)                      AGE
public-cr18e61e63c6e94b658596ca93d087eed9-alb1   LoadBalancer   10.xxx.xx.xxx    169.xx.xxx.xxx 80:30416/TCP,443:32668/TCP   109d
  2. Open the ALB config map. 
    kubectl edit configmap ibm-cloud-provider-ingress-cm -n kube-system
  3. Add the non-default HTTP and HTTPS ports to the config map. Replace <port> with the HTTP or HTTPS port that you want to open.

    By default, ports 80 and 443 are open. If you want to keep 80 and 443 open, you must also include them in addition to any other TCP ports you specify in the `public-ports` field. If you enabled a private ALB, you must also specify any ports you want to keep open in the `private-ports` field. For more information, see Opening ports in the Ingress ALB.

    apiVersion: v1
kind: ConfigMap
data:
 public-ports: <port1>;<port2>
metadata:
 creationTimestamp: 2017-08-22T19:06:51Z
 name: ibm-cloud-provider-ingress-cm
 namespace: kube-system
 resourceVersion: "1320"
 selfLink: /api/v1/namespaces/kube-system/configmaps/ibm-cloud-provider-ingress-cm
 uid: <uid>
  4. Verify that your ALB is re-configured with the non-default ports. 
    kubectl get service -n kube-system
    Your CLI output looks similar to the following: 
    NAME                                             TYPE           CLUSTER-IP       EXTERNAL-IP    PORT(S)                      AGE
public-cr18e61e63c6e94b658596ca93d087eed9-alb1   LoadBalancer   10.xxx.xx.xxx  169.xx.xxx.xxx <port1>:30776/TCP,<port2>:30412/TCP   109d
  5. Configure your Ingress to use the non-default ports when routing incoming network traffic to your services. Use the sample YAML file in this reference.
  6. Update your ALB configuration. 
    kubectl apply -f myingress.yaml
  7. Open your preferred web browser to access your app. Example: https://<ibmdomain>:<port>/<service_path>/

HTTP redirects to HTTPS (redirect-to-https)

{: #redirect-to-https}

Convert insecure HTTP client requests to HTTPS. {:shortdesc}

Description
You set up your Ingress ALB to secure your domain with the IBM-provided TLS certificate or your custom TLS certificate. Some users might try to access your apps by using an insecure http request to your ALB domain, for example http://www.myingress.com, instead of using https. You can use the redirect annotation to always convert insecure HTTP requests to HTTPS. If you do not use this annotation, insecure HTTP requests are not converted into HTTPS requests by default and might expose unencrypted confidential information to the public.



Redirecting HTTP requests to HTTPS is disabled by default.

Sample Ingress resource YAML
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
 name: myingress
 annotations:
   ingress.bluemix.net/redirect-to-https: "True"
spec:
 tls:
 - hosts:
   - mydomain
   secretName: mytlssecret
 rules:
 - host: mydomain
   http:
     paths:
     - path: /
       backend:
         serviceName: myservice
         servicePort: 8080

HTTP Strict Transport Security (hsts)

{: #hsts}

Description
HSTS instructs the browser to only access a domain by using HTTPS. Even if the user enters or follows a plain HTTP link, the browser strictly upgrades the connection to HTTPS.
Sample Ingress resource YAML
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: myingress
  annotations:
    ingress.bluemix.net/hsts: enabled=true maxAge=<31536000> includeSubdomains=true
spec:
  tls:
  - hosts:
    - mydomain
    secretName: mysecret
  rules:
  - host: mydomain
    http:
      paths:
      - path: /service1_path
        backend:
          serviceName: myservice1
          servicePort: 8443
      - path: /service2_path
        backend:
          serviceName: myservice2
          servicePort: 8444
Understanding the annotation components
Idea icon Understanding the annotation components
enabled Use true to enable HSTS.
maxAge Replace <31536000> with an integer representing how many seconds a browser will cache sending requests straight to HTTPS. The default is 31536000, which is equal to 1 year.
includeSubdomains Use true to tell the browser that the HSTS policy also applies to all subdomains of the current domain. The default is true.

Mutual authentication (mutual-auth)

{: #mutual-auth}

Configure mutual authentication for the ALB. {:shortdesc}

Description
Configure mutual authentication of downstream traffic for the Ingress ALB. The external client authenticates the server and the server also authenticates the client by using certificates. Mutual authentication is also known as certificate-based authentication or two-way authentication.

Use the `mutual-auth` annotation for SSL termination between the client and the Ingress ALB. Use the [`ssl-services` annotation](#ssl-services) for SSL termination between the Ingress ALB and the back-end app.
Pre-requisites
  • You must have a valid mutual authentication secret that contains the required ca.crt. To create a mutual authentication secret, see the steps at the end of this section.
  • To enable mutual authentication on a port other than 443, [configure the ALB to open the valid port](cs_ingress.html#opening_ingress_ports) and then specify that port in this annotation. Do not use the `custom-port` annotation to specify a port for mutual authentication.
Sample Ingress resource YAML
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: myingress
annotations:
  ingress.bluemix.net/mutual-auth: "secretName=<mysecret> port=<port> serviceName=<servicename1>,<servicename2>"
spec:
  tls:
  - hosts:
    - mydomain
    secretName: mytlssecret
  rules:
  - host: mydomain
    http:
      paths:
      - path: /
        backend:
          serviceName: myservice
          servicePort: 8080
Understanding the annotation components
Idea icon Understanding the annotation components
secretName Replace <mysecret> with a name for the secret resource.
port Replace <port> with the ALB port number.
serviceName Replace <servicename> with the name of one or more Ingress resources. This parameter is optional.

To create a mutual authentication secret:

  1. Generate a key and certificate in one of the following ways:
    • Generate a certificate authority (CA) cert and key from your certificate provider. If you have your own domain, purchase an official TLS certificate for your domain. Make sure the CN External link icon is different for each certificate.
    • For testing purposes, you can create a self-signed certificate by using OpenSSL. For more information, see this self-signed SSL certificate tutorial External link icon.
      1. Create a ca.key. 
        openssl genrsa -out ca.key 1024
        {: pre}
      2. Use the key to create a ca.crt. 
        openssl req -new -x509 -key ca.key -out ca.crt
        {: pre}
      3. Use the ca.crt to create a self-signed certificate. 
        openssl x509 -req -in example.org.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out example.org.crt
        {: pre}
  2. Convert the cert into base-64 External link icon.
  3. Create a secret YAML file using the cert. 
    apiVersion: v1
kind: Secret
metadata:
  name: ssl-my-test
type: Opaque
data:
  ca.crt: <ca_certificate>
    {: codeblock}
  4. Create the certificate as a Kubernetes secret. 
    kubectl create -f ssl-my-test
    {: pre}

SSL services support (ssl-services)

{: #ssl-services}

Allow HTTPS requests and encrypt traffic to your upstream apps. {:shortdesc}

Description
When your Ingress resource configuration has a TLS section, the Ingress ALB can handle HTTPS-secured URL requests to your app. By default, the ALB terminates the TLS termination and decrypts the request before using the HTTP protocol to forward the traffic to your apps. If you have apps that require the HTTPS protocol and need traffic to be encrypted, use the `ssl-services` annotation. With the `ssl-services` annotation, the ALB terminates the external TLS connection, then creates a new SSL connection between the ALB and the app pod. Traffic is re-encrypted before it is sent to the upstream pods.

If your back-end app can handle TLS and you want to add additional security, you can add one-way or mutual authentication by providing a certificate that is contained in a secret.

Use the `ssl-services` annotation for SSL termination between the Ingress ALB and the back-end app. Use the [`mutual-auth` annotation](#mutual-auth) for SSL termination between the client and the Ingress ALB.
Sample Ingress resource YAML
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: <myingressname>
  annotations:
    ingress.bluemix.net/ssl-services: |
      ssl-service=<myservice1> ssl-secret=<service1-ssl-secret>;
      ssl-service=<myservice2> ssl-secret=<service2-ssl-secret>
spec:
  tls:
  - hosts:
    - mydomain
    secretName: mysecret
  rules:
  - host: mydomain
    http:
      paths:
      - path: /service1_path
        backend:
          serviceName: myservice1
          servicePort: 8443
      - path: /service2_path
        backend:
          serviceName: myservice2
          servicePort: 8444
Understanding the annotation components
Idea icon Understanding the annotation components
ssl-service Replace <myservice> with the name of the service that requires HTTPS. Traffic is encrypted from the ALB to this app's service.
ssl-secret If your back-end app can handle TLS and you want to add additional security, replace <service-ssl-secret> with the one-way or mutual authentication secret for the service.
  • If you provide a one-way authentication secret, the value must contain the trusted.crt from the upstream server. To create a one-way secret, see the steps at the end of this section.
  • If you provide a mutual authentication secret, the value must contain the required ca.crt and ca.key that your app is expecting from the client. To create a mutual authentication secret, see the steps at the end of this section.

If you do not provide a secret, insecure connections are permitted. You might choose to omit a secret if want to test the connection and do not have certificates ready, or if your certificates are expired and you want to allow insecure connections.

To create a one-way authentication secret:

  1. Get the certificate authority (CA) key and certificate from your upstream server.

  2. Convert the cert into base-64 External link icon.

  3. Create a secret YAML file using the cert.

    apiVersion: v1
kind: Secret
metadata:
  name: ssl-my-test
type: Opaque
data:
  trusted.crt: <ca_certificate>

    {: codeblock}

    If you want to also enforce mutual authentication for upstream traffic, you can provide a client.crt and client.key in addition to the trusted.crt in the data section. {: tip}

  4. Create the certificate as a Kubernetes secret.

    kubectl create -f ssl-my-test

    {: pre}


**To create a mutual authentication secret:**
  1. Generate a key and certificate in one of the following ways:
    • Generate a certificate authority (CA) cert and key from your certificate provider. If you have your own domain, purchase an official TLS certificate for your domain. Make sure the CN External link icon is different for each certificate.
    • For testing purposes, you can create a self-signed certificate by using OpenSSL. For more information, see this self-signed SSL certificate tutorial External link icon.
      1. Create a ca.key. 
        openssl genrsa -out ca.key 1024
        {: pre}
      2. Use the key to create a ca.crt. 
        openssl req -new -x509 -key ca.key -out ca.crt
        {: pre}
      3. Use the ca.crt to create a self-signed certificate. 
        openssl x509 -req -in example.org.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out example.org.crt
        {: pre}
  2. Convert the cert into base-64 External link icon.
  3. Create a secret YAML file using the cert. 
    apiVersion: v1
kind: Secret
metadata:
  name: ssl-my-test
type: Opaque
data:
  ca.crt: <ca_certificate>
    {: codeblock}
  4. Create the certificate as a Kubernetes secret. 
    kubectl create -f ssl-my-test
    {: pre}

Istio annotations

{: #istio-annotations}

Istio services (istio-services)

{: #istio-services}

Route traffic to Istio-managed services. {:shortdesc}

Description

This annotation works only with Istio 0.7 and earlier.

If you have Istio-managed services, you can use a cluster ALB to route HTTP/HTTPS requests to the Istio Ingress controller. The Istio Ingress controller then routes the requests to the app services. In order to route traffic, you must make changes to the Ingress resources for both the cluster ALB and the Istio Ingress controller.

In the Ingress resource for the cluster ALB, you must:
  • specify the `istio-services` annotation
  • define the service path as the actual path the app listens on
  • define the service port as the port of the Istio Ingress controller

In the Ingress resource for the Istio Ingress controller, you must:
  • define the service path as the actual path the app listens on
  • define the service port as the HTTP/HTTPS port of the app service that is exposed by the Istio Ingress controller
Sample Ingress resource YAML for the cluster ALB
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: myingress
  annotations:
    ingress.bluemix.net/istio-services: "enable=true serviceName=<myservice1> istioServiceNamespace=<istio-namespace> istioServiceName=<istio-ingress-service>"
spec:
  tls:
  - hosts:
    - mydomain
    secretName: mytlssecret
  rules:
  - host: mydomain
    http:
      paths:
      - path: </myapp1>
        backend:
          serviceName: <myservice1>
          servicePort: <istio_ingress_port>
      - path: </myapp2>
        backend:
          serviceName: <myservice2>
          servicePort: <istio_ingress_port>
Understanding the YAML file components
Idea icon Understanding the YAML file components
enable To enable traffic routing to Istio-managed services, set to True.
serviceName Replace <myservice1> with the name of the Kubernetes service that you created for your Istio-managed app. Separate multiple services with a semi-colon (;). This field is optional. If you do not specify a service name, then all Istio-managed services are enabled for traffic routing.
istioServiceNamespace Replace <istio-namespace> with the Kubernetes namespace where Istio is installed. This field is optional. If you do not specify a namespace, then the istio-system namespace is used.
istioServiceName Replace <istio-ingress-service> with the name of the Istio Ingress service. This field is optional. If you do not specify the Istio Ingress service name, then service name istio-ingress is used.
path For each Istio-managed service that you want to route traffic to, replace </myapp1> with the back-end path that the Istio-managed service listens on. The path must correspond to the path that you defined in the Istio Ingress resource.
servicePort For each Istio-managed service that you want to route traffic to, replace <istio_ingress_port> with port of the Istio Ingress controller.
Usage

  1. Deploy your app. The example resources provided in these steps use the Istio sample app called BookInfo External link icon, which can be found in the istio-0.7.1/samples/bookinfo/kube repository.

    kubectl apply -f bookinfo.yaml -n istio-system

    {: pre}

  2. Set up Istio routing rules for the app. For example, in the Istio sample app called BookInfo, routing rules for each microservice External link icon are defined in the route-rule-all-v1.yaml file.

  3. Expose the app to the Istio Ingress controller by creating an Istio Ingress resource. The resource allows Istio features, such as monitoring and route rules, to be applied to traffic entering the cluster. For example, the following resource for the BookInfo app is pre-defined in the bookinfo.yaml file:

    apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: istio-ingress-resource
  annotations:
    kubernetes.io/ingress.class: "istio"
spec:
  rules:
  - http:
      paths:
      - path: /productpage
        backend:
          serviceName: productpage
          servicePort: 9080
      - path: /login
        backend:
          serviceName: productpage
          servicePort: 9080
      - path: /logout
        backend:
          serviceName: productpage
          servicePort: 9080
      - path: /api/v1/products.*
        backend:
          serviceName: productpage
          servicePort: 9080

    {: codeblock}

  4. Create the Istio Ingress resource.

    kubectl create -f istio-ingress-resource.yaml -n istio-system

    {: pre} You app is connected to the Istio Ingress controller.

  5. Get the IBM Ingress subdomain and Ingress secret for your cluster. The subdomain and secret are pre-registered for your cluster and are used as a unique public URL for your app.

    ibmcloud ks cluster-get <cluster_name_or_ID>

    {: pre}

  6. Connect the Istio Ingress controller to the IBM Ingress ALB for your cluster by creating an IBM Ingress resource. Example for the BookInfo app:

    apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ibm-ingress-resource
  annotations:
    ingress.bluemix.net/istio-services: "enabled=true serviceName=productpage istioServiceName=istio-ingress-resource"
spec:
  tls:
  - hosts:
    - mycluster-459249.us-south.containers.mybluemix.net
    secretName: mycluster-459249
  rules:
  - host: mycluster-459249.us-south.containers.mybluemix.net
    http:
      paths:
      - path: /productpage
        backend:
          serviceName: productpage
          servicePort: 9080
      - path: /login
        backend:
          serviceName: productpage
          servicePort: 9080
      - path: /logout
        backend:
          serviceName: productpage
          servicePort: 9080
      - path: /api/v1/products.*
        backend:
          serviceName: productpage
          servicePort: 9080

    {: codeblock}

  7. Create the IBM ALB Ingress resource.

    kubectl apply -f ibm-ingress-resource.yaml -n istio-system

    {: pre}

  8. In a browser, go to https://<hostname>/frontend to view the app web page.


Proxy buffer annotations

{: #proxy-buffer}

Client response data buffering (proxy-buffering)

{: #proxy-buffering}

Use the buffer annotation to disable the storage of response data on the ALB while the data is sent to the client. {:shortdesc}

Description
The Ingress ALB acts as a proxy between your back-end app and the client web browser. When a response is sent from the back-end app to the client, the response data is buffered on the ALB by default. The ALB proxies the client response and starts sending the response to the client at the client's pace. After all data from the back-end app is received by the ALB, the connection to the back-end app is closed. The connection from the ALB to the client remains open until the client receives all data.



If buffering of response data on the ALB is disabled, data is immediately sent from the ALB to the client. The client must be able to handle incoming data at the pace of the ALB. If the client is too slow, data might get lost.

Response data buffering on the ALB is enabled by default.

Sample Ingress resource YAML
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
 name: myingress
 annotations:
   ingress.bluemix.net/proxy-buffering: "enabled=<false> serviceName=<myservice1>"
spec:
 tls:
 - hosts:
   - mydomain
   secretName: mytlssecret
 rules:
 - host: mydomain
   http:
     paths:
     - path: /
       backend:
         serviceName: myservice
         servicePort: 8080
Understanding the annotation components
Idea icon Understanding the annotation components
enabled To disable response data buffering on the ALB, set to false.
serviceName Replace <myservice1> with the name of the Kubernetes service that you created for your app. Separate multiple services with a semi-colon (;). This field is optional. If you do not specify a service name, then all services use this annotation.

Proxy buffers (proxy-buffers)

{: #proxy-buffers}

Configure the number and size of proxy buffers for the ALB. {:shortdesc}

Description
Set the number and size of the buffers that read a response for a single connection from the proxied server. The configuration is applied to all of the services in the Ingress host unless a service is specified. For example, if a configuration such as serviceName=SERVICE number=2 size=1k is specified, 1k is applied to the service. If a configuration such as number=2 size=1k is specified, 1k is applied to all of the services in the Ingress host.

If you get the error message `upstream sent too big header while reading response header from upstream`, the upstream server in your backend sent a header size that is larger than the default limit. Increase the size for both proxy-buffers and [proxy-buffer-size](#proxy-buffer-size).

Sample Ingress resource YAML
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
 name: proxy-ingress
 annotations:
   ingress.bluemix.net/proxy-buffers: "serviceName=<myservice> number=<number_of_buffers> size=<size>"
spec:
 tls:
 - hosts:
   - mydomain
   secretName: mytlssecret
 rules:
 - host: mydomain
   http:
     paths:
     - path: /
       backend:
         serviceName: myservice
         servicePort: 8080
Understanding the annotation components
Idea icon Understanding the annotation components
serviceName Replace <myservice> with the name for a service to apply proxy-buffers.
number Replace <number_of_buffers> with a number, such as 2.
size Replace <size> with the size of each buffer in kilobytes (k or K), such as 1K.

Proxy buffer size (proxy-buffer-size)

{: #proxy-buffer-size}

Configure the size of the proxy buffer that reads the first part of the response. {:shortdesc}

Description
Set the size of the buffer that reads the first part of the response that is received from the proxied server. This part of the response usually contains a small response header. The configuration is applied to all of the services in the Ingress host unless a service is specified. For example, if a configuration such as serviceName=SERVICE size=1k is specified, 1k is applied to the service. If a configuration such as size=1k is specified, 1k is applied to all of the services in the Ingress host.
Sample Ingress resource YAML
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
 name: proxy-ingress
 annotations:
   ingress.bluemix.net/proxy-buffer-size: "serviceName=<myservice> size=<size>"
spec:
 tls:
 - hosts:
   - mydomain
   secretName: mytlssecret
 rules:
 - host: mydomain
   http:
     paths:
     - path: /
       backend:
         serviceName: myservice
         servicePort: 8080
Understanding the annotation components
Idea icon Understanding the annotation components
serviceName Replace <myservice> with the name of a service to apply proxy-buffers-size.
size Replace <size> with the size of each buffer in kilobytes (k or K), such as 1K.

Proxy busy buffers size (proxy-busy-buffers-size)

{: #proxy-busy-buffers-size}

Configure the size of proxy buffers that can be busy. {:shortdesc}

Description
Limit the size of any buffers that are sending a response to the client while the response is not yet fully read. In the meantime, the rest of the buffers can read the response and, if needed, buffer part of the response to a temporary file. The configuration is applied to all of the services in the Ingress host unless a service is specified. For example, if a configuration such as serviceName=SERVICE size=1k is specified, 1k is applied to the service. If a configuration such as size=1k is specified, 1k is applied to all of the services in the Ingress host.
Sample Ingress resource YAML
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
 name: proxy-ingress
 annotations:
   ingress.bluemix.net/proxy-busy-buffers-size: "serviceName=<myservice> size=<size>"
spec:
 tls:
 - hosts:
   - mydomain
   secretName: mytlssecret
 rules:
 - host: mydomain
   http:
     paths:
     - path: /
       backend:
         serviceName: myservice
         servicePort: 8080
Understanding the annotation components
Idea icon Understanding the annotation components
serviceName Replace <myservice> with the name of a service to apply proxy-busy-buffers-size.
size Replace <size> with the size of each buffer in kilobytes (k or K), such as 1K.

Request and response annotations

{: #request-response}

Add server port to host header (add-host-port)

{: #add-host-port}

Description
Add the `:server_port` to the host header of a client request before forwarding the request to your back-end app.
Sample Ingress resource YAML
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
 name: myingress
 annotations:
   ingress.bluemix.net/add-host-port: "enabled=<true> serviceName=<myservice>"
spec:
 tls:
 - hosts:
   - mydomain
   secretName: mytlssecret
 rules:
 - host: mydomain
   http:
     paths:
     - path: /
       backend:
         serviceName: myservice
         servicePort: 8080
Understanding the annotation components
Idea icon Understanding the annotation components
enabled To enable setting of server_port for the host, set to true.
serviceName Replace <myservice> with the name of the Kubernetes service that you created for your app. Separate multiple services with a semi-colon (;). This field is optional. If you do not specify a service name, then all services use this annotation.

Additional client request or response header (proxy-add-headers, response-add-headers)

{: #proxy-add-headers}

Add extra header information to a client request before sending the request to the back-end app or to a client response before sending the response to the client. {:shortdesc}

Description
The Ingress ALB acts as a proxy between the client app and your back-end app. Client requests that are sent to the ALB are processed (proxied) and put into a new request that is then sent to your back-end app. Similarly, back-end app responses that are sent to the ALB are processed (proxied) and put into a new response that is then sent to the client. Proxying a request or response removes HTTP header information, such as the user name, that was initially sent from the client or back-end app.

If your back-end app requires HTTP header information, you can use the proxy-add-headers annotation to add header information to the client request before the request is forwarded by the ALB to the back-end app. If the client web app requires HTTP header information, you can use the response-add-headers annotation to add header information to the response before the response is forwarded by the ALB to the client web app.
  • For example, you might need to add the following X-Forward header information to the request before it is forwarded to your app: 
    proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    To add the X-Forward header information to the request that is sent to your app, use the `proxy-add-headers` annotation in the following way: 
    ingress.bluemix.net/proxy-add-headers: |
  serviceName=<myservice1> {
  Host $host;
  X-Real-IP $remote_addr;
  X-Forwarded-Proto $scheme;
  X-Forwarded-For $proxy_add_x_forwarded_for;
  }

The response-add-headers annotation does not support global headers for all services. To add a header for all service responses at a server level, you can use the [server-snippets annotation](#server-snippets).

Sample Ingress resource YAML
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: myingress
  annotations:
    ingress.bluemix.net/proxy-add-headers: |
      serviceName=<myservice1> {
      <header1> <value1>;
      <header2> <value2>;
      }
      serviceName=<myservice2> {
      <header3> <value3>;
      }
    ingress.bluemix.net/response-add-headers: |
      serviceName=<myservice1> {
      <header1>:<value1>;
      <header2>:<value2>;
      }
      serviceName=<myservice2> {
      <header3>:<value3>;
      }
spec:
  tls:
  - hosts:
    - mydomain
    secretName: mytlssecret
  rules:
  - host: mydomain
    http:
      paths:
      - path: /service1_path
        backend:
          serviceName: <myservice1>
          servicePort: 8080
      - path: /service2_path
        backend:
          serviceName: <myservice2>
          servicePort: 80
Understanding the annotation components
Idea icon Understanding the annotation components
service_name Replace <myservice> with the name of the Kubernetes service that you created for your app.
<header> The key of the header information to add to the client request or client response.
<value> The value of the header information to add to the client request or client response.

Client response header removal (response-remove-headers)

{: #response-remove-headers}

Remove header information that is included in the client response from the back-end end app before the response is sent to the client. {:shortdesc}

Description
The Ingress ALB acts as a proxy between your back-end app and the client web browser. Client responses from the back-end app that are sent to the ALB are processed (proxied), and put into a new response that is then sent from the ALB to the client web browser. Although proxying a response removes http header information that was initially sent from the back-end app, this process might not remove all back-end app specific headers. Remove header information from a client response before the response is forwarded from the ALB to the client web browser.
Sample Ingress resource YAML
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: myingress
  annotations:
    ingress.bluemix.net/response-remove-headers: |
      serviceName=<myservice1> {
      "<header1>";
      "<header2>";
      }
      serviceName=<myservice2> {
      "<header3>";
      }
spec:
  tls:
  - hosts:
    - mydomain
    secretName: mytlssecret
  rules:
  - host: mydomain
    http:
      paths:
      - path: /service1_path
        backend:
          serviceName: <myservice1>
          servicePort: 8080
      - path: /service2_path
        backend:
          serviceName: <myservice2>
          servicePort: 80
Understanding the annotation components
Idea icon Understanding the annotation components
service_name Replace <myservice> with the name of the Kubernetes service that you created for your app.
<header> The key of the header to remove from the client response.

Client request body size (client-max-body-size)

{: #client-max-body-size}

Set the maximum size of the body that the client can send as part of a request. {:shortdesc}

Description
To maintain the expected performance, the maximum client request body size is set to 1 megabyte. When a client request with a body size over the limit is sent to the Ingress ALB, and the client does not allow data to be divided, the ALB returns a 413 (Request Entity Too Large) HTTP response to the client. A connection between the client and the ALB is not possible until the size of the request body is reduced. When the client allows data to be split up into multiple chunks, data is divided into packages of 1 megabyte and sent to the ALB.



You might want to increase the maximum body size because you expect client requests with a body size that is greater than 1 megabyte. For example, you want your client to be able to upload large files. Increasing the maximum request body size might impact the performance of your ALB because the connection to the client must stay open until the request is received.

Some client web browsers cannot display the 413 HTTP response message properly.

Sample Ingress resource YAML
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
 name: myingress
 annotations:
   ingress.bluemix.net/client-max-body-size: size=<size>
spec:
 tls:
 - hosts:
   - mydomain
   secretName: mytlssecret
 rules:
 - host: mydomain
   http:
     paths:
     - path: /
       backend:
         serviceName: myservice
         servicePort: 8080
Understanding the annotation components
Idea icon Understanding the annotation components
<size> The maximum size of the client response body. For example, to set the maximum size to 200 megabyte, define 200m. You can set the size to 0 to disable the check of the client request body size.

Large client header buffers (large-client-header-buffers)

{: #large-client-header-buffers}

Set the maximum number and size of buffers that read large client request headers. {:shortdesc}

Description
Buffers that read large client request headers are allocated only by demand: If a connection is transitioned into the keepalive state after the end-of-request processing, these buffers are released. By default, the buffer size is equal to 8K bytes. If a request line exceeds the set maximum size of one buffer, the 414 Request-URI Too Large error is returned to the client. Additionally, if a request header field exceeds the set maximum size of one buffer, the 400 Bad Request error is returned to the client. You can adjust the maximum number and size of buffers that are used for reading large client request headers.
Sample Ingress resource YAML
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
 name: myingress
 annotations:
   ingress.bluemix.net/large-client-header-buffers: "number=<number> size=<size>"
spec:
 tls:
 - hosts:
   - mydomain
   secretName: mytlssecret
 rules:
 - host: mydomain
   http:
     paths:
     - path: /
       backend:
         serviceName: myservice
         servicePort: 8080
Understanding the annotation components
Idea icon Understanding the annotation components
<number> The maximum number of buffers that should be allocated to read large client request header. For example, to set it to 4, define 4.
<size> The maximum size of buffers that read large client request header. For example, to set it to 16 kilobytes, define 16k. The size must end with a k for kilobyte or m for megabyte.

Service limit annotations

{: #service-limit}

Global rate limits (global-rate-limit)

{: #global-rate-limit}

Limit the request processing rate and number of connections per a defined key for all services. {:shortdesc}

Description
For all services, limit the request processing rate and the number of connections per a defined key that are coming from a single IP address for all paths of the selected backends.
Sample Ingress resource YAML
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: myingress
  annotations:
    ingress.bluemix.net/global-rate-limit: "key=<key> rate=<rate> conn=<number_of_connections>"
spec:
  tls:
  - hosts:
    - mydomain
    secretName: mytlssecret
  rules:
  - host: mydomain
    http:
      paths:
      - path: /
        backend:
          serviceName: myservice
          servicePort: 8080
Understanding the annotation components
Idea icon Understanding the annotation components
key To set a global limit for incoming requests based on the zone or service, use `key=zone`. To set a global limit for incoming requests based on the header, use `X-USER-ID key=$http_x_user_id`.
rate Replace <rate> with the processing rate. Enter a value as a rate per second (r/s) or rate per minute (r/m). Example: 50r/m.
number-of_connections Replace <conn> with the number of connections.

Service rate limits (service-rate-limit)

{: #service-rate-limit}

Limit the request processing rate and the number of connections for specific services. {:shortdesc}

Description
For specific services, limit the request processing rate and the number of connections per a defined key that are coming from a single IP address for all paths of the selected backends.
Sample Ingress resource YAML
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: myingress
  annotations:
    ingress.bluemix.net/service-rate-limit: "serviceName=<myservice> key=<key> rate=<rate> conn=<number_of_connections>"
spec:
  tls:
  - hosts:
    - mydomain
    secretName: mytlssecret
  rules:
  - host: mydomain
    http:
      paths:
      - path: /
        backend:
          serviceName: myservice
          servicePort: 8080
Understanding the annotation components
Idea icon Understanding the annotation components
serviceName Replace <myservice> with the name of the service for which you want to limit the processing rate.
key To set a global limit for incoming requests based on the zone or service, use `key=zone`. To set a global limit for incoming requests based on the header, use `X-USER-ID key=$http_x_user_id`.
rate Replace <rate> with the processing rate. To define a rate per second, use r/s: 10r/s. To define a rate per minute, use r/m: 50r/m.
number-of_connections Replace <conn> with the number of connections.