Proposal: define custom hostname for functions

[alexellis]

Expected Behaviour

Within the function definition we should provide an alternative configuration for when someone wants to serve a website or api from a function such as:

Imagine a microservice/function named giving-page

It would be exposed as per:

gw:port/function/giving-page

• Optional user-friend URLs

If a user would like a more user-friendly or custom URL than what they currently have by default

• Added security when managing / issuing cookies

Cookies issued to gw:port/function/name1 may also be accessible by gw:port/function/name2, by allowing custom domains we can have more separation at the domain level

• Make this more streamlined / simple for users

User currently need to do this via an ingress path in Kubernetes or a custom configuration for their reverse proxy of choice.

This proposal would allow a custom domain to be routed such as:

giving-page.my-domain.com
giving-page.com

The way we add arbitrary metadata to functions is via annotations, such as how we define topics for the connector SDK / triggers.

functions:
  giving-page:
    image: ae/annotations:0.1
    annotations:
      url: giving-page.com

The gateway would look for a Host header, and if that matched a function queried via the /system/functions/ API (to be cached in-memory) then the traffic would be forwarded to the function using a proxy and any other gateway endpoints would not respond. i.e. you would not be able to call giving-page.com/system/functions for instance or giving-page.com/ui.

A query and cache for /system/functions is already being built up and consumed when scale_from_zero is enabled on the gateway. It makes available a list of functions and how many readyReplicas they have. This could be extended, or a single mechanism could populate this cache and a second cache.

I don't think that multiple URLs (hosts) should be specified at this point, since they will point to the same function.

In production use, the user would still have to manage and configure DNS records and TLS certificates. This change would only be concerned with reverse-proxying requests to functions.

Current Behaviour

• Define an ingress path pointing to the gateway.
• Create a custom reverse-proxy rule i.e. in nginx

[alexellis]

@LucasRoesler @stefanprodan

The DNS variant is probably the lowest hanging fruit?

[LucasRoesler]

If I understand the suggestion correctly, the core of this implementation is to add a Host header check during the function invocation.

Given the above and reiterating some of what you said, I understand that we need to

• add a "search by annotation" to the /system/functions, to be implemented in the providers
• add a in-memory cache to the gateway to cache the "search by annotation"
• add a cache TTL of X seconds, (30s?, more? less?)
• add a cache invalidation when a function is deployed?

Do we want to make this optional via a flag? I am a little worried about adding an additional call to the provider for every function invocation, even with caching. Do we always expect the gateway to always be deployed as single replica? If the replica count is increased, then it is possible that waiting for the cache to populate would really make the initial function invocations slow and if the usage is low enough or TTL long enough, this would then impact all function calls

I am also worried about the cache invalidation. What happens when the function is re-deployed? What if someone is using the openfaas-operator? The gateway won't always see the deployment and could miss a cache invalidation event. For example, if the url annotation is changed, the gateway will potentially serve the wrong function or 404s.

[alexellis]

If you're worried about this, then I can't be doing a very good job of explaining it.

Let me try to introduce a diagram to help.

Users are unsure how to map pay.example.com to their function that serves that API i.e. the pay function mounted at http://gateway.openfaas:8080/function/pay. The upshot is that they now have to manage some "host path" mappings in their Ingress definitions.

I.e. pay.example.com => http://gateway.openfaas:8080/function/pay

We can make this easier for them to manage and have them just map it like this:

I.e. pay.example.com => http://gateway.openfaas:8080

Then when we detect the Host: header of pay.example.com (already present from the client's request) we can use that to match the correct function and route accordingly.

[stefanprodan]

Maybe URL is not the best representation of this feature, I would call it custom host name.

[LucasRoesler]

I like the idea of having the ingress handle host name => gateway routing. And the diagram is really clear.

My only concern is around the caching you originally mentioned.

Would the simplest configuration be to allow the admin to pass a list of allowed host names and then allow matching functions as subdomains of those hosts?

E.g. --allowed-host=example.com --allowed-host=team1.example.com --allowed-host=team2.example.com would allow pay.example.com => http://gateway.openfaas:8080/function/pay and pay.team1.example.com => http://gateway.openfaas:8080/function/pay

If we want to allow additional host to be added via annotations on functions this could simply be scrapped from the function and added to the list from the flags during gateway startup? You want it during gateway startup to ensure that it is up-to-date after a rolling deployment or if the container is rescheduled. Run it on a schedule in the background of the gateway and it would keep it up-to-date and allow HA deployments of the gateway.

[alexellis]

#1082 (comment)

^ Yes 👍

Changed that to "custom hostname"

Here's an example of what some users are doing today:

• Example 1:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: api
  namespace: openfaas
  labels:
    app: faas-netes
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /function/$1
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    certmanager.k8s.io/issuer: acme-issuer
spec:
  tls:
  - hosts:
    - api.example.com
    secretName: api-tls-certificate
  rules:
  - host: api.example.com
    http:
      paths:
      - path: /(.*)
        backend:
          serviceName: gateway
          servicePort: 8080

• Example 2

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: www-ingress
  namespace: openfaas
  labels:
    app: faas-netes
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /function/www/$1
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    certmanager.k8s.io/issuer: acme-issuer
spec:
  tls:
  - hosts:
    - example.com
    secretName: www-tls-certificate
  rules:
  - host: example.com
    http:
      paths:
      - path: /(.*)
        backend:
          serviceName: gateway
          servicePort: 8080

Sample from Brian Woodward on Slack.

I don't think this would go away, but it wouldn't have the path mappings anymore.

[MarcusNoble]

This sounds pretty much like what I'm looking for but I'm curious as to why the Ingress points to the gateway service rather than directly at the functions service? My guess is it's something to do with the scale to zero behaviour?

The way I had envisioned it was you have a function defined something like this:

functions:
  giving-page:
    image: example/example:0.1
    ingress:
      host: giving-page.com

When this is created in Kubernetes this code (https://github.com/openfaas/faas-netes/blob/master/handlers/deploy.go) would check for the presence of the ingress field and generate an ingress resource pointing at the service it creates.

I would prefer not to have it interpreted from annotations unless a strict schema is used such as com.openfaas.ingress.url so there is no chance of clash.

[alexellis]

I would suggest you have a look at how OpenFaaS Cloud works, it may be suitable for your use-case. A single ingress definition is created and points at the edge-router service. A wild-card DNS entry is then used to map and the edge-router creates the appropriate function URLs for invocations.

• edge-router - subdomain per user, could be adapted to sub-domain per function
• conceptual architecture diagram

1-1 mapping between functions and Kubernetes Ingress records would not be practical for a number of reasons.

• The gateway is a required middleware for all function invocations
• The gateway applies authz
• The gateway tracks metrics and call-id
• The gateway provides asynchronous capabilities
• The gateway controls horizontal pod scaling
• The gateway metrics are used by faas-idler for scale to zero
• The gateway provides scale from zero

If you simply wish to use the gateway to deploy your service endpoint and want none of the functionality of OpenFaaS beyond that, then feel free to create individual ingress records, this would be counter to the design and ethos of the architecture and is not recommended.

Typically when people create Ingress records they re-write the host path and point to the gateway. This is something that we are exploring above, but without the automatic creation and management of Ingress records.

Ingress does not suit everybody and OpenFaaS does not target only a single platform. Ref: https://blog.alexellis.io/the-power-of-interfaces-openfaas/

• Often many customisations are required for the specific IngressController and / or certificate issuer - i.e. Nginx / cert-manager
• Ambassador does not support Ingress
• Istio does not use Ingress

[MarcusNoble]

Typically when people create Ingress records they re-write the host path and point to the gateway. This is something that we are exploring above, but without the automatic creation and management of Ingress records.

Ah yes! This is actually all we need. Thanks!

[alexellis]

I've started some work on this and would love feedback:

https://github.com/openfaas-incubator/ingress-operator

[MarcusNoble]

@alexellis Do you want feedback here on this issue or should I open issue on that repo?

[alexellis]

Slack would be ideal in the #kubernetes channel.

[alexellis]

The FunctionIngress CRD is now documented at: https://docs.openfaas.com/reference/ssl/kubernetes-with-cert-manager/

[alexellis]

I've published a blog / tutorial that walks through this in more detail. https://www.openfaas.com/blog/custom-domains-function-ingress/

Would be good to get feedback from those users who needed this feature.

[brandonkal]

What about async though?
pay.example.com => http://gateway.openfaas:8080

[alexellis]

/lock: resolved through https://github.com/openfaas/ingress-operator