Skip to content

rancher

Jump to bottom
arnaud gaboury edited this page Sep 6, 2018 · 1 revision

Rancher

Rancher is a tool built to simplify Docker orchestration and management. In Rancher, everything (like containers, networks or images ) is an API resource with a process lifecycle. Containers, images, networks, and accounts are all API resources with their own process li

Basic notions

Microservice

Instead of building a single monstrous, monolithic application, the idea is to split your application into set of smaller, interconnected services.

The rise of microservices has been a remarkable advancement in application development and deployment. With microservices, an application is developed, or refactored, into separate services that “speak” to one another in a well-defined way – via APIs, for instance. Each microservice is self-contained, each maintains its own data store (which has significant implications), and each can be updated independently of others.

Moving to a microservices-based approach makes app development faster and easier to manage, requiring fewer people to implement more new features. Changes can be made and deployed faster and easier. An application designed as a collection of microservices is easier to run on multiple servers with load balancing, making it easy to handle demand spikes and steady increases in demand over time, while reducing downtime caused by hardware or software problems.

API

The term API stands for Application Programming Interface. The term can be used to describe the features of a library, or how to interact with it. Your favorite library may have "API Documentation" which documents which functions are available, how you call them, which arguments are required, etc.

However, these days, when people refer to an API they are most likely referring to an HTTP API, which can be a way of sharing application data over the internet.

For example, let's say we have an application that allows you to view,create, edit, and delete widgets. We could create an HTTP API that allows you to perform these functions:

http://example.com/view_widgets

http://example.com/create_new_widget?name=Widgetizer

http://example.com/update_widget?id=123&name=Foo

http://example.com/delete_widget?id=123

A problem has started to arise when everyone starts implementing their own APIs. Without a standard way of naming URLs, you always have to refer to the documentation to understand how the API works. One API might have a URL like /view_widgets whereas another API might use /widget. REST comes to rescue us from this mess.

REST

REST stands for Representational State Transfer. This is a term to describe a standard way of creating HTTP APIs.

HTTP methods

There are technically 8 different HTTP methods:

GET
POST
PUT
DELETE
OPTIONS
HEAD
TRACE
CONNECT

Examples of REST

Let's look at a few examples of what makes an API RESTful. Using our widgets example again...

If we wanted to view all widgets, the URL would look like this:

GET http://example.com/widgets

Create a new widget by posting the data:

POST http://example.com/widgets

Data: name = Foobar

To view a single widget we "get" it by specifying that widget's id:

GET http://example.com/widgets/123

Update that widget by "putting" the new data:

PUT http://example.com/widgets/123

Data: name = New name color = blue

Delete that widget:

DELETE http://example.com/widgets/123

In the above examples, REST URLs use a consistent naming scheme. When interacting with an API, we are almost always manipulating some sort of object. In our examples, this is a Widget. In REST terminology, this is called a Resource. The first part of the URL is always the plural form of the resource:

/widgets

This is always used when referring to this collection of resources ("list all" and "add one" actions). When working with a specific resource, we add the ID to the URL:

/widgets/123

HTTP Status Codes

Another important part of REST is responding with the correct status code for the type of request that was made. If you're new to HTTP status codes, heres a quick summary. When you make an HTTP request, the server will respond with a code which corresponds to whether or not the request was successful and how the client should proceed. There are four different levels of codes:

2xx = Success
3xx = Redirect
4xx = User error
5xx = Server error

Here's a list of the most important status codes: Success codes:

200 - OK (the default)
201 - Created
202 - Accepted (often used for delete requests)

User error codes:

400 - Bad Request (generic user error/bad data)
401 - Unauthorized (this area requires you to log in)
404 - Not Found (bad URL)
405 - Method Not Allowed (wrong HTTP method)
409 - Conflict (i.e. trying to create the same resource with a PUT request)

API response formats

JSON has quickly become the format of choice for REST APIs. It has a lightweight, readable syntax that can be easily manipulated. So when a user of our API makes a request and specifies JSON as the format they would prefer:

GET /widgets

Accept: application/json

...our API will return an array of widgets formatted as JSON:

[
  {
    id: 123,
    name: 'Simple Widget'
  },
  {
    id: 456,
    name: 'My other widget'
  }
]

API authentication

We use token based authentication. The user logs in with their username and password and the application responds with a unique token that the user can use for future requests. This token can be passed onto the application so that the user can revoke that token later if they choose to deny that application further access.

There is a standard way of doing this that has become very popular. It's called OAuth. Specifically, version 2 of the OAuth standard.

API gateway

For most microservices‑based applications, it makes sense to implement an API Gateway, which acts as a single entry point into a system. The API Gateway is responsible for request routing, composition, and protocol translation. It provides each of the application's clients with a custom API.

Endpoint

Each microservice would have a public endpoint:

https://serviceName.api.company.name

This URL would map to the microservice’s load balancer, which distributes requests across the available instances.

The Microservices Architecture pattern significantly impacts the relationship between the application and the database. Rather than sharing a single database schema with other services, each service has its own database schema.

Service discovery

The API Gateway needs to know the location (IP address and port) of each microservice with which it communicates. Application services have dynamically assigned locations. Also, the set of instances of a service changes dynamically because of autoscaling and upgrades. Consequently, the API Gateway, like any other service client in the system, needs to use the system’s service discovery mechanism.

Below is an example of a service definition with image emilevauge/whoamI in Rancher:

# kubectl -n default describe service myosinfo
Name:              myosinfo
Namespace:         default
Labels:            <none>
Annotations:       field.cattle.io/targetWorkloadIds=["deployment:default:myosinfo"]
                   targetWorkloadIdNoop=true
Selector:          workload.user.cattle.io/workloadselector=deployment-default-myosinfo
Type:              ClusterIP
IP:                None
Port:              default  42/TCP
TargetPort:        42/TCP
Endpoints:         10.42.0.50:42
Session Affinity:  None
Events:            <none>

NOTES:

  • This describes a Service object named “myosinfo” which targets TCP port 42 on any Pod with the workloadID_MyService=true selector.
  • This Service will also be assigned an IP address, called the cluster IP, which is used by the service proxies.
  • Service map incoming port 42 (in host) to 42 targetPort (in container).
  • Endpoint is 10.42.0.50:42

In Rancher, the service is also called a DNS Record and can be found in the Service Discovery. In our example above, a new service called myosinfo has been created in the service discovery menu.

Service registry

The API Gateway must be able to query the service registry, which is a database of all microservice instances and their locations. The network location of a service instance is registered with the service registry when it starts up. It is removed from the service registry when the instance terminates.

A service registry needs to be highly available and up to date. Clients can cache network locations obtained from the service registry. However, that information eventually becomes out of date and clients become unable to discover service instances. Consequently,a service registry consists of a cluster of servers that use a replication protocol to maintain consistency.

etcd is a highly available, distributed, consistent, key-value store that is used for shared configuration and service discovery. One notable project that use etcd is Kubernetes.

Server-Side Discovery Pattern

The client makes a request to a service via a load balancer. The load balancer queries the service registry and routes each request to an available service instance. The AWS Elastic Load Balancer (ELB) is an example of a server-side discovery router. Some deployment environments such as Kubernetes run a proxy on each host in the cluster. The proxy plays the role of a server-side discovery load balancer. In order to make a request to a service, a client routes the request via the proxy using the host’s IP address and the service’s assigned port. The proxy then transparently forwards the request to an available service instance running somewhere in the cluster.

RBAC

Role-Based Access Control (“RBAC”) uses the “rbac.authorization.k8s.io” API group to drive authorization decisions, allowing admins to dynamically configure policies through the Kubernetes API. It is activated on our cluster and we need to pass the following option to helm when deploying from catalogue.

rbac.create=true

For example, in case of nginx ingress controller, this parameter shall be set in the configmap menu.

Fllow this guide to configure rbac in the cluster. The two common use cases are :

  • create a user with limited access
  • enable Helm

Rancher server

here is a script to setup Rancher server with Nginx running.

DNS entry

We created a DNS A record, pointing to the IP addresses of our Linux host, dahlia, for rancher.thetradinghall.com hosted at Hurricane Electric.

  • Verify
# nslookup rancher.thetradinghall.com 
Server:		192.168.1.254 <-- infomaniak server
Address:	192.168.1.254#53

Non-authoritative answer:
Name:	rancher.thetradinghall.com
Address: 83.166.150.131

NOTE:

  • Rancher will not have access control configured and your UI and API will be available to anyone who has access to your IP. We recommend configuring access control.
  • Most issues may come from misconfigured iptables.

Installation

Access Control

Access Control is how Rancher limits the users who have the access permissions to your Rancher instance. By default, Access Control is not configured. This means anyone who has the IP address of your Rancher instance will be able to use it and access the API.

The first account that authenticates with Rancher will become the admin of the account

Infrastructure services

When you first log in to Rancher, you are automatically in a Default environment. The default cattle environment template has been selected for this environment to launch infrastructure services. These infrastructure services are required to be launched to take advantage of Rancher’s benefits like dns, metadata, networking, and health checks. These infrastructure stacks can be found in Stacks -> Infrastructure. These stacks will be in an unhealthy state until a host is added into Rancher. After adding a host, it is recommended to wait until all the infrastructure stacks are active before adding services.

Environment

When starting Rancher, each environment is based on an environment template. An environment template allows users to define a different combination of infrastructure services to be deployed. The infrastructure services includes but not limited to container orchestration (i.e. cattle, kubernetes, mesos, swarm), networking, rancher services, i.e healthcheck, dns, metadata, scheduling, service discovery and storage.

Rancher supports grouping resources into multiple environments. Each environment starts with a set of infrastructure services defined by the environment template used to create the environment. All hosts and any Rancher resources, such as containers, infrastructure services, and so on are created in and belong to an environment.

An environment template allows users to define a different combination of infrastructure services to be deployed. The infrastructure services includes but not limited to container orchestration (i.e. cattle, kubernetes, mesos, swarm), networking, rancher services (i.e healthcheck, dns, metadata, scheduling, service discovery and storage.

Build High availability cluster

High-availability clusters (also known as HA clusters or fail-over clusters) are groups of computers that support server applications that can be reliably utilized with a minimum amount of down-time. They operate by using high availability software to harness redundant computers in groups or clusters that provide continued service when system components fail.

Below instructions creates a new Kubernetes cluster that’s dedicated to running Rancher in a high-availability (HA) configuration.

Virtual machines

We have:

  • 1 VM dedicated to Rancher server and Nginx. It is not part of our cluster

  • 3 VM dedicated to etcd:

  • 2 VM dedicated to controller

  • 1 VM dedicated to worker

Linux hosts

Follow these instructions to provision the Linux hosts. Pur VM run Fedora.

Most important is to start with a clean and fresh new install, no old docker images or settings. Iptables and SELinux are the two more sensitive materials.

Load balancer

Deployment

A Deployment controller provides declarative updates for Pods and ReplicaSets. In next section we will work with a concrete case, the Hello-world API.

Rancher UI

Kubernetes uses a .yaml file for deployment loading informations into the cluster. Rancher UI allow to import/export/modify these files.

1- Under the Workloads menu, click Import YAML

apiVersion: apps/v1beta1
kind: Deployment
metadata: 
  name: hello-world-deployment
spec: 
  replicas: 1
  template: 
    metadata: 
      labels: 
        app: hello-world
    spec: 
      containers: 
        - image: "gokul93/hello-world:latest"
          imagePullPolicy: Always
          name: hello-world-container
          ports: 
            - containerPort: 8080

This will create a new Hello-world-deployment workload.

2- Under the Service Discovery menu, click Import YAML

apiVersion: v1
kind: Service
metadata: 
  name: hello-world-svc
spec: 
  ports: 
     -  port: 8080
        protocol: TCP
        targetPort: 8080
  selector: 
    app: hello-world
  type: NodePort

NOTE:

Under the Service Discovery menu, when adding a Record, a good choice is to check The set of pods which match a selector and when clicking the Add selector button, add a Label named app with the app name as Value.

CLI

In this case, nothing will appear in the GUI dashboard

We run the following command with the hello-world.yaml:

# kubectl apply -f hello-world.yaml.

Another solution is to call an URL:

# kubectl create -f https://github.com/gokulchandra/k8s-ingress-setup/blob/master/hello-world.yaml

Manage deployments

kubectl get deployments list all deployments (or workload).

Once the app is deployed, watch its status. Example for Hello-world:

#  kubectl get service hello-world-svc --namespace test
NAME              TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
hello-world-svc   NodePort   10.43.236.218   <none>        8080:30767/TCP   24m

Helm

Helm is a tool for managing Kubernetes charts. Charts are packages of pre-configured Kubernetes resources. Once Helm repo is activated, find apps in the Catalog Rancher UI section. Think of it like apt/yum/homebrew for Kubernetes. It also provides other useful features, such as templating and dependency management.

Chart

Helm uses a packaging format called charts.

To get all .yaml files for a package, do the following:

# helm search traefik
NAME          	VERSION	DESCRIPTION                                       
stable/traefik	1.29.1 	A Traefik based Kubernetes ingress controller w...
# helm fetch stable/traefik
....
# ls
traefik-1.29.1.tgz

Once untar:

# tree
.
├── Chart.yaml
├── README.md
├── templates
│   ├── acme-pvc.yaml
│   ├── configmap.yaml
│   ├── dashboard-ingress.yaml
│   ├── dashboard-service.yaml
│   ├── default-cert-secret.yaml
│   ├── deployment.yaml
│   ├── dns-provider-secret.yaml
│   ├── _helpers.tpl
│   ├── NOTES.txt
│   ├── poddisruptionbudget.yaml
│   ├── rbac.yaml
│   └── service.yaml
└── values.yaml

Helm has two parts: a client called Helm and a service named Tiller running on the Kubernetes cluster.

Tiller uses configmaps to store information for each deployment. To find out what it actually stores, we can run the following command:

# kubectl get configmaps traefik -n ckuster-management -o yaml
apiVersion: v1
data:
  traefik.toml: |
    # traefik.toml
    logLevel = "INFO"
    defaultEntryPoints = ["http", "httpn"]
    [entryPoints]
      [entryPoints.http]
      address = ":80"
      compress = true
      [entryPoints.httpn]
      address = ":8880"
      compress = true
    [kubernetes]
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"traefik.toml":"# traefik.toml\nlogLevel = \"INFO\"\ndefaultEntryPoints = [\"http\", \"httpn\"]\n[entryPoints]\n  [entryPoints.http]\n  address = \":80\"\n  compress = true\n  [entryPoints.httpn]\n  address = \":8880\"\n  compress = true\n[kubernetes]\n"},"kind":"ConfigMap","metadata":{"annotations":{},"labels":{"app":"traefik","chart":"traefik-1.28.1","heritage":"Tiller","release":"traefik"},"name":"traefik","namespace":"cluster-management"}}
  creationTimestamp: 2018-05-03T15:26:07Z
  labels:
    app: traefik
    chart: traefik-1.28.1
    heritage: Tiller
    io.cattle.field/appId: traefik
    release: traefik
  name: traefik
  namespace: cluster-management
  resourceVersion: "304498"
  selfLink: /api/v1/namespaces/cluster-management/configmaps/traefik
  uid: 4d299f85-4ee6-11e8-a2b1-fa163e603fa2

Configmaps themselves are stored in etcd. To remove the Helm release completely, use helm delete --purge Myapp.

https://www.nclouds.com/blog/simplify-kubernetes-deployments-helm-part-3-creating-configmaps-secrets/

Secret and Configmap

Secrets (for confidential data) and ConfigMaps (for non-confidential data).

# kubectl get secret and # kubectl get configmap.

Kubernetes let’s you mount ConfigMaps and Secrets as files. Unlike environment variables, if these files change the new files will be pushed to the running pods without needing a restart, so they are a lot more powerful. You can also map multiple files to a single ConfigMap or Secret, and mount them all at once as a directory!

Certificates

We need a TSL Certificate. A self-signed certificate may be appropriate if you do not have a domain name associated with your server and for instances where the encrypted web interface is not user-facing. If you do have a domain name, in many cases it is better to use a CA-signed certificate (like Let's encrypt).

Note: A self-signed certificate will encrypt communication between your server and any clients. However, because it is not signed by any of the trusted certificate authorities included with web browsers, we cannot use the certificate to validate the identity of our server automatically.

For practical and testing purpose, we will create our own self-signed certificate.

Build self-signed certificate

TLS/SSL works by using a combination of a public certificate and a private key. The SSL key is kept secret on the server. It is used to encrypt content sent to clients. The SSL certificate is publicly shared with anyone requesting the content. It can be used to decrypt the content signed by the associated SSL key.

Here are two links which explain in detail: here and there.

Creating OpenSSL x509 certificates

NOTE : the key need to be PEM-encoded PKCS#1 to avoid this bug. In short, there are two formats for keys: "BEGIN RSA PUBLIC KEY" is PKCS#1

"BEGIN PUBLIC KEY" is PKCS#8

This stackoverflow thread explains everything and how to convert one format to the other one.

Build steps:

1- create needed directory

# mkdir /etc/ssl/private
# chmod 700 /etc/ssl/private

2- create a self-signed key and certificate pair with OpenSSL. Below command options will create both a key file and a certificate

# openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/ssl/private/nginx-selfsigned.key -out /etc/ssl/certs/nginx-selfsigned.crt

3- convert the private key from PKCS#8 to PKCS#1 (not needed when bug is corrected)

openssl rsa -in /etc/ssl/private/nginx-selfsigned.key -out /etc/ssl/private/nginx-selfsigned-rsa.key

4- create a strong Diffie-Hellman group

openssl dhparam -out /etc/ssl/certs/dhparam.pem 2048

5- create secret with CLI

# kubectl --namespace=test create secret tls hello-world-cert --key /etc/ssl/private/nginx-selfsigned.key --cert /etc/ssl/certs/nginx-selfsigned.crt
# kubectl --namespace=test  create secret generic hello-world-dhparam --from-file=/etc/ssl/certs/dhparam.pem

6- verify

 # kubectl --namespace=test get secret hello-world-cert
NAME               TYPE                DATA      AGE
hello-world-cert   kubernetes.io/tls   2         27s
# kubectl --namespace=test get secret hello-world-dhparam
NAME                  TYPE      DATA      AGE
hello-world-dhparam   Opaque    1         2

Certificat manager

cert-manager is a Kubernetes add-on to automate the management and issuance of TLS certificates from various issuing sources.

Ingress nginx controller

Install

Rancher use the NGINX controller built around the Kubernetes Ingress resource that uses ConfigMap to store the NGINX configuration. Use Helm deployment via the catalogue menu.

NOTE: configurable parameters rbac.create=TRUE

After the controller is deployed, we have:

1- one workload

2- one service

3- a configmap

We need to create a secret with a TLS certificate and a key for the default server in NGINX, as described above.

Default back-end

Default backend is a special service endpoint which will handle the traffic that arrives at the ingress and does not match any of the configured routes in the ingress route map. It is the default service that Nginx falls backs to if it cannot route a request successfully.

Basically a default backend exposes two URLs:

  • serves a 404 page at /
  • serves 200 on a /healthz

Networking

By default, all environment templates have the Network Services enabled.

Test inter node communication

  • list nodes with their public IP
# kubectl -n kube-system  get pods --output=wide 
NAME                                  READY     STATUS    RESTARTS   AGE       IP             NODE
canal-265pb                           3/3       Running   0          5d        10.52.11.177   etcd1
canal-57ms9                           3/3       Running   0          5d        10.52.11.179   etcd2
canal-jkkvl                           3/3       Running   0          5d        10.52.11.193   control1
canal-nzzss                           3/3       Running   0          5d        10.52.11.191   etcd3
canal-wcth8                           3/3       Running   1          5d        10.52.16.22    worker
canal-zwm5c                           3/3       Running   0          5d        10.52.11.199   control2
  • run ping command from one pod
# kubectl -n kube-system exec -it canal-265pb -- ping -c 3 10.52.11.199 
Defaulting container name to calico-node.
Use 'kubectl describe pod/canal-265pb -n kube-system' to see all of the containers in this pod.
PING 10.52.11.199 (10.52.11.199) 56(84) bytes of data.
64 bytes from 10.52.11.199: icmp_seq=1 ttl=64 time=0.905 ms
64 bytes from 10.52.11.199: icmp_seq=2 ttl=64 time=0.541 ms
64 bytes from 10.52.11.199: icmp_seq=3 ttl=64 time=0.468 ms

--- 10.52.11.199 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2046ms
rtt min/avg/max/mdev = 0.468/0.638/0.905/0.191 ms

We see above that pods can communicate with their external IP.

CNI

Rancher implements a CNIframework, which provides the ability to select different network drivers within Rancher. To leverage the CNI framework, an environment is required to use the Network Services infrastructure service deployed. These services host internal DNS server and manage routing to published ports on the host (via iptables).

Any container launched in managed network will have the Rancher managed IP (default subnet: 10.42.0.0/16). By default, all containers within the same environment are reachable via the managed network. All containers in the managed network are able to communicate with each other regardless of which host the container was deployed on. Most of Rancher’s features, such as load balancers or DNS service, require the service to be in the managed network.

Overlay Network

Platforms like Kubernetes assume that each container (pod) has a unique, routable IP inside the cluster. The advantage of this model is that it removes the port mapping complexities that come from sharing a single host IP.

Flannel is responsible for providing a layer 3 IPv4 network between multiple nodes in a cluster. Flannel does not control how containers are networked to the host, only how the traffic is transported between hosts. However, flannel does provide a CNI plugin for Kubernetes and a guidance on integrating with Docker.

Load balancer

https://rancher.com/docs/rancher/v2.x/en/installation/load-balancing-config/

Traefik

Traefik is a modern HTTP reverse proxy and load balancer made to deploy microservices with ease.

Traefik ressources

All below elements are writen on a traefik.yml file

1- Ingress controller deployment

2- one service before Traefik

3- one service for Traefik web UI

4- one configmap

Volumes

Rancher uses etcd as datastore. The persistent data will be stored in /var/lib/rancher inside the container. You can bind mount a host volume to this location to preserve data on the host it is running on.

Command to use:

docker run -d -p 80:80 -p 443:443 \
  -v /host/rancher:/var/lib/rancher \
  rancher/rancher

List existing volumes:

# docker volume ls
DRIVER              VOLUME NAME
local               042d1a6dd6...

Inspect volume:

# docker volume inspect VolID
[
    {
        "CreatedAt": "2018-05-02T12:19:12Z",
        "Driver": "local",
        "Labels": null,
        "Mountpoint": "/var/lib/docker/volumes/VolID/_data",
        "Name": "VolID",
        "Options": {},
        "Scope": "local"
    }

Bind Mount

When mounting folders and files from host they will override the ones in the container. As example, etc/Myconf in host bind/mounted on /etc/Myconf on container will then make the container use the configuration file from the host. Thus it allow to write configuration directly from the host.

  1. open the volume menu

  2. Volune name:    VolID

  3. Path on the node:    /Path/on/host

  4. Mount point:    /Path/on/container

  5. Subpath in Volume:    ??

Longhorn

ISCSI

ISCSI is an acronym for Internet Small Computer Systems Interface, an Internet Protocol (IP)-based storage networking standard for linking data storage facilities. It provides block-level access to storage devices by carrying SCSI commands over a TCP/IP network.

Host

Hosts are the most basic unit of resource within Rancher and is represented as any Linux server, virtual or physical.

Meta data

Metadat service

Rancher OS

All consoles except the default (busybox) console are persistent. Persistent console means that the console container will remain the same and preserves changes made to its filesystem across reboots. If a container is deleted/rebuilt, state in the console will be lost except what is in the persisted directories.

/home
/opt
/var/lib/docker
/var/lib/rancher

TEST

To see if the setup is correct, here is a little app to run. This app is called whoamI and its image can be found here.

  1. In rancher UI workload, add

    • name: mywho
    • docker image: jwilder/whoami
    • port mapping: Nodes running a pod Source: 8080 Container: 8000

  2. On container

    • run $ curl $(hostname --all-ip-addresses | awk '{print $1}'):8080. It will return I'm mywho-855976b9b5-ddx8g (in our case, mywho-855976b9b5-ddx8g is the application name)
    • run curl localhost:8080, same output
    • on a browser, enter public IP:8080, same output

NOTE: you can veryify no service has been created for this app.

Resources

Clone this wiki locally