-
Notifications
You must be signed in to change notification settings - Fork 2
/
SERVICE_MESH_LAB.html
225 lines (172 loc) · 9.87 KB
/
SERVICE_MESH_LAB.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
I - setup LAB
You must setup a lab OCP 4.2+ to deploy Service Mesh. You can use lab from opentlc:
- OpenShift 4 Service Mesh Lab
In our case we will use a standard OpenShift platform to start from the installation procedure.
You can use service mesh foundations lab guide from learning platform.
In this procedure we will only make use of lab guide scripts, but will go through another path.
$ git clone https://github.com/thoraxe/lab-ossm.git
II - Deploy Service Mesh Operator
1 - Install operators
We will make use of the official procedure:
https://docs.openshift.com/container-platform/4.5/service_mesh/service_mesh_install/installing-ossm.html
Steps:
- Installing the Elasticsearch Operator
- Installing the Jaeger Operator
- Installing the Kiali Operator
- Installing the Red Hat OpenShift Service Mesh Operator
2 - Deploy Service Mesh control plane
$ oc new-project my-smcp
In the following control plane configuration is a basic configuration
$ oc create -f control_plane.yaml
In order to apply a custom configuration for each component we recommend to follow the guides in official documentation:
https://docs.openshift.com/container-platform/4.5/service_mesh/service_mesh_install/customizing-installation-ossm.html#ossm-cr-example_customizing-installation-ossm
The must important element to retain, is to configure jeager in a production elasticsearch rather than keeping all-in-one mode for dev, or demo purprose.
3 - Create ServiceMeshMemberRoll
Once control plane is deployed and running, administrator needs to configure wich namespaces need to become part of the cluster.
A CRD can be created to list all the namespace members of the cluster.
Please notice that users who dont have admin role can create ServiceMeshMember to add projects into the Service Mesh cluster.
administrator can then give the privileges to do so.
https://docs.openshift.com/container-platform/4.5/service_mesh/service_mesh_install/installing-ossm.html#ossm-member-roll-create_installing-ossm
$ oc create -f member_roll.yaml
3 - Deep dive in security
Community istio make use of envoy as side proxy to apply routing rules. Envoy proxy uses cap_net_admin privileges with init sidecars to manipulate iptables of nodes when application runs.
Red Hat Service Mesh consider this configuration as a security vulnerability, thus it makes use of istio-cni as a deamonset named istio-node.
This deamon set is only owned by cluster-admin, and it's this component that allows to manipulate the iptables of the underlaying nodes.
$ oc get daemonset istio-node -n openshift-operators
4 - Deploy the sample lab
$ oc new-project my-tutorial
$ oc create -n my-tutorial -f curl.yaml
$ oc create -n my-tutorial -f customer.yaml
$ oc create -n my-tutorial -f preference.yaml
$ oc create -n my-tutorial -f recommendation.yaml
5 - Envoy injector
Once the lab deployed, you can remark that all service mesh pods had a envoy side car proxy injected automatically.
$ oc get pod -l app=customer -o yaml| grep -i 'name: istio-envoy'
This is thank to the kubernetes webhooks mutation mechanism, which is implemented by the istio operator in our use case.
$ oc get mutatingwebhookconfiguration istio-sidecar-injector-my-smcp
III - Routing, Rules and Policies
1 - Gateway, VirtualService & Destination Rules
Istio control plane exposes an ingress gateway to the cluster, in order to extend the configuration to route the traffic into our applications,
We need first to create a gateway, and a VirtualService.
$ oc create -f default_routing_demo.yaml
The gateway will make the application accessible from outside the cluster on a specific host.
$ oc get gateway customer-gateway -o yaml
The VirtualService will applies a routing rules and redirects to the specific backend.
$ oc get virtualservice customer -o yaml
Request customer service
$ export MY_TUTORIAL_GATEWAY=$(oc get route my-tutorial -n my-smcp -o 'jsonpath={.spec.host}')
$ curl $MY_TUTORIAL_GATEWAY/customer
2 - A/B testing (with Weighted Routing configuration)
Check Gateway, Virtual Service you can see that traffic is splitted 80% to V1, 20% V2, 0% V3
---
- route:
- destination:
host: recommendation
subset: v1
weight: 80
- destination:
host: recommendation
subset: v2
weight: 20
---
Check that all requests are distributed accoding to configuration, and take a look on Kiali dashbord to observe
$ while true; do curl $MY_TUTORIAL_GATEWAY/recommendation; sleep 1; done
recommendation v1 from '5fcfd9bb6c-kg6qb': 180
recommendation v2 from '7fd6798df9-zfphz': 120
In this case we are calling recommendation directly to check the behaviour, ideally we could separate the rules in two different
virtual services. But for demo purpose we configured it that way. In next exemple, we'll create an additional virtual service
to validate the call through customer service
3 - Canary Release
$ oc create -f recommendation_canary.yaml
Check that all requests are distributed accoding to configuration, and take a look on Kiali dashbord to observe
$ while true; do curl -H "user-location: Boston" $MY_TUTORIAL_GATEWAY/customer; sleep 1; done
customer => preference => recommendation v2 from '7fd6798df9-zfphz': 163
customer => preference => recommendation v2 from '7fd6798df9-zfphz': 164
$ while true; do curl $MY_TUTORIAL_GATEWAY/customer; sleep 1; done
customer => preference => recommendation v1 from '5fcfd9bb6c-kg6qb': 203
customer => preference => recommendation v1 from '5fcfd9bb6c-kg6qb': 204
4 - Mirroring traffic
$ oc apply -f recommendation_mirroring.yaml
$ oc get pods -l app=recommendation,version=v3 --no-headers | awk '{print $1}'
recommendation-v3-666b94f647-2b75h
$ oc logs -f recommendation-v3-666b94f647-2b75h -c recommendation
...
Oct 02, 2020 12:59:08 PM com.redhat.developer.demos.recommendation.rest.RecommendationResource getRecommendations
INFO: recommendation request from 666b94f647-2b75h: 105
$ while true; do curl $MY_TUTORIAL_GATEWAY/customer; sleep 1; done
customer => preference => recommendation v2 from '7fd6798df9-zfphz': 175
customer => preference => recommendation v2 from '7fd6798df9-zfphz': 176
$ oc logs -f recommendation-v3-666b94f647-2b75h -c recommendation
...
Oct 02, 2020 1:02:11 PM com.redhat.developer.demos.recommendation.rest.RecommendationResource getRecommendations
INFO: recommendation request from 666b94f647-2b75h:
---
Remarque:
For the next part, in order to simplify the demonstration, we will proceed with simple routing rules. Thus, let's client first
The policies already created and apply a simple config.
$ oc delete destinationrule recommendation
$ oc delete virtualservice customer
$ oc delete
Let's create simple configuration
$ oc create -f sample_routing.yaml
Check all ok
$ while true; do curl $MY_TUTORIAL_GATEWAY; sleep 1; done
customer => preference => recommendation v2 from '7fd6798df9-zfphz': 181
customer => preference => recommendation v3 from '666b94f647-2b75h': 112
customer => preference => recommendation v1 from '5fcfd9bb6c-kg6qb': 206
---
IV - Advanced configuration
1 - Retries, Circuit breaker & Timeout
Istio allows to configure retries, timeouts, or circuit breaker mechanisms when calling a misbehaving service.
By default a retry mechanism is applied, and the service is removed from the backend cluster until it's fixed.
Break recommendation v2
$ export RECO_POD=$(oc get pod -n my-tutorial -l 'app=recommendation,version=v2' -o jsonpath='{..metadata.name}')
$ export RECO_IP=$(oc get pod -n my-tutorial $RECO_POD -o jsonpath='{.status.podIP'})
$ oc exec -n my-tutorial -c recommendation $RECO_POD -- curl -s $RECO_IP:8080/misbehave
Check service will not call v2 any more
$ while true; do curl $MY_TUTORIAL_GATEWAY; sleep 1; done
customer => preference => recommendation v1 from '5fcfd9bb6c-kg6qb': 230
customer => preference => recommendation v3 from '666b94f647-2b75h': 137
Fix v2
$ oc exec -n my-tutorial -c recommendation $RECO_POD -- curl -s $RECO_IP:8080/behave
$ while true; do curl $MY_TUTORIAL_GATEWAY; sleep 1; done
customer => preference => recommendation v2 from '7fd6798df9-zfphz': 210
customer => preference => recommendation v3 from '666b94f647-2b75h': 141
customer => preference => recommendation v1 from '5fcfd9bb6c-kg6qb': 235
2 - Circuit breaker //TODO: TEST IT
$ oc apply -f recommendation_circuit_breaker.yaml
//TODO RateLimit, Fault Injection
V - Security
1 - mTLS
Apply preference mtls
$ oc create -f preference_mtls.yaml
Note: It may take some time before the mesh begins to enforce the policy. If *you do not see the expected behaviors, wait a few more moments and then try *again. Patience is a virtue.
Check traffic from external pod (outside the service mesh cluster), and watch Kiali
$ oc get pods -l app=curl --no-headers | awk '{print $1}'
curl-98648bc8b-fdnz6
Out side the mesh (curl -> preference)
$ curl preference:8080
curl: (56) Recv failure: Connection reset by peer
Inside the mesh (customer -> preference)
$ curl customer:8080
customer => preference => recommendation v2 from '7fd6798df9-zfphz': 193
Clean up
$ oc delete DestinationRule preference-destination-rule
$ oc delete Policy preference-mutualtls
2 - Authentication
It is possible to authenticate to a service mesh using jwt token or rbac rules. In addition, there is many different implementation options
using Mixer adapater, EnvoyFilter (in recent versions) or Red Hat Service Mesh built-in option.
In this example we'll go throug a jwt token authentication.
Create policy for service customer
$ oc create -f customer_jwt.yaml
Check call is refused
$ while true; do curl $MY_TUTORIAL_GATEWAY; sleep 1; done
Origin authentication failed.
Authorise call with token
TOKEN=$(curl https://raw.githubusercontent.com/istio/istio/release-1.1/security/tools/jwt/samples/demo.jwt -s)
$ while true; do curl --header "Authorization: Bearer $TOKEN" $MY_TUTORIAL_GATEWAY -s; sleep 1; done
customer => preference => recommendation v1 from '5fcfd9bb6c-kg6qb': 228
customer => preference => recommendation v2 from '7fd6798df9-zfphz': 204
Clean up
oc delete policy jwt-example
//TODO OPA, 3SCALE samples