Improve global-sidecar request forwarding capabilities #47

cywang1905 · 2022-04-19T10:20:51Z

Problem Solved

Global-sidecar would calculate the destination address to be forwarded based on the request header "Slime-Orig-Dest" when forwarding the request previously. When the request header "Slime-Orig-Dest" is empty, forwarding is rejected.

This PR extends the use of the global-sidecar scenario. When the request header "Slime-Orig-Dest" is empty, global-sidecar uses the request.host and the port on which the global-sidecar accepts the request as the destinatiuon address. Then request forwards directly, without generating an accesslog metric.

* The picture is not accurate enough. Traffic forwarded by global-sidecar is routed through the sidecarProxy when it reaches the destination, unless traffic hijacking has been disabled.

This allows business traffic to be directed to the global-sidecar for correct forwarding when there is a problem with some business sidecarProxy. This implies some sense of disaster recovery capability.

Notes

general.wormholePort defines requests of which service ports should be forwarded, connected by ";"
replicas of global-sidecar can be defined in component.globalSidecar.replicas
resources of global-sidecar can be defined in component.globalSidecar.resources

Usage Example

---
apiVersion: config.netease.com/v1alpha1
kind: SlimeBoot
metadata:
  name: lazyload
  namespace: mesh-operator
spec:
  image:
    pullPolicy: Always
    repository: registry.cn-hangzhou.aliyuncs.com/slimeio/slime-lazyload
    tag: master-78f3d09_linux_amd64-dirty_e09cde4
  module:
    - name: lazyload-test
      kind: lazyload
      enable: true
      general:
        wormholePort: # replace to your application service ports, and extend the list in case of multi ports
          - "9080"
          - "9090"
      global:
        misc:
          globalSidecarMode: cluster
          metricSourceType: accesslog
  component:
    globalSidecar:
      enable: true
      sidecarInject:
        enable: true # should be true
        mode: pod
        labels:
          sidecar.istio.io/inject: "true"
      replicas: 2
      resources:
        requests:
          cpu: 200m
          memory: 200Mi
        limits:
          cpu: 400m
          memory: 400Mi
      image:
        repository: registry.cn-hangzhou.aliyuncs.com/slimeio/slime-global-sidecar
        tag: master-5d96f4d_linux_amd64
      probePort: 28888

api/v1alpha1/fence_module.proto

charts/templates/cluster-global-sidecar.yaml

YonkaFang · 2022-04-22T09:30:08Z

charts/templates/cluster-global-sidecar.yaml

@@ -95,19 +95,25 @@ spec:
 serviceAccountName: global-sidecar
 containers:
 - name: global-sidecar
+ env:
+ - name: wormholePorts
+ value: {{ join ";" $f.wormholePort | quote }}


, joined items are more common

change to comma

YonkaFang · 2022-04-22T09:31:04Z

charts/templates/cluster-global-sidecar.yaml

@@ -95,19 +95,25 @@ spec:
 serviceAccountName: global-sidecar
 containers:
 - name: global-sidecar
+ env:
+ - name: wormholePorts


WORMHOLE_PORTS

YonkaFang · 2022-04-22T09:31:31Z

cmd/proxy/main.go


 log "github.com/sirupsen/logrus"

 "slime.io/slime/modules/lazyload/pkg/proxy"
 )

+const WormholePorts = "wormholePorts"


EnvWormholePorts

YonkaFang · 2022-04-22T09:32:26Z

cmd/proxy/main.go

- log.Println("Starting health check on :8080")
- if err := http.ListenAndServe(":8080", handler); err != nil {
+ log.Println("Starting health check on :18080")
+ if err := http.ListenAndServe(":18080", handler); err != nil {


no hardcode. pass via env/config and etc...

add env "PROBE_PORT", default 8080

YonkaFang · 2022-04-22T09:38:45Z

cmd/proxy/main.go

- log.Println("Starting proxy server on", *addr)
- if err := http.ListenAndServe(*addr, handler); err != nil {
- log.Fatal("ListenAndServe:", err)
+ p, err := strconv.Atoi(whPort)


prepare int ports in advance

YonkaFang · 2022-04-22T09:40:45Z

pkg/proxy/http.go

@@ -72,7 +53,33 @@ func (p *Proxy) ServeHTTP(w http.ResponseWriter, req *http.Request) {
 }
 }
 }
- log.Infof("received request, reqHost: %s", reqHost)
+ log.Infof("proxy received request, reqHost: %s", reqHost)


Infof? debug or trace level

change to debug level

YonkaFang · 2022-04-22T09:41:17Z

cmd/proxy/main.go

+ }
+ handler := &proxy.Proxy{WormholePort: p}
+
+ addr := flag.String("addr"+strconv.Itoa(idx), "0.0.0.0:"+whPort, "The addr of the application.")


remove addr

YonkaFang · 2022-04-22T09:42:26Z

pkg/proxy/http.go

+ origDestIp = reqHost
+ }
+ }
+ log.Infof("proxy forward request to: %s:%d", origDestIp, origDestPort)


debug level

YonkaFang · 2022-04-22T09:49:12Z

cmd/proxy/main.go

+ flag.Parse()
+
+ if idx != len(whPorts)-1 {
+ go handler.StartServer(addr)


only the last server failure will cause the app to exit?

change, main func will exit when all web servers are closed

pkg/proxy/http.go

YonkaFang · 2022-04-24T06:37:22Z

charts/templates/cluster-global-sidecar.yaml

@@ -95,19 +95,29 @@ spec:
 serviceAccountName: global-sidecar
 containers:
 - name: global-sidecar
+ env:
+ - name: PROBE_PORT
+ value: {{ default 8080 $gs.probePort | quote }}


this default port value is very likely to conflict with business svc port. we should replace it with an uncommon one.

change to 18181

YonkaFang · 2022-04-24T06:38:21Z

charts/templates/cluster-global-sidecar.yaml

 livenessProbe:
 failureThreshold: 3
 httpGet:
 path: /healthz/live
- port: 8080
+ port: {{ default 8080 $gs.probePort }}


can we use a var to hold this and avoid writing it everywhere?

cywang1905 requested a review from YonkaFang April 19, 2022 10:20

YonkaFang reviewed Apr 19, 2022

View reviewed changes

api/v1alpha1/fence_module.proto Outdated Show resolved Hide resolved

YonkaFang reviewed Apr 19, 2022

View reviewed changes

charts/templates/cluster-global-sidecar.yaml Outdated Show resolved Hide resolved

cywang1905 force-pushed the master branch 2 times, most recently from dc580b1 to 84a6744 Compare April 22, 2022 03:44

cywang1905 changed the title ~~New feature: disaster recovery based on global-sidecar~~ Improve global-sidecar request forwarding capabilities Apr 22, 2022

cywang1905 added the enhancement New feature or request label Apr 22, 2022

cywang1905 force-pushed the master branch 2 times, most recently from 3690966 to 70a905e Compare April 22, 2022 06:25

YonkaFang reviewed Apr 22, 2022

View reviewed changes

cywang1905 force-pushed the master branch from 70a905e to cd78089 Compare April 24, 2022 06:28

YonkaFang reviewed Apr 24, 2022

View reviewed changes

Improve global-sidecar request forwarding capabilities

916aefd

cywang1905 force-pushed the master branch from cd78089 to 916aefd Compare April 24, 2022 06:47

YonkaFang approved these changes Apr 24, 2022

View reviewed changes

YonkaFang merged commit 5d96f4d into slime-io:master Apr 24, 2022

cywang1905 added new feature and removed enhancement New feature or request labels Oct 26, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve global-sidecar request forwarding capabilities #47

Improve global-sidecar request forwarding capabilities #47

cywang1905 commented Apr 19, 2022 •

edited

Loading

YonkaFang Apr 22, 2022

cywang1905 Apr 24, 2022

YonkaFang Apr 22, 2022

cywang1905 Apr 24, 2022

YonkaFang Apr 22, 2022

cywang1905 Apr 24, 2022

YonkaFang Apr 22, 2022

cywang1905 Apr 24, 2022

YonkaFang Apr 22, 2022

cywang1905 Apr 24, 2022

YonkaFang Apr 22, 2022

cywang1905 Apr 24, 2022 •

edited

Loading

YonkaFang Apr 22, 2022

cywang1905 Apr 24, 2022

YonkaFang Apr 22, 2022

cywang1905 Apr 24, 2022

YonkaFang Apr 22, 2022

cywang1905 Apr 24, 2022

YonkaFang Apr 24, 2022

cywang1905 Apr 24, 2022

YonkaFang Apr 24, 2022

Improve global-sidecar request forwarding capabilities #47

Improve global-sidecar request forwarding capabilities #47

Conversation

cywang1905 commented Apr 19, 2022 • edited Loading

Problem Solved

Notes

Usage Example

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cywang1905 Apr 24, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cywang1905 commented Apr 19, 2022 •

edited

Loading

cywang1905 Apr 24, 2022 •

edited

Loading