Add traffic mirroring for Istio service mesh #311
Conversation
|
I marked this as work-in-progress as I think it needs some high-level review and deciding what adjustments need to be made to the approach. This implementation creates a traffic mirroring pre-stage for canary deployments. It does this by adding a field to the canary spec called Mirror. If true, we run a pre-stage where we tell Istio to mirror traffic to the canary. Here's an example prometheus plot:
The top teal line is primary, the red line is mirrored traffic to canary, the green line is shifted traffic to canary. First question I have is what should be the appropriate behavior if the service mesh doesn't support mirroring? I implemented and tested for Istio but some meshes & SMI don't support mirroring. |
|
@andrewjjenkins I think mirroring should be enabled for Blue/Green deployments as well.
For SMI and others, mirroring should be off be default, the code looks ok as it this. Thanks |
|
I have changed the way promotion works in #310 I'll merge it soon, please rebase with master after the merge, as some changes could conflict with your branch. |
The mirror option will be used to tell routers to configure traffic mirroring. Implement mirror for GetRoutes and SetRoutes for Istio. For other routers, GetRoutes always returns mirror == false, and SetRoutes ignores mirror. After this change there is no behavior change because no code sets mirror true (yet). Enhanced TestIstioRouter_SetRoutes and TestIstioRouter_GetRoutes.
SetupMocks() currently takes a bool switch that tells it to configure against either a shifting canary or an A-B canary. I'll need a third canary that has mirroring turned on so I changed this to an interface that just takes the canary to configure (and configs the default shifting canary if you pass nil).
andrewjjenkins
force-pushed
the
mirror
branch
from
6a07fb1 to
3fa892c
Compare
|
OK, rebased in #310, reduced the log message you pointed out to Infof and deleted Still remaining: Blue/Green implementation and I think I should add this to the docs as well. |
andrewjjenkins
changed the title
|
@stefanprodan I think I've addressed your comments & work that I'm aware of.
The B/G implementation: if "mirror: true" then mirror traffic for the entire canary analysis phase. Open to suggestions here if you had a different idea on how it should work. Thanks & I think it's ready for you to take a look. |
Traffic mirroring is a pre-stage for canary deployments. When mirroring is enabled, at the beginning of a canary deployment traffic is mirrored to the canary instead of shifted for one canary period. The service mesh should mirror by copying the request and sending one copy to the primary and one copy to the canary; only the response from the primary is sent to the user. The response from the canary is only used for collecting metrics. Once the mirror period is over, the canary proceeds as usual, shifting traffic from primary to canary until complete. Added TestScheduler_Mirroring unit test.
Traffic mirroring for blue/green will mirror traffic for the entire canary analysis phase of the blue/green deployment.
|
Sorry for the delay, added to those three copies of the CRD and rebased in (see e.g. e384b03#diff-5614edd054a6dc84c5fab6ccd18b9600 ). |
|
Thanks, I don't see a merge button so I'll wait for you to click it. |
Traffic mirroring is a pre-stage for canary deployments. When mirroring
is enabled, at the beginning of a canary deployment traffic is mirrored
to the canary instead of shifted for one canary period. The service
mesh should mirror by copying the request and sending one copy to the
primary and one copy to the canary; only the response from the primary
is sent to the user. The response from the canary is only used for
collecting metrics.
Once the mirror period is over, the canary proceeds as usual, shifting
traffic from primary to canary until complete.
Traffic mirroring can also be used for Blue/Green deployments Fix: #313