SourceIP preservation with weave-net not possible

[sh0rez]

What happened:
Attempting to enable SourceIP preservation as described here using externalTrafficPolicy: Local fails, as the IP-address seen by the pod is always the weave ip of the node.

What you expected to happen:
The client-ip should be preserved and the pod should be able to access it.

How to reproduce it (as minimally and precisely as possible):

• Setup a basic pharos cluster with weave-net (default), no fancy config
• follow the steps over here https://kubernetes.io/docs/tutorials/services/source-ip/#source-ip-for-services-with-type-nodeport

Anything else we need to know?:
This should be related to weaveworks/weave#2924, as weave masquerades the local traffic by default. This is highly undesired, as applications like ingress-controllers need the source-ip for subnetting, rate-limiting, etc.
According to the issue linked above, this should be fixable by adding the env-var NO_MASQ_LOCAL=1 over here:

pharos-cluster/lib/pharos/resources/weave/daemon-set.yml.erb

Lines 18 to 29 in 88162bb

	containers:
	- name: weave
	command:
	- /home/weave/launch.sh
	env:
	- name: HOSTNAME
	valueFrom:
	fieldRef:
	apiVersion: v1
	fieldPath: spec.nodeName
	- name: CONN_LIMIT
	value: "0"

Environment:

• Pharos version (use pharos --version): 2.1.2
• Cloud provider or hardware configuration: KVM
• OS (e.g. from /etc/os-release; both for the client and for the used nodes): ubuntu1604

cluster.yml:

hosts:
  # masters
  - address: "10.0.0.10"
    role: master

  # workers
  - address: "10.0.0.20"
    role: worker

network:
  provider: weave

[jnummelin]

Maybe this needs to be set as configurable option that is still disabled by default. I mean there's quite a few setups out there that actually might expect/depend on the way it currently works and now if we change this to default to no masquerading, we might break existing clusters.

I'll cook up a PR to address this.