[NET-9376] Fix issue with multiple APIGW

[jm96441n]

Changes proposed in this PR

• Update role with correct policy when creating a new gateway
• this works on upgrade process

How I've tested this PR

• have kubectl on your system

1. clone the following gist: https://gist.github.com/jm96441n/c0645d345c4a92f8b5ea6af272a19d63
2. set the imageK8s field in the consul_values.yaml file to be hashicorp/consul-k8s-control-plane:1.4
3. run the start.sh script
4. run the following in your terminal

export CONSUL_HTTP_ADDR="127.0.0.1:8501"
export CONSUL_HTTP_SSL=true
export CONSUL_HTTP_SSL_VERIFY=false
export CONSUL_HTTP_TOKEN=$(kubectl get --namespace consul secrets/consul-consul-bootstrap-acl-token --template={{.data.token}} | base64 -d)

1. in a tool like k9s see that api-gateway-2 is in an error state
2. open consul at https://localhost:8501 (you'll need to allow the insecure traffic to bypass the warning)
3. copy the value of CONSUL_HTTP_TOKEN to your clipboard and use that to login to the UI
4. go to the roles tab and view the role for the api-gateway-2 and see the policy it is using is the api-gateway-token-policy which grants write access to the first gateway created
5. on this branch run make docker-dev
6. update the imageK8s field to be consul-k8s-control-plane:local
7. in your terminal run: helm upgrade --install consul hashicorp/consul -f ./consul/consul_values.yaml -n consul --create-namespace --wait
8. once that finishes in a tool like k9s you should see all the api-gateways spin up correctly
9. run kubectl port-forward service/consul-consul-ui 8501:443 -n consul & in your terminal to re-establish the port-forward to the consul ui
10. repeat steps 2/3/4 to open the consul ui and view the role for api-gateway-2 then you should see it is using the specific policy for this gateway that grants it write access to itself

How I expect reviewers to test this PR

read the code
run the above steps

Checklist

• Tests added
• CHANGELOG entry added

[NiniOak]

Why do you need 2 changelogs?

[jm96441n]

because I committed this one on the wrong branch 😂

[NiniOak]

LGTM.
Test cases:

• No Upgrade: imageK8S: consul-k8s-control-plane:local.
  RESULT: two API Gateways successfully deployed
• Direct version upgrade: imageK8S: hashicorp/consul-k8s-control-plane:1.4 > imageK8S: consul-k8s-control-plane:local
  RESULT: two API Gateways successfully deployed
• SKIP version upgrade: imageK8S: hashicorp/consul-k8s-control-plane:1.3 > imageK8S: consul-k8s-control-plane:local
  RESULT: two API Gateways successfully deployed