Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Problem with HTTPS, CloudFlare and X-Forwarded-Port header #6358

Closed
tpoindessous opened this issue Oct 21, 2020 · 53 comments
Closed

Problem with HTTPS, CloudFlare and X-Forwarded-Port header #6358

tpoindessous opened this issue Oct 21, 2020 · 53 comments
Labels
help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. kind/bug Categorizes issue or PR as related to a bug. lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. triage/accepted Indicates an issue or PR is ready to be actively worked on.

Comments

@tpoindessous
Copy link

NGINX Ingress controller version: 0.40.2

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.0", GitCommit:"9e991415386e4cf155a24b1da15becaa390438d8", GitTreeState:"clean", BuildDate:"2020-03-26T06:16:15Z", GoVersion:"go1.14", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.13-gke.401", GitCommit:"eb94c181eea5290e9da1238db02cfef263542f5f", GitTreeState:"clean", BuildDate:"2020-09-09T00:57:35Z", GoVersion:"go1.13.9b4", Compiler:"gc", Platform:"linux/amd64"}

Environment: GKE

  • Cloud provider or hardware configuration: GKE
  • OS (e.g. from /etc/os-release): GKE
  • Kernel (e.g. uname -a): GKE
  • Install tools: Kustomize
  • Others: CloudFlare Proxy mode

What happened:

Hi, I have this setup

CloudFlare with HTTPS and Proxy mode => GKE nginx-ingress-controller in HTTP => myApp.

When I got to my website in HTTPS mode, myApp receives this request :

GET / HTTP/1.1
Host: myApp.mydomain.com
X-Request-ID: XXXXXXX
X-Real-IP: myIP
X-Forwarded-For: myIP
X-Forwarded-Host: myApp.mydomain.com
X-Forwarded-Port: 80
X-Forwarded-Proto: https
X-Scheme: https
X-Original-Forwarded-For: myIP

So, because of X-Forwarded-Port: 80, myApp thinks that original request was in HTTPS BUT on port 80 :( and then, links generated on the response are not OK.

What you expected to happen:

I would like to receive a X-Forwarded-Port: 443 header.

How to reproduce it:
My conf :

  body-size: "20m"
  # Cloudflare IP ranges which you can find online
  proxy-real-ip-cidr: "173.245.48.0/20,103.21.244.0/22,103.22.200.0/22,103.31.4.0/22,141.101.64.0/18,108.162.192.0/18,190.93.240.0/20,188.114.96.0/20,197.234.240.0/22,198.41.128.0/17,162.158.0.0/15,104.16.0.0/12,172.64.0.0/13,131.0.72.0/22,2400:cb00::/32,2606:4700::/32,2803:f800::/32,2405:b500::/32,2405:8100::/32,2a06:98c0::/29,2c0f:f248::/32"
  use-forwarded-headers: "true"
  forwarded-for-header: "CF-Connecting-IP"

Anything else we need to know:

I can't verify the original request (and so, headers added by CloudFlare) which arrives in nginx-ingress-controller, as I don't have tcpdump installed on this container. If you know how to dump the original request , I could dump it.

/kind bug

@tpoindessous tpoindessous added the kind/bug Categorizes issue or PR as related to a bug. label Oct 21, 2020
@tpoindessous
Copy link
Author

Hi !

I found a way to do a tcpdump in nginx-ingress-controllers.

CloudFlare doesn't send a X-Forwarded-Port Header.

I think this is related to this :

set $pass_server_port $server_port;

This variable is set to the server port.

More critical is that I think this variable is not overwritable in general server-snippet nor in annotation in the Ingress.

for example, nginx.conf when I added this variable to annotation :

        server {
                server_name myApp.mydomain.com ;
                listen 80;
                listen [::]:80;
                set $proxy_upstream_name "-";
                proxy_set_header X-Forwarded-Port "443";
                location / {
                        proxy_set_header X-Forwarded-Port       $pass_port;

@kolorful
Copy link
Contributor

kolorful commented Oct 27, 2020

I tried adding these on Ingress object but didn't change the header at all either. (Based on the "fix" in #3481 (comment))

annotations:
  nginx.ingress.kubernetes.io/configuration-snippet: |
    more_set_input_headers "X-Forwarded-Port: 443"

and

annotations:
  nginx.ingress.kubernetes.io/configuration-snippet: |
    more_clear_input_headers "X-Forwarded-Port"

I verified they showed up in nginx.conf but had no effect.

@kolorful
Copy link
Contributor

Seems like this has somewhat been discussed in terms of the we can't override this header. #3481 (comment)

@aledbf do you think it would make sense to add an config to let user override "X-Forwarded-Port" header, or do you know why more_set_input_headers would not work as claimed in #3481 (comment) thanks.

@Dohbedoh
Copy link

Did you configure CloudFlare with the Full(strict) mode ? In this mode CloudFlare should talk to Ingress Nginx through port 443 which I think would cause Ingress Nginx to set the X-Forwarded-Port header properly.

@toredash
Copy link
Contributor

toredash commented Nov 3, 2020

Did you configure CloudFlare with the Full(strict) mode ? In this mode CloudFlare should talk to Ingress Nginx through port 443 which I think would cause Ingress Nginx to set the X-Forwarded-Port header properly.

This.

Please verify that CloudFlare it set to Full or Full (strict) mode. Found under SSL-TLS => Overview:
image

@tpoindessous
Copy link
Author

Hi !

@Dohbedoh @toredash , no I didn't set Full mode, so my nginx-ingress-controller is listening in HTTP mode.

This is why nginx set X-Forwarded-Port to port 80 and this is why I want to override this (to delete this header).

Thanks !

@ailurarctos
Copy link

ailurarctos commented Nov 4, 2020

I think there is an issue with the way the Lua code sets pass_server_port (which pass_port is later derived from):

if ngx.var.http_x_forwarded_port then
ngx.var.pass_server_port = ngx.var.http_x_forwarded_port
end

Ideally if ngx.var.http_x_forwarded_port is not set but ngx.var.http_x_forwarded_proto is then it would fall back to choosing a port based on ngx.var.http_x_forwarded_proto like this:

    if ngx.var.http_x_forwarded_port then
      ngx.var.pass_server_port = ngx.var.http_x_forwarded_port
    elseif ngx.var.http_x_forwarded_proto == "http" then
      ngx.var.pass_server_port = 80
    elseif ngx.var.http_x_forwarded_proto == "https" then
      ngx.var.pass_server_port = 443
    end

@Dohbedoh
Copy link

Dohbedoh commented Nov 4, 2020

@ailurarctos this would solves the problem but it is not clear to me if this is an acceptable / standard behavior ? It seems to me that CloudFlare SSL should set X-Forwarded-Port header accordingly and for some reason they don't. But I might be wrong ?
Maybe we should add more flexibility to users to allow overriding / setting those headers to solve such particular scenarios. WDYT ?

@toredash
Copy link
Contributor

toredash commented Nov 4, 2020

CloudFlare does not set X-Forwarded-Port according to their docs: https://support.cloudflare.com/hc/en-us/articles/200170986-How-does-Cloudflare-handle-HTTP-Request-headers-

@tpoindessous
I think you need to set a location based snippet, as such:

nginx.ingress.kubernetes.io/configuration-snippet: |
  proxy_set_header X-Forwarded-Port 443;

I assume you are not using a HTTPS Load balancer in GKE ?

@ailurarctos
Copy link

ailurarctos commented Nov 4, 2020

@toredash if you use a snippet to set the X-Forwarded-Port header it ends up appending instead of replacing.

Ideally if the upstream loadbalancer is not sending an X-Forwarded-Port header and ingress-nginx is configured to use-forwarded-headers then either it would 1) not send an X-Forwarded-Port at all or 2) derive the X-Forwarded-Port from the X-Forwarded-Proto. Right now it is sending an X-Forwarded-Port that is the port NGINX itself is listening on, which means the resulting X-Forwarded-* headers are a mixture of NGINX and the upstream loadbalancer.

@ailurarctos
Copy link

Maybe we should add more flexibility to users to allow overriding / setting those headers to solve such particular scenarios. WDYT ?

@Dohbedoh if there is a way to override or set those headers that would work for me.

@tpoindessous
Copy link
Author

tpoindessous commented Nov 4, 2020 via email

@hobti01
Copy link
Contributor

hobti01 commented Nov 5, 2020

@tpoindessous
I think you need to set a location based snippet, as such:

nginx.ingress.kubernetes.io/configuration-snippet: |
  proxy_set_header X-Forwarded-Port 443;

This does not work as expected because the value is appended, resulting in header:

X-Forwarded-Port: 80,443

Edit: We are attempting to use

nginx.ingress.kubernetes.io/configuration-snippet: |
  more_set_input_headers 'X-Forwarded-Port 443';

but it does not overwrite the Header.

@aledbf
Copy link
Member

aledbf commented Nov 5, 2020

but it does not overwrite the Header.

This doesn't work because the template contains proxy_set_header X-Forwarded-Port and the directive more_set_input_headers is executed in the same nginx phase.

This does not work as expected because the value is appended, resulting in header:

this works as expected for the same reason. Multiple proxy_set_header for the same header appends the values

@aledbf
Copy link
Member

aledbf commented Nov 5, 2020

Changing the value of the header X-Forwarded-Port without using a custom template can be done with a plugin.
https://github.com/kubernetes/ingress-nginx/tree/master/rootfs/etc/nginx/lua/plugins

The content of such a thing is trivial:

local ngx = ngx

local _M = {}

function _M.rewrite()
  if ngx.var.http_cf_connecting_ip then
    ngx.log(ngx.ERR, "Changing x-forwarded-port to 443")
    ngx.var.pass_port = 443
  end
end

return _M
curl localhost -H 'CF-Connecting-IP: 1.1.1'


Hostname: http-svc-6b7fcd49cc-jb27g

Pod Information:
	node name:	kind-control-plane
	pod name:	http-svc-6b7fcd49cc-jb27g
	pod namespace:	default
	pod IP:	10.244.0.11

Server values:
	server_version=nginx: 1.12.2 - lua: 10010

Request Information:
	client_address=10.244.0.7
	method=GET
	real path=/
	query=
	request_version=1.1
	request_scheme=http
	request_uri=http://localhost:8080/

Request Headers:
	accept=*/*
	cf-connecting-ip=1.1.1
	host=localhost
	user-agent=curl/7.68.0
	x-forwarded-for=172.18.0.1
	x-forwarded-host=localhost
	x-forwarded-port=443
	x-forwarded-proto=http
	x-real-ip=172.18.0.1
	x-request-id=dc00f8bf88bc383b786a227b7d2657e4
	x-scheme=http

Request Body:
	-no body in request-

The condition to change the variable can check for any other header (like limiting the change to a particular host)
Using a configmap is possible to mount the plugin as a file https://github.com/kubernetes/ingress-nginx/blob/master/charts/ingress-nginx/values.yaml#L396-L404

@tpoindessous
Copy link
Author

Thanks @aledbf !

@Uysim
Copy link

Uysim commented Jan 14, 2021

This problem is driving me nuts. Any solution yet?

@fejta-bot
Copy link

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Apr 14, 2021
@tpoindessous
Copy link
Author

/remove-lifecycle stale

@k8s-ci-robot k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Apr 14, 2021
@iamNoah1
Copy link
Contributor

Hi @tpoindessous @kolorful @Uysim, can you guys can confirm that the issue still exists? Also with newer versions of ingress-nginx?

@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle stale
  • Mark this issue or PR as rotten with /lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Sep 26, 2021
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

@k8s-ci-robot k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Oct 26, 2021
@clglavan
Copy link

Hi @tpoindessous @kolorful @Uysim, can you guys can confirm that the issue still exists? Also with newer versions of ingress-nginx?

I can confirm this still happens on nginx 1.0.1

@Uysim
Copy link

Uysim commented Oct 27, 2021

@clglavan I don't have info in my hand. My application has been moved.

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Apr 10, 2022
@tpoindessous
Copy link
Author

/remove-lifecycle stale

@k8s-ci-robot k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Apr 26, 2022
@dhinesh-babu-bankbuddy
Copy link

@brsolomon-deloitte thanks this is working fine in the newer 4.0.19 chart.
The plugins didn't seem to load unless i explicitly set the plugins in the controllers configmap.

From what i can see by browsing the nginx.conf generated, This issue seems to arise because all the X-Forwarded headers are set after the server/configuration snippet. Would moving these X-Forwarded header definition above those definitions solve them?

@iamNoah1
Copy link
Contributor

@brsolomon-deloitte @dhinesh-babu-bankbuddy I could not completely follow up on your conversation. Do you still consider this a bug?

@guilhem
Copy link
Contributor

guilhem commented Jun 15, 2022

From what i can see by browsing the nginx.conf generated, This issue seems to arise because all the X-Forwarded headers are set after the server/configuration snippet. Would moving these X-Forwarded header definition above those definitions solve them?

This is why I pushed this PR #7542

@brsolomon-deloitte @dhinesh-babu-bankbuddy I could not completely follow up on your conversation. Do you still consider this a bug?

Workaround proposed can't be used when mixing HTTP / HTTPS.

@iamNoah1
Copy link
Contributor

ok thanks for clarifying. BTW when looking at this issue there is no PR linked, which is why you should fill the template when opening a PR, so that issue and PR are linked. Anyway, I pingee @rikatz to follow up here :)

@iamNoah1
Copy link
Contributor

#7542

@DavidKittleSEL
Copy link

Here is illustration of a full working solution with Helm and ingress-nginx chart version 4.0.13 / K8s 1.21.

First, add new template, ingress-nginx/templates/controller-configmap-lua.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    {{- include "ingress-nginx.labels" . | nindent 4 }}
    app.kubernetes.io/component: controller
    {{- with .Values.controller.labels }}
    {{- toYaml . | nindent 4 }}
    {{- end }}
  name: {{ include "ingress-nginx.fullname" . }}-custom-lua
  namespace: {{ .Release.Namespace }}
data:
  main.lua: |
    local ngx = ngx

    local _M = {}

    function _M.rewrite()
      ngx.var.pass_port = 443
      ngx.var.pass_access_scheme = "https"
    end

    return _M

Then add custom values.yaml:

  plugins: "rewrite_fwd_headers"

  extraVolumeMounts:
    ## Additional volumeMounts to the controller main container.
    - name: cm-volume-lua-plugin
      mountPath: /etc/nginx/lua/plugins/rewrite_fwd_headers

  extraVolumes:
    ## Additional volumes to the controller pod.
    - name: cm-volume-lua-plugin
      configMap:
        name: ingress-nginx-custom-lua

Your upstream apps should now see

* `X-Forwarded-Proto` -> https

* `X-Forwarded-Scheme` -> https

* `X-Forwarded-Port` -> 443

Note there's a slight defect in this setup, in that values.yaml hardcodes the configmap name even though that name is configurable through the template. (It could just be hardcoded there also.)

I just wanted to mention that with the most recent ingress-nginx helm chart this solution needs to be modified to work correctly.
In "values.yaml" plugins: "rewrite_fwd_headers" needs to be a child of controller.config. Thanks @brsolomon-deloitte for this workaround!

@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle stale
  • Mark this issue or PR as rotten with /lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Sep 21, 2022
@s7an-it
Copy link

s7an-it commented Oct 2, 2022

@DavidKittleSEL, doesn't work on helm chart 4.2.5 with Jenkins. I get internal calls redirected to http not respecting the proto call.

@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

@k8s-ci-robot k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Nov 1, 2022
@k8s-triage-robot
Copy link

The issue has been marked as an important bug and triaged.
Such issues are automatically marked as frozen when hitting the rotten state
to avoid missing important bugs.

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle frozen

@k8s-ci-robot k8s-ci-robot added lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. and removed lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. labels Nov 1, 2022
@hasonhai
Copy link

hasonhai commented Dec 1, 2022

Is there any plan for this error?
X-Forwarded-Port should be configurable from the annotations.
I saw that the protocol and the scheme can be set with the annotations, but don't know why the port is missed out.
Currently we have to hardcode the forward header port within the template file. It would be not friendly with the mixed http and https traffic.

                                ### hardcoded x-forwarded-port for Keycloak
                                #{{ $proxySetHeader }} X-Forwarded-Port       $pass_port;
                                #{{ $proxySetHeader }} X-Forwarded-Proto      $pass_access_scheme;
                                #{{ $proxySetHeader }} X-Forwarded-Scheme     $pass_access_scheme;
                                {{ $proxySetHeader }} X-Forwarded-Port       "443";
                                {{ $proxySetHeader }} X-Forwarded-Proto      "https";
                                {{ $proxySetHeader }} X-Forwarded-Scheme     "https";
                                ### End of modification

@gab-despreslaberge
Copy link

I just wanted to mention that with the most recent ingress-nginx helm chart this solution needs to be modified to work correctly. In "values.yaml" plugins: "rewrite_fwd_headers" needs to be a child of controller.config. Thanks @brsolomon-deloitte for this workaround!

Just to add a bit to this, using the bitnami chart, this is what we did:

controller:
  config:
    plugins: "rewrite_fwd_headers"
  extraVolumeMounts:
    - name: cm-volume-lua-plugin
      mountPath: /etc/nginx/lua/plugins/rewrite_fwd_headers
  extraVolumes:
    - name: cm-volume-lua-plugin
      configMap:
        name: ingress-nginx-custom-lua

@thgruiz
Copy link

thgruiz commented Mar 31, 2023

having the same problem here (and with keycloak too)...
the solution still this this 'custom plugin', really?

@xgt001
Copy link

xgt001 commented Apr 6, 2023

@gab-despreslaberge thank you, this is the solution that worked. Can confirm with ingress-nginx 4.2.0.

@Schachte
Copy link

Not sure if this is affecting anyone else here, but OOTB on latest helm chart and ingress-nginx, this works fine for TLS routes (can't speak to non-tls/tls mixture) for me.

The thing that was affecting me was that I was getting a 404 when my Cloudflare DNS reverse proxy setting was toggled on for my DNS A-record, but it worked when I toggled it off. This was because I had a Cloudflare Worker that had bound all HTTP routes to it on the same DNS zone for *.mydomain.com. This forced all traffic to route through my worker and not my TCP load balancer on GKE.

The solution was to add an explicit HTTP route on my Workers Routes and set the worker to "Disabled" so it would just target my LB instead of any CF Worker.

@strongjz
Copy link
Member

@tpoindessous is this still an issue? Did you test it on the latest like @Schachte talks about?

@tpoindessous
Copy link
Author

tpoindessous commented Apr 26, 2024 via email

@longwuyuan
Copy link
Contributor

I tested on controller v1.10.1 and I saw X-Forwarded-Port retained at 443 as per my curl request.

I will close this for now as no more tests can be done from users as posted earlier. If there is a problem found in the controller and can be reproduced on minikube or kind cluster, please re-opn the issue and provide the reproduce steps.

/close

@k8s-ci-robot
Copy link
Contributor

@longwuyuan: Closing this issue.

In response to this:

I tested on controller v1.10.1 and I saw X-Forwarded-Port retained at 443 as per my curl request.

I will close this for now as no more tests can be done from users as posted earlier. If there is a problem found in the controller and can be reproduced on minikube or kind cluster, please re-opn the issue and provide the reproduce steps.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@xgt001
Copy link

xgt001 commented Aug 1, 2024

I am using the Lua plugin based on the above and it works fine. However, is there a more elegant way of doing this as we don't want to maintain a custom plugin for our needs?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. kind/bug Categorizes issue or PR as related to a bug. lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Development

No branches or pull requests