add crlset exceptions to network audit script

[diracdeltas]

follow-up to #2160 - the proxied requests should be whitelisted for 0.58.x; otherwise the build will continue to fail npm run test-security

[bbondy]

ETA on fix? Will have to revert the relevant PRs if no fix soon.

[diracdeltas]

waiting on issue 447 in brave/devops, hopefully done tomorrow. fyi the relevant PR is in 0.58 and above

…

On Wed, Nov 28, 2018 at 16:41 Brian R. Bondy ***@***.***> wrote: ETA on fix? Will have to revert the relevant PRs if no fix soon. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#2271 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAhjFpKiGgHCUmDAlgI7I60oApRF5se3ks5uzy0egaJpZM4Y4Vof> .

[bkero]

Hi there. I've been working on this. Here is my understanding and what seems to be the problem and what the current status is. The tl;dr is that I am unsure if it is possible to implement request #447 as requested with fastly, and I'm talking to their support to see if it is possible somehow, or to add support.

crlsets2.brave.com is a proxy service ran in fastly that routes all requests to redirector.gvt1.com. This is used to attempt to audit all outbound network connections during build. Unlike crlsets1/3, this one contained HTTP 301 and 302 messages. According to https://github.com/brave/devops/issues/447 we need to absorb and hide all HTTP redirects on the proxy itself and only serve HTTP 200s or errors. https://github.com/brave/devops/pull/449 was proposed as a solution for it.

I spent about 4 hours the subsequent evening and 4 more hours the day after attempting to find out that provided PR was not working. The result was HTTP 500 Service Unavailable returned by Fastly when attempting to access the following URL in the browser. curl -L and wget both work.

The request itself goes as follows:

http://crlsets2.brave.com/edgedl/release2/chrome_component/J89nUqIt9Hg_4846/4846_all_crl-set-16477547484184933434.data.crx3

which gets redirected to:
https://redirector.gvt1.com/edgedl/release2/chrome_component/J89nUqIt9Hg_4846/4846_all_crl-set-16477547484184933434.data.crx3

which gets redirected to:
https://r2---sn-hufvjvgx-ntqe.gvt1.com/edgedl/release2/chrome_component/J89nUqIt9Hg_4846/4846_all_crl-set-16477547484184933434.data.crx3?cms_redirect=yes&mip=218.214.11.167&mm=28&mn=sn-hufvjvgx-ntqe&ms=nvh&mt=1543595289&mv=u&pl=16&shardbypass=yes

Attempting to find out the actual result of the HTTP 500 led me to set up a remote logging to a S3 bucket in our AWS sandbox account with a new restricted username and access key, since Fastly only supports remote logging. This ultimately didn't end up revealing anything new. Only standard HTTP entries but no tracebacks or error messages.

That led me with the only option being to examine the VCL for problems. Sleuthing on the internet for a few hours around this problem showed that many organizations used code similar to pull devops/449 to achieve similar goals with Varnish. This made me wonder why it was not working.

Fastly is a thinly veiled service around a caching proxy called Varnish. Varnish uses a configuration language that compiles down to C in VCL. There are a few built-in functions that Varnish uses to route traffic and make decisions. Fastly's boilerplate: https://docs.fastly.com/vcl/custom-vcl/creating-custom-vcl/

vcl_recv is a function that is executed when a client initiates a connection and a request is made, but before the backend servers are contacted or cache lookups done. We're not interested in this.

vcl_deliver is a function that is executed right before the content is served to the client. Unfortunately it does not have access to any variables from the backend response. Fastly also fails to allow vcl_deliver to return an error as in the Varnish 2.1.5 documentation.

vcl_fetch is called after a document is successfully retrieved from a backend. This ended up being the function that we wanted. However, what I discovered on the second day after much print debugging is that this doesn't seem to be called for a HTTP 301 or 302. Even simple boilerplate code such as this fails to trigger:

sub vcl_fetch {
#FASTLY fetch
error 418 "I'm a teapot"
}

At this point I was pretty frustrated and went to talk to Fastly staff on IRC. They confirmed they understood the issue, and that I'd need to email support@fastly.com to ask if this is even possible. So current status is that I've reached out to Fastly support to attempt to resolve the matter. I've provided our goal and my reduced test case, and will update this issue when I hear back.

In the mean time I'm attempting to use a local copy of their Varnish build to reproduce locally, where I will have much better debugging tools to use.