[Sharethrough] Add CCPA support

[maphe]

List of changes:

• forward Regs.Ext.us_privacy signal to the ad server

[maphe]

@SyntaxNode Thank you. I addressed changes that are not related to gzip. Waiting for your guidelines on gzip.

[SyntaxNode]

@maphe Excellent. I'm currently investigating why gzip automatic decompression isn't working. I'll let you know what I find.

[maphe]

@SyntaxNode, just checking in, any update here?

[SyntaxNode]

Yes. I am unable to reproduce the issue. In my tests locally, the Go http library properly requests gzip encoded content and decodes it with the same gzip.NewReader helper in this PR. Can you share a bit more about your test setup and the errors you saw?

[maphe]

Sure, here's what I did:

• point adapters.sharethrough.endpoint from prod to our test environment: http://btlr.sharethrough.com/FGMrCMMc/v1 -> http://ad-faker.sharethrough.com/FGMrCMMc/v1
• our ad-faker service works the same as our prod ad server, as it only returns gzip encoded response if it received the Accept-Encoding: gzip header
• I confirmed that if I don't add manually/explicitly the header to the request built into MakeRequests, the ad-faker service does not receive any gzip related header, that's where I assumed there's nothing in the core prebid-server that "decorates" the request adding any headers
• then if I manually add the header, the ad-faker receives it and return a gzip encoded response that has the Content-Encoding: gzip header, but then response.Body in MakeBids is just not usable until I decod it manually (hence this PR), that's where I assumed the core prebid-server does not intercept the response and decode the body if it's gzip'd

Let me know if you need more info or if I can help any further

[SyntaxNode]

@maphe Thank you. I re-ran my tests locally with the following setup:

• Sharethough adapter url set to the ad-faker test environment.
• A single imp banner request targeting only the sharethough adapter.

I stepped through the golang http code and could see the gzip header being added to the request and the gzip response properly decoded. I also ran the test using a Fiddler proxy (manually added proxy code locally) and could see the gzip encoding. Here's the capture of the traffic to/from Sharethrough from my sample request:

POST http://ad-faker.sharethrough.com/FGMrCMMc/v1?adRequestAt=2020-05-01T15%3A37%3A32.469011Z&bidId=some-impression-id&consent_required=false&consent_string=&height=500&instant_play_capable=true&placement_key=some-placement&stayInIframe=false&strVersion=7&stxuid=some-buyer-id&supplyId=FGMrCMMc&width=600 HTTP/1.1
Host: ad-faker.sharethrough.com
User-Agent: insomnia/7.1.1
Content-Length: 54
Accept: application/json
Content-Type: application/json;charset=utf-8
Origin: 
Referer: prebid.org
X-Forwarded-For: ::1
Accept-Encoding: gzip

{"tmax":5000,"deadline":"2020-05-01T15:37:37.468955Z"}

and the response headers (since the body is gzipped)

HTTP/1.1 200 OK
Access-Control-Allow-Credentials: true
Content-Encoding: gzip
Content-Type: application/json; charset=utf-8
Date: Fri, 01 May 2020 15:37:33 GMT
ETag: W/"14e5-fdbcopYKqYsAZtYE5qByhN0+RRo"
Server: nginx/1.16.1
Vary: Origin, Accept-Encoding
X-Powered-By: Express
Content-Length: 2002
Connection: keep-alive

How are you verifying the outgoing request to the Sharethrough servers is not using gzip?

[maphe]

@SyntaxNode, you're right, I'm not sure where but I must have missed something, gonna work on removing the custom logic soon.

Just for my own curiosity/sanity, do you happen to know when the gzip handling got introduced in pbs core, roughly? We initially introduced gzip in our adapter like several month ago and it broke, we had to roll it back, and this feature got benched for a while for some reasons until recently. So i'm just trying to figure out if the core got updated in between those events or if we just missed something in the first place. Thanks

[SyntaxNode]

@maphe There have been no changes to PBS Core recently (past 6 months) which directly affected outgoing gzip compression. This behavior is handled by the Go libraries. We have upgraded the Go version, so maybe something got fixed that was broken before? I'm not sure. I don't believe PBS Core has ever turned off the Go gzip feature.

Glad it's working well now though. :)

[maphe]

@SyntaxNode, thanks for the details, I updated the PR cleaning up all gzip stuff. Now it's only adding CCPA.

[guscarreon]

LGTM