No longer proxy RTD ads through RTD servers #7506

davidfischer · 2020-09-23T22:50:38Z

Instead of proxying ads through readthedocs.org/api/..., hit EthicalAds directly. This has a lot of performance advantages from a server scalability perspective since there won't be any process on RTD servers waiting on ad responses.
This relies on additional data (whether ad free, community ads only, keywords) being sent in ~~the Footer API response or in having~~ a new API endpoint.
Rely on viewport detection in the ad client (Add viewport detection using the Verge module ethical-ad-client#29) which will be present in the beta version
Removes the footer ad type and the fixed footer ad types entirely. Instead, we will place a regular ad in the footer if the sidebar would push the ad off the screen.
From a timing perspective, when the DOM is ready, we immediately query the new RTD user data endpoint and concurrently load the ethical ad client. When the RTD user data endpoint returns, that's when we request an ad. The performance of this shouldn't be significantly worse than our current code.

Open Questions/To-Do

We might bring back the sustainability API but instead of proxying the connection to the ad server, it just returns the data necessary to display an ad. (Edit: we did this)
- To get the functionality on par with our existing system, we also want to send keywords related to the project to the ad server. This is used for content targeting.
- We have a few projects on RTD where revenue is shared. These have specific publisher IDs which we will also need to sent to the ad server.
The ad client only works for image or text ads. Any other custom types might need a modification. The ad client appends v1 to either the image or text ad type. (Edit: added in Add keywords and campaign types to the client ethical-ad-client#31)
There's some additional functionality I stripped out in the initial implementation to pick other ad types (footer ads, mostly). This will need to come back with some modifications to support the ad client. (Edit: these features are back)

Screenshot

- Instead, hit EthicalAds directly - Relies on additional data sent through the footer API

ericholscher

This seems like a good start. I see the issues arounds special casing "publishers within a publisher" and some of the other integration data we were doing on the server side. I don't have a great solution tho, but I think it'll be useful to figure out if we can.

ericholscher · 2020-09-23T22:57:33Z

readthedocs/api/v2/views/footer_views.py

+    def is_ad_free_user(self, user):
+        if not settings.USE_PROMOS:
+            return True
+        if user.is_authenticated and hasattr(user, 'gold') and hasattr(user, 'goldonce') and (user.gold.exists() or user.goldonce.exists()):


I think we want goldonce or gold uses right, not and?

I believe the logic is correct. It is an OR except first checking that the attributes exist. Since this API can be called even when the gold app or the donate app are not installed, this extra logic is necessary here.

I think Eric is right.

In the case that hasattr(user, 'gold') -> True and user.gold.exists() -> True, but hasattr(user, 'goldonce') -> False it won't enter the IF block but it seems that it's currently ad free.

However, I'm not sure if it's possible that gold exists but not goldonce, tho.

Maybe this change?

if user.is_authenticated and ((hasattr(user, 'gold') and user.gold.exists()) or (hasattr(user, 'goldonce') and user.goldonce.exists())):

ericholscher · 2020-09-23T22:58:28Z

readthedocs/api/v2/views/footer_views.py

+    def is_ad_free_project(self, project):
+        if not settings.USE_PROMOS:
+            return True
+        if project and hasattr(project, 'gold_owners') and (project.gold_owners.exists() or project.ad_free):


Won't this also make ad_free projects not work if they don't have gold_owners?

The check hasattr(project, 'gold_owners') just checks whether that object is present, not whether there actually is a gold owner. This is needed because the gold app may not be installed.

Gotcha -- we should probably just check for that explicitly (if 'donate' in settings.INSTALLED_APPS), or at least add a comment. It isn't clear that's the reasoning from the code.

When this if's gets complicated on the or things, I try to split per each case to avoid confusions, like:

if not settings.USE_PROMOS: # commercial site return True if project.ad_free: # marked manually as ad free return True if project and hasattr(project, 'gold_owners') and project.gold_owners.exists(): # project's owner is gold member return True return False

You end up with multiple exits where the output is the same, True, but it's clearer to read, IMO.

After discussing, we're going to check settings.INSTALLED_APPS.

ericholscher · 2020-09-23T23:02:36Z

readthedocs/core/static-src/core/js/doc-embed/footer.js

@@ -78,6 +79,10 @@ function init() {
            }
            injectFooter(data);
            setupBookmarkCSRFToken();
+
+            if (!data.ad_free_user && !data.ad_free_project) {
+                sponsorship.init();


I imagine this is going to add a decent bit of latency to ad viewing, I wonder if we should do the ads request always, but vary the display based on this data?

It will add some latency, but I'm not sure how you can actually do what you're saying. Either you add the ethicalads.js meaning you want an ad or you don't. To determine that, you need to know some additional data about the user or project. Having a stripped down API that just gets the project/user data would likely be faster although there would be an additional concurrent API call.

I think we do the ad request, but we can hide it with CSS until we confirm we want to show it. That way as soon as the footer responds, we can display the ad, instead of doing another full round trip.

I don't think we should be hiding ads with CSS and showing them based on API responses. I think this will lead to complication. Instead, perhaps there's a way to add some integration points to the ad client so we can control when requests for an ad are made.

I think we can start with this approach for now. But I do think trying to reduce latency on the ad display is important. The only way to reduce latency is to do the ad load on initial page load, and then display it later based on data from the server. If we don't do that, it doesn't matter what approach we take, it will be quite slow.

I don't think we should be hiding ads with CSS and showing them based on API responses

Is it possible to do the request for the ad, save all the data needed (view URL, click URL, image URL, text, etc) without creating the HTML element to display it yet, until we receive the response from the footer and there create the HTML element and show it?

I assume the ad-client is not thought to work like that at this point, but I think we could have the best of both ideas with lower latency.

Yea, this is what we settled on 👍

ericholscher · 2020-09-23T23:05:44Z

readthedocs/core/static-src/core/js/doc-embed/sponsorship.js

-        };
+        container = $('<div />').attr('style', 'text-align:center').appendTo(selector);
+        $('<div />')
+            .attr('data-ea-publisher', "readthedocs")


We could include the publisher group in the footer response to adjust this for our revshare folks, but that does require blocking this on the server response.

I don't think you can initialize ads without first getting the project/user data. All the decisions you want to make require that data.

One idea: 99% of our pageviews aren't revshare or gold users, so I still think loading the ad JS makes sense at pageload, but we can hide the ad display until we get the footer data back. That will keep ads at the same latency as now, and we just need to re-request them on revshare projects. We could even dump some metadata into the built HTML for revshare projects to be able to read that data prior to the initial request.

This seems like it'll get us almost every ad view in the same latency as the previous approach, if not faster.

ericholscher · 2020-09-24T18:10:13Z

We talked about this on a call, and the proposed path forward:

Have revshare users add the EA div to their docs directly, so we know the publisher
Update the ad client so it can fetch ads, but not display them
Add a lightweight ads api that just returns gold user status
Request the ad & footer at pageload, then only display the ad once we confirm it isn't a gold user.

- Relies on ad client for viewport detection

- The block can be in the footer

davidfischer · 2020-10-13T04:19:34Z

This has been updated and I updated the description. It is ready for a review.

ericholscher

Looks good 🎉

Looks like eslint is failing, so just need to fix that up.

readthedocs/core/static-src/core/js/doc-embed/sponsorship.js

davidfischer · 2020-10-15T22:45:11Z

This is testing out really well. Here's how you can test it:

Set USE_PROMOS = True in readthedocs/settings/docker_compose.py
Build the latest static assets on RTD npm run build
Load ads on a built project. Verify that the view (https://server.ethicalads.io/proxy/view/...) is triggered only if the ad is in the viewport. The ad should be loaded after the gold user/project is checked.

ericholscher · 2020-10-16T00:03:18Z

Tested this locally and it looks 💯

No longer proxy RTD ads through RTD servers

a39c94e

- Instead, hit EthicalAds directly - Relies on additional data sent through the footer API

davidfischer added the PR: work in progress Pull request is not ready for full review label Sep 23, 2020

davidfischer requested a review from ericholscher September 23, 2020 22:50

ericholscher reviewed Sep 23, 2020

View reviewed changes

davidfischer added 4 commits October 12, 2020 14:48

Use the new sustainability data API

ba33c09

- Relies on ad client for viewport detection

Merge branch 'master' into davidfischer/stop-proxying-ads

4c4da0c

Remove logging

0c4f8a9

Re-add the keep us sustainable block

45f9e69

- The block can be in the footer

ericholscher approved these changes Oct 13, 2020

View reviewed changes

readthedocs/core/static-src/core/js/doc-embed/sponsorship.js Show resolved Hide resolved

Fix eslint issues

2b17800

Build production JS

56b5852

davidfischer merged commit c91da7f into master Oct 19, 2020

davidfischer deleted the davidfischer/stop-proxying-ads branch October 19, 2020 21:55

davidfischer removed the PR: work in progress Pull request is not ready for full review label Oct 20, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

No longer proxy RTD ads through RTD servers #7506

No longer proxy RTD ads through RTD servers #7506

davidfischer commented Sep 23, 2020 •

edited

Loading

ericholscher left a comment

ericholscher Sep 23, 2020

davidfischer Sep 23, 2020

humitos Sep 24, 2020 •

edited

Loading

ericholscher Sep 23, 2020

davidfischer Sep 23, 2020

ericholscher Sep 24, 2020 •

edited

Loading

humitos Sep 24, 2020

davidfischer Oct 6, 2020

ericholscher Sep 23, 2020

davidfischer Sep 23, 2020

ericholscher Sep 24, 2020

davidfischer Sep 24, 2020

ericholscher Sep 24, 2020 •

edited

Loading

humitos Sep 24, 2020

ericholscher Sep 24, 2020

ericholscher Sep 23, 2020

davidfischer Sep 23, 2020

ericholscher Sep 24, 2020

ericholscher commented Sep 24, 2020

davidfischer commented Oct 13, 2020

ericholscher left a comment •

edited

Loading

davidfischer commented Oct 15, 2020

ericholscher commented Oct 16, 2020

No longer proxy RTD ads through RTD servers #7506

No longer proxy RTD ads through RTD servers #7506

Conversation

davidfischer commented Sep 23, 2020 • edited Loading

Open Questions/To-Do

Screenshot

ericholscher left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

humitos Sep 24, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ericholscher Sep 24, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ericholscher Sep 24, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ericholscher commented Sep 24, 2020

davidfischer commented Oct 13, 2020

ericholscher left a comment • edited Loading

Choose a reason for hiding this comment

davidfischer commented Oct 15, 2020

ericholscher commented Oct 16, 2020

davidfischer commented Sep 23, 2020 •

edited

Loading

humitos Sep 24, 2020 •

edited

Loading

ericholscher Sep 24, 2020 •

edited

Loading

ericholscher Sep 24, 2020 •

edited

Loading

ericholscher left a comment •

edited

Loading