Drop of impressions on SSPs & adserver requests migration 1.X

[Deimos01]

Type of issue

Question to people who did the migration from Prebid 0.X to 1.X.
Are you facing a drop of impression requests with some SSPs (for the same inventory size) ?
Are you facing a drop of adcalls to your adserver with the new Prebid (for the same inventory size) ?

Description

On some SSPs we are seeing a significatif drop of impressions requested for the same inventory size:
-50% on Sovrn, -40% on Indexexchange,-33% on One by Aol, -20% on Smartadserver, -10% on Appnexus ...
And what is really weird, the adserver calls are dropping too for our network (-25% to -50%). We have a bidderTimeout at 2000ms and a "Master timeout" for the adserver calls at 2300 ms which means that all the calls to our adserver are automatically triggered after the Master timeout, with or without a Prebid winner.

We are using Prebid.js since the version 0.4.0 and it's the first time we are experiencing a large drop of adserver requests because of a new Prebid version. We did hundreds of different tests during the last 10 days, and it's not coming from a specific device, browser, geo or internet connection speed ... It's not a problem of adapter codes because some adapters are compatible with 1.X version since a while. But we are not able to catch a bug with the new version, everything seems to be working well during our tests.
It seems to be a global issue related to new Prebid core because the only thing which is working is to rollback on our 0.34 version. It's not related to GPDR or the ConsentModule also.
So I'm wondering if there is a deep issue with the new version of Prebid during the execution time ? or a really vicious bug hard to catch ?

Steps to reproduce

Test page is useless, the code execution is working well. It's just a open question to all the publishers.

Expected results

No impressions or adserver calls drop

Actual results

Impressions & adserver calls drop

Platform details

All browsers, all geos, all devices, 1.11 & 1.12 Prebid version. SmartAdserver as adserver.

Thank you,

[GosuDari]

Hey we switched over to 1.11.0/1.12.0 for gpdr suport as well and tested a lot and didn't see any issues until we rolled the changes out. We see like Deimos01 wrote the same issues with drop of ssps and even ad server calls. We have for now reverted to the 0.34.x version, hope the issue will be found and resolved soon, but its hard to find out what causes the issue

[YOzaz]

Happening here as well... still investigating.

[Saigredan]

We are facing similar issues. Make sure your bidder configurations are valid for current Prebid release. It doesn't solve the problem entirely though.

[jaiminpanchal27]

@Deimos01 Can you share some url's where we can debug ?

[Deimos01]

@jaiminpanchal27 Sure : Demo Page 1.12. Do not hesitate if you need more ressources (adserver reports, SSPs reports or other testpages).
@Saigredan our bidder configurations are valid. It's a global issue.
In the meantime, we decided to rollback the prebid version to 0.34 on our network.

Thank you,

[sultanboss]

Having the same issue. It's only filling a small percentage of placements and has a huge revenue impact.

Yes, this is a global issue.

[jaiminpanchal27]

@Deimos01 @YOzaz @sultanboss
I have pushed a fix for targeting which might have caused this issue.
Can you guys create a prebid build using branch https://github.com/prebid/Prebid.js/tree/getAdservertargeting-bugfix and test it if it solves the issue.

Let me know if you need any help with building prebid.js with this branch.

[Deimos01]

Thanks @jaiminpanchal27
That new Prebid version is now live on one of our websites.
I will keep you informed tomorrow.

[dcaminer]

@jaiminpanchal27 am I getting it right that the problem you fixed occurred only on the subsequent auction on the same unit? I.e. for refresh impressions or on websites using single page applications?

We've seen a major drop of impressions but we're pretty sure that the problem affects only non-refresh auctions. Moreover, it doesn't affect only prebid. DFP's reporting showed a significant decrease in the number of impressions but also in "total code served count". We also ran some a/b tests and apparently, the problem manifests itself only when there are a significant number of units and bidders on the page. We've tried limiting the number of units or the number of bidders, and both approaches worked. The "total code served count" and monetized impression as reported by DFP were the same in both variations.

It would seem to me that there is some problem with performance in the new version of prebid, that manifests itself only when there are many banners & bidders on the page.

@Deimos01 @YOzaz @sultanboss can you perhaps share some more details on home banners you have on the page and how many bidders you have in your setup? For us, it's 7 units and 6 bidders.

[jaiminpanchal27]

@dcaminer Yes you are right.
To add more here, some publishers do not use pbjs.setTargetingForGPTAsync function but set targeting with custom code and use pbjs.getAdserverTargeting function. I fixed the bug with pbjs.getAdserverTargeting in that branch. If results are good than will create PR and merge.

Thanks for sharing the results of you a/b tests. I haven't looked at the scenario you explained above.
Can you share your url where you have significant number of units and bidders ?

[YOzaz]

Hi guys. In our case, amount of requests is pretty much the same - maybe because we're using 100% of lazy loaded ads, and GPT async mode.
However, bid level is catastrophic - e.g. Sharethrough went down from ~50% to ~2%. We suspect that's something to do with generated ConsentString (GDPR), but nothing sure yet.

[Deimos01]

@jaiminpanchal27 Sorry but the new Prebid version with the fix on pbjs.getAdserverTargeting function is not the origin of our problem. The bug is still here.

To do a quick recap of our investigation :

• The problem has been noted on the following versions of Prebid : 1.3, 1.7, 1.11 , 1.12, 1.13 and 1.14-pre from @jaiminpanchal27. We didn't try all the other 1.X versions.
• It affects only non-refresh auctions (@dcaminer) but it seems not to be linked with the numbers of Adunits or bidders on the page. We are facing the problem with websites with only 2 Adunits et 3 bidders and also for websites with 6 adunits and 7 bidders.
• It's not linked to the ConsentModule or GDPR because we are testing Prebid versions without the ConsentModule and since 05/19/2018
• We are not able to find any errors in the console from Prebid when we are debbuging the websites
• And what is really interesting, it affects websites more than others. Websites with top geo-audiences from South America, Africa or Asia are really more impacted than websites with North America or Europe top geo-audiences.

So we are thinking that issue is coming from the new way to call the bidders in the 1.X Prebid versions, it seems to be slower for some geos. So maybe we are reaching the timeout more often than with the legacy version ?
That would explain the drop of impressions for some SSP and also the drop of adserver requests ( the visitor leave the page before the adserver call because SSP are too long to answer). It's just an idea, what do you think guys ?

[jaiminpanchal27]

Have been testing with @Deimos01 page with both legacy and 1.x by throttling network.
On Slow 3G/Fast 3g
In 1.x: See lot of xhr requests getting cancelled after 1st auction is completed.
In legacy: Its working fine.

The reason for this can be :
In 1.x we added a timeout on xhr. timeout value will be the value passed in requestObj http://prebid.org/dev-docs/publisher-api-reference.html#module_pbjs.requestBids or if that is not passed we use bidderTimeout. We did this because in 1.x we are not adding a no bid response to the auction and hence no reason to continue the xhr request if it responds after timeout.

1.x has a concept of concurrent auctions, you may have too many xhr requests triggered at the same time if you have significant number of auctions running at the same time. It also depends on number of adunits/bidders configured. Browsers can at max only handle 6 requests at at time. See Here

@dcaminer did some a/b tests and observed same. #2648 (comment)

Also based on last comment by @Deimos01 he is facing drop in geos with slowers internet connections. This will make xhr request fail.

Increasing bidderTimeout for now may solve this problem.

[jaiminpanchal27]

@Deimos01 What timeout are you using ? Can you increase and test ?

[dcaminer]

@jaiminpanchal27

1. On the initial load, we run a single auction. I.e. we do single requestBids call and pass all the units as the parameter. I don't think we're running concurrent auctions
2. I'm pretty sure prebid was dropping slow requests prior 1.x https://github.com/prebid/Prebid.js/blob/legacy/src/ajax.js#L86 But prior 1.x it did this silently, without any notifications. Hence the difference you're seeing. Ofc in 0.34 some bidders could send requests w/o using ajax module, but still, only those responses that were received prior t/o were sent to the ad server. AFAIK Prebid 1.x introduced no changes in this regard, the only exception is that now all requests are sent by Prebid and not by adapters.
3. There are certain limitations that slow down bid responses as more units/bidders are added. But the same limitations apply to 0.34 as well. If in Prebid 1.x it takes more time for bidders to respond that should be considered as a bug. Raising timeout is not the solution, as it has obvious downsides.

[Deimos01]

@jaiminpanchal27 as I said in my initial post, we have a 2000ms bidderTimeout (which is very long). We tried to push the bidderTimeout to 3000ms (which is insane) without seeing significant improvements. And it's not the solution. How the same endpoint URL of a SSP could answer slower depending of the Prebid version ?

The first reason (timeout on xhr) seems legit but it shouldn't impact the adserver requests.

But what you are saying about "concept of concurrent auctions" is really interesting.
It means that if there are more than 2 adUnits with 3 bidders configured on the page (+6 xhr requests), we will face latency because of that "new concept of concurrent auctions" ?
What is the purpose of that new improvement ?

Thanks,

[jaiminpanchal27]

@dcaminer I did not mean raising timeout is the permanent solution. I agree its a bug and we are debugging in depth as we speak.

@Deimos01 You won't face latency with just 2 adUnits and 3 bidders. I don't have a test page to show stalling of requests.

I agree 3000ms is very very high. Are you also testing on this https://www.themoneytizer.com/demo/prebid_timeout?pbjs_debug=true page

Thanks for helping out here. One last thing i have for now is building prebid with https://github.com/prebid/Prebid.js/blob/master/modules/pre1api.js

[dbemiller]

How the same endpoint URL of a SSP could answer slower depending of the Prebid version ?

There's a subtlety here which I don't think is coming across very well in the comments...

Consider the following scenario:

• The browser/device allows at most 6 concurrent XHR connections
• You want to run two auctions, both with 6 bidders and a timeout of 2000ms
• In the first auction, 1 bidder returns in 100ms, and the rest return in 1900ms.

In both versions of Prebid, the auction timer starts when the first XHR call executes.

In Prebid 0.x, the first auction must complete before the second one starts. The second auction starts at 1900ms, and every bidder in both auctions returns within the timeout.

In Prebid 1.x, Prebid tries to start all the adapters from all the auctions at once. However, the browser queues XHR calls because it only allows 6 concurrent connections. When the first adapter from the first auction returns (100ms), an adapter from the second auction is allowed to make a call.

In this case, the second auction's timer starts at 100ms, when the first XHR call in that auction executes. Since the first auction continues to use 5 connections for 1800ms more, the other adapters in the second auction only get 100ms to complete before the auction ends.

This wouldn't affect @dcaminer, since they're only running a single auction... but it could definitely increase the timeout rate for anyone using concurrent auctions.

[YOzaz]

We've just pushed a manual queuing system at our end to avoid concurrent Prebid.js auctions. This should basically simulate pre-1.0 auction system.
Will leave it running for a few days and will come back with results.
I was seeing "xhr cancelled" messages in console before, not anymore - but still seeing some "xhr timeout after 500ms", while prebid config is set to have a timeout of 2000ms. Weird.

[jaiminpanchal27]

@YOzaz You can use pre1api module to enable legacy features with 1.x

[Deimos01]

@jaiminpanchal27 the 1.12.0 Prebid version with the pre1api module activated is now live on one of our websites.
The test page is updated too.
I will keep you informed tomorrow.

[Deimos01]

@jaiminpanchal27 I'm really sorry (again) but the Prebid version with the pre1api module activated is not fixing our problem. We have exactly the same problem & metrics as we had with the 1.11 - 1.12 - 1.13 - 1.14 Prebid version ...

[gramorris]

No problems across 10 sites here. We do have an auction queue in place that we've retained from pre-1.0 so if it is a concurrent auction issue we wouldn't experience it.

[pdramos1]

@gramorris when you say an auction queue what do you mean? Are you saying you are using the pre1api module? Or, you have something specific or some other method of queueing requests?

[gramorris]

@pdramos1 we have our own auction queue implemented outside of PreBid

[lebrunm]

Hello, I identified something similar last week because of the GDPR module: it appears the timeout of the module do not behave as usual timeouts but blocks everything (no Prebid request sent to SSPs etc). In other term, if your Prebid setup has a timeout of 1000ms while the GDPR timeout is set to 1000ms, you should see a massive drop of ad requests to SSPs (~90-100%).

I hope that helps ;)

[dmitriyshashkin]

@lebrunm perhaps you've set "allowAuctionWithoutConsent" to false. Then such behavior would be expected

[lebrunm]

@YOzaz No, we never tried the bidderTimeout param.

[mkendall07]

hey all.

If you still are having this issue can you please link your site, info about the setup and what you have tried to resolve it? It's odd that some folks are reporting success with various setting but others are not. So would like to look for some more patterns.

Feel free to email me directly as well.

[janvdc]

After all, I'm not 100% sure that we've managed to fix it on our side. It's a lot better, but I don't know if the revenue is back to 100% of 0.34. We're going to run a proper A/B test for a few days and I'll report back.

@lebrunm, could you please post a code example of how you're calling pbjs.requestBids()? I'm not clear on how it works in combination with pbjs.setTargetingForGPTAsync() and pbjs.initAdserverSet = true. Do we still need to call these separately, or is this handled by requestBids()? Thank you!

[lebrunm]

@janvdc The following piece of code from the Getting Started page is still required. Updating your implementation with the Prebid timeout can be done with "pbjs.requestBids({ timeout: YOUR_TIMEOUT })" as follows:

        pbjs.que.push(function() {
            pbjs.addAdUnits(adUnits);
            pbjs.requestBids({
                timeout: YOUR_TIMEOUT,
                bidsBackHandler: initAdserver
            });
        });

        function initAdserver() {
            if (pbjs.initAdserverSet) return;
            pbjs.initAdserverSet = true;
            googletag.cmd.push(function() {
                pbjs.que.push(function() {
                    pbjs.setTargetingForGPTAsync();
                    googletag.pubads().refresh();
                });
            });
        }

Again, I do know whether this helped to fix the current issue (drop of impressions) since - as said in my above comments - we released 2 major changes at the same time.

[janvdc]

@lebrunm Thanks for the more detailed explanation. We've just turned on an A/B test that also incorporates your fix. I'll report back when we got the results.

[janvdc]

Our A/B test is complete and I can say with confidence that all of the changes above applied together have fixed the issue. We are now earning more with 1.15 than with 0.34. The following are the results of our test, conducted on US users only:

pbjs_world.js => Prebid 0.34
pbjs_eu.js => Prebid 1.15

Many thanks to everyone in this thread who helped us figure it out. Special shoutout to @Slind14 for his awesome Assertive Yield analytics platform that helped us enormousely with the A/B test.

[mercuryyy]

@janvdc

Can you share the exact setup you used for pbjs_eu.js ?
Can you also run a test on pb 1.19.0 ?

[janvdc]

@mercuryyy

Sure, you can find our configs here:

1.15 => https://www.ancient.eu/js/pbjs_eu.js?v=1534296564
0.34 => https://www.ancient.eu/js/pbjs_world.js?v=1534296564

We're going to upgrade to 0.19 soon, and see if that makes any additional difference.

[Deimos01]

Hello guys,

I don't want to kill the mood but the bug isn't resolved.
As I said before, some geos are more affected than others.
We did a test with the setup of @lebrunm + 1.19 Prebid version and we are still facing a massive drop of impressions for :

• all south american countries (from -90% to -60%)
• India (-90%)
• Indonesia (-80%)
• Malaysia (-80%)
• Philippines (-90%)

And probably other exotic countries are impacted.
@janvdc & @lebrunm, could you please share some geo/internet speed reports ?

Thank you

[lebrunm]

@Deimos01 I could share some data with you on Monday. Can you please confirm you're still not using the Prebid GDPR module?

[Deimos01]

@lebrunm Many thanks for the prompt reply and yes the GDPR module is not activated and not linked to this issue.

[janvdc]

@Deimos01 We've only A/B tested in the US so far. We could now include other countries, too. I'll report back to you as soon as I know more.

[lebrunm]

@Deimos01 Below is the evolution of the the Prebid impression ratio per country (AI / AR):

(with Prebid 0.84 => Prebid 1.15 => Prebid 1.15 w/ fix)

• Brazil: 45% => 20% => 50%
• Chile: 45% => 25% => 45%
• China: 30% => 3% => 23%
• France: minor impact (-5%)
• Germany: minor impact (-5%)
• India: 25% => 8% => 25%
• Indonesia: 25% => 8% => 40%
• Malaysia: 40% => 8% => 45%
• Philippines: 30% => 5% => 30%
• United Kingdom: 45% => 35% => 55%
• United States: 48% => 22% => 43%

We realize only today that we were impacted in such proportion in the USA. In other western countries, the impact on the Prebid impression ratio was almost invisible. Surprisingly, we've even improved this ratio by 10% in the UK!
Anyway, for the so-called "exotic countries" you listed @Deimos01, the above figures confirm we're now back to normal ;)

[mkendall07]

Hey all. Thanks for the information. I going to go ahead and close this issue out. If you are still seeing a problem please feel free to open a new issue but we believe that most problems have been resoled. I've opened #3004 to increase the default timeoutBuffer.

@Deimos01
I believe your issue is related to your implementation. Feel free to continue the conversation offline. Thanks again all!

[headerbidding]

Just so I understand it correctly: Is "timeoutBuffer: 400" the fix for the dropped impressions?

The reason I am asking is that I added "timeoutBuffer: 400" to my script on August 4 (using prebid 1.16) and impressions started to go up 13 days later (August 17).

(I am not sure if the server's cache control (10 days) + browser caching is responsible for the delay?)
(I also later on August 8 upgraded from prebid 1.16 to 1.19 )
I just like to better understand why impressions suddenly went up.

[mkendall07]

@headerbidding

Just so I understand it correctly: Is "timeoutBuffer: 400" the fix for the dropped impressions?

I believe it's a large contributing factor yes. I think also concurrency plays a role but to a lesser extend. It will depend on how many bidders you have and how you are configured etc.

Could definitely be caused by CDN caching. Hard to say exactly though without other data points.

[generic10one]

Can someone please explain what does 'timeoutBuffer' do exactly ?