User ID module not getting data from GDPR consent module

[EskelCz]

Bug or unhandled corner case of implementation

Description

We're using OneTrust CMP (to provide us with consent string from TCF API v2) first as a fast loaded stub (tcf.stub.js), which gets filled later on. This works great for bid requests, adapters seem to be getting the consent correctly. Problem comes with passing the string into user ID module ID5. The module is not getting the consent data, therefore it's sending an empty string.

I think it might be caused by the delayed CMP loading, but it's just a guess at the moment.

Steps to reproduce

1. Add iab (TCF v2) compliant CMP, ideally OneTrust, which is using the stub loading method
2. Add latest Prebid with id5 module and GDPR consent module
3. Configure according to the example at the bottom of this page: https://console.id5.io/docs/public/prebid

Test page

We have a demo page to test both the CMP and ID5 sync: https://demo.cpex.cz/cmp/
Open it with network debugger open and (if you don't already have id5 cookie) you should see the id5 server request: https://id5-sync.com/g/v1/250.json?1puid=&gdpr=0&gdpr_consent=

Expected results

The module should wait for the consent data, the same way as bidder adapters seem to do. The id5 network request should therefore contain gdpr=1 and the consent string.

Actual results

The consent is not present.

Platform details

I'm testing on Prebid 3.23.0, chrome, mac os

[smenzer]

Thanks @EskelCz for writing this up.

Prebid team - I've debugged this a bit and I can see that calls to getId() are actually happening before the call to the cmpCallMap, which means the userId module has no way of getting consent. I can't tell why this is happening, though, just that it is.

[EskelCz]

I should also add that we had this issue even with TCF 1.1 and our in-house built CMP which was loaded more synchronously, so it's probably a problem specifically with the consentModule->ID5 interaction.

[smenzer]

For whoever picks this up...I think this is more general than just ID5, I think it's the entire userId module that is impacted

[SKOCHERI]

Debugging with the https://demo.cpex.cz/cmp/ it does not seem to be an issue with Prebid.js.

Due the the way Prebid.js is integrated in your page, when prebid initializes after initial load (using pbjs.processQueue();), the configs are not set yet. At this point no prebid modules including userId are initialized.
However, during the course of your page load, cpexAnalyticsAdaptor.js makes an explicit call to prebidjs's userId module which cause initialization outside of usual flow for PrebidJS. Due to this explicit call, Prebid.js has no way of maintining module dependencies. Following is the code in cpexAnalyticsAdaptor.js which does this.
line number 27
const userIds = $$PREBID_GLOBAL$$.getUserIds();

When the bid request is made, the configured modules are called in order but userId retains the initial GDPR consent and will not reinitialize.

A simple fix for you would be to remove the explicite call to getUserIds() in cpexAnalyticsAdaptor.js and try.
Once done, you need to make sure that only once all the initializations are done.

One way to achieve this would to be build your cpexAnalyticsAdaptor as a a hook maybe.

[EskelCz]

@SKOCHERI Thank you, I wasn't aware that the internal flow wouldn't allow for something like that.
I agree that a hook will be a better solution for analytics.

[EskelCz]

This came back to bite us. One publisher used a library that reads user ids through prebid (pbjs.getUserIds), which if it runs early enough will basically disable the user id module. This can easily happen without our knowledge.
I'd say it's highly undesirable and too fragile a design. Now that user ids are so important, working with them could use some polish. What do you think?

[smenzer]

I can think of two potential options here, although there are probably several other ways to handle this.

1. Add a flag that indicates if the user ids have been initialized through the normal prebid internal flow. If that flag is set, then calls to getUserIds() will succeed, else they won't. That flag can be checked prior to calling it. The downside is you'd have to poll the variable until it was true.

if (pbjs.isUserIdModuleInitialized === true) {
   let myUsers = pbjs.getUserIds();
}

2. Add a callback handler so that prebid can register those that want to get user ids when they're ready. We could add a timeout here in case the user id module takes too long to be ready.

let myUsers;
pbjs.getUserIdsWhenAvailable(function(userIds) {myUsers = userIds; }));

[patmmccann]

This is potentially relevant to publishers using a script to collect ids for a gam encrypted signal?

[EskelCz]

Personally I would prefer a promise that would resolve once the IDs are available

[smenzer]

Personally I would prefer a promise that would resolve once the IDs are available

Yep that works too!

[bretg]

Patrick and I discussed. Prebid.js must support a way that the page can call the following two functions and be guaranteed that the answer is valid -- i.e. that the module has been initialized:

1. getUserIds()
2. getUserIdsAsEids()

We propose that the initialization of the userID subsystem is considered 'complete' when these three conditions are met:

• setConfig({userId})
• all id sub-modules have been called
• If GDPR Consent module is present, the CMP has either responded or timed out

The implementation of this could be a callback or promise - at the engineer's discretion.

[dgirardi]

At the moment consent data is only resolved at the start of an auction. Coincidentally this will change with #8185; but in the meanwhile, fixing this means that the public ID API (getUserIds, getUserIdsAsEids, refreshUserIds) will be stuck doing nothing until an auction is started - if the GDPR/USP modules are enabled.

[EskelCz]

@dgirardi I would say it's better to do nothing than break things in unexpected ways. We even found example of some IDs being read in time and others not.

[EskelCz]

@dgirardi I just tested the new getUserIdsAsync function and I must say I'm a bit confused. It takes 3 seconds to resolve, called in bidsBackHandler... meaning after consent and all ids should be long present. That doesn't fit the description by bretg:

We propose that the initialization of the userID subsystem is considered 'complete' when these three conditions are met:

• setConfig({userId})
• all id sub-modules have been called
• If GDPR Consent module is present, the CMP has either responded or timed out

Any idea what's the holdup?

[dgirardi]

@EskelCz what is your configuration? if you have userSync: {syncDelay: <some time here>}, the ID subsystems won't start initializing until the auction has completed. With userSync: { auctionDelay: <some time here>}, they will start with the auction. I am not sure that this makes sense, but I tried to keep the logic that was there before.

[EskelCz]

No delays.
Just consentManagement.gdpr, currency, schain, and a few ids and analytics modules.
I guess I'll make a demo.

[dgirardi]

@EskelCz - with neither delay, ID submodules start init immediately after the end of the auction. Can you try userSync: {auctionDelay: 1}?

from the documentation it's not clear to me that these effects are intentional, so we can consider changing it.

[EskelCz]

@dgirardi Interesting, that works. WIth the delay: 1 setting there's just 0-400ms delay (usually 0), without it it's always about 3 seconds. That would be a hard sell to our publishers frankly.

[dgirardi]

@EskelCz I opened #8311 to discuss this

[EskelCz]

@dgirardi I'm wondering, how can userId submodules init after the auction, when the auction itself needs userIds from them?

[patmmccann]

it doesn't 'need' them unless the publisher configures it to be so

[dgirardi]

@EskelCz auctions work with partially initialized IDs - effectively with what you get from getUserIds(). Most of the time that will contain IDs that were stored in cookies / localstorage after previous auctions; some submodules also do not need to make a network call to retrieve an ID, just for tracking (but, as far as I can tell, the userID module cannot tell the difference). In general, auctions start when IDs can still be missing.

[EskelCz]

@dgirardi Interesting, but that's still weird because why would then getUserIdsAsync be slower, when it should return the same thing - the partially initialized IDs.

[dgirardi]

@EskelCz I am confused now, you can get the partial IDs with getUserIds(); which will not guarantee a complete set of IDs. If the async version did the same I don't see the point of it?

[EskelCz]

@dgirardi The synchronous version caused a bug (#5447 (comment)) when called before auction, I thought about the async version as just a way to prevent that from happening. Relying on Prebid to figure out it's internal state and timing.

[dgirardi]

@EskelCz I think you mean that you're not interested in knowing when init is complete, but only in when Prebid decides it has enough to start the auction? (the combination of gdpr is ready / setConfig was called / locally stored IDs have been retrieved).

At the moment that would be equivalent of calling getUserIds() from an AUCTION_INIT event handler. Should we add a way to get that as a promise?

[dgirardi]

@EskelCz if you mean you don't want to call getUserIds() because if you do it at the wrong time it will cause that bug, that was addressed by removing that side effect - it's safe to use at any point, but before those three conditions it will return an empty object.

[EskelCz]

@dgirardi I guess you're right, the event handler should suffice. I wasn't sure about the exact event. Thank you