[Master Feature] Support IAB Consent Framework using amp-consent

[jasti]

At the moment, consent within amp-consent is binary. Either the user can accept consent for all vendors on the page or reject consent for all vendors on the page.
This feature provides the ability for a user to accept or reject consent on a per vendor basis. Further, this could also be enhanced to support consent on a per vendor and per purpose basis.
for e.g. a user should be able to give consent at a granular level to Vendor A for the purpose of analytics but reject consent to Vendor A for the purpose of ads.
Master issue is #15651

AMP will provide the generic framework for any 3rd party CMP to directly integrate with amp-consent to manage consent workflows.

If you are a CMP, please get in touch with us on this issue.

[jeffnordlund-ghost]

Hey, I work for a CMP and would love to know how you are planning to implement the IAB support and am happy to provide any feedback if needed.

[jasti]

@jeffnordlund-ghost thanks! We've made some progress and @zhouyx is working on documenting the design for community feedback. @zhouyx please post design here when ready.

[fchristant]

I think this feature is much needed in order to meet GDPR requirements which explicitly states that a user must be able to give individual consent on a per processing purpose basis.

I don't really understand how the idea is vendor driven, imho it is purpose driven. A user gives consent to a data processing purpose, not to a specific vendor doing the processing? I suppose it is possible to have two vendors on the same page executing the same processing purpose, whereby a user can give consent to each vendor, but that seems a bit of an edge case to me.

[felixarntz]

Hey there! I have been working on a cookie consent plugin for WordPress that supports AMP via amp-consent, and I'd like for it to support more granular cookie control as well (it already does for non-AMP sites). I agree with @fchristant that I'd expect those controls to work per cookie category / purpose, not vendor. Per-vendor management might be useful as way, in that case I think both should be covered.

As an example, here is what cookie types my cookie consent allows to be specifically allowed/disallowed:

• Functional (automatically-checked, can't be disabled when accepting cookies)
• Preferences
• Analytics
• Marketing

[jasti]

@fchristant thanks for the feedback. We were worried initially about being able to have the capacity to store vendor and purpose information in local storage but further analysis shows we can. So we'll simply rely on the CMP passing AMP with the "daisy bit" as defined by the IAB: https://github.com/InteractiveAdvertisingBureau/GDPR-Transparency-and-Consent-Framework/blob/master/v1.1%20Implementation%20Guidelines.md#consent-management-provider-guidelines- which supports both vendor and purpose.

[jeffnordlund-ghost]

Yeah, just leaving that generic would be best and just set up a communication mechanism for that information. Currently the IAB supports consent by purpose, or by vendor, but the vendor is all or nothing (no granular purposes on a vendor). But, rumor has it they are looking into the granular purposes by vendor support (not really sure why they didn't go that way to start).

For us we are supporting consent via the iframe component because our UI is so rich and we would need to basically re-write it to natively support amp-consent. So please keep that in mind and enable this through the message flow for the iframe.

Since the native IAB __cmp interface is able to support cross-frame messaging that should be relatively straight-forward to accomplish.

[jasti]

@jeffnordlund-ghost We weren't planning on supporting the IAB version from an external iframe because the external iframe will then need to be page render blocking which is overall a pretty bad experience of the user from a latency standpoint.

We'd recommend considering adding native support unless you can think of alternatives that ensure good performance. @zhouyx anything else to add?

[jeffnordlund-ghost]

I don't understand why that would be any more blocking then normal consent. I assume you are going to store the consent string in local storage as you do the consent marker now, correct? Also, the __cmp interface is designed to be non-blocking to page render. It will prevent ads from firing until a response is returned, but the page can render as normal. It just queues up the calls made to it until they can be answered. It shouldn't be that difficult to just set up a message interaction to route those calls to a calling iframe.

The problem we have with the standard amp-consent is it's just too simplistic. We deal with huge global brands where they support dozens of languages and most countries. Our platform simplifies that for them, centralizing everything from messaging to keeping their vendor lists up-to-date. It just wouldn't be possible to do that in the typical amp-consent tag, there is too much dynamic stuff happening.

[jasti]

I agree that it's not different from the current amp-consent solution but the external iframe solution was meant as a stop-gap solution for publishers who have simplistic consent gathering UIs.
For CMPs, we recommend native support. Note that you will have full control over the UI in this new version. I'd suggest we continue discussing this topic once @zhouyx publishes a design for how CMPs can integrate?

[jeffnordlund-ghost]

Sure. I was not aware you had plans to make the consent UI support more robust. I would very much like to see what is on the roadmap for that so we can plan accordingly.

[zhouyx]

Yes we are planning to support the standard IAB purpose list and vendor list. But currently has no plan to support the granular purposes by vendor because it requires much space to store the consent and from the feedback we gather from some CMP they always collect purposes for all set of vendors. Please let us know if there's requirement for more granular purposes per vendor.

@jeffnordlund-ghost As @jasti pointed out, we don't have a good way to ensure good performance via iframe. Which is very important especially given the UI dialog is blocking user from consuming the page. Could you give us an example of your UI? And we can see what's the best way to support your use case. Thank you!

[jeffnordlund-ghost]

You can go to a number of places to see our consent ui. https://www.cosmopolitan.com/uk/, www.vmware.com, www.crownpeak.com, cbs.com, most nestle sites. Some of them require you come in from an EU country to see the UI though.

Our UI isn't dramatically different from other CMP's. But it is fully customizable (colors, fonts, size, which buttons are shown, what those buttons look like, etc.) and it automatically picks up on things like language settings and current geo so our customers can deploy a common script on every one of their properties and then manage everything from our interface. We also have a process to automatically scan their pages and detect vendors so they don't have to keep updating their vendor lists. So the more of that dynamic nature that can be supported the easier it is for our customers to deploy and manage. But most of that requires scripts to function.

Additionally, we need to be able to do reporting. Currently we report by dropping image pings with the information we need for reporting. But reporting is crucial for compliance with regulations like the GDPR. Our current scripts will log a number of actions, including consent, with enough information to satisfy the EU regulators if necessary.

[zhouyx]

Thank you @jeffnordlund-ghost This is really helpful.

We also have a process to automatically scan their pages and detect vendors so they don't have to keep updating their vendor lists. So the more of that dynamic nature that can be supported the easier it is for our customers to deploy and manage. But most of that requires scripts to function.

This is an interesting requirement that we haven't heard from other CMP before. I wonder how do you scan the page to detect vendors, especially given that some vendors are not directly included in the page but requested by another vendor. What kind of information do you expect from AMP to help you achieve that? Because even with the iframe approach you won't be able to run script on the parent page.

Our current scripts will log a number of actions, including consent, with enough information to satisfy the EU regulators if necessary.

AMP do supports some sort of reporting. For example it send a POST request to the configured endpoint with user's decision. Is that sufficient?

[jeffnordlund-ghost]

In regards to the page scanning there isn't anything the AMP framework needs to do to support that. But what that means is we get a frequently changing list of vendors and that is what we need to support. So some way to pull in the list of vendors/tags (both IAB and non-IAB) would be really helpful. If our customers need to re-add that to their AMP pages as a static list that can become a maintenance nightmare. The IAB is planning to support something that might make this easier, a publisher.json file that allows the list of vendors to be stored in a json file. Support for something like that might be enough to enable this.

It would be nice if you supported some additional events. We frequently get requests from customers for things like opt-in rates. So we need to store both the number of impressions as well as the number of users who accepted or declined. They ask us for details on which vendors are being opt-ed out of, or how often the user elects to opt-out of all vendors (for some scenarios). Our expectation is they will ask for similar reporting for the IAB platform: which purposes are enabled/disabled, how often does a user opt out of everything, how often do they even look at the choices, etc. If you could make the reporting more generic instead of just listing specific actions that would give people the flexibility to report on what is important to their customers.

Also, it would be really nice if you were flexible enough to support GET requests as well as POST's. We have our reporting set up the same way as many advertisers do where we just drop image pings, which are all GET requests obviously.

Other nice to have: Support for a settings file. This would allow our customers to deploy a central amp script and let the settings dictate things such as which geo's need consent, which vendors/tech are on the page, what does the consent dialog look like. Things like that.

[zhouyx]

@jeffnordlund-ghost Thank you for the information. Again this is really helpful!

To support different analytics events, and different request method, we are planning to support a CMP based template. Then you can use or component to collect analytics with much flexibility.

Other nice to have: Support for a settings file. This would allow our customers to deploy a central amp script and let the settings dictate things such as which geo's need consent, which vendors/tech are on the page, what does the consent dialog look like. Things like that.

Yes we have onConsentHref that's supposed to support features like this.

[jeffnordlund-ghost]

I assume onConsentHref is something new since I don't see any docs for it yet.

If you can start to build in support for GET requests that would be great. We don't use servers for 99% of our stuff, just using our CDN to distribute the scripts and settings. This makes our scripts faster and our uptime is basically 100% as opposed to using a server. So supporting that would be great for us.

[janwinkler]

We are a CMP too, how can we contribute?

[consentmanager]

:-)

[jawadst]

@jasti To double down on Jeff's comments and list a few more requirements that we (as a CMP) have for our regular web/in-app SDKs and that are actively used by publishers today:

1. Ability to control the workflow (when to collect consent, what the different banners / popups look like and how the user goes from one to the other, different languages, etc.). As you mentioned, being able to provide a ready-to-go template per CMP would go a long way vs the existing situation where every publisher needs to rebuild the consent UI.
2. Use our own list of vendors and purposes (we rely heavily on the list provided by the IAB but also need to support custom vendors and purposes that are not part of the IAB). This might also require being able to store more information than just the IAB consent string.
3. Be able to condition the loading of tags at the vendor level for a given set of purposes so this can be driven by the consent string and does not necessarily need to be a CMP-specific approach. It can rely on the existing data-block-on-consent attribute, assuming it gets the adequate level of granularity.
4. Reporting of all events to our back-end (consent status, clicks on various elements of the UI, etc.)
5. Support for different ways of giving consent. Depending on the country and the DPA, publishers might want to collect consent on scroll, navigation or click on the banner.

Also happy to help review any documentation/design that would be targeting CMPs.

[zhouyx]

@jawadst Thank you for the information. At this point we are leaning to let CMP implement the logic within an iframe, and then AMP runtime can help to store and pass the consent string to components on the page.

Since you mentioned that you will be supporting vendors that's not necessary of of the IAB list. Could you please provide more information on what exactly are you storing, and passing to vendors. My concerns are

1. AMP has very strict rules on the size of storage that can be used for this purpose. I want to make sure the consent string you are going to store is space efficient.
2. Your consent string can be passed to every vendors that integrates with AMP. I want to make sure that all vendors can understand the response from CMP.

Thanks

[consentmanager]

@zhouyx storing info on server end: probably possible (e.g. give the user a cookie with an id and save the consent string on servers) but not yet supported by the iab framework

[jawadst]

@zhouyx Storing consent on the server-side is possible independently from the IAB framework but most CMPs would rather not have to store anything server-side for availability and speed questions. Especially in AMP, it is extremely important to guarantee that everything is stored directly on the device to avoid extra HTTP requests and increased latency for the user (especially when the CMP views are likely to be blocking for the user).

[jeffnordlund-ghost]

the biggest problem with storing info server-side as a CMP is identifying the user the information belongs to. without some kind of specific user identifier it isn't possible to map a server request to the consent information and your standard browser request doesn't contain enough information to accurately identify a specific user from random sites on the internet. that, and performance, are the main reasons we use either cookies or local storage for this information.

[zhouyx]

Especially in AMP, it is extremely important to guarantee that everything is stored directly on the device to avoid extra HTTP requests and increased latency for the user (especially when the CMP views are likely to be blocking for the user).

@jawadst This is a great point. I know some would still want to make a request for additional information even with stored value. Is that true for your case? Can I assume that once there's previous stored consent string, AMP runtime can simply use that without sending request to CMP?

without some kind of specific user identifier it isn't possible to map a server request to the consent information

@jeffnordlund-ghost You're correct it's impossible to identify user on server side right now. But it can be supported if we feel server side storage is the way to go. What concerns me more is that do you feel it's OK for you to store an user's identifier as a CMP in general?

[mobiluck]

@zhouyx the amp runtime should/must load the cmp code every time on every page, even if the value is already stored. the code will be providing the functionality/API for vendors, hence its necessary to load it even in cases where consent is already given

[jeffnordlund-ghost]

No, that's kind of the point of not doing things server side. If we do that then we are forced to store some kind of user identifier and we have no desire to do that. All we want to do is provide users a mechanism to exercise their choices and then communicate those choices to the publisher.

[hhhjort]

Good to see AMP moving in the direction of the IAB consent framework. One use case I haven't seen explicitly mentioned is an RTC call to a vendor that will then proxy additional calls to additional vendors. (Example: prebid) That will require passing the IAB consent string (or the same information in some other format) in the RTC call so the vendor can properly support the GDPR requirements in whatever it does with the request.

[jeffnordlund-ghost]

I was under the impression that forwarding the consent string through to downstream calls is the responsibility of the vendor. So an auction vendor would be responsible for sending the string to the auction winner. I don't think there is anything special that would be required in AMP to support that.

[hhhjort]

Well, the vendor needs the consent string to be able to forward it. If the vendor is being called via AMP RTC, then AMP needs to be able to provide the consent string in the RTC call for the vendor to have it.

[jasti]

@keithwrightbos are there plans for DoubleClick to pass the consent string to RTC vendors?

[jdelhommeau]

Hi @jasti ,

Is there any update on supporting consent string passed in RTC config? Seems to be a blocker for prebid support in AMP.

[zhouyx]

Hello @jdelhommeau I talked to @jasti today on this topic. We can definitely add support to pass consent string along with the RTC request.

Trying to understand your requirement better? Are you working with the prebid server, that can consume the consent string along with the RTC request.

[hhhjort]

I imagine the best way to support this is to have a macro that will be populated with the IAB consent string if it is available. Ideally also a GDPR flag to signify if the viewer is in a location where the GDPR applies. If these two can be placed in the query string, it will not be difficult to modify prebid server to read and make use of them.

[Facens]

@jasti what's the status of this task? Was there any progress?

[zhouyx]

The support is already in prod. Please refer to the doc here for integration guidelines. Closing the issue.

[mgriego]

This is still in experimental status?

[zhouyx]

No it's not. Good point, I'll update the doc. Thank you. #23049

[jeteve]

I don't see why this is closed? @zhangsu the document you're referring to simply shows how to delegate the ternary 'google only personalised ads' consent to a publisher's own interface (via the iFrame). It doesn't help implementing AIB's TCF's granularity of Purpose and Vendors choices, nor the page level API compatible vendors would look for.

Am I wrong?

[zhouyx]

Hello @jeteve Thank you for your question.

AMP only provides a tool for publisher along with consent management platform and vendors to collect and maintain their user consent. As you mentioned, the support is provided via iframe. AMP helps store and pass consent information from the iframe. But it's up to the publisher or the CMP to provide IAB's granular support within the iframe.

We do have API for vendors to retrieve the stored consent information. However because AMP decides to not enforce any format on the stored consent information. The API we provide couldn't handle more granular information, and that's by design. It's up to the CMP and vendor to agree on a standard, which could be IAB standard or any other format.

The AMP's support works when the CMP decides to collect granular consent information from user. AMP will pass and stored the information to vendor, where they can interpret the info. Does this make sense.

Thanks.

[adamread]

I also don't understand how this FR has been addressed. According to the docs, it's still not possible to enable ads or analytics on a granular basis. Both can only use data-block-on-consent and it's an all or nothing toggle. Collecting granular consent data is only half the requirement and it's not only useful if you can apply the user's decisions to the current page.
From interactions with potential AMP developers, a way to set and retrieve key-values with amp-consent and then use them with data-block-on-consent to selectively allow amp-analytics components is the missing part of this FR.

[martinschierle]

+1 we still hear many requests for a better integration here.

[zhouyx]

@adamread @martinschierle, you're right data-block-on-consent only supports the a global decision. Granular decision is passed via the consent string, and then AMP assumes that it's the 3rd party vendor's responsibility to interpret the consent string and behave accordingly.

Can I ask if you're working with a CMP or you're customizing the page behavior as a publisher. Thank you

[zhouyx]

Saw #26735, let's move the conversation there. Thank you