amp-consent: Expand the storage limit

[jawadst]

What's the issue?

When a CMP iframe sends consent information to AMP (consent-response message sent via postMessage), the maximum string length is 150.

The size of the IAB consent string can vary when users make more granular choices due to the encoding of the string and the difficulty of encoding granular choices (vs encoding a yes or a no for all vendors). See https://github.com/InteractiveAdvertisingBureau/GDPR-Transparency-and-Consent-Framework/blob/master/Consent%20string%20and%20vendor%20list%20formats%20v1.1%20Final.md#vendor-consent-string-format- for the encoding of the consent string.

When encoding fully granular choices, a consent string has a size of 143 characters today. This grows with the number of vendors that are part of the IAB framework so we will soon be over 150 and not be able to store the IAB consent string in the allocated storage.
As is, the storage is too small to accommodate all variations of the current IAB consent strings (TCFv1) which means that the user choices might sometimes not be correctly stored and applied. When the string is too long, the consent UI keeps popping up on every page that the user visits.

With the new version of the IAB Transparency and Consent Framework (TCF v2) that Google has joined and should implement in its ad products, the string will get a little bit longer to accommodate for new signals (see https://github.com/InteractiveAdvertisingBureau/GDPR-Transparency-and-Consent-Framework/blob/master/TCFv2/IAB%20Tech%20Lab%20-%20Consent%20string%20and%20vendor%20list%20formats%20v2.md#tc-string-format).

I'd recommend increasing the storage limit for the consent status to 300. That would be plenty for the current consent string. I am not completely sure about the size of the v2.

cc @janwinkler (I am sure you'll run into this issue soon enough!)
cc @zhouyx @torch2424

How do we reproduce the issue?

Run

window.parent.postMessage({
    type: 'consent-response',
    info: 'string longer than 150 characters',
}, '*');

from an iframe loaded by amp-consent

What browsers are affected?

The issue does not depend on the browser

Which AMP version is affected?

All versions are affected (since the addition of support for external CMPs)

[zhouyx]

Thank you @jawadst for reporting. This is a very solid request.
Can I ask about the 143 characters when encoding fully granular choices? Is that the maximum number that we can get today with tcf-v1 and the current vendor list? Thanks.

[andresilveirah]

@zhouyx do you have any update on this? I think a client of ours just faced the same issue.

[zhouyx]

@andresilveirah. Yes the AMP team agreed to loose the storage limit and improve the localStorage caching logic instead. Do you have an estimate on what is a good size limit? Thanks

[Shiron2302]

@zhouyx Any news on when this will be implemented? I would say around 200-250? At the moment I see we are going around 155

[zhouyx]

200 sounds good, it will take 200 * 2 (utf8Encode) * 4/3 (base64) = 533 bytes.

should be a minor change. I can work on a quick fix to this.

[janwinkler]

200 will only be sufficient for a very short time. some TCF v1 strings are already bigger than 200byte and with TCF v2 (march 2020) the strings will defenetly get bigger than 200.

[zhouyx]

Ideally we would be fine with the size if we improve the cache logic in the AMP viewer. However the team currently doesn't have bandwidth for the improvement in quarter.
Would increasing it to 250 bytes help?

[Facens]

@zhouyx I'm a contributor to Tech Lab, where the TCF specs are defined. You can play with the V2 string here: https://iabtcf.com/#/encode
This item is critical because right now using AMP+TCF breaks completely due to the above.

[janwinkler]

For my understanding AMP will always be service specific consent, hence the string wil be shorter. anyhow. 250 bytes are still "easy to reach" ...

[zhouyx]

I've increased the limit to 200 bytes for now.

Please let us know what is a desired limit? Due to the implementation of localStorage when AMP is embedded in AMP viewer, we have to have a threshold.

[jawadst]

@zhouyx Thanks for this change, very helpful!
Is this live in production yet or will it be released at a later date?

As for the ideal limit, I don't think we know yet. 300 would be more comfortable for the TCFv1 (as stated by @janwinkler , we already have v1 strings bigger in size than 200) and the TCF v2 will have increased size.

Here is an example of a TCFv2 string with size 409 (and that's a pretty standard string when the user says Yes to everything with the current vendor list): COvVQBZOvVQBZCAAAAENAPCAAP_AAP_AAAAAFoEUQQgAIQwgIwQABAEAAAAOIAACAIAAAAQAIAgEAACEAAAAAgAQBAAAAAAAGBAAgAAAAAAAFAAECAAAgAAQARAEQAAAAAJAAIAAgAAAYQEAAAQmAgBC3ZAYzUwLQIoghAAQhhARggACAIAAAAcQAAEAQAAAAgAQBAIAAEIAAAABAAgCAAAAAAAMCABAAAAAAAAKAAIEAABAAAgAiAIgAAAAASAAQABAAAAwgIAAAhMBACFuyAxmpgAA.IFoEUQQgAIQwgIwQABAEAAAAOIAACAIAAAAQAIAgEAACEAAAAAgAQBAAAAAAAGBAAgAAAAAAAFAAECAAAgAAQARAEQAAAAAJAAIAAgAAAYQEAAAQmAgBC3ZAYzUw

I'd say 600-700 would be more comfortable for the v2 but it's hard to be sure at this point.

[zhouyx]

The change should be released to product next week.

Thank you for the information. I agree we need to further loose the size limit. I see two things that can help

1. Get rid of the base64 encoding to prevent increasing the string
2. Better storage caching logic.

[jawadst]

@zhouyx Thanks for ETA, very helpful.

On the base64 encoding, are you referring to the consent string itself being base64-encoded or to AMP applying base64-encoding on top of the consent string returned by a CMP?

If you are referring to the consent string, I don't think we could get rid of the base64 encoding. The consent string is actually a binary format so base64 is a pretty natural representation for this purpose. Another possible representation would be a series of 0 and 1s (000111000...) but I do not think that it would be more efficient.
Although it is a big string, it is pretty efficient in its format as it encodes a lot of information in a reasonable amount of space.

[zhouyx]

I'm referring to AMP applying the base64 encoding. We encode all stored value because we don't want to pass plain string around, but since consent string is already encoded maybe that can be an exception.

[jawadst]

@zhouyx Makes sense, my bad!

[Facens]

@zhouyx We've briefly discussed this in the iab Tech Lab and 4kb would be the ideal size limit as it's the same size limit cookies have. In terms of real use case, 1kb should instead be plenty enough for the foreseeable future. If increasing the size is problematic, please let us know and we'll discuss this more thoroughly; in this case, any indications on the trade offs on your side would be useful.

[Facens]

@zhouyx Bumping this up because the deadline for TCF v2 is getting closer and 200 bytes are not enough for the v2 string, so increasing the size becomes unavoidable. As I've said in my earlier comment, 1kb is minimum viable option, while 4kb would be ideal. Can we get this prioritized?

[zhouyx]

Thank you @Facens for bumping this up. There were some discussion around this topic with multiple teams last week, and the AMP team decided to expand the raw string size limit to 1kb to accommodate the TCF v2 requirement.

More context on the size limit can be found here.
Right now the size limit Google AMP Viewer has is 2kb for each entry(based on origin). I did a brief calculation. With four AMP supported storage sets, assuming each storage key is 5 bytes (customized by publisher), the consent string being 1kb, and the additional consent meta data being 30 chars in length. The base64 encoded string adds up to 1732 chars which is under 2kb, but close. Update from the Google AMP Viewer that they're open to expanding the 2kb limit on their side if the space turn out to be not sufficient.
An alternative is to drop the stored value encoding, which can help save 25% of the space.