Topics module: Initial Topics iframe implementation

[pm-nitin-nimbalkar]

Type of change

• Bugfix

• Feature

• New bidder adapter

• Code style update (formatting, local variables)

• Refactoring (no functional changes, no api changes)

• Build related changes

• CI related changes

• Does this change affect user-facing APIs or examples documented on http://prebid.org?

• Other

Description of change

This change is the proposal of iframe implementation which is the enhancements of existing module under topicsFpdModule.js where different bidders will call the topic API under their domain to fetch the topics for respective domain and the segment data will be part of ORTB request under user.data object.

Enhancements are discussed in this thread #8741

Note: This is rough draft and initial proposal of iframe implementation to discuss about the change.

Implementation

• Configured topics under userSync object to read the iframeURL for particular bidder on publisher domain. This would override the default configuration. If these configurations are not present on the page, default configuration of bidders iframe which will call topics API. Default SSP rotation is kept as 1 as default configuration which can be override by adding configuration inside topics object having "maxTopicCaller" as key on page as shown below configuration examples

Note: IframeURL for each domain is dummy URL used for to give showcase how to configure the URL

pbjs.setConfig({
            userSync: {
                ...,
                topics: { 
                        maxTopicCaller: 3, // SSP rotation 
                        bidders: [{
                            bidder: 'pubmatic',
                            iframeURL: 'https://pubmatic.com:8080/topics/fpd/topic.html', // dummy URL
                            expiry: 7 // Configurable expiry days
                        },{
                            bidder: 'rubicon',
                            iframeURL: 'https://rubicon.com:8080/topics/fpd/topic.html', // dummy URL
                            expiry: 7 // Configurable expiry days
                        },{
                            bidder: 'appnexus',
                            iframeURL: 'https://appnexus.com:8080/topics/fpd/topic.html', // dummy URL
                            expiry: 7 // Configurable expiry days
                        }]
                    }
                ....
            }
})

• Topics data will be loaded for particular domain from iframe and sent back to first party domain or publisher website. This data will be cached in localstorage to make it available for bidders to consume in their APIs

• Data will be cached in 'prebid:topics'

• This Data will be read from cache(localstorage) and will be part of bidder's respective API under user.data object
  For Eg:

"user":{
  "gender":"M",
  "geo":{
     "lat":40.712775,
     "lon":-74.005973
  },
  "yob":1982,
  "data":[
     {
        "ext":{
           "segtax":600,
           "segclass":"2206021246"
        },
        "segment":[
           {
              "id":"45"
           }
        ],
        "name":"timesofindia.com"
     },
     {
        "ext":{
           "segtax":600,
           "segclass":"2206021246"
        },
        "segment":[
           {
              "id":"265"
           },
           {
              "id":"45"
           }
        ],
        "name":"ads.pubmatic.com"
     }
  ]
},

Open questions

1. How SSP Rotations works ?

• Does it means we need to call any random 3 bidders to load iframe and make topics data for that domain/api caller to make it available for bidder request.

• What happens if we configured more than 3 bidders (for Eg: 7-8) which will load iframe for 7-8 bidders which might cause latency and concerning for publisher or first party domain.

2. Expiry date for cached data should be added for each bidder or 1 expiry for all bidders? Should publisher be given control to refresh the cached data by setting the expiry

3. How iframe should be maintained for each bidder? Should it be in this repository itself ?

4. Cached Topics data in local-storage needs to be encrypted (Base64) ?

Other information

[pm-nitin-nimbalkar]

@patmmccann Can you please help us review the changes and if suggestions/comments regarding the implementation of iframe. Let us know if you have any questions.
Note: This is WIP PR so that we can discuss about the proposal/approach if it looks good to go ahead.

[patmmccann]

Open questions
How SSP Rotations works ?
Does it means we need to call any random 3 bidders to load iframe and make topics data for that domain/api caller to make it >available for bidder request.

What happens if we configured more than 3 bidders (for Eg: 7-8) which will load iframe for 7-8 bidders which might cause >latency and concerning for publisher or first party domain.

I think SSP rotation would only call one of the frames at random per pageview; maybe with a 'callAll' override?

Expiry date for cached data should be added for each bidder or 1 expiry for all bidders? Should publisher be given control to >refresh the cached data by setting the expiry

Expiration should be fairly aggressive I think; as the topics API itself is fairly aggressive with expiration.

How iframe should be maintained for each bidder? Should it be in this repository itself ?

I think the iframes in the default rotation should be in the repo likely; and publisher could override the array with their own array in config.

Cached Topics data in local-storage needs to be encrypted (Base64) ?`

I am not sure if we need to encrypt anything, the topics are meant to be passed around freely it seems. I am not sure of any security risk from exposing them.

[pm-nitin-nimbalkar]

Open questions
How SSP Rotations works ?
Does it means we need to call any random 3 bidders to load iframe and make topics data for that domain/api caller to make it >available for bidder request.

What happens if we configured more than 3 bidders (for Eg: 7-8) which will load iframe for 7-8 bidders which might cause >latency and concerning for publisher or first party domain.

I think SSP rotation would only call one of the frames at random per pageview; maybe with a 'callAll' override?
Regarding "I think SSP rotation would only call one of the frames at random per page view; maybe with a 'callAll' override?"

Regarding SSP rotation, Does adding config ("maxTopicsCaller": 3) inside the setConfig under topics make sense which gives more power to publishers to provide count of bidders from where it get the topics from those bidders domain ?
This is configurable as suggested, this can even configure as 1 (Bidder).

For Eg. Over here, Adding 2 as count is configured for maxTopicsCaller and 3 bidders has been configured but only random 2 bidders will call the frames. If this "maxTopicsCaller" property is missing all together in config, it will load the frames for all 3 bidders ( as replacement of the one what was suggested with callAll )

pbjs.setConfig({
            userSync: {
                ...,
                topics: { 
                        bidders: {
                           "maxTopicsCaller": 2,
                            "pubmatic" : {
                                iframeURL: "https://ads.pubmatic.com/topics/fpd/topic-pubmatic.html" // Will be hosted on pubmatic domain
                            },
                            "rubicon" : {
                                iframeURL: "https://rubicon.com:8080/topics/fpd/topic-rubicon.html" // Will be hosted on rubicon domain
                            },
                            "appnexus" : {
                                iframeURL: "https://appnexus.com:8080/topics/fpd/topic-appnexus.html" // Will be hosted on appnexus domain
                            }
                        }
                    }
                ....
            }
})

Note: Also, this is being said in case of SSP rotation, random iframes of third party API caller will be loaded, but cached topics data of third party api caller will be available in user.data object(SDA data) rather than only topic data loaded for iframe for that page view at that particular time.

@patmmccann Please let me know your view on the config and above note if any?

Expiry date for cached data should be added for each bidder or 1 expiry for all bidders? Should publisher be given control to >refresh the cached data by setting the expiry

Expiration should be fairly aggressive I think; as the topics API itself is fairly aggressive with expiration.

How iframe should be maintained for each bidder? Should it be in this repository itself ?

I think the iframes in the default rotation should be in the repo likely; and publisher could override the array with their own array in config.

@patmmccann Can you help to give more details on this ? Thanks

Cached Topics data in local-storage needs to be encrypted (Base64) ?`

I am not sure if we need to encrypt anything, the topics are meant to be passed around freely it seems. I am not sure of any security risk from exposing them.

[patmmccann]

I think the iframes in the default rotation should be in the repo likely; and publisher could override the array with their own array in config.

@patmmccann Can you help to give more details on this ? Thanks

I would think that in this module we would have an array of topics callers' iframe locations, but that array could be overridden by the publisher.

[burk504]

Expiry date for cached data should be added for each bidder or 1 expiry for all bidders? Should publisher be given control to >refresh the cached data by setting the expiry

Expiration should be fairly aggressive I think; as the topics API itself is fairly aggressive with expiration.

My opinion is that Topics are designed to have a 3 week lifespan by definition and that should be the default life of them here. That said, I think a publisher should have the capability to change that timeframe easily.

This also means that each iframe should have a chance to be fired once every 7 days. That way each iframe has a rolling set of topics they have been added to storage. On the 4th 7 day period, the iframe would call again and have the older one from 4 weeks ago get deleted...leaving the storage with a rolling 3 weeks of data from each individual vendor iframe

[pm-nitin-nimbalkar]

Expiry date for cached data should be added for each bidder or 1 expiry for all bidders? Should publisher be given control to >refresh the cached data by setting the expiry

Expiration should be fairly aggressive I think; as the topics API itself is fairly aggressive with expiration.

My opinion is that Topics are designed to have a 3 week lifespan by definition and that should be the default life of them here. That said, I think a publisher should have the capability to change that timeframe easily.

This also means that each iframe should have a chance to be fired once every 7 days. That way each iframe has a rolling set of topics they have been added to storage. On the 4th 7 day period, the iframe would call again and have the older one from 4 weeks ago get deleted...leaving the storage with a rolling 3 weeks of data from each individual vendor iframe

@patmmccann @dgirardi Any comments on the suggestion given by @burk504 ?

[patmmccann]

Jeff's comments sound reasonable to me; I think the more configurable the choices, the better off we are. Prebid's goals are to discover the utility of this interface and to allow publishers and community members to participate in the feedback the browser folks are seeking. I suspect most publishers will want to hit the api every pageview however, and prefer current results to any stored results; I don't anticipate a delay or withheld calling option to be popular.

[abhinavsinha001]

This is what I understand and recommend for basic implementation:

1. We should cache and send only latest topics, the caching is just to reduce Iframe based communication overhead
2. In case any of the caller is not called for more than 3 weeks then it's cached topics should expire in 3 weeks or publishers can define a custom expiry.
3. If we want to cache topics over a dynamic rolling window then lets take that as separate PR
   i. Will need to set expectations throughout the pipe that we are sending extra topics.
   ii. Will have to decide on ordering / notifying which topics are recent vs cached.

[pm-nitin-nimbalkar]

@patmmccann @dgirardi I have added the changes which Abhinav have suggested.

1. Caching the topics for specific for default 3 weeks (21 days). Meaning if topics are older than 21 days, it will be deleted and no longer will be sent in oRTB request. Configuration maintain in topicsFPDModule.

2. Publisher can override the cache expiry time from 21 days to custom day to send the recent topics. Please find below configuration which needs to be on the page.

pbjs.setConfig({
            userSync: {
                ...,
                topics: { 
                        maxTopicCaller: 3, // SSP rotation 
                        bidders: [{
                            bidder: 'pubmatic',
                            iframeURL: 'https://pubmatic.com:8080/topics/fpd/topic.html', // dummy URL
                            expiry: 7 // Configurable expiry days
                        },{
                            bidder: 'rubicon',
                            iframeURL: 'https://rubicon.com:8080/topics/fpd/topic.html', // dummy URL
                            expiry: 7 // Configurable expiry days
                        },{
                            bidder: 'appnexus',
                            iframeURL: 'https://appnexus.com:8080/topics/fpd/topic.html', // dummy URL
                            expiry: 7 // Configurable expiry days
                        }]
                    }
                ....
            }
})

where expiry and maxTopicCaller( Numbers of iframes count which will be loaded ) can be configured which will be override the default configuration set in the topicsFpdModule.
Default configuration in the Module are as follows

• maxTopicCaller is set to 1 (That means random 1 iframe will be loaded which will called Topics API)
• expiry is set to 21 days ( Topics which are cached and older than 21 days will be deleted from cache to fetch recent topics and stop sending stale topics date to oRTB request)

I have also added topics_iframe.html which will be hosted on bidders domain which will call the Topics API just for reference.

Can you guys help us to review the changes and suggest if somethings need to change or needs to be added.

[patmmccann]

@nitin0610 is this ready to come out of draft status?

[pm-nitin-nimbalkar]

@nitin0610 is this ready to come out of draft status?

@patmmccann @dgirardi
Just wanted to check if the code looks good after confirmation we will work on the PR to make it available for merge by removing the draft.
Anything which you can suggest need changes in terms of business logic and code.
Let us(PubMatic) know if you have questions for the same.

[dgirardi]

Please add unit tests; I'd especially like to see the expiration logic tested. Also, how should this be treated for GDPR purpose 1? The implementation as it is will not require any vendor consent, is that appropriate?

[patmmccann]

Purpose 3 is building a personalized ads profile, that's what the topics api does

[patmmccann]

Please make sure the module respects the device access control flag which many publishers set in opt out scenarios

[patmmccann]

@nitin0610 could you clarify what is still in progress? This is looking very near finished to me

[pm-nitin-nimbalkar]

@nitin0610 could you clarify what is still in progress? This is looking very near finished to me

Hi @patmmccann, thanks for approving the changes/request.

Currently we are working on test these feature end to end throughly due to recent changes of GDPR, device access changes. No dev work is pending for this feature.

We will be planning to complete this ASAP and will remove the In progress note from heading.

Also we are working on the documentation for the same.

Thanks.

• Nitin(PubMatic)

[pm-nitin-nimbalkar]

@patmmccann We have completed the testing. PR is ready to get merged. We have approvals too.

[patmmccann]

Thanks! @nitin0610

I saw you mention the documentation but now I'm having trouble finding it

[pm-nitin-nimbalkar]

@patmmccann Please find below PR related to documentation
prebid/prebid.github.io#4222