PTOW-2: Update the pubx analytics adapter.

[nathan-pubx]

Ticket: PTOW-2

Task: update the pubx analytics adapter, to make use of more modern approaches and to consume more data.

Implements:

• Using beacons to send data to pubx endpoints.
  • The beacon preserves the content type of the data.
• Deep copy of the data is sent to pubx, and is not modified throughout the auction (done).
  • The data sent back is cloned before send.
  • Each of the properties present in the auction is either not taken from the auction itself, or taken as a deep copy
• Sample the auction and winning bids together.
  • The sampling uses a hash of the auction id, so that wins and auctions are sampled together.
  • The sampling is not done on the page level, to avoid skewing sampling towards "repeat" auctions, which are less valuable.
• verification of the rank of the auction on the page.
  • as auctions are held, their rank in time is preserved.
  • This auction rank is not specific to an adslot, but global to the page.
  • I have observed consistent behaviour that an auction can have multiple winning bids, each independently rendered. In this case the auction rank is not incremented/repeated, as no auction is held to promote a new winning bid to render.
• Google object data (key value pairs)
  • The google ad server data is constructed before the bidwon event from prebid
  • i was unable to observe any occasion where GAM rendering did not trigger a prebid win event
  • the gam data is tied to an ad slot, and as mentioned above, single auctions were observed rendering many winning bids into the same slot, making tracking the specific GPT KVPs for a bid impossible. The KVPs associated to the winning bid are the KVPs available at the time of the winning bid event firing in prebid.
• PMAC data
  • as requested via slack, the PMAC data is loaded from session storage, and expected to be partitioned by auction id.
  • no data manipulation was done on the pmac data from local storage
• user ids
  • captured from the prebid globals. The types of user ids only are recorded, but the schema could be expanded to support more user information in future.
• Floor price in case of no bids
  • there is a wonderful variable in the priceFloors.js module, _floorDataForAuction, which contains all information, derived or otherwise, regarding the price floor information of an auction, up to 3 seconds after the auction ends. Try as i might, i am unable to record the data from this meta variable.
  • i am pulling price data from the ad unit bids of the prebid auction. The "bid" property of an ad unit is declared by the publisher, and then augmented by the price floors module later in the auction lifecycle. In the event that no bids occur, the ad units will still exist, and the "bid" property of the ad unit, which defines the bidders that should provide responses, is also still defined.
• concurrency support
  • I observed several cases of multiple auctions running concurrently across the ad units of a page. Typically it seems that rather than bundle multiple ad units into one auction, each ad unit has a separate auction. We used to aggregate all bids for "an auction" under one list, and then send the array of bids back as part of the same auction. This meant that regularly, the bids attributed to a single auction were actually part of many auctions, and so the floor data we sent back was messed up (being derived from the first bid)

Not implemented:

• viewability
  • i attempted to use the gpt viewbility module, and because auctions can have multiple winning bids, but the gpt event can only track at the slot level, it was not possible to create a mapping between a bid and a "view event"
  • attempting to use the intersection observer didn't yield sensible results. I attempted to update the insersection observers data in realtime before the send to pubx, which didn't produce good/reliable/accurate viewability results, and attempting to leave the data to the initial data recorded by the observer was similarly inaccurate.
  • attempting to "construct" a viewable event (i.e 50% in view for 1 second) was also entirely inaccurate, since the slots typically render for about a second before the ad is shown.

[tej656]

check for floorData exists from any of the bids

[nathan-pubx]

done; we now check each ad unit for an appropriate bid payload and take the first configured floor data (so incorrect ad unit setups are avoided)

[tej656]

we can remove this duplicate data and keep the adunits in auctionDetail

[nathan-pubx]

done, the lambda endpoint that processes this data doesn't explicitly require it, and can construct it from elsewhere in the payload

[tej656]

this can happen before auctionEnd as well, so better to get it the same way we get it in auctionEnd or save it during the auctionInit itself.

[nathan-pubx]

After double checking the code, it is technically possible for a publisher to operate such that they render the first bid that is returned by prebid, without waiting for any other bids to be returned, or for the end of the auction. This would be a publisher error, and would take quite a substantial amount of work for a publisher to do. (If you look at src/Prebid.js, src/auction.js and src/adRendering.js, you'll see that before prebid will render an ad, it expects to have accepted a suitable bid request for the ad slot, which in turn is managed by the auction_init event and registered as a callback in the running auction itself. As far as i can tell, there are no hooks available to "cancel" the running auction and interject a winning bid callback before the auction has ended, without implicitly ending the auction as a side effect of running the bidwon callback.)

Without operating in a malicious way, this case is "impossible", and so after discussion with PHN i've left this logic as-is.

[tej656]

filter for relevant pubx and hb_pb keys only here

[nathan-pubx]

done; we are now including only the "winning bid" hb_ keys, plus anything starting with "pubx-"

[tej656]

instead of sending the data for every auction, we can batch the data and call it once it exceeds certain size.

[nathan-pubx]

the lambda function which receives this data is not ready to handle batch data, and beacons are not designed for batching, rather they are designed for very small payloads as the browser session "unloads". Additionally, this method of sending data via beacon is consistent with other approaches in pubx:
https://github.com/Pubx-ai/stream/pull/6/files
So, i have made no changes

[tej656]

@nathan-pubx beacon supports payload upto 64KB. Even in the existing code base we batch and send logs when visibility change event is triggered(i.e when user leaves page) and not for every auction. lets add batching to this PR right away. In the mean time this PR gets approved in prebid, we will be updating the lambda's to support this.

if (hasBrowserSupport()) {
    document.addEventListener("visibilitychange", () => {
      if (document.visibilityState === "hidden") {
        pushLogs(aucErrorLogs, ERROR_LOG_API);
        pushLogs(pageErrorLogs, ERROR_LOG_API);
        pushLogs(aucEventLogs, EVENT_LOG_API);
      }
    });
  }

[nathan-pubx]

@tej656 (and @pnhegde , since you also asked a question about batching data)

I assume/hope this is in reference to this, more recent, pull request, for remote logging in the stream repo?

In which case, i've pushed review actions to implement similar behaviour, plus an additional test to check the batching logic. It's worth noting explicitly (although i'm sure you're aware) that the batch behaviour is not yet supported by the lambda function which is expected to process the payload. Additionally, I haven't changed the payload at all, so it can definitely be further optimised for smaller data transfer and marginally reduced cost. Let me know if you'd like me to make additional changes.

[pnhegde]

@nathan-pubx Thanks. We have an update for the lambda to handle the batched data already.

[pnhegde]

@nathan-pubx The above list of comments are from a PR review call we conducted last week and includes suggestion from all of us. Let me know if you want to connect over a call and discuss further.

Also, I would like to extend the requirements to include another field in the captured data which gives consent info. Probably we can make use of this - https://docs.prebid.org/dev-docs/publisher-api-reference/getConsentMetadata.html .

[nathan-pubx]

@nathan-pubx The above list of comments are from a PR review call we conducted last week and includes suggestion from all of us. Let me know if you want to connect over a call and discuss further.

Also, I would like to extend the requirements to include another field in the captured data which gives consent info. Probably we can make use of this - https://docs.prebid.org/dev-docs/publisher-api-reference/getConsentMetadata.html .

Done, but please check that you're happy with the payload. Additionally, i've added changes to address https://pubai.atlassian.net/browse/PUB-2160

[pnhegde]

@tej656 @nathan-pubx Is the beacon batching the data and making the request? Looks like it's not. Should we add visibilitychange event here?

[pnhegde]

@nathan-pubx Would you mind raising this PR again? I made some mistakes while merging and reverted. Now can't see the the updates anywhere.