Revise Account Configuration storage

[bretg]

There have been several recent additions to account-configuration parameters (bid_validations, analytics_config) and there are a couple more coming, e.g. endpoint_config, block_config. This issue aims to clarify the interface for adding this kind of config to Prebid Server.

1. We would like establish an internal configuration model, documented on docs.prebid.org and consistent across PBS-Java and PBS-Go.
2. Each host company can map a mysql schema to this internal configuration model using JSON query functions. This decouples a host company's schema from the capabilities of account config. This basically exists already, but more by accident than by well-documented purpose.
3. When we add a new account-level config in the future, the internal config model will be clear, and host companies can decide to use it or not by just setting their sql to a static/null value.

Here's a draft of the proposed config model:

{
  "status": "active",
  "auction": {
    "price-granularity": "low",
    "banner-cache-ttl": 100,
    "video-cache-ttl": 100,
    "truncate-target-attr": 20,
    "default-integration": "web",
    "bid-validations": {
      "banner-creative-max-size": "enforce"
    },
    "events": {
       // see https://docs.google.com/document/d/1iiKnqZjqpnAvtsP3jVyW66hTySLUmgWtl9_7Z9Wx3oM/edit?ts=60504d04#
    }
  },
  "privacy": {
    "enforce-ccpa": true,
    "enforce-gdpr": true,
    "gdpr": {
      "enabled": true,
      "integration-enabled": {
        "web": true,
        "app": true,
        "amp": true
      },
      "purposes": {
           ...
      },
      "special-features": {
           ...
      }
    }
  },
  "analytics": {
    "modules": {
      "rubicon-analytics": {
        "sampling-factor": 1,
        "event-types TBD": {
          // event logging rules will land here once defined
        }
      }
    }
  },
  "cookie-sync": {
    "default-limit": 4,
    "max-limit": 8,
    "default-coop-sync": false
  }
}

A couple of features to note:

• endpoint config can be collected together (i.e. auction options and cookie-sync options)
• the top level privacy object contains all the GDPR/CCPA/etc options
• the top level analytics object makes room for supplying multiple analytics adapters with different config

If the host company has broken most of these into separate columns in their DB, then the account-query host config might be something like:

settings:
  database:
    type: mysql
    account-query: 

SELECT 
    uuid, 
    JSON_MERGE_PATCH(config, JSON_OBJECT(
        'status', status,
        'auction', JSON_OBJECT(
            'price-granularity', price_granularity,
            'banner-cache-ttl', banner_cache_ttl,
            'video-cache-ttl', video_cache_ttl,
            'truncate-target-attr', truncate_target_attr,
            'default-integration', default_integration,
            'bid-validations', bid_validations,
            'events', JSON_OBJECT(
                'enabled', NOT NOT(events_enabled)
            )
        ), 
        'privacy', JSON_OBJECT(
            'enforce-ccpa', NOT NOT(enforce_ccpa),
            'enforce-gdpr', NOT NOT(enforce_gdpr),
            'gdpr', tcf_config
        ), 
        'analytics', JSON_OBJECT(
            'modules', JSON_OBJECT(
                'rubicon-analytics', JSON_OBJECT(
                    'sampling-factor', analytics_sampling_factor,
                    'legacy-config', analytics_config
                )
            )
        )
    )) as consolidated_config 
FROM accounts_account 
WHERE uuid = %ACCOUNT_ID% 
LIMIT 1;

With this clarified model of account config, here are the steps PBS-Java and PBS-Go would take to add another object:

1. Open an issue describing the feature, including where it goes in the configuration model
2. The configuration model becomes well-documented on docs.prebid.org
3. When PBS-Java/Go implement the account config feature, it's highlighted in the release notes so Host Companies know they have to adjust their account-query configuration. They may choose a static default value or may choose to build out the column in their DB.

FWIW - Magnite is planning to put most new values in a generic 'config' JSON column.

[laurb9]

I like this. It aligns the storage format for files, HTTP API and database across all storable objects (stored requests, imps, accounts). It allows for easy NoSQL integration as well if the enabled field is stored in json also and not alongside it. It no longer requires a complicated SQL query to be provided for configuration.

Postgres has a binary json format (jsonb) which stores parsed json and allows indexing, querying and updating of json subfields.

Given the complexity of working with databases in PBS-Go and PBS-Java I would go one step further and suggest we could move all the database functionality out of both PBS and into a separate microservice implementing the HTTP API already supported by both. This data service can have multiple modules for querying the data (SQL, noSQL, even files) as well as its own cache, and can scale and manage its connections independently.

[SyntaxNode]

Sounds good to me. Easy for me to say though, PBS-Go does not have a db fetcher for account information.

I would go one step further and suggest we could move all the database functionality out of both PBS and into a separate microservice implementing the HTTP API already supported by both.

I very much like the idea of separating out the database layer, such that PBS doesn't force hosts to use our specific schema and need to handle migrations. The http layer is much cleaner and easily allows for a microservice to sit behind it. At the same time, PBS has long supported a built-in database layer and I think it's reasonable to continue to evolve that offering. Removing it completely would force hosts to redesign their systems and that doesn't seem fair.

I would like to see us continue with built-in database support but also encourage new hosts to create a microservice if they want to do something different or support a new database type. I imaging a lot of folks would like to use a nosql store at this point, so we could build that out as a separate service should other hosts want it. Not that I'm volunteering.

[bretg]

Revised the focus of this issue.

Basically we don't want to decide what schema a host company should be using. Instead, we want to better document a "config model" that's consistent across Java and Go, and well-documented. Then host companies can use the account-query config to map their schema to this config model.

[ShriprasadM]

@bretg : Can we update the link to point to PRD, mentioned in events object of proposed config model above - #1672 (comment)

[bretg]

Will review the status of this general account config update with the PBS-Java engineers tomorrow, but in the meantime, the "Event Config" section of https://docs.google.com/document/d/1a-U1DytiCTtacM27qH3OZwhPa7bDwQzfpVRHDhyikWw/edit#heading=h.wg49xad6oyg6 is the documentation I think you're looking for.

I offer to meet with you guys again before you kickoff the project so we wind up on the same page. Prebid-Slack me if interested.

[bretg]

• this issue tracks a general account config change
• PBS-core receives a "config" column in the DB results that stores many attributes and merges all the columns together into one internal config representation
• specifically for events, we recommend placing the account event config (Section 3.1.1-3 of the Events doc) at auctions.events.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[bretg]

This is done in PBS-Java. Documentation about the final JSON format is in https://github.rp-core.com/ContainerTag/prebid-server-java/blob/master-rubicon/docs/application-settings.md

PBS-Host companies can define a query which maps their specific database columns to the PBS internal JSON model as shown in that doc.

[SyntaxNode]

We would like establish an internal configuration model, documented on docs.prebid.org and consistent across PBS-Java and PBS-Go.

@bretg The Prebid Server Committee did not progress to define a standard across both PBS-Go and PBS-Java. Is this still desirable?

Otherwise, PBS-Go currently only accepts json for account config. Database fetching is not currently supported, but would be expected to follow this pattern and return only json when eventually implemented

[SyntaxNode]

Closing this since it's complete for PBS-Java and not applicable for PBS-Go. PBS-Go is working on a new account retrieval implementation which will import json as described in this issue.