Merge branch 'master' into bid-response-reducer

docs/application-settings.md

@@ -24,7 +24,9 @@ There are two ways to configure application settings: database and file. This do
 - `default-integration` - Default integration to assume.
 - `analytics-config.auction-events.<channel>` - defines which channels are supported by analytics for this account
 - `bid-validations.banner-creative-max-size` - Overrides creative max size validation for banners.
+- `status` - allows to mark account as `active` or `inactive`.
 
+Here are the definitions of the "purposes" that can be defined in the GDPR setting configurations:
 ```
 Purpose   | Purpose goal                    | Purpose meaning for PBS (n\a - not affected)  
 ----------|---------------------------------|---------------------------------------------
@@ -43,22 +45,26 @@ sf1       | Precise geo                     | Verifies user opt-in. If the user
 sf2       | Fingerprinting                  | n\a
 ```
 
-## File application setting
+## Setting Account Configuration in Files
 
 In file based approach all configuration stores in .yaml files, path to which are defined in application properties.
 
 ### Configuration in application.yaml
 
-```
+The general idea is that you'll place all the account-specific settings in a separate YAML file and point to that file.
+
+```yaml
 settings:
   filesystem:
     settings-filename: <directory to yaml file with settings>
 ```
 ### File format
 
-```
+Here's an example YAML file containing account-specific settings:
+
+```yaml
 accounts:
-  - id: 14062
+  - id: 1111
     bannerCacheTtl: 100
     videoCacheTtl: 100
     eventsEnabled: true
@@ -70,6 +76,7 @@ accounts:
     analytics-config:
       auction-events:
         amp: true
+    status: active
     gdpr:
       enabled: true
       integration-enabled:
@@ -152,53 +159,69 @@ accounts:
       purpose-one-treatment-interpretation: ignore
 ```
 
-
-## Database application setting
+## Setting Account Configuration in the Database
+
+In database approach account properties are stored in database table.
 
-In database approach account properties are stored in database table with name accounts_account.
+SQL query for retrieving account is configurable and can be specified in [application configuration](config-app.md). 
+Requirements for the SQL query stated below.
 
 ### Configuration in application.yaml
-```
+
+```yaml
 settings:
   database:
-    type: <mysql or postgres>
     pool-size: 20
-    type: mysql
+    type: <mysql or postgres>
     host: <host>
     port: <port>
+    account-query: <SQL query for account>
 ```
 
-### Table description 
+### Configurable SQL query for account requirements
 
-Query to create accounts_account table:
+The general approach is that each host company can set up their database however they wish, so long as the configurable query run by
+Prebid Server returns expected data in the expected order. Here's an example configuration:
 
+```yaml
+settings:
+  database:
+    type: mysql
+    account-query: SELECT uuid, price_granularity, banner_cache_ttl, video_cache_ttl, events_enabled, enforce_ccpa, tcf_config, analytics_sampling_factor, truncate_target_attr, default_integration, analytics_config, bid_validations, status FROM accounts_account where uuid = ? LIMIT 1
 ```
-'CREATE TABLE `accounts_account` (
-`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
-`uuid` varchar(40) NOT NULL,
-`price_granularity` enum('low','med','high','auto','dense','unknown') NOT NULL DEFAULT 'unknown',
-`granularityMultiplier` decimal(9,3) DEFAULT NULL,
-`banner_cache_ttl` int(11) DEFAULT NULL,
-`video_cache_ttl` int(11) DEFAULT NULL,
-`events_enabled` bit(1) DEFAULT NULL,
-`enforce_ccpa` bit(1) DEFAULT NULL,
-`enforce_gdpr` bit(1) DEFAULT NULL,
-`tcf_config` json DEFAULT NULL,
-`analytics_sampling_factor` tinyint(4) DEFAULT NULL,
-`truncate_target_attr` tinyint(3) unsigned DEFAULT NULL,
-`default_integration` varchar(64) DEFAULT NULL,
-`analytics_config` varchar(512) DEFAULT NULL,
-`bid_validations` json DEFAULT NULL,
-`status` enum('active','inactive') DEFAULT 'active',
-`updated_by` int(11) DEFAULT NULL,
-`updated_by_user` varchar(64) DEFAULT NULL,
-`updated` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
-PRIMARY KEY (`id`),
-UNIQUE KEY `uuid` (`uuid`))
-ENGINE=InnoDB DEFAULT CHARSET=utf8'
+
+The SQL query for account must:
+* return following columns, with specified type, in this order:
+    * account ID, string
+    * price granularity, string
+    * banner cache TTL, integer
+    * video cache TTL, integer
+    * events enabled flag, boolean
+    * enforce CCPA flag, boolean
+    * TCF configuration, JSON string, see below
+    * analytics sampling factor, integer
+    * maximum targeting attribute size, integer
+    * default integration value, string
+    * analytics configuration, JSON string, see below
+    * status, string. Expected values: "active", "inactive", NULL. Only "inactive" has any effect and only when settings.enforce-valid-account is on.
+* specify a special single `%ACCOUNT_ID%` placeholder in the `WHERE` clause that will be replaced with account ID in 
+runtime
+
+It is recommended to include `LIMIT 1` clause in the query because only the very first result returned will be taken.
+
+If a host company doesn't support a given field, or they have a different table name, they can just update the query with whatever values are needed. e.g.
+
+```yaml
+settings:
+  database:
+    type: mysql
+    account-query: SELECT uuid, 'med', banner_cache_ttl, video_cache_ttl, events_enabled, enforce_ccpa, tcf_config, 0, null, default_integration, '{}', '{}' FROM myaccountstable where uuid = ? LIMIT 1
 ```
+### Configuration Details
+
+#### TCF configuration JSON
 
-where tcf_config column is json with next format
+Here's an example of the value that the `tcf_config` column can take:
 
 ```json
 {
@@ -311,18 +334,62 @@ where tcf_config column is json with next format
 }
 ```
 
-and bid_validations column is json with next format
+#### Bid Validations configuration JSON
+
+The `bid_validations` column is json with this format:
 
 ```json
 {
   "banner-creative-max-size": "enforce"
 }
 ```
 
-Query used to get an account:
+Valid values are:
+- "skip": don't do anything about creative max size for this publisher
+- "warn": if a bidder returns a creative that's larger in height or width than any of the allowed sizes, log an operational warning.
+- "enforce": if a bidder returns a creative that's larger in height or width than any of the allowed sizes, reject the bid and log an operational warning.
+
+#### Analytics Validations configuration JSON
+
+The `analytics_config`  configuration column format:
+
+```json
+{
+  "auction-events": {
+    "web": true,   // the analytics adapter should log auction events when the channel is web
+    "amp": true,   // the analytics adapter should log auction events when the channel is AMP
+    "app": false   // the analytics adapter should not log auction events when the channel is app
+  }
+}
 ```
-SELECT uuid, price_granularity, banner_cache_ttl, video_cache_ttl, events_enabled, enforce_ccpa, tcf_config, analytics_sampling_factor, truncate_target_attr, default_integration, analytics_config, bid_validations 
-FROM accounts_account where uuid = ?
-LIMIT 1
 
+#### Creating the accounts table
+
+Traditionally the table name used by Prebid Server is `accounts_account`. No one remembers why. But here's SQL 
+you could use to create your table:
+
+```sql
+'CREATE TABLE `accounts_account` (
+    `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
+    `uuid` varchar(40) NOT NULL,
+    `price_granularity` enum('low','med','high','auto','dense','unknown') NOT NULL DEFAULT 'unknown',
+    `granularityMultiplier` decimal(9,3) DEFAULT NULL,
+    `banner_cache_ttl` int(11) DEFAULT NULL,
+    `video_cache_ttl` int(11) DEFAULT NULL,
+    `events_enabled` bit(1) DEFAULT NULL,
+    `enforce_ccpa` bit(1) DEFAULT NULL,
+    `enforce_gdpr` bit(1) DEFAULT NULL,
+    `tcf_config` json DEFAULT NULL,
+    `analytics_sampling_factor` tinyint(4) DEFAULT NULL,
+    `truncate_target_attr` tinyint(3) unsigned DEFAULT NULL,
+    `default_integration` varchar(64) DEFAULT NULL,
+    `analytics_config` varchar(512) DEFAULT NULL,
+    `bid_validations` json DEFAULT NULL,
+    `status` enum('active','inactive') DEFAULT 'active',
+    `updated_by` int(11) DEFAULT NULL,
+    `updated_by_user` varchar(64) DEFAULT NULL,
+    `updated` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
+PRIMARY KEY (`id`),
+UNIQUE KEY `uuid` (`uuid`))
+ENGINE=InnoDB DEFAULT CHARSET=utf8'
 ```

docs/config-app.md

@@ -78,9 +78,10 @@ Removes and downloads file again if depending service cant process probably corr
 - `auction.cache.expected-request-time-ms` - approximate value in milliseconds for Cache Service interacting. This time will be subtracted from global timeout.
 - `auction.cache.only-winning-bids` - if equals to `true` only the winning bids would be cached. Has lower priority than request-specific flags.
 - `auction.generate-bid-id` - whether to generate seatbid[].bid[].ext.prebid.bidid in the OpenRTB response.
-- `auction.id-generator-type` - if generate-bid-id is on, then this defines how the ID should be generated. Currently onlye `uuid` is supported.
+- `auction.generate-source-tid` - whether to generate bidrequest.source.tid in the OpenRTB request.
 - `auction.validations.banner-creative-max-size` - enables creative max size validation for banners. Possible values: `skip`, `enforce`, `warn`. Default is `skip`.
 - `auction.validations.secure-markup` - enables secure markup validation. Possible values: `skip`, `enforce`, `warn`. Default is `skip`.
+- `auction.host-schain-node` - defines global schain node that will be appended to `request.source.ext.schain.nodes` passed to bidders
 
 ## Amp (OpenRTB)
 - `amp.default-timeout-ms` - default operation timeout for OpenRTB Amp requests.
@@ -244,6 +245,7 @@ For database data source available next options:
 - `settings.database.user` - database user.
 - `settings.database.password` - database password.
 - `settings.database.pool-size` - set the initial/min/max pool size of database connections.
+- `settings.database.account-query` - the SQL query to fetch account.
 - `settings.database.stored-requests-query` - the SQL query to fetch stored requests.
 - `settings.database.amp-stored-requests-query` - the SQL query to fetch AMP stored requests.
 - `settings.database.stored-responses-query` - the SQL query to fetch stored responses.
@@ -260,6 +262,24 @@ For HTTP data source available next options:
 For account processing rules available next options:
 - `settings.enforce-valid-account` - if equals to `true` then request without account id will be rejected with 401.
 
+It is possible to specify default account configuration values that will be assumed if account config have them 
+unspecified or missing at all. Example:
+```yaml
+settings:  
+  default-account-config:
+    events-enabled: true
+    enforce-ccpa: true
+    gdpr: '{"enabled": true}'
+    analytics-sampling-factor: 1
+    default-integration: pbjs
+    analytics-config: '{"auction-events":{"amp":true}}'
+```
+See [application settings](application-settings.md) for full reference of available configuration parameters.
+Be aware that individual configuration values will not be merged with concrete 
+account values if they exist in account configuration but account value will completely replace the default value. For 
+example, if account configuration defines `gdpr` field, it will completely replace `settings.default-account-config.gdpr` 
+value in the final account configuration model.
+
 For caching available next options:
 - `settings.in-memory-cache.ttl-seconds` - how long (in seconds) data will be available in LRU cache.
 - `settings.in-memory-cache.cache-size` - the size of LRU cache.

docs/metrics.md

@@ -89,10 +89,10 @@ Following metrics are collected and submitted if account is configured with `bas
 
 Following metrics are collected and submitted if account is configured with `detailed` verbosity:
 - `account.<account-id>.requests.type.(openrtb2-web,openrtb-app,amp,legacy)` - number of requests received from account with `<account-id>` broken down by type of incoming request
-- `account.<account-id>.<bidder-name>.request_time` - timer tracking how long did it take to make a request to `<bidder-name>` when incoming request was from `<account-id>` 
-- `account.<account-id>.<bidder-name>.bids_received` - number of bids received from `<bidder-name>` when incoming request was from `<account-id>`
-- `account.<account-id>.<bidder-name>.requests.(gotbids|nobid)` - number of requests made to `<bidder-name>` broken down by result status  when incoming request was from `<account-id>`
 - `account.<account-id>.requests.rejected` - number of rejected requests caused by incorrect `accountId`
+- `account.<account-id>.adapter.<bidder-name>.request_time` - timer tracking how long did it take to make a request to `<bidder-name>` when incoming request was from `<account-id>` 
+- `account.<account-id>.adapter.<bidder-name>.bids_received` - number of bids received from `<bidder-name>` when incoming request was from `<account-id>`
+- `account.<account-id>.adapter.<bidder-name>.requests.(gotbids|nobid)` - number of requests made to `<bidder-name>` broken down by result status  when incoming request was from `<account-id>`
 
 ## General Prebid Cache metrics
 - `prebid_cache.requests.ok` - timer tracking how long did successful cache requests take

pom.xml

@@ -4,7 +4,7 @@
 
     <groupId>org.prebid</groupId>
     <artifactId>prebid-server</artifactId>
-    <version>1.52.0-SNAPSHOT</version>
+    <version>1.54.0-SNAPSHOT</version>
 
     <name>prebid-server</name>
     <description>Prebid Server (Server-side Header Bidding)</description>

src/main/java/org/prebid/server/auction/AmpRequestFactory.java

@@ -636,7 +636,7 @@ private ExtRequestTargeting createTargetingWithDefaults(ExtRequestPrebid prebid)
 
         final Boolean includeFormat = !isTargetingNull ? targeting.getIncludeformat() : null;
 
-        return ExtRequestTargeting.builder()
+        return (isTargetingNull ? ExtRequestTargeting.builder() : targeting.toBuilder())
                 .pricegranularity(outgoingPriceGranularityNode)
                 .mediatypepricegranularity(mediaTypePriceGranularity)
                 .includewinners(includeWinners)