Programmatic Guaranteed implementation

docs/config-app.md

@@ -189,6 +189,26 @@ Also, each bidder could have its own bidder-specific options.
 - `admin-endpoints.logging-httpinteraction.on-application-port` - when equals to `false` endpoint will be bound to `admin.port`.
 - `admin-endpoints.logging-httpinteraction.protected` - when equals to `true` endpoint will be protected by basic authentication configured in `admin-endpoints.credentials` 
 
+- `admin-endpoints.tracelog.enabled` - if equals to `true` the endpoint will be available.
+- `admin-endpoints.tracelog.path` - the server context path where the endpoint will be accessible.
+- `admin-endpoints.tracelog.on-application-port` - when equals to `false` endpoint will be bound to `admin.port`.
+- `admin-endpoints.tracelog.protected` - when equals to `true` endpoint will be protected by basic authentication configured in `admin-endpoints.credentials` 
+
+- `admin-endpoints.deals-status.enabled` - if equals to `true` the endpoint will be available.
+- `admin-endpoints.deals-status.path` - the server context path where the endpoint will be accessible.
+- `admin-endpoints.deals-status.on-application-port` - when equals to `false` endpoint will be bound to `admin.port`.
+- `admin-endpoints.deals-status.protected` - when equals to `true` endpoint will be protected by basic authentication configured in `admin-endpoints.credentials` 
+
+- `admin-endpoints.lineitem-status.enabled` - if equals to `true` the endpoint will be available.
+- `admin-endpoints.lineitem-status.path` - the server context path where the endpoint will be accessible.
+- `admin-endpoints.lineitem-status.on-application-port` - when equals to `false` endpoint will be bound to `admin.port`.
+- `admin-endpoints.lineitem-status.protected` - when equals to `true` endpoint will be protected by basic authentication configured in `admin-endpoints.credentials` 
+
+- `admin-endpoints.e2eadmin.enabled` - if equals to `true` the endpoint will be available.
+- `admin-endpoints.e2eadmin.path` - the server context path where the endpoint will be accessible.
+- `admin-endpoints.e2eadmin.on-application-port` - when equals to `false` endpoint will be bound to `admin.port`.
+- `admin-endpoints.e2eadmin.protected` - when equals to `true` endpoint will be protected by basic authentication configured in `admin-endpoints.credentials` 
+
 - `admin-endpoints.credentials` - user and password for access to admin endpoints if `admin-endpoints.[NAME].protected` is true`.
 
 ## Metrics
@@ -394,3 +414,38 @@ If not defined in config all other Health Checkers would be disabled and endpoin
 - `analytics.pubstack.buffers.size-bytes` - threshold in bytes for buffer to send events. 
 - `analytics.pubstack.buffers.count` - threshold in events count for buffer to send events
 - `analytics.pubstack.buffers.report-ttl-ms` - max period between two reports.
+
+## Programmatic Guaranteed Delivery
+- `deals.planner.plan-endpoint` - planner endpoint to get plans from.
+- `deals.planner.update-period` - cron expression to start job for requesting Line Item metadata updates from the Planner.
+- `deals.planner.plan-advance-period` - cron expression to start job for advancing Line Items to the next plan.
+- `deals.planner.retry-period-sec` - how long (in seconds) to wait before re-sending a request to the Planner that previously failed with 5xx HTTP error code.
+- `deals.planner.timeout-ms` - default operation timeout for requests to planner's endpoints.
+- `deals.planner.register-endpoint` - register endpoint to get plans from.
+- `deals.planner.register-period-sec` - time period (in seconds) to send register request to the Planner.
+- `deals.planner.username` - username for planner BasicAuth.
+- `deals.planner.password` - password for planner BasicAuth.
+- `deals.delivery-stats.delivery-period` - cron expression to start job for sending delivery progress to planner.
+- `deals.delivery-stats.cached-reports-number` - how many reports to cache while planner is unresponsive.
+- `deals.delivery-stats.timeout-ms` - default operation timeout for requests to delivery progress endpoints.
+- `deals.delivery-stats.username` - username for delivery progress BasicAuth.
+- `deals.delivery-stats.password` - password for delivery progress BasicAuth.
+- `deals.delivery-stats.line-items-per-report` - max number of line items in each report to split for batching. Default is 25.
+- `deals.delivery-stats.reports-interval-ms` - interval in ms between consecutive reports. Default is 0.
+- `deals.delivery-stats.batches-interval-ms` - interval in ms between consecutive batches. Default is 1000.
+- `deals.delivery-stats.request-compression-enabled` - enables request gzip compression when set to true.
+- `deals.delivery-progress.line-item-status-ttl-sec` - how long to store line item's metrics after it was expired.
+- `deals.delivery-progress.cached-plans-number` -  how many plans to store in metrics per line item.
+- `deals.delivery-progress.report-reset-period`- cron expression to start job for closing current delivery progress and starting new one.
+- `deals.delivery-progress-report.competitors-number`- number of line items top competitors to send in delivery progress report.
+- `deals.user-data.user-details-endpoint` - user Data Store endpoint to get user details from.
+- `deals.user-data.win-event-endpoint` - user Data Store endpoint to which win events should be sent.
+- `deals.user-data.timeout` - time to wait (in milliseconds) for User Data Service response.
+- `deals.user-data.user-ids` - list of Rules for determining user identifiers to send to User Data Store.
+- `deals.max-deals-per-bidder` - maximum number of deals to send to each bidder.
+- `deals.alert-proxy.enabled` - enable alert proxy service if `true`.
+- `deals.alert-proxy.url` - alert service endpoint to send alerts to.
+- `deals.alert-proxy.timeout-sec` - default operation timeout for requests to alert service endpoint.
+- `deals.alert-proxy.username` - username for alert proxy BasicAuth.
+- `deals.alert-proxy.password` - password for alert proxy BasicAuth.
+- `deals.alert-proxy.alert-types` - key value pair of alert type and sampling factor to send high priority alert.

docs/deals.md

@@ -0,0 +1,152 @@
+# Deals
+
+## Planner and Register services
+
+### Planner service
+
+Periodically request Line Item metadata from the Planner. Line Item metadata includes:
+1. Line Item details
+2. Targeting
+3. Frequency caps
+4. Delivery schedule
+
+### Register service
+
+Each Prebid Server instance register itself with the General Planner with a health index
+(QoS indicator based on its internal counters like circuit breaker trip counters, timeouts, etc.)
+and KPI like ad requests per second.
+
+Also allows planner send command to PBS admin endpoint to stored request caches and tracelogs.
+
+### Planner and register service configuration 
+
+```yaml
+planner:
+   register-endpoint: <register endpoint>
+   plan-endpoint: <planner endpoint>
+   update-period: "0 */1 * * * *"
+   register-period-sec: 60
+   timeout-ms: 8000
+   username: <username for BasicAuth>
+   password: <password for BasicAuth>
+```
+
+## Deals stats service
+
+Supports sending reports to delivery stats serving with following metrics:
+
+1. Number of client requests seen since start-up
+2. For each Line Item
+- Number of tokens spent so far at each token class within active and expired plans
+- Number of times the account made requests (this will be the same across all LineItem for the account)
+- Number of win notifications
+- Number of times the domain part of the target matched
+- Number of times impressions matched whole target
+- Number of times impressions matched the target but was frequency capped
+- Number of times impressions matched the target but the fcap lookup failed
+- Number of times LineItem was sent to the bidder
+- Number of times LineItem was sent to the bidder as the top match
+- Number of times LineItem came back from the bidder
+- Number of times the LineItem response was invalidated 
+- Number of times the LineItem was sent to the client
+- Number of times the LineItem was sent to the client as the top match
+- Array of top 10 competing LineItems sent to client
+
+### Deals stats service configuration
+
+```yaml
+delivery-stats:
+  endpoint: <delivery stats endpoint>
+  delivery-period: "0 */1 * * * *"
+  cached-reports-number: 20
+  line-item-status-ttl-sec: 3600
+  timeout-ms: 8000
+  username: <username for BasicAuth>
+  password: <password for BasicAuth>
+```
+
+## Alert service
+
+Sends out alerts when PBS cannot talk to general planner and other critical situations. Alerts are simply JSON messages
+over HTTP sent to a central proxy server.
+
+```yaml
+  alert-proxy:
+    enabled: truew
+    timeout-sec: 10
+    url: <aler service endpoint url>
+    username: <username for BasicAuth>
+    password: <password for BasicAuth>
+    alert-types: 
+       <type of allert>: <sampling factor>
+       pbs-planner-empty-response-error: 15
+```
+
+## GeoLocation service
+
+This service currently has 1 implementation:
+- MaxMind
+
+In order to support targeting by geographical attributes the service will provide the following information:
+
+1. `continent` - Continent code
+2. `region` - Region code using ISO-3166-2
+3. `metro` - Nielsen DMAs
+4. `city` - city using provider specific encoding
+5. `lat` - latitude from -90.0 to +90.0, where negative is south
+6. `lon` - longitude from -180.0 to +180.0, where negative is west
+
+### GeoLocation service configuration for MaxMind
+
+```yaml
+geolocation:
+  enabled: true
+  type: maxmind
+  maxmind:
+    remote-file-syncer:
+      download-url: <url to maxmind database>
+      save-filepath: <save-filepath>
+      tmp-filepath: <tmp-filepath>
+      retry-count: 3
+      retry-interval-ms: 3000
+      timeout-ms: 300000
+      update-interval-ms: 0
+      http-client:
+        connect-timeout-ms: 2500
+        max-redirects: 3
+```
+
+## User Service
+
+This service is responsible for:
+- Requesting user targeting segments and frequency capping status from the User Data Store
+- Reporting to User Data Store when users finally see ads to aid in correctly enforcing frequency caps
+
+### User service configuration
+
+```yaml
+  user-data:
+    win-event-endpoint: <win-url-endpoint>
+    user-details-endpoint: <user-deatils-endpoint>
+    timeout: 1000
+    user-ids:
+      - location: rubicon
+        source: uid
+        type: khaos
+```
+1. khaos, adnxs - types of the ids that will be specified in requests to User Data Store
+2. source - source of the id, the only supported value so far is “uids” which stands for uids cookie
+3. location - where exactly in the source to look for id
+
+## Device Info Service
+
+DeviceInfoService returns  device-related attributes based on User-Agent for use in targeting:
+- deviceClass: desktop, tablet, phone, ctv
+- os: windows, ios, android, osx, unix, chromeos
+- osVersion
+- browser: chrome, firefox, edge, safari
+- browserVersion
+
+## See also
+
+- [Configuration](config.md)

docs/endpoints/deals-status.md

@@ -0,0 +1,10 @@
+# Deals Status
+This endpoint is available on admin port called /pbs-admin/deals-status
+
+## `GET /pbs-admin/deals-status` 
+
+Giving read-only access to current Line Items status, progress and aggregated metrics.
+
+### Sample request
+
+`GET http://prebid.site.com/pbs-admin/deals-status` 

docs/endpoints/lineitem-status.md

@@ -0,0 +1,18 @@
+# Line Item status
+
+This endpoint is available on admin port called `/pbs-admin/lineitem-status`.
+
+Giving read-only access to defined in parameters line item. Contains information about active delivery schedule,
+ready at timestamp, spent tokens number and pacing frequency in milliseconds.
+
+## `GET /pbs-admin/lineitem-status?id=<lineItemId>`
+
+### Query parameters:
+
+This endpoint supports the following query parameters:
+
+`id` - line item id indicate a the line item about which information is needed.
+
+### Sample request
+
+`GET http://prebid.site.com/pbs-admin/lineitem-status?id=lineItemId1`

docs/endpoints/tracelog.md

@@ -0,0 +1,29 @@
+# Tracelog Endpoint
+
+This endpoint is available on admin port called `/pbs-admin/tracelog`.
+
+## POST `/pbs-admin/tracelog`
+
+ Allows to configure logging level for specific account, line item and bidder code during some defined time period.
+
+### Query parameters:
+
+This endpoint supports the following query parameters:
+
+ 1. `account` - specified an account for which logging level should be changed. (Not required, no default value) 
+ 2. `lineItemId` - specified a lineItemId for which logging level should be changed. (Not required, no default value) 
+ 3. `bidderCode`- specified a bidderCode for which logging level should be changed. (Not required, no default value) 
+ 4. `level` - specified a log level to which logs should be updated. Allowed values are `info`, `warn`, `trace`,
+ `error`, `fatal`, `debug`. Default value if not defined is `error`. (Not required) 
+ 5. `duration` - time in seconds during which changes will be applied. (Required).
+
+At least one of `account`, `lineItemId` or `bidderCode` should be specified. If more than one specified,
+logic conjuction (and operation) is applied to parameters.
+
+### Request samples
+
+`GET http://prebid.site.com/pbs-admin/tracelog?account=1234&duration=100` - updates logging level to `error` level for account 1234
+for 100 seconds.
+
+`GET http://prebid.site.com/pbs-admin/tracelog?account=1234&bidder=rubicon&lineItemId=lineItemId1&level=debug&duration=100` - updates
+logging level to warn, for account = 1234 and bidder = rubicon and lineItemId = lineItemId1 for 100 seconds.

docs/metrics.md

@@ -135,3 +135,29 @@ Following metrics are collected and submitted if account is configured with `det
 - `analytics.<reporter-name>.(auction|amp|video|cookie_sync|event|setuid).timeout` - number of event requests, failed with timeout cause
 - `analytics.<reporter-name>.(auction|amp|video|cookie_sync|event|setuid).err` - number of event requests, failed with errors
 - `analytics.<reporter-name>.(auction|amp|video|cookie_sync|event|setuid).badinput` - number of event requests, rejected with bad input cause
+
+## win notifications
+- `win_notifications` - total number of win notifications.
+- `win_requests` - total number of requests sent to user service for win notifications.
+- `win_request_preparation_failed` - number of request failed validation and were not sent.
+- `win_request_time` - latency between request to user service and response for win notifications.
+- `win_request_failed` - number of failed request sent to user service for win notifications.
+- `win_request_successful` - number of successful request sent to user service for win notifications.
+
+## user details
+- `user_details_requests` - total number of requests sent to user service to get user details.
+- `user_details_request_preparation_failed` - number of request failed validation and were not sent.
+- `user_details_request_time` - latency between request to user service and response to get user details.
+- `user_details_request_failed` - number of failed request sent to user service to get user details.
+- `user_details_request_successful` -  number of successful request sent to user service to get user details.
+
+## Programmatic guaranteed metrics
+- `pg.planner_lineitems_received` - number of line items received from general planner.
+- `pg.planner_requests` - total number of requests sent to general planner.
+- `pg.planner_request_failed` - number of failed request sent to general planner.
+- `pg.planner_request_successful` - number of successful requests sent to general planner.
+- `pg.planner_request_time` - latency between request to general planner and its successful (200 OK) response.
+- `pg.delivery_requests` - total number of requests to delivery stats service.
+- `pg.delivery_request_failed` - number of failed requests to delivery stats service.
+- `pg.delivery_request_successful` - number of successful requests to delivery stats service.
+- `pg.delivery_request_time` - latency between request to delivery stats and its successful (200 OK) response.

pom.xml

@@ -55,6 +55,7 @@
         <mockito.version>2.23.4</mockito.version>
         <assertj.version>3.8.0</assertj.version>
         <wiremock.version>2.26.3</wiremock.version>
+        <awaitility.version>4.0.1</awaitility.version>
         <jetty.version>9.4.43.v20210629</jetty.version>
         <restassured.version>3.0.6</restassured.version>
         <h2.version>1.4.196</h2.version>
@@ -88,6 +89,10 @@
             <artifactId>spring-boot-starter</artifactId>
             <version>${spring.boot.version}</version>
         </dependency>
+        <dependency>
+            <groupId>org.springframework.boot</groupId>
+            <artifactId>spring-boot-starter-aop</artifactId>
+        </dependency>
         <dependency>
             <groupId>javax.annotation</groupId>
             <artifactId>javax.annotation-api</artifactId>
@@ -304,6 +309,12 @@
             <version>${assertj.version}</version>
             <scope>test</scope>
         </dependency>
+        <dependency>
+            <groupId>org.awaitility</groupId>
+            <artifactId>awaitility</artifactId>
+            <version>${awaitility.version}</version>
+            <scope>test</scope>
+        </dependency>
         <dependency>
             <groupId>org.springframework.boot</groupId>
             <artifactId>spring-boot-starter-test</artifactId>

src/main/java/org/prebid/server/analytics/model/NotificationEvent.java

@@ -17,6 +17,8 @@ public class NotificationEvent {
 
     Account account;
 
+    String lineItemId;
+
     String bidder;
 
     Long timestamp;