Introduce a framework for hooks at different stages of request proces…

…sing (#1345)

* First cut of established hook interfaces (#1195)

* Insert Entrypoint hook invocation logic (#1203)

* Insert bidder request hooks invocation logic (#1206)

* Insert auction response hooks invocation logic (#1208)

* Add Module interface and HookCatalog utility (#1199)

* Add hooks execution plan to application and account configuration (#1211)

* Insert raw auction request hooks invocation logic  (#1210)

* Merge master into hook framework (#1228)

* Prebid Server prepare release 1.61.0

* Prebid Server prepare for next development iteration

* Make generation of Bid Adm as a first step to remove duplicating call (#1214)

* Support boolean for bidrequest.ext.prebid.debug together with integer for backward compatibility (#1160)

* Update setuid format parameter and make pixel depends on cookie type (#997)

* Refactored: Extracted Ortb2RequestFactory (#1224)

* Implement usersync method choosing according to filter settings from request (#1130)

Co-authored-by: rpanchyk <rpanchyk@rubiconproject.com>
Co-authored-by: Braslavskyi Andrii <abraslavskyi@rubiconproject.com>
Co-authored-by: Sergii Chernysh <schernysh@users.noreply.github.com>

* Insert processed bidder response hooks invocation logic (#1234)

* Inject RawBidderResponseStage (#1239)

* Flesh out machinery for running hooks following execution plan (#1227)

* Flesh out machinery for running entrypoint hooks

* Unit tests for hooks execution machinery

* Move HookCatalog up in package hierarchy

* Make checkstyle happy

* Split hook execution functionality into smaller and manageable pieces

* Extend hook execution onto raw auction request stage

* Move HookCatalog close to other classes responsible for hooks execution

* Put hooks-related interface implementations to more granular package structure

* Implement bidder request hooks execution

* Implement auction response hooks execution

* Simplify stage executor bootstrapping

* Check that response phase hooks are not able to cause request rejection

* Implement (raw and processed) bidder response hooks execution

* Introduce module context - a way for hook implementations within single module to pass arbitrary data between each other

* Put module context management in one place

* Replace duplication with delegation in invocation contexts implementations

* Processed auction request stage (#1241)

* Basic integration tests for hooks framework (#1250)

* Sample module for integration testing

* Integration test specifically for hooks invocation

* Add sample raw auction request hook to integration test

* Add sample processed auction request hook to integration test

* Add sample bidder request hook to integration test

* Add sample raw bidder response hook to integration test

* Add sample processed bidder response hook to integration test

* Add sample auction response hook to integration test

* Simplify integration test for hooks framework by using a separate account

* Add tests for rejection scenarios at different stages

* Handle whole request rejection scenarios (#1268)

* Drop synchronous groups support (#1278)

* Split host-level execution plan into two parts: one is always executed, another could be overridden by account configuration (#1282)

* Populate auction response with hooks debug and trace output (#1292)

* Implement hooks metrics submission (#1305)

* Expose analytics tags from hooks to analytics adapters (#1311)

* Account for stages that may produce several outcomes (per-bidder) (#1321)

* Rename HttpRequestWrapper to HttpRequestContext and use own models in it and hooks interfaces (#1330)

* Rename HttpRequestWrapper to HttpRequestContext

* Use own model for passing query parameters and headers around

* Make CaseInsensitiveMultiMap immutable

* Add tests for CaseInsensitiveMultiMap

* Get rid of cookies map on HttpRequestContext

* Properly handle and log the errors happened during interaction with hooks (#1340)

* Validate host and default account execution plan

* Tests for unknown hooks validation at startup

* Re-design HookCatalog to make it more generic

* Log and write into hooks trace the errors happened when applying hooks invocation results

* Install and deploy original PBS JAR instead of repackaged Spring Boot fat JAR

* Avoid tampering with impid in integration test for hooks (#1349)

* Avoid tampering with impid in integration test for hooks

* Tidy up BidResponseCreator after refactoring

Co-authored-by: rpanchyk <rpanchyk@rubiconproject.com>
Co-authored-by: rpanchyk <rpanchyk@users.noreply.github.com>
Co-authored-by: Dmitriy <DimaGarbar49@gmail.com>
Co-authored-by: Braslavskyi Andrii <abraslavskyi@rubiconproject.com>

pom.xml

@@ -558,6 +558,7 @@
                 <artifactId>spring-boot-maven-plugin</artifactId>
                 <version>${spring.boot.version}</version>
                 <configuration>
+                    <attach>false</attach>
                     <executable>true</executable>
                 </configuration>
                 <executions>

src/main/java/com/iab/openrtb/response/BidResponse.java

@@ -1,12 +1,10 @@
 package com.iab.openrtb.response;
 
-import com.fasterxml.jackson.databind.node.ObjectNode;
 import lombok.Builder;
 import lombok.Value;
+import org.prebid.server.proto.openrtb.ext.response.ExtBidResponse;
 
-import java.util.Comparator;
 import java.util.List;
-import java.util.Objects;
 
 /**
  * This object is the top-level bid response object (i.e., the unnamed outer
@@ -21,7 +19,7 @@
  * for not bidding, just a {@link BidResponse} object is returned with a reason
  * code in the {@code nbr} attribute.
  */
-@Builder
+@Builder(toBuilder = true)
 @Value
 public class BidResponse {
 
@@ -61,19 +59,5 @@ public class BidResponse {
     /**
      * Placeholder for bidder-specific extensions to OpenRTB.
      */
-    ObjectNode ext;
-
-    public static final Comparator<BidResponse> COMPARATOR = (left, right) -> {
-        if (Objects.isNull(left)
-                || left.getSeatbid().isEmpty()
-                || left.getSeatbid().get(0).getBid().isEmpty()
-                || Objects.isNull(right)) {
-            return -1;
-        }
-        if (right.getSeatbid().isEmpty() || right.getSeatbid().get(0).getBid().isEmpty()) {
-            return 1;
-        }
-        return left.getSeatbid().get(0).getBid().get(0).getPrice()
-                .compareTo(right.getSeatbid().get(0).getBid().get(0).getPrice());
-    };
+    ExtBidResponse ext;
 }

src/main/java/org/prebid/server/auction/BidResponsePostProcessor.java

@@ -3,9 +3,9 @@
 import com.iab.openrtb.request.BidRequest;
 import com.iab.openrtb.response.BidResponse;
 import io.vertx.core.Future;
-import io.vertx.ext.web.RoutingContext;
 import org.prebid.server.cookie.UidsCookie;
 import org.prebid.server.cookie.proto.Uids;
+import org.prebid.server.model.HttpRequestContext;
 import org.prebid.server.settings.model.Account;
 
 /**
@@ -17,15 +17,18 @@ public interface BidResponsePostProcessor {
     /**
      * This method is called when auction is finished.
      *
-     * @param context     represents initial web request
+     * @param httpRequest represents initial web request
      * @param uidsCookie  auction request {@link Uids} container
      * @param bidRequest  original auction request
      * @param bidResponse auction result
      * @param account     {@link Account} fetched from request
      * @return a {@link Future} with (possibly modified) auction result
      */
-    Future<BidResponse> postProcess(RoutingContext context, UidsCookie uidsCookie, BidRequest bidRequest,
-                                    BidResponse bidResponse, Account account);
+    Future<BidResponse> postProcess(HttpRequestContext httpRequest,
+                                    UidsCookie uidsCookie,
+                                    BidRequest bidRequest,
+                                    BidResponse bidResponse,
+                                    Account account);
 
     /**
      * Returns {@link NoOpBidResponsePostProcessor} instance that just does nothing to original auction result.
@@ -39,8 +42,12 @@ static BidResponsePostProcessor noOp() {
      */
     class NoOpBidResponsePostProcessor implements BidResponsePostProcessor {
         @Override
-        public Future<BidResponse> postProcess(RoutingContext context, UidsCookie uidsCookie, BidRequest bidRequest,
-                                               BidResponse bidResponse, Account account) {
+        public Future<BidResponse> postProcess(HttpRequestContext httpRequest,
+                                               UidsCookie uidsCookie,
+                                               BidRequest bidRequest,
+                                               BidResponse bidResponse,
+                                               Account account) {
+
             return Future.succeededFuture(bidResponse);
         }
     }