From d9c133fa7231d6e287d19024f64a7a31c8ac4d4a Mon Sep 17 00:00:00 2001 From: Mikko Karjalainen Date: Thu, 24 May 2018 11:24:31 +0100 Subject: [PATCH 1/8] Add request/response transformation examples into plugin-scenarios.md. --- docs/developer-guide/plugins-scenarios.md | 259 +++++++++++++--------- 1 file changed, 151 insertions(+), 108 deletions(-) diff --git a/docs/developer-guide/plugins-scenarios.md b/docs/developer-guide/plugins-scenarios.md index 80d73fde30..ec3d6d0375 100644 --- a/docs/developer-guide/plugins-scenarios.md +++ b/docs/developer-guide/plugins-scenarios.md @@ -4,130 +4,173 @@ ### Synchronously transforming requests -Transforming request object synchronously is trivial. Just transform the HttpRequest object via the HttpRequest.Builder -object, and pass it to the chain.proceed(), as the example demonstrates. - - import com.hotels.styx.api.HttpInterceptor; - import com.hotels.styx.api.HttpRequest; - import com.hotels.styx.api.HttpResponse; - - public class FooAppendingInterceptor implements HttpInterceptor { - @Override - public Observable intercept(HttpRequest request, Chain chain) { - return chain.proceed( - request.newBuilder() - .header("X-Foo", "Bar") - .build()); - } +Transforming request object synchronously is trivial. Just call `request.newBuilder()` to +create a new `HttpRequest.Builder` object. It has already been initialised with the copy of +the original `request`. Modify the builder as desired, consturct a new version, and pass +it to the `chain.proceed()`, as the example demonstrates: + +```java +import com.hotels.styx.api.HttpRequest; +import com.hotels.styx.api.HttpResponse; +import com.hotels.styx.api.StyxObservable; +import com.hotels.styx.api.plugins.spi.Plugin; + +import static com.hotels.styx.api.HttpHeaderNames.USER_AGENT; + +public class SyncRequestPlugin implements Plugin { + @Override + public StyxObservable intercept(HttpRequest request, Chain chain) { + return chain.proceed( + request.newBuilder() + .header(USER_AGENT, "Styx/1.0 just testing plugins") + .build() + ); } +} + +``` ### Synchronously transforming response -In this example, we will synchronously transom the response by adding an "X-Foo" header to it. +This example demonstrates how to synchronously transform a HTTP response. We will +use a `StyxObservable` `map` method to add an "X-Foo" header to the response. + +```java +import com.hotels.styx.api.HttpInterceptor; +import com.hotels.styx.api.HttpRequest; +import com.hotels.styx.api.HttpResponse; +import com.hotels.styx.api.StyxObservable; +import com.hotels.styx.api.plugins.spi.Plugin; + +public class SyncResponsePlugin implements Plugin { + @Override + public StyxObservable intercept(HttpRequest request, HttpInterceptor.Chain chain) { + return chain.proceed(request) + .map(response -> response.newBuilder() + .header("X-Foo", "bar") + .build() + ); + } +} -Use Rx Java map() to transform a response object synchronously. +``` + +### Asynchronously transform request object -Remember that chain.proceed() returns an Rx Observable of HttpResponse. We'll use map() to register a callback that Java -Rx framework calls back once the response becomes available. In the example below, it is the lambda expression -HttpResponse -> HttpResponse that is called when response is available. The lambda expression constructs a new -HttpResponse object with the new header added. +Sometimes it is necessary to transform the request asynchronously. For example, may need to +look up external key value stores to parametrise the request transformation. The example below +shows how to modify a request URL path based on an asynchronous lookup to an external database. + +```java +import com.hotels.styx.api.HttpInterceptor; +import com.hotels.styx.api.HttpRequest; +import com.hotels.styx.api.HttpResponse; +import com.hotels.styx.api.StyxObservable; +import com.hotels.styx.api.Url; +import com.hotels.styx.api.plugins.spi.Plugin; -The map() itself returns a new Rx Observable, satisfying the type signature for intercept() method. -Therefore it can be returned as such from intercept(). +import java.util.concurrent.CompletableFuture; - import com.hotels.styx.api.HttpInterceptor; - import com.hotels.styx.api.HttpRequest; - import com.hotels.styx.api.HttpResponse; - - public class FooAppendingInterceptor implements HttpInterceptor { - @Override - public Observable intercept(HttpRequest request, Chain chain) { - return chain.proceed(request) - .map(response -> response.newBuilder() - .header("X-Foo", "Bar") - .build()); - } - } - - -### Asynchronously transform request object +public class AsyncRequestInterceptor implements Plugin { -Sometimes it is necessary to transform the request asynchronously. For example, it may be necessary to consult an -external service(s) like databases and such to look up necessary information for the request transformation. In this case: -1. Perform a non-blocking call to an asynchronous API. It is best to wrap such API behind Rx Java observables. -2. Call the chain.proceed(request) when the the asynchronous operation completes. -3. Because chain.proceed() itself is an asynchronous operation returning Observable[HttpResponse], it needs to be called from inside Rx flatMap() operator. - -In the following example, the interceptor either allows or rejects the request based on a query to an external service. -The API for external service is wrapped behind requestAccessQueryAsynch() method. Normally a method like this would -initiate a transaction to the external service, and return a Future, Promise, Rx Observable, etc. of the outcome. -For sake of simplicity, here it returns an Observable.just(true). - -The flatMap() operator allows us to register a function that itself performs an asynchronous operation. In our example, -requestAccessQueryAsynch() returns back an Observable[Boolean] indicating if the access is granted or not. When the -access query completes, Rx framework calls out to the registered function, which in this example is a lambda expression -of type Boolean => Observable[HttpResponse]. The lambda expression transforms the access grant outcome into a HttpResponse -object. This is an asynchronous transformation, because the lambda expression calls out to asynchronous chain.proceed(). - -Our lambda expression looks into the access grant outcome. When true, the request is proxied onwards by a call to -chain.proceed(request). Otherwise a FORBIDDEN response is returned. - - import com.hotels.styx.api.HttpInterceptor; - import com.hotels.styx.api.HttpRequest; - import com.hotels.styx.api.HttpResponse; - ... - - public class AsyncRequestDelayInterceptor implements HttpInterceptor { - private static final HttpResponse rejected = HttpResponse.Builder.response(FORBIDDEN).build(); - @Override - public Observable intercept(HttpRequest request, Chain chain) { - return requestAccessQueryAsync(request) - .flatMap((allowed) -> allowed ? chain.proceed(request) : Observable.just(rejected)); - } + @Override + public StyxObservable intercept(HttpRequest request, HttpInterceptor.Chain chain) { - private Observable requestAccessQueryAsync(HttpRequest request) { - // do something asynchronous - // ... - return Observable.create((downstream) -> { - CompletableFuture.supplyAsync(() -> { - downstream.onNext("foo"); - downstream.onComplete(); - }); - }); - } + return StyxObservable.of(request) // (1) + .flatMap(na -> asyncUrlReplacement(request.url())) // (2) + .map(newUrl -> request.newBuilder() // (5) + .url(newUrl) + .build()) + .flatMap(chain::proceed); // (6) } - + + private static StyxObservable asyncUrlReplacement(Url url) { + return StyxObservable.from(pathReplacementService(url.path())) // (4) + .map(newPath -> new Url.Builder(url) + .path(newPath) + .build()); + } + + private static CompletableFuture pathReplacementService(String url) { + return CompletableFuture.completedFuture("/replacement/path"); // (3) + } +} +``` + +Step 1. For asynchronous transformation we'll start by constructing a response observable. +We can construct it with any initial value, but in this example we'll just use the `request`. +But it doesn't have to be a `request` as it is available from the closure. + +Step 2. We will call the `asyncUrlReplacement`, and bind it to the response observable using +`flatMap` operator. The `asyncUrlReplacement` wraps a call to the remote service and converts +the outcome into a `StyxObservable`. + +Step 3. A call to `pathReplacementService` makes a non-blocking call to the remote key/value store. +Well, at least we pretend to call the key value store, but in this example we'll just return a +completed future of a constant value. + +Step 4. `CompletableFuture` needs to be converted to `StyxObservable` so that the operation can +be bound to the response observable created previously in step 1. + +Step 5. The eventual outcome from the `asyncUrlReplacement` yields a new, modified URL instance. +We will map this to a new `HttpRequest` instance replaced URL. + +Step 6. Finally, we will bind the outcome of `chain.proceed` into the response observable. +Remember that `chain.proceed` returns an `Observable` it is therefore +interface compatible and can be `flatMap`'d to the response observable. The resulting +response observable chain is returned from the `intercept`. + + ### Asynchronously transform response object -Processing a HTTP response asynchronously is very similar to processing response synchronously. Instead of Rx map() -operator, you'll use a flatMap(). In this example, we assume asyncOperation() makes a network call, performs I/O, -or off-loads response processing to a separate thread pool. For this reason it returns an Observable[HttpResponse] -so that it can emit a transformed response once completed. +This example demonstrates asynchronous response processing. Here we pretend that `callTo3rdParty` method +makes a non-blocking request to retrieve string that is inserted into a response header. + +A `callTo3rdParty` returns a `CompletableFuture` which asynchronously produces a string value. We +will call this function when the HTTP response is received. + +```java +import com.hotels.styx.api.HttpInterceptor; +import com.hotels.styx.api.HttpRequest; +import com.hotels.styx.api.HttpResponse; +import com.hotels.styx.api.StyxObservable; +import com.hotels.styx.api.plugins.spi.Plugin; + +import java.util.concurrent.CompletableFuture; + +public class AsyncResponseInterceptor implements Plugin { + private static final String X_MY_HEADER = "X-My-Header"; + + @Override + public StyxObservable intercept(HttpRequest request, HttpInterceptor.Chain chain) { + return chain.proceed(request) // (1) + .flatMap(response -> // (3) + StyxObservable + .from(callTo3rdParty(response.header(X_MY_HEADER).orElse("default"))) // (1) + .map(value -> // (4) + response.newBuilder() + .header(X_MY_HEADER, value) + .build()) + ); + } + + private static CompletableFuture callTo3rdParty(String myHeader) { + return CompletableFuture.completedFuture("value"); + } +} +``` -The Rx framework calls asyncOperation() once the the response becomes available from the adjacent plugin. A call to -asyncOperation() triggers another asynchronous operation, whose result becomes available via the returned observable, -which is linked to the Observable chain using the Rx flatMap() operator. +Step 1. We start by calling `chain.proceed(request)` to obtain a response observable. + +Step 2. A `callTo3rdParty` returns a CompletableFuture. We use `StyxObservable.from` to convert it +into a `StyxObservable` so that it can be bound to the response observable. + +Step 3. The `flatMap` operator binds `callToThirdParty` into the response observable. + +Step 4. We will transform the HTTP response by inserting an outcome of `callToThirdParty`, or `value`, +into the response headers. - import rx.Observable; - import rx.schedulers.Schedulers; - import rx.lang.scala.ImplicitFunctionConversions.*; - import com.hotels.styx.api.HttpInterceptor; - import com.hotels.styx.api.HttpRequest; - import com.hotels.styx.api.HttpResponse; - import com.hotels.styx.api.HttpInterceptor.Chain; - ... - - public class AsyncResponseProcessingPlugin implements HttpInterceptor { - public Observable intercept(HttpRequest request, Chain chain) { - return chain.proceed(request).flatMap(this::asyncOperation); - } - - private Observable asyncOperation(HttpResponse response) { - return Observable.just(response); - } - } - ## Transforming HTTP Content HTTP content can be processed in a streaming fashion, or it can be aggregated into one big blob and decoded into a From 5ee7243d44c2cfb2b7a482bff250c27841e522fe Mon Sep 17 00:00:00 2001 From: Mikko Karjalainen Date: Fri, 25 May 2018 09:09:30 +0100 Subject: [PATCH 2/8] Update plugin-scenarios.md with 1.0 API. --- docs/developer-guide/plugins-scenarios.md | 253 ++++++++++++---------- 1 file changed, 139 insertions(+), 114 deletions(-) diff --git a/docs/developer-guide/plugins-scenarios.md b/docs/developer-guide/plugins-scenarios.md index ec3d6d0375..f8bf7a62d3 100644 --- a/docs/developer-guide/plugins-scenarios.md +++ b/docs/developer-guide/plugins-scenarios.md @@ -171,134 +171,159 @@ Step 4. We will transform the HTTP response by inserting an outcome of `callToTh into the response headers. -## Transforming HTTP Content +## HTTP Content Transformations -HTTP content can be processed in a streaming fashion, or it can be aggregated into one big blob and decoded into a -business domain object. Here we will explain how to do this. And as always, this can be done in a synchronous or -asynchronous fashion. +Styx exposes HTTP messages to interceptors as streaming `HttpRequest` and `HttpResponse` messages. +In this form, the interceptors can process the content in a streaming fashion. That is, they they +can look into, and modify the content as it streams through. -Content transformation always involves creating a new copy of a proxied HTTP object and overriding its body content. -Because the body itself is an Observable, it can be transformed using similar techniques as demonstrated for -the HTTP headers above. The example below transforms the HTTP response content in as streaming fashion using a -transformContent() function. The new response is created by invoking a response.newBuilder() method. Using the -resulting builder, the response body is overridden. +Alternatively, streaming messages can be aggregated into a `FullHttpRequest` or `FullHttpResponse` +messages. The full HTTP message body is then available at interceptor's disposal. Note that content +aggregation is always an asynchronous operation. This is because Styx must wait until all content +has been received. - chain.proceed(request) - .map((HttpResponse response) -> response.newBuilder() - .body(response.body().content().map(this::transformContent)) - .build() - ); +### Aggregating Content into Full Messages -As always, the content can be transformed both synchronously or asynchronously. In this document we will explore all the options. +```java +import com.hotels.styx.api.HttpRequest; +import com.hotels.styx.api.HttpResponse; +import com.hotels.styx.api.StyxObservable; +import com.hotels.styx.api.plugins.spi.Plugin; -Content can be decoded to a business domain object for further processing. +public class RequestAggregationPlugin implements Plugin { + private static final int MAX_CONTENT_LENGTH = 100000; -### Decode content into business domain objects + @Override + public StyxObservable intercept(HttpRequest request, Chain chain) { + return request.toFullRequest(MAX_CONTENT_LENGTH) + .map(fullRequest -> fullRequest.newBuilder() + .body(new byte[0], true) + .build()) + .flatMap(fullRequest -> chain.proceed(fullRequest.toStreamingRequest())); + } +} +``` -Decoding content into a business domain object is always an asynchronous operation. This is because Styx must wait -until it has received enough HTTP content to attempt decoding content. +`maxContentBytes` is a safety valve that prevents Styx from accumulating too much content +and possibly running out of memory. Styx only accumulates up to `maxContentBytes` of content. +`toFullRequest` fails when the content stream exceeds this amount, and Styx emits a +`ContentOverflowException`, -HttpResponse object has a decode() method that can be used to decode HTTP response content into business domain objects. -The decode() method has the following type: - public Observable> decode(Function decoder, int maxContentBytes) - -The decode() takes two arguments: a decoder function, and a maxContentBytes integer. It returns an Observable> -which is underlines its asynchronous nature. The decode() accumulates entire HTTP response content into a single aggregated -byte buffer, and then calls the supplied decoder method passing in the aggregated buffer as a parameter. The decoder -is a function, or typically a lambda expression supplied by the plugin, that converts the aggregated http content into -a business domain object of type T. Styx will take care of managing all aspects related to direct memory buffers, such -as releasing them if necessary. +### Synchronously transforming streaming request content -The maxContentBytes is a safety valve that prevents styx from accumulating too much content. Styx will only accumulate -up to maxContentBytes of content. If the content exceeds this amount, Styx will emit a ContentOverflowException. -This is to prevent out of memory exceptions in face of very large responses. +In this rather contrived example we will transform HTTP response content to +upper case letters. -Because decode() itself is an asynchronous operation, the plugin must call it inside a flatMap() operator. -When the HTTP response has arrived, and its content decoded, a DecodedResponse instance is emitted. The DecodedResponse -contains the decoded business domain object along with a HTTP response builder that can be used for further transformations. +Streaming `HttpRequest` content is a byte buffer stream. The stream can be accessed +with a `request.body()` method and its data type is `StyxObservable`. -In the example below, we will map over the decodedResponse to add a new "bytes_aggregated" header to the response, containing -a string length (as opposed to content length) of the received response content. Note that it is necessary to add the -response body back to the new response by calling decodedResponse.responseBuilder().body(). +Because `toUpperCase` is non-blocking transformation we can compose it to the original +byte stream using a `map` operator. - - @Override - public Observable intercept(HttpRequest request, Chain chain) { - return chain.proceed(request) - .flatMap(response -> response.decode((byteBuf) -> byteBuf.toString(UTF_8), maxContentBytes)) - .map(decodedResponse -> decodedResponse.responseBuilder() - .header("test_plugin", "yes") - .header("bytes_aggregated", decodedResponse.body().length()) - .body(decodedResponse.body()) - .build()); - } - -### Synchronously transform streaming request content - -In the following example, we will transform the HTTP response content to upper case letters. - -Because the transformation to upper case does not involve IO or blocking operations, it can be done synchronously with -Rx map() operator. First we create a new observable called toUpperCase which contains the necessary transformation on -the body observable. Then we create a new HTTP request object passing in the transformed observable as a new body. - -Notice that the lambda expression for the map() operator receives a ByteBuf buf as an argument. We must assume it is -a reference counted buffer, and to avoid leaks we must call buf.release() before returning a copy from the lambda expression. - - import com.hotels.styx.api.HttpRequest; - import com.hotels.styx.api.HttpResponse; - import io.netty.buffer.ByteBuf; - import rx.Observable; - import static com.google.common.base.Charsets.UTF_8; - import static io.netty.buffer.Unpooled.copiedBuffer; - - ... - - @Override - public Observable intercept(HttpRequest request, Chain chain) { - Observable toUpperCase = request.body() - .content() - .map(buf -> { - String transformed = buf.toString(UTF_8).toUpperCase(); - buf.release(); - return copiedBuffer(transformed, UTF_8); - }); - - return chain.proceed( - request.newBuilder() - .body(toUpperCase) - .build() - ); - } - -### [DRAFT] Replace the HTTP response content with a new body and discarding the existing one +The mapping function decodes the original byte buffer `buf` to an UTF-8 encoded +Java `String`, and copies its upper case representation into a newly created byte buffer. +The old byte buffer `buf` is discarded. -In the following example the plugin replaces the entire HTTP response content with a custom response content. +Because `buf` is a Netty reference counted `ByteBuf`, we must take care to decrement its +reference count by calling `buf.release()`. -In this scenario the plugin creates a new HttpResponse based on the existing one. However changing entirely the response -content can be a source of memory leaks if the current content is not programmatically released. This is because the -Observable content resides in the direct memory whereas the rest of the plugin will be in the heap. If the content will -not be manually released it will remain in the direct memory and eventually it will cause Styx to fail with a: +```java +public class MyPlugin extends Plugin { + @Override + public StyxObservable intercept(HttpRequest request, Chain chain) { + StyxObservable toUpperCase = request.body() + .map(buf -> { + buf.release(); + return copiedBuffer(buf.toString(UTF_8).toUpperCase(), UTF_8); + }); + + return chain.proceed( + request.newBuilder() + .body(toUpperCase) + .build() + ); + } +} +``` -java.lang.OutOfMemoryError: Direct buffer memory +### Asynchronously transforming streaming request content -In order to avoid memory leaks it is necessary to release the existing ByteBuf by using the doOnNext() operator which -creates a new Observable with a side-effect behavior. In this specific case the side-effect is a call to a function which -releases the reference count. Make sure to subscribe to this new Observable otherwise the function will not be executed. - - import com.hotels.styx.api.HttpRequest; - import com.hotels.styx.api.HttpResponse; - import com.hotels.styx.api.plugins.spi.Plugin; - import io.netty.util.ReferenceCountUtil; - import rx.Observable; - - ... - - @Override - public Observable intercept(HttpRequest request, Chain chain) { - Observable responseObservable = chain.proceed(request); - return responseObservable.map(response -> { - response.body().content().doOnNext(byteBuf -> ReferenceCountUtil.release(byteBuf)).subscribe(); - return response.newBuilder().body("Custom HTTP response content").build(); - }); - } +Asynchronous content stream transformation is very similar to the synchronous transformation. +The only difference is that asynchronous transformation must be composed with a `flatMap` +instead of `map`. Otherwise the same discussion apply. As always it is important to take care +of the reference counting. + +```java +public class AsyncRequestContentTransformation implements Plugin { + + @Override + public StyxObservable intercept(HttpRequest request, Chain chain) { + StyxObservable contentMapping = request.body() + .flatMap(buf -> { + String content = buf.toString(UTF_8); + buf.release(); + return sendToRemoteServer(content) + .map(value -> copiedBuffer(value, UTF_8)); + }); + + return chain.proceed( + request.newBuilder() + .body(contentMapping) + .build() + ); + } + + StyxObservable sendToRemoteServer(String buf) { + return StyxObservable.of("modified 3rd party content"); + } +} +``` + +### Synchronously transforming streaming response content + +```java +public class AsyncResponseContentStreamTransformation implements Plugin { + @Override + public StyxObservable intercept(HttpRequest request, Chain chain) { + return chain.proceed(request) + .map(response -> { + StyxObservable contentMapping = response.body() + .map(buf -> { + buf.release(); + return copiedBuffer(buf.toString(UTF_8).toUpperCase(), UTF_8); + }); + return response.newBuilder() + .body(contentMapping) + .build(); + }); + } +} +``` + +### Asynchronously transforming streaming response content + +```java +public class AsyncResponseContentStreamTransformation implements Plugin { + @Override + public StyxObservable intercept(HttpRequest request, Chain chain) { + return chain.proceed(request) + .map(response -> { + StyxObservable contentMapping = response.body() + .flatMap(buf -> { + String content = buf.toString(UTF_8); + buf.release(); + return sendToRemoteServer(content) + .map(value -> copiedBuffer(value, UTF_8)); + }); + return response.newBuilder() + .body(contentMapping) + .build(); + }); + } + + StyxObservable sendToRemoteServer(String buf) { + return StyxObservable.of("modified 3rd party content"); + } +} +``` \ No newline at end of file From 3d233687f5e3ec5138e44c715658611d0f48e15b Mon Sep 17 00:00:00 2001 From: Mikko Karjalainen Date: Fri, 25 May 2018 15:35:15 +0100 Subject: [PATCH 3/8] Add some words of warning to the plugin-scenarios.md. --- docs/developer-guide/plugins-scenarios.md | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) diff --git a/docs/developer-guide/plugins-scenarios.md b/docs/developer-guide/plugins-scenarios.md index f8bf7a62d3..ac5a0e9a07 100644 --- a/docs/developer-guide/plugins-scenarios.md +++ b/docs/developer-guide/plugins-scenarios.md @@ -27,7 +27,6 @@ public class SyncRequestPlugin implements Plugin { ); } } - ``` ### Synchronously transforming response @@ -52,7 +51,6 @@ public class SyncResponsePlugin implements Plugin { ); } } - ``` ### Asynchronously transform request object @@ -182,6 +180,7 @@ messages. The full HTTP message body is then available at interceptor's disposal aggregation is always an asynchronous operation. This is because Styx must wait until all content has been received. + ### Aggregating Content into Full Messages ```java @@ -210,6 +209,23 @@ and possibly running out of memory. Styx only accumulates up to `maxContentBytes `ContentOverflowException`, +### Transformations on Streaming HTTP response content + +Streaming HTTP body content can be transformed both synchronously and asynchronously. +However there are some pitfalls you need to know: + + - Reference counting. Styx exposes the content stream as an observable of reference counted + Netty `ByteBuf` objects. Ensure the reference counts are correctly decremented when + buffers are transformed. + + - Continuity (or discontinuity) of Styx content observable. Each content tranformation with + `map` or `flatMap` is a composition of some source observable. So is each content transformation + linked to some source observable, an ultimate source being the Styx server core. + It is the consumer's responsibility to ensure this link never gets broken. That is, you + are not allowed to just substitute the content observable with another one, unless it composes + to the previous content source. + + ### Synchronously transforming streaming request content In this rather contrived example we will transform HTTP response content to From 30b0a435f1ab2475795784749aa9df97cf62eff5 Mon Sep 17 00:00:00 2001 From: Mikko Karjalainen Date: Fri, 25 May 2018 16:30:23 +0100 Subject: [PATCH 4/8] Add instrumentation. --- .../netty/connectionpool/NettyToStyxResponsePropagator.java | 1 + .../src/test/scala/com/hotels/styx/proxy/ProxySpec.scala | 4 ++++ 2 files changed, 5 insertions(+) diff --git a/components/client/src/main/java/com/hotels/styx/client/netty/connectionpool/NettyToStyxResponsePropagator.java b/components/client/src/main/java/com/hotels/styx/client/netty/connectionpool/NettyToStyxResponsePropagator.java index be456cd35b..18e86c44b8 100644 --- a/components/client/src/main/java/com/hotels/styx/client/netty/connectionpool/NettyToStyxResponsePropagator.java +++ b/components/client/src/main/java/com/hotels/styx/client/netty/connectionpool/NettyToStyxResponsePropagator.java @@ -139,6 +139,7 @@ protected void channelRead0(ChannelHandlerContext ctx, Object msg) throws Except if (msg instanceof io.netty.handler.codec.http.HttpResponse) { io.netty.handler.codec.http.HttpResponse nettyResponse = (io.netty.handler.codec.http.HttpResponse) msg; if (nettyResponse.getDecoderResult().isFailure()) { + LOGGER.warn("Http Response Failure. ", nettyResponse.decoderResult()); emitResponseError(new BadHttpResponseException(origin, nettyResponse.getDecoderResult().cause())); return; } diff --git a/system-tests/e2e-suite/src/test/scala/com/hotels/styx/proxy/ProxySpec.scala b/system-tests/e2e-suite/src/test/scala/com/hotels/styx/proxy/ProxySpec.scala index 0047e32fe6..575d714cca 100644 --- a/system-tests/e2e-suite/src/test/scala/com/hotels/styx/proxy/ProxySpec.scala +++ b/system-tests/e2e-suite/src/test/scala/com/hotels/styx/proxy/ProxySpec.scala @@ -121,8 +121,12 @@ class ProxySpec extends FunSpec .addHeader(HOST, styxServer.proxyHost) .build() + println("Sending request, waiting for response ...") + val resp = decodedRequest(req) + println("Got response ...") + recordingBackend.verify(headRequestedFor(urlPathEqualTo("/bodiless"))) assert(resp.status() == OK) From 867add0f004d1f6f628d68886a2f3288573469d8 Mon Sep 17 00:00:00 2001 From: Mikko Karjalainen Date: Tue, 29 May 2018 09:03:58 +0100 Subject: [PATCH 5/8] Add instrumentation to ProxySpec. Remove unused imports. --- .../NettyConnectionFactory.java | 5 +++ .../NettyToStyxResponsePropagator.java | 2 ++ .../netty/connectors/HttpResponseWriter.java | 4 +-- .../conf/logback/logback-debug-instrument.xml | 16 +++++++++ .../com/hotels/styx/proxy/ProxySpec.scala | 33 +++++++++++++++---- 5 files changed, 52 insertions(+), 8 deletions(-) create mode 100644 system-tests/e2e-suite/src/test/resources/conf/logback/logback-debug-instrument.xml diff --git a/components/client/src/main/java/com/hotels/styx/client/netty/connectionpool/NettyConnectionFactory.java b/components/client/src/main/java/com/hotels/styx/client/netty/connectionpool/NettyConnectionFactory.java index e50bd68948..2dafb814dd 100644 --- a/components/client/src/main/java/com/hotels/styx/client/netty/connectionpool/NettyConnectionFactory.java +++ b/components/client/src/main/java/com/hotels/styx/client/netty/connectionpool/NettyConnectionFactory.java @@ -34,6 +34,7 @@ import io.netty.channel.ChannelPipeline; import io.netty.handler.codec.http.HttpClientCodec; import io.netty.handler.codec.http.HttpContentDecompressor; +import io.netty.handler.logging.LoggingHandler; import io.netty.handler.ssl.SslContext; import rx.Observable; @@ -106,6 +107,8 @@ private class Initializer extends ChannelInitializer { protected void initChannel(Channel ch) { ChannelPipeline pipeline = ch.pipeline(); + pipeline.addLast("logging-handler-1", new LoggingHandler()); + if (sslContext != null) { pipeline.addLast("ssl", sslContext.newHandler(ch.alloc())); } @@ -114,6 +117,8 @@ protected void initChannel(Channel ch) { if (httpConfig.compress()) { pipeline.addLast("decompressor", new HttpContentDecompressor()); } + + pipeline.addLast("logging-handler-2", new LoggingHandler()); } } diff --git a/components/client/src/main/java/com/hotels/styx/client/netty/connectionpool/NettyToStyxResponsePropagator.java b/components/client/src/main/java/com/hotels/styx/client/netty/connectionpool/NettyToStyxResponsePropagator.java index 18e86c44b8..73980c2fe5 100644 --- a/components/client/src/main/java/com/hotels/styx/client/netty/connectionpool/NettyToStyxResponsePropagator.java +++ b/components/client/src/main/java/com/hotels/styx/client/netty/connectionpool/NettyToStyxResponsePropagator.java @@ -136,6 +136,8 @@ private void scheduleResourcesTearDown(ChannelHandlerContext ctx) { protected void channelRead0(ChannelHandlerContext ctx, Object msg) throws Exception { ensureContentProducerIsCreated(ctx); + LOGGER.warn("channelRead0: {}", msg); + if (msg instanceof io.netty.handler.codec.http.HttpResponse) { io.netty.handler.codec.http.HttpResponse nettyResponse = (io.netty.handler.codec.http.HttpResponse) msg; if (nettyResponse.getDecoderResult().isFailure()) { diff --git a/components/server/src/main/java/com/hotels/styx/server/netty/connectors/HttpResponseWriter.java b/components/server/src/main/java/com/hotels/styx/server/netty/connectors/HttpResponseWriter.java index a34e2b12d5..92562e995a 100644 --- a/components/server/src/main/java/com/hotels/styx/server/netty/connectors/HttpResponseWriter.java +++ b/components/server/src/main/java/com/hotels/styx/server/netty/connectors/HttpResponseWriter.java @@ -21,6 +21,7 @@ import io.netty.channel.ChannelFutureListener; import io.netty.channel.ChannelHandlerContext; import io.netty.handler.codec.http.DefaultHttpContent; +import io.netty.handler.codec.http.HttpUtil; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import rx.Subscriber; @@ -32,7 +33,6 @@ import java.util.concurrent.atomic.AtomicLong; import static com.hotels.styx.api.StyxInternalObservables.toRxObservable; -import static io.netty.handler.codec.http.HttpHeaders.setTransferEncodingChunked; import static io.netty.handler.codec.http.LastHttpContent.EMPTY_LAST_CONTENT; import static java.util.Objects.requireNonNull; @@ -154,7 +154,7 @@ private void completeIfAllSent(CompletableFuture future) { private ChannelFuture writeHeaders(HttpResponse response) { io.netty.handler.codec.http.HttpResponse nettyResponse = responseTranslator.toNettyResponse(response); if (!(response.contentLength().isPresent() || response.chunked())) { - setTransferEncodingChunked(nettyResponse); + HttpUtil.setTransferEncodingChunked(nettyResponse, true); } return nettyWriteAndFlush(nettyResponse); diff --git a/system-tests/e2e-suite/src/test/resources/conf/logback/logback-debug-instrument.xml b/system-tests/e2e-suite/src/test/resources/conf/logback/logback-debug-instrument.xml new file mode 100644 index 0000000000..c73326ab8d --- /dev/null +++ b/system-tests/e2e-suite/src/test/resources/conf/logback/logback-debug-instrument.xml @@ -0,0 +1,16 @@ + + + + + %d{HH:mm:ss.SSS} ProxySpecLog [%thread] %-5level %logger{36} - %msg%n + + + + + + + + + + + diff --git a/system-tests/e2e-suite/src/test/scala/com/hotels/styx/proxy/ProxySpec.scala b/system-tests/e2e-suite/src/test/scala/com/hotels/styx/proxy/ProxySpec.scala index 575d714cca..4f1c0c68af 100644 --- a/system-tests/e2e-suite/src/test/scala/com/hotels/styx/proxy/ProxySpec.scala +++ b/system-tests/e2e-suite/src/test/scala/com/hotels/styx/proxy/ProxySpec.scala @@ -30,21 +30,26 @@ import com.hotels.styx.api.support.HostAndPorts._ import com.hotels.styx.api.{FullHttpResponse, HttpRequest} import com.hotels.styx.client.StyxHeaderConfig.STYX_INFO_DEFAULT import com.hotels.styx.support.backends.FakeHttpServer -import com.hotels.styx.support.configuration.{HttpBackend, Origin, Origins} +import com.hotels.styx.support.configuration +import com.hotels.styx.support.configuration._ import com.hotels.styx.support.matchers.IsOptional.{isValue, matches} import com.hotels.styx.support.matchers.RegExMatcher.matchesRegex import com.hotels.styx.support.server.UrlMatchingStrategies.urlStartingWith import com.hotels.styx.{DefaultStyxConfiguration, MockServer, StyxProxySpec} import org.hamcrest.MatcherAssert.assertThat import org.scalatest.{BeforeAndAfter, FunSpec} +import com.hotels.styx.support.ResourcePaths.fixturesHome class ProxySpec extends FunSpec with StyxProxySpec - with DefaultStyxConfiguration + // with DefaultStyxConfiguration with BeforeAndAfter { - val mockServer = new MockServer(0) - - val recordingBackend = FakeHttpServer.HttpStartupConfig().start() + val styxConfig: StyxBaseConfig = configuration.StyxConfig( + proxyConfig = ProxyConfig(Connectors(HttpConnectorConfig(port = 8080), null)), + logbackXmlLocation = fixturesHome(this.getClass, "/conf/logback/logback-debug-instrument.xml") + ) + val recordingBackend = FakeHttpServer.HttpStartupConfig(port = 10001).start() + val mockServer = new MockServer(10002) override protected def beforeAll(): Unit = { super.beforeAll() @@ -98,7 +103,6 @@ class ProxySpec extends FunSpec mockServer.stub("/http10", responseSupplier(() => response().build())) - val req = new HttpRequest.Builder(GET, "/http10") .addHeader(HOST, styxServer.proxyHost) .build() @@ -116,6 +120,15 @@ class ProxySpec extends FunSpec .stub(urlPathEqualTo("/bodiless"), aResponse.withStatus(200)) .stub(head(urlPathEqualTo("/bodiless")), aResponse.withStatus(200)) + /* + * As a HEAD it should not have had a body. + * + * However Styx keeps the `transfer-encoding: chunked` header. + * + * For this reason, Netty client codec (HttpObectDecoder) removes the `transfer-encoding: chunked` + * header from the message. + * + */ it("should respond to HEAD with bodiless response") { val req = new HttpRequest.Builder(HEAD, "/bodiless") .addHeader(HOST, styxServer.proxyHost) @@ -134,6 +147,14 @@ class ProxySpec extends FunSpec } it("should remove body from the 204 No Content responses") { + /* + * Doesn't work as intended. This is because WireMock doesn't include + * the HTTP header in the first place. + * + * Therefore there is no content in the messages that styx could + * remove. + * + */ recordingBackend .stub(urlPathEqualTo("/204"), aResponse.withStatus(204).withBody("I should not be here")) From d23b7cf1a0c2307da009914d7b63cdef3a54d437 Mon Sep 17 00:00:00 2001 From: Mikko Karjalainen Date: Wed, 30 May 2018 17:17:18 +0100 Subject: [PATCH 6/8] Fix typos and remove unnecessary code changes. --- .../NettyConnectionFactory.java | 4 -- .../NettyToStyxResponsePropagator.java | 2 - docs/developer-guide/plugins-scenarios.md | 58 ++++++++++--------- 3 files changed, 30 insertions(+), 34 deletions(-) diff --git a/components/client/src/main/java/com/hotels/styx/client/netty/connectionpool/NettyConnectionFactory.java b/components/client/src/main/java/com/hotels/styx/client/netty/connectionpool/NettyConnectionFactory.java index 2dafb814dd..794f4f56a9 100644 --- a/components/client/src/main/java/com/hotels/styx/client/netty/connectionpool/NettyConnectionFactory.java +++ b/components/client/src/main/java/com/hotels/styx/client/netty/connectionpool/NettyConnectionFactory.java @@ -107,8 +107,6 @@ private class Initializer extends ChannelInitializer { protected void initChannel(Channel ch) { ChannelPipeline pipeline = ch.pipeline(); - pipeline.addLast("logging-handler-1", new LoggingHandler()); - if (sslContext != null) { pipeline.addLast("ssl", sslContext.newHandler(ch.alloc())); } @@ -117,8 +115,6 @@ protected void initChannel(Channel ch) { if (httpConfig.compress()) { pipeline.addLast("decompressor", new HttpContentDecompressor()); } - - pipeline.addLast("logging-handler-2", new LoggingHandler()); } } diff --git a/components/client/src/main/java/com/hotels/styx/client/netty/connectionpool/NettyToStyxResponsePropagator.java b/components/client/src/main/java/com/hotels/styx/client/netty/connectionpool/NettyToStyxResponsePropagator.java index 73980c2fe5..18e86c44b8 100644 --- a/components/client/src/main/java/com/hotels/styx/client/netty/connectionpool/NettyToStyxResponsePropagator.java +++ b/components/client/src/main/java/com/hotels/styx/client/netty/connectionpool/NettyToStyxResponsePropagator.java @@ -136,8 +136,6 @@ private void scheduleResourcesTearDown(ChannelHandlerContext ctx) { protected void channelRead0(ChannelHandlerContext ctx, Object msg) throws Exception { ensureContentProducerIsCreated(ctx); - LOGGER.warn("channelRead0: {}", msg); - if (msg instanceof io.netty.handler.codec.http.HttpResponse) { io.netty.handler.codec.http.HttpResponse nettyResponse = (io.netty.handler.codec.http.HttpResponse) msg; if (nettyResponse.getDecoderResult().isFailure()) { diff --git a/docs/developer-guide/plugins-scenarios.md b/docs/developer-guide/plugins-scenarios.md index ac5a0e9a07..0da536e319 100644 --- a/docs/developer-guide/plugins-scenarios.md +++ b/docs/developer-guide/plugins-scenarios.md @@ -4,9 +4,12 @@ ### Synchronously transforming requests -Transforming request object synchronously is trivial. Just call `request.newBuilder()` to +Transforming a request object synchronously is trivial. By "synchronously" we mean in the +same thread, in non-blocking fashion. + +Just call `request.newBuilder()` to create a new `HttpRequest.Builder` object. It has already been initialised with the copy of -the original `request`. Modify the builder as desired, consturct a new version, and pass +the original `request`. Modify the builder as desired, construct a new version, and pass it to the `chain.proceed()`, as the example demonstrates: ```java @@ -32,7 +35,7 @@ public class SyncRequestPlugin implements Plugin { ### Synchronously transforming response This example demonstrates how to synchronously transform a HTTP response. We will -use a `StyxObservable` `map` method to add an "X-Foo" header to the response. +use a `StyxObservable.map` method to add an "X-Foo" header to the response. ```java import com.hotels.styx.api.HttpInterceptor; @@ -55,8 +58,8 @@ public class SyncResponsePlugin implements Plugin { ### Asynchronously transform request object -Sometimes it is necessary to transform the request asynchronously. For example, may need to -look up external key value stores to parametrise the request transformation. The example below +Sometimes it is necessary to transform the request asynchronously. For example, you may need to +look up external key value stores to parameterise the request transformation. The example below shows how to modify a request URL path based on an asynchronous lookup to an external database. ```java @@ -74,46 +77,43 @@ public class AsyncRequestInterceptor implements Plugin { @Override public StyxObservable intercept(HttpRequest request, HttpInterceptor.Chain chain) { - return StyxObservable.of(request) // (1) - .flatMap(na -> asyncUrlReplacement(request.url())) // (2) - .map(newUrl -> request.newBuilder() // (5) + return asyncUrlReplacement(request.url()) // (1) + .map(newUrl -> request.newBuilder() // (4) .url(newUrl) .build()) - .flatMap(chain::proceed); // (6) + .flatMap(chain::proceed); // (5) } private static StyxObservable asyncUrlReplacement(Url url) { - return StyxObservable.from(pathReplacementService(url.path())) // (4) + return StyxObservable.from(pathReplacementService(url.path())) // (3) .map(newPath -> new Url.Builder(url) .path(newPath) .build()); } private static CompletableFuture pathReplacementService(String url) { - return CompletableFuture.completedFuture("/replacement/path"); // (3) + // Pretend to make a call here: + return CompletableFuture.completedFuture("/replacement/path"); // (2) } } ``` -Step 1. For asynchronous transformation we'll start by constructing a response observable. -We can construct it with any initial value, but in this example we'll just use the `request`. -But it doesn't have to be a `request` as it is available from the closure. - -Step 2. We will call the `asyncUrlReplacement`, and bind it to the response observable using -`flatMap` operator. The `asyncUrlReplacement` wraps a call to the remote service and converts -the outcome into a `StyxObservable`. +Step 1. We call the `asyncUrlReplacement`, which returns a `StyxObservable`. +The `asyncUrlReplacement` wraps a call to the remote service and converts +the outcome into a `StyxObservable`, which is the basis for our response observable. -Step 3. A call to `pathReplacementService` makes a non-blocking call to the remote key/value store. +Step 2. A call to `pathReplacementService` makes a non-blocking call to the remote key/value store. Well, at least we pretend to call the key value store, but in this example we'll just return a completed future of a constant value. -Step 4. `CompletableFuture` needs to be converted to `StyxObservable` so that the operation can -be bound to the response observable created previously in step 1. +Step 3. `CompletableFuture` is converted to `StyxObservable`, so that other asynchronous +operations like `chain.proceed` can be bound to it later on. -Step 5. The eventual outcome from the `asyncUrlReplacement` yields a new, modified URL instance. -We will map this to a new `HttpRequest` instance replaced URL. +Step 4. The eventual outcome from the `asyncUrlReplacement` yields a new, modified URL instance. +We'll transform the `request` by substituting the URL path with the new one. +This is a quick synchronous operation so we'll do it in a `map` operator. -Step 6. Finally, we will bind the outcome of `chain.proceed` into the response observable. +Step 5. Finally, we will bind the outcome from `chain.proceed` into the response observable. Remember that `chain.proceed` returns an `Observable` it is therefore interface compatible and can be `flatMap`'d to the response observable. The resulting response observable chain is returned from the `intercept`. @@ -144,7 +144,7 @@ public class AsyncResponseInterceptor implements Plugin { return chain.proceed(request) // (1) .flatMap(response -> // (3) StyxObservable - .from(callTo3rdParty(response.header(X_MY_HEADER).orElse("default"))) // (1) + .from(thirdPartyHeaderService(response.header(X_MY_HEADER).orElse("default"))) // (1) .map(value -> // (4) response.newBuilder() .header(X_MY_HEADER, value) @@ -152,7 +152,8 @@ public class AsyncResponseInterceptor implements Plugin { ); } - private static CompletableFuture callTo3rdParty(String myHeader) { + private static CompletableFuture thirdPartyHeaderService(String myHeader) { + // Pretend to make a call here: return CompletableFuture.completedFuture("value"); } } @@ -177,8 +178,9 @@ can look into, and modify the content as it streams through. Alternatively, streaming messages can be aggregated into a `FullHttpRequest` or `FullHttpResponse` messages. The full HTTP message body is then available at interceptor's disposal. Note that content -aggregation is always an asynchronous operation. This is because Styx must wait until all content -has been received. +aggregation is always an asynchronous operation. This is because the streaming HTTP message is +exposing the content, in byte buffers, as it arrives from the network, and Styx must wait until +all content has been received. ### Aggregating Content into Full Messages From 0f6e1af2e2b02f8de381ccd2e78b44eb0c9a96ff Mon Sep 17 00:00:00 2001 From: Mikko Karjalainen Date: Wed, 30 May 2018 17:22:00 +0100 Subject: [PATCH 7/8] Back off unnecessary instrumentation. --- .../NettyConnectionFactory.java | 1 - .../NettyToStyxResponsePropagator.java | 1 - .../netty/connectors/HttpResponseWriter.java | 4 +- .../conf/logback/logback-debug-instrument.xml | 16 -------- .../com/hotels/styx/proxy/ProxySpec.scala | 37 +++---------------- 5 files changed, 8 insertions(+), 51 deletions(-) delete mode 100644 system-tests/e2e-suite/src/test/resources/conf/logback/logback-debug-instrument.xml diff --git a/components/client/src/main/java/com/hotels/styx/client/netty/connectionpool/NettyConnectionFactory.java b/components/client/src/main/java/com/hotels/styx/client/netty/connectionpool/NettyConnectionFactory.java index 794f4f56a9..e50bd68948 100644 --- a/components/client/src/main/java/com/hotels/styx/client/netty/connectionpool/NettyConnectionFactory.java +++ b/components/client/src/main/java/com/hotels/styx/client/netty/connectionpool/NettyConnectionFactory.java @@ -34,7 +34,6 @@ import io.netty.channel.ChannelPipeline; import io.netty.handler.codec.http.HttpClientCodec; import io.netty.handler.codec.http.HttpContentDecompressor; -import io.netty.handler.logging.LoggingHandler; import io.netty.handler.ssl.SslContext; import rx.Observable; diff --git a/components/client/src/main/java/com/hotels/styx/client/netty/connectionpool/NettyToStyxResponsePropagator.java b/components/client/src/main/java/com/hotels/styx/client/netty/connectionpool/NettyToStyxResponsePropagator.java index 18e86c44b8..be456cd35b 100644 --- a/components/client/src/main/java/com/hotels/styx/client/netty/connectionpool/NettyToStyxResponsePropagator.java +++ b/components/client/src/main/java/com/hotels/styx/client/netty/connectionpool/NettyToStyxResponsePropagator.java @@ -139,7 +139,6 @@ protected void channelRead0(ChannelHandlerContext ctx, Object msg) throws Except if (msg instanceof io.netty.handler.codec.http.HttpResponse) { io.netty.handler.codec.http.HttpResponse nettyResponse = (io.netty.handler.codec.http.HttpResponse) msg; if (nettyResponse.getDecoderResult().isFailure()) { - LOGGER.warn("Http Response Failure. ", nettyResponse.decoderResult()); emitResponseError(new BadHttpResponseException(origin, nettyResponse.getDecoderResult().cause())); return; } diff --git a/components/server/src/main/java/com/hotels/styx/server/netty/connectors/HttpResponseWriter.java b/components/server/src/main/java/com/hotels/styx/server/netty/connectors/HttpResponseWriter.java index 92562e995a..a34e2b12d5 100644 --- a/components/server/src/main/java/com/hotels/styx/server/netty/connectors/HttpResponseWriter.java +++ b/components/server/src/main/java/com/hotels/styx/server/netty/connectors/HttpResponseWriter.java @@ -21,7 +21,6 @@ import io.netty.channel.ChannelFutureListener; import io.netty.channel.ChannelHandlerContext; import io.netty.handler.codec.http.DefaultHttpContent; -import io.netty.handler.codec.http.HttpUtil; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import rx.Subscriber; @@ -33,6 +32,7 @@ import java.util.concurrent.atomic.AtomicLong; import static com.hotels.styx.api.StyxInternalObservables.toRxObservable; +import static io.netty.handler.codec.http.HttpHeaders.setTransferEncodingChunked; import static io.netty.handler.codec.http.LastHttpContent.EMPTY_LAST_CONTENT; import static java.util.Objects.requireNonNull; @@ -154,7 +154,7 @@ private void completeIfAllSent(CompletableFuture future) { private ChannelFuture writeHeaders(HttpResponse response) { io.netty.handler.codec.http.HttpResponse nettyResponse = responseTranslator.toNettyResponse(response); if (!(response.contentLength().isPresent() || response.chunked())) { - HttpUtil.setTransferEncodingChunked(nettyResponse, true); + setTransferEncodingChunked(nettyResponse); } return nettyWriteAndFlush(nettyResponse); diff --git a/system-tests/e2e-suite/src/test/resources/conf/logback/logback-debug-instrument.xml b/system-tests/e2e-suite/src/test/resources/conf/logback/logback-debug-instrument.xml deleted file mode 100644 index c73326ab8d..0000000000 --- a/system-tests/e2e-suite/src/test/resources/conf/logback/logback-debug-instrument.xml +++ /dev/null @@ -1,16 +0,0 @@ - - - - - %d{HH:mm:ss.SSS} ProxySpecLog [%thread] %-5level %logger{36} - %msg%n - - - - - - - - - - - diff --git a/system-tests/e2e-suite/src/test/scala/com/hotels/styx/proxy/ProxySpec.scala b/system-tests/e2e-suite/src/test/scala/com/hotels/styx/proxy/ProxySpec.scala index 4f1c0c68af..0047e32fe6 100644 --- a/system-tests/e2e-suite/src/test/scala/com/hotels/styx/proxy/ProxySpec.scala +++ b/system-tests/e2e-suite/src/test/scala/com/hotels/styx/proxy/ProxySpec.scala @@ -30,26 +30,21 @@ import com.hotels.styx.api.support.HostAndPorts._ import com.hotels.styx.api.{FullHttpResponse, HttpRequest} import com.hotels.styx.client.StyxHeaderConfig.STYX_INFO_DEFAULT import com.hotels.styx.support.backends.FakeHttpServer -import com.hotels.styx.support.configuration -import com.hotels.styx.support.configuration._ +import com.hotels.styx.support.configuration.{HttpBackend, Origin, Origins} import com.hotels.styx.support.matchers.IsOptional.{isValue, matches} import com.hotels.styx.support.matchers.RegExMatcher.matchesRegex import com.hotels.styx.support.server.UrlMatchingStrategies.urlStartingWith import com.hotels.styx.{DefaultStyxConfiguration, MockServer, StyxProxySpec} import org.hamcrest.MatcherAssert.assertThat import org.scalatest.{BeforeAndAfter, FunSpec} -import com.hotels.styx.support.ResourcePaths.fixturesHome class ProxySpec extends FunSpec with StyxProxySpec - // with DefaultStyxConfiguration + with DefaultStyxConfiguration with BeforeAndAfter { - val styxConfig: StyxBaseConfig = configuration.StyxConfig( - proxyConfig = ProxyConfig(Connectors(HttpConnectorConfig(port = 8080), null)), - logbackXmlLocation = fixturesHome(this.getClass, "/conf/logback/logback-debug-instrument.xml") - ) - val recordingBackend = FakeHttpServer.HttpStartupConfig(port = 10001).start() - val mockServer = new MockServer(10002) + val mockServer = new MockServer(0) + + val recordingBackend = FakeHttpServer.HttpStartupConfig().start() override protected def beforeAll(): Unit = { super.beforeAll() @@ -103,6 +98,7 @@ class ProxySpec extends FunSpec mockServer.stub("/http10", responseSupplier(() => response().build())) + val req = new HttpRequest.Builder(GET, "/http10") .addHeader(HOST, styxServer.proxyHost) .build() @@ -120,26 +116,13 @@ class ProxySpec extends FunSpec .stub(urlPathEqualTo("/bodiless"), aResponse.withStatus(200)) .stub(head(urlPathEqualTo("/bodiless")), aResponse.withStatus(200)) - /* - * As a HEAD it should not have had a body. - * - * However Styx keeps the `transfer-encoding: chunked` header. - * - * For this reason, Netty client codec (HttpObectDecoder) removes the `transfer-encoding: chunked` - * header from the message. - * - */ it("should respond to HEAD with bodiless response") { val req = new HttpRequest.Builder(HEAD, "/bodiless") .addHeader(HOST, styxServer.proxyHost) .build() - println("Sending request, waiting for response ...") - val resp = decodedRequest(req) - println("Got response ...") - recordingBackend.verify(headRequestedFor(urlPathEqualTo("/bodiless"))) assert(resp.status() == OK) @@ -147,14 +130,6 @@ class ProxySpec extends FunSpec } it("should remove body from the 204 No Content responses") { - /* - * Doesn't work as intended. This is because WireMock doesn't include - * the HTTP header in the first place. - * - * Therefore there is no content in the messages that styx could - * remove. - * - */ recordingBackend .stub(urlPathEqualTo("/204"), aResponse.withStatus(204).withBody("I should not be here")) From 923a72de0a506959a034b2fcfdc8c73c1bc69283 Mon Sep 17 00:00:00 2001 From: Mikko Karjalainen Date: Wed, 30 May 2018 17:25:57 +0100 Subject: [PATCH 8/8] Paraphrase plugin-scenarios.md document. --- docs/developer-guide/plugins-scenarios.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/developer-guide/plugins-scenarios.md b/docs/developer-guide/plugins-scenarios.md index 0da536e319..da03227f0a 100644 --- a/docs/developer-guide/plugins-scenarios.md +++ b/docs/developer-guide/plugins-scenarios.md @@ -177,7 +177,7 @@ In this form, the interceptors can process the content in a streaming fashion. T can look into, and modify the content as it streams through. Alternatively, streaming messages can be aggregated into a `FullHttpRequest` or `FullHttpResponse` -messages. The full HTTP message body is then available at interceptor's disposal. Note that content +messages. The full HTTP message body is then available for the interceptor to use. Note that content aggregation is always an asynchronous operation. This is because the streaming HTTP message is exposing the content, in byte buffers, as it arrives from the network, and Styx must wait until all content has been received.