Merge pull request #33 from TomPallister/develop

merge newest code

build-and-release-unstable.ps1

@@ -1 +1,2 @@
-./build.ps1 -target BuildAndReleaseUnstable
+./build.ps1 -target BuildAndReleaseUnstable
+exit $LASTEXITCODE

build-and-run-tests.ps1

@@ -1 +1,2 @@
-./build.ps1 -target RunTests
+./build.ps1 -target RunTests
+exit $LASTEXITCODE

build.cake

@@ -102,17 +102,10 @@ Task("Version")
 		}
 	});
 
-Task("Restore")
+Task("Compile")
 	.IsDependentOn("Clean")
 	.IsDependentOn("Version")
 	.Does(() =>
-	{	
-		DotNetCoreRestore(slnFile);
-	});
-
-Task("Compile")
-	.IsDependentOn("Restore")
-	.Does(() =>
 	{	
 		var settings = new DotNetCoreBuildSettings
 		{
@@ -140,7 +133,7 @@ Task("RunUnitTests")
 				new OpenCoverSettings()
 				{
 					Register="user",
-					ArgumentCustomization=args=>args.Append(@"-oldstyle -returntargetcode")
+					ArgumentCustomization=args=>args.Append(@"-oldstyle -returntargetcode -excludebyattribute:*.ExcludeFromCoverage*")
 				}
 				.WithFilter("+[Ocelot*]*")
 				.WithFilter("-[xunit*]*")
@@ -199,6 +192,9 @@ Task("RunAcceptanceTests")
 		var settings = new DotNetCoreTestSettings
 		{
 			Configuration = compileConfig,
+			ArgumentCustomization = args => args
+				.Append("--no-restore")
+				.Append("--no-build")
 		};
 
 		EnsureDirectoryExists(artifactsForAcceptanceTestsDir);
@@ -212,6 +208,9 @@ Task("RunIntegrationTests")
 		var settings = new DotNetCoreTestSettings
 		{
 			Configuration = compileConfig,
+			ArgumentCustomization = args => args
+				.Append("--no-restore")
+				.Append("--no-build")
 		};
 
 		EnsureDirectoryExists(artifactsForIntegrationTestsDir);
@@ -474,4 +473,4 @@ private bool ShouldPublishToUnstableFeed(string filter, string branchName)
 		Information("Branch " + branchName + " will not be published to the unstable feed");
 	}
 	return publish;	
-}
+}

docs/features/administration.rst

@@ -6,32 +6,23 @@ using bearer tokens that you request from Ocelot iteself. This is provided by th
 `Identity Server <https://github.com/IdentityServer/IdentityServer4>`_ project that I have been using for a few years now. Check them out.
 
 In order to enable the administration section you need to do a few things. First of all add this to your
-initial configuration.json. The value can be anything you want and it is obviously reccomended don't use
+initial Startup.cs. 
+
+The path can be anything you want and it is obviously reccomended don't use
 a url you would like to route through with Ocelot as this will not work. The administration uses the
 MapWhen functionality of asp.net core and all requests to {root}/administration will be sent there not 
 to the Ocelot middleware.
 
-.. code-block:: json
-
-    "GlobalConfiguration": {
-        "AdministrationPath": "/administration"
-    }
-
-This will get the admin area set up but not the authentication. 
-Please note that this is a very basic approach to 
-this problem and if needed we can obviously improve on this!
-
-You need to set 3 environmental variables. 
-
-    ``OCELOT_USERNAME``
-
-        This need to be the admin username you want to use with Ocelot.
-    ``OCELOT_HASH``
-    ``OCELOT_SALT``
-        The hash and salt of the password you want to use given hashing algorythm. When requesting bearer tokens for use with the administration api you will need to supply username and password. In order to create a hash and salt of your password please check out HashCreationTests.should_create_hash_and_salt() this technique is based on [this](https://docs.microsoft.com/en-us/aspnet/core/security/data-protection/consumer-apis/password-hashing)
-        using SHA256 rather than SHA1.
+The secret is the client secret that Ocelot's internal IdentityServer will use to authenticate requests to the administration API. This can be whatever you want it to be!
 
+.. code-block:: csharp
 
+    public virtual void ConfigureServices(IServiceCollection services)
+    {
+        services
+            .AddOcelot(Configuration)
+            .AddAdministration("/administration", "secret");
+    }
 
 Now if you went with the configuration options above and want to access the API you can use the postman scripts
 called ocelot.postman_collection.json in the solution to change the Ocelot configuration. Obviously these 
@@ -40,7 +31,6 @@ will need to be changed if you are running Ocelot on a different url to http://l
 The scripts show you how to request a bearer token from ocelot and then use it to GET the existing configuration and POST 
 a configuration.
 
-
 Administration running multiple Ocelot's
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 If you are running multiple Ocelot's in a cluster then you need to use a certificate to sign the bearer tokens used to access the administration API.
@@ -59,21 +49,17 @@ Administration API
 
 **POST {adminPath}/connect/token**
 
-This gets a token for use with the admin area using the username and password we talk about setting above. Under the hood this calls into an IdentityServer hosted within Ocelot.
+This gets a token for use with the admin area using the client credentials we talk about setting above. Under the hood this calls into an IdentityServer hosted within Ocelot.
 
 The body of the request is form-data as follows
 
 ``client_id`` set as admin
 
-``client_secret`` set as secret
+``client_secret`` set as whatever you used when setting up the administration services.
 
 ``scope`` set as admin
 
-``username`` set as whatever you used
-
-``password`` set aswhatever you used
-
-``grant_type`` set as password
+``grant_type`` set as client_credentials
 
 **GET {adminPath}/configuration**
 

docs/features/logging.rst

@@ -3,11 +3,11 @@ Logging
 
 Ocelot uses the standard logging interfaces ILoggerFactory / ILogger<T> at the moment. 
 This is encapsulated in  IOcelotLogger / IOcelotLoggerFactory with an implementation 
-for the standard asp.net core logging stuff at the moment. 
+for the standard asp.net core logging stuff at the moment. This is because Ocelot add's some extra info to the logs such as request id if it is configured.
 
-There are a bunch of debugging logs in the ocelot middlewares however I think the 
-system probably needs more logging in the code it calls into. Other than the debugging
-there is a global error handler that should catch any errors thrown and log them as errors.
+There is a global error handler that should catch any exceptions thrown and log them as errors.
+
+Finally if logging is set to trace level Ocelot will log starting, finishing and any middlewares that throw an exception which can be quite useful.
 
 The reason for not just using bog standard framework logging is that I could not 
 work out how to override the request id that get's logged when setting IncludeScopes 

docs/features/raft.rst

@@ -0,0 +1,45 @@
+Raft (EXPERIMENTAL DO NOT USE IN PRODUCTION)
+============================================
+
+Ocelot has recenely integrated `Rafty <https://github.com/TomPallister/Rafty>`_ which is an implementation of Raft that I have also been working on over the last year. This project is very experimental so please do not use this feature of Ocelot in production until I think it's OK.
+
+Raft is a distributed concensus algorythm that allows a cluster of servers (Ocelots) to maintain local state without having a centralised database for storing state (e.g. SQL Server). 
+
+In order to enable Rafty in Ocelot you must make the following changes to your Startup.cs.
+
+.. code-block:: csharp
+
+    public virtual void ConfigureServices(IServiceCollection services)
+    {
+         services
+            .AddOcelot(Configuration)
+            .AddAdministration("/administration", "secret")
+            .AddRafty();
+    }
+
+In addition to this you must add a file called peers.json to your main project and it will look as follows
+
+.. code-block:: json
+
+    {
+        "Peers": [{
+                "HostAndPort": "http://localhost:5000"
+            },
+            {
+                "HostAndPort": "http://localhost:5002"
+            },
+            {
+                "HostAndPort": "http://localhost:5003"
+            },
+            {
+                "HostAndPort": "http://localhost:5004"
+            },
+            {
+                "HostAndPort": "http://localhost:5001"
+            }
+        ]
+    }
+
+Each instance of Ocelot must have it's address in the array so that they can communicate using Rafty.
+
+Once you have made these configuration changes you must deploy and start each instance of Ocelot using the addresses in the peers.json file. The servers should then start communicating with each other! You can test if everything is working by posting a configuration update and checking it has replicated to all servers by getting there configuration.

docs/features/requestid.rst

@@ -4,25 +4,57 @@ Request Id / Correlation Id
 Ocelot supports a client sending a request id in the form of a header. If set Ocelot will
 use the requestid for logging as soon as it becomes available in the middleware pipeline. 
 Ocelot will also forward the request id with the specified header to the downstream service.
-I'm not sure if have this spot on yet in terms of the pipeline order becasue there are a few logs
-that don't get the users request id at the moment and ocelot just logs not set for request id
-which sucks. You can still get the framework request id in the logs if you set 
-IncludeScopes true in your logging config. This can then be used to match up later logs that do
-have an OcelotRequestId.
 
-In order to use the requestid feature in your ReRoute configuration add this setting
+You can still get the asp.net core request id in the logs if you set 
+IncludeScopes true in your logging config.
+
+In order to use the reques tid feature you have two options.
+
+*Global*
+
+In your configuration.json set the following in the GlobalConfiguration section. This will be used for all requests into Ocelot.
 
 .. code-block:: json
 
+   "GlobalConfiguration": {
     "RequestIdKey": "OcRequestId"
+  }
+
+I reccomend using the GlobalConfiguration unless you really need it to be ReRoute specific.
 
-In this example OcRequestId is the request header that contains the clients request id.
+*ReRoute*
 
-There is also a setting in the GlobalConfiguration section which will override whatever has been
-set at ReRoute level for the request id. The setting is as fllows.
+If you want to override this for a specific ReRoute add the following to configuration.json for the specific ReRoute.
 
 .. code-block:: json
 
     "RequestIdKey": "OcRequestId"
 
-It behaves in exactly the same way as the ReRoute level RequestIdKey settings.
+Once Ocelot has identified the incoming requests matching ReRoute object it will set the request id based on the ReRoute configuration.
+
+This can lead to a small gotcha. If you set a GlobalConfiguration it is possible to get one request id until the ReRoute is identified and then another after that because the request id key can change. This is by design and is the best solution I can think of at the moment. In this case the OcelotLogger will show the request id and previous request id in the logs.
+
+Below is an example of the logging when set at Debug level for a normal request..
+
+.. code-block:: bash
+
+    dbug: Ocelot.Errors.Middleware.ExceptionHandlerMiddleware[0]
+          requestId: asdf, previousRequestId: no previous request id, message: ocelot pipeline started,
+    dbug: Ocelot.DownstreamRouteFinder.Middleware.DownstreamRouteFinderMiddleware[0]
+          requestId: asdf, previousRequestId: no previous request id, message: upstream url path is {upstreamUrlPath},
+    dbug: Ocelot.DownstreamRouteFinder.Middleware.DownstreamRouteFinderMiddleware[0]
+          requestId: asdf, previousRequestId: no previous request id, message: downstream template is {downstreamRoute.Data.ReRoute.DownstreamPath},
+    dbug: Ocelot.RateLimit.Middleware.ClientRateLimitMiddleware[0]
+          requestId: asdf, previousRequestId: no previous request id, message: EndpointRateLimiting is not enabled for Ocelot.Values.PathTemplate,
+    dbug: Ocelot.Authorisation.Middleware.AuthorisationMiddleware[0]
+          requestId: 1234, previousRequestId: asdf, message: /posts/{postId} route does not require user to be authorised,
+    dbug: Ocelot.DownstreamUrlCreator.Middleware.DownstreamUrlCreatorMiddleware[0]
+          requestId: 1234, previousRequestId: asdf, message: downstream url is {downstreamUrl.Data.Value},
+    dbug: Ocelot.Request.Middleware.HttpRequestBuilderMiddleware[0]
+          requestId: 1234, previousRequestId: asdf, message: setting upstream request,
+    dbug: Ocelot.Requester.Middleware.HttpRequesterMiddleware[0]
+          requestId: 1234, previousRequestId: asdf, message: setting http response message,
+    dbug: Ocelot.Responder.Middleware.ResponderMiddleware[0]
+          requestId: 1234, previousRequestId: asdf, message: no pipeline errors, setting and returning completed response,
+    dbug: Ocelot.Errors.Middleware.ExceptionHandlerMiddleware[0]
+          requestId: 1234, previousRequestId: asdf, message: ocelot pipeline finished,

docs/features/routing.rst

@@ -62,4 +62,33 @@ In order to change this you can specify on a per ReRoute basis the following set
 
 This means that when Ocelot tries to match the incoming upstream url with an upstream template the
 evaluation will be case sensitive. This setting defaults to false so only set it if you want 
-the ReRoute to be case sensitive is my advice!
+the ReRoute to be case sensitive is my advice!
+
+Catch All
+^^^^^^^^^
+
+Ocelot's routing also supports a catch all style routing where the user can specify that they want to match all traffic if you set up your config like below the request will be proxied straight through (it doesnt have to be url any placeholder name will work). 
+
+.. code-block:: json
+
+    {
+        "DownstreamPathTemplate": "/{url}",
+        "DownstreamScheme": "https",
+        "DownstreamPort": 80,
+        "DownstreamHost" "localhost",
+        "UpstreamPathTemplate": "/{url}",
+        "UpstreamHttpMethod": [ "Get" ]
+    }
+
+The catch all has a lower priority than any other ReRoute. If you also have the ReRoute below in your config then Ocelot would match it before the catch all. 
+
+.. code-block:: json
+
+    {
+        "DownstreamPathTemplate": "/",
+        "DownstreamScheme": "https",
+        "DownstreamPort": 80,
+        "DownstreamHost" "10.0.10.1",
+        "UpstreamPathTemplate": "/",
+        "UpstreamHttpMethod": [ "Get" ]
+    }

docs/index.rst

@@ -1,29 +1,7 @@
 Welcome to Ocelot
 =================
 
-This project is aimed at people using .NET running 
-a micro services / service orientated architecture 
-that need a unified point of entry into their system.
-
-In particular I want easy integration with 
-IdentityServer reference and bearer tokens. 
-
-We have been unable to find this in my current workplace
-without having to write our own Javascript middlewares 
-to handle the IdentityServer reference tokens. We would
-rather use the IdentityServer code that already exists
-to do this.
-
-Ocelot is a bunch of middlewares in a specific order.
-
-Ocelot manipulates the HttpRequest object into a state specified by its configuration until 
-it reaches a request builder middleware where it creates a HttpRequestMessage object which is 
-used to make a request to a downstream service. The middleware that makes the request is 
-the last thing in the Ocelot pipeline. It does not call the next middleware. 
-The response from the downstream service is stored in a per request scoped repository 
-and retrived as the requests goes back up the Ocelot pipeline. There is a piece of middleware 
-that maps the HttpResponseMessage onto the HttpResponse object and that is returned to the client.
-That is basically it with a bunch of other features.
+Thanks for taking a look at the Ocelot documentation. Please use the left hand nav to get around. I would suggest taking a look at introduction first.
 
 .. toctree::
    :maxdepth: 2
@@ -46,6 +24,7 @@ That is basically it with a bunch of other features.
    features/authentication
    features/authorisation
    features/administration
+   features/raft
    features/caching
    features/qualityofservice
    features/claimstransformation 

docs/introduction/bigpicture.rst

@@ -1,4 +1,38 @@
 Big Picture
 ===========
 
-Coming soon...
+Ocleot is aimed at people using .NET running 
+a micro services / service orientated architecture 
+that need a unified point of entry into their system.
+
+In particular I want easy integration with 
+IdentityServer reference and bearer tokens. 
+
+Ocelot is a bunch of middlewares in a specific order.
+
+Ocelot manipulates the HttpRequest object into a state specified by its configuration until 
+it reaches a request builder middleware where it creates a HttpRequestMessage object which is 
+used to make a request to a downstream service. The middleware that makes the request is 
+the last thing in the Ocelot pipeline. It does not call the next middleware. 
+The response from the downstream service is stored in a per request scoped repository 
+and retrived as the requests goes back up the Ocelot pipeline. There is a piece of middleware 
+that maps the HttpResponseMessage onto the HttpResponse object and that is returned to the client.
+That is basically it with a bunch of other features.
+
+The following are configuration that you use when deploying Ocelot.
+
+Basic Implementation
+^^^^^^^^^^^^^^^^^^^^
+.. image:: ../images/OcelotBasic.jpg
+
+With IdentityServer
+^^^^^^^^^^^^^^^^^^^
+.. image:: ../images/OcelotIndentityServer.jpg
+
+Multiple Instances
+^^^^^^^^^^^^^^^^^^
+.. image:: ../images/OcelotMultipleInstances.jpg
+
+With Consul
+^^^^^^^^^^^
+.. image:: ../images/OcelotMultipleInstancesConsul.jpg