Concurrent requests with different services, using consul, returns the same service and then 404 #2119

minnocenti901 · 2024-07-10T09:29:12Z

Expected Behavior

Ocelot should request from Consul the correct service for every route

Actual Behavior

If concurrent requests, ocelot uses the same servicename for every concurrent request

Steps to Reproduce the Problem

On consul I have 2 services registered:
ProjectsService with url http://localhost:5013
CustomersService with url http://localhost:5004

Ocelot is running on http://localhost:5280

Global configuration:

  "GlobalConfiguration": {
    "RequestIdKey": "ApiGateway",
    "ServiceDiscoveryProvider": {
      "Type": "Consul",
      "Port": "8502",
      "Host": "192.168.0.204"
    }
  }

Routes configuration:

{
  "UpstreamHttpMethod": ["GET","POST","PUT","DELETE"],
  "UpstreamPathTemplate": "/projects/{all}",
  "DownstreamPathTemplate": "/{all}",
  "ServiceName": "ProjectsService"
},
{
  "UpstreamHttpMethod": ["GET","POST","PUT","DELETE"],
  "UpstreamPathTemplate": "/customers/{all}",
  "DownstreamPathTemplate": "/{all}",
  "ServiceName": "CustomersService"
}

Then, at startup, I make 3 different requests at the same time:

GET http://localhost:5280/projects/api/projects
GET http://localhost:5280/projects/api/industrialSectors
GET http://localhost:5280/customers/api/customers

This is the log in console with LogLevel Debug

dbug: Ocelot.Errors.Middleware.ExceptionHandlerMiddleware[0]
      requestId: 0HN50O7EO7FED:00000001, previousRequestId: No PreviousRequestId, message: 'ocelot pipeline started'
dbug: Ocelot.Errors.Middleware.ExceptionHandlerMiddleware[0]
      requestId: 0HN50O7EO7FEE:00000001, previousRequestId: No PreviousRequestId, message: 'ocelot pipeline started'
dbug: Ocelot.Errors.Middleware.ExceptionHandlerMiddleware[0]
      requestId: 0HN50O7EO7FEC:00000001, previousRequestId: No PreviousRequestId, message: 'ocelot pipeline started'
dbug: Ocelot.DownstreamRouteFinder.Middleware.DownstreamRouteFinderMiddleware[0]
      requestId: 0HN50O7EO7FEC:00000001, previousRequestId: No PreviousRequestId, message: 'Upstream URL path is '/projects/api/projects'.'
dbug: Ocelot.DownstreamRouteFinder.Middleware.DownstreamRouteFinderMiddleware[0]
      requestId: 0HN50O7EO7FED:00000001, previousRequestId: No PreviousRequestId, message: 'Upstream URL path is '/customers/api/customers'.'
dbug: Ocelot.DownstreamRouteFinder.Middleware.DownstreamRouteFinderMiddleware[0]
      requestId: 0HN50O7EO7FEE:00000001, previousRequestId: No PreviousRequestId, message: 'Upstream URL path is '/projects/api/industrialSectors'.'
dbug: Ocelot.DownstreamRouteFinder.Middleware.DownstreamRouteFinderMiddleware[0]
      requestId: 0HN50O7EO7FEE:00000001, previousRequestId: No PreviousRequestId, message: 'downstream templates are /{all}'
dbug: Ocelot.DownstreamRouteFinder.Middleware.DownstreamRouteFinderMiddleware[0]
      requestId: 0HN50O7EO7FEC:00000001, previousRequestId: No PreviousRequestId, message: 'downstream templates are /{all}'
dbug: Ocelot.DownstreamRouteFinder.Middleware.DownstreamRouteFinderMiddleware[0]
      requestId: 0HN50O7EO7FED:00000001, previousRequestId: No PreviousRequestId, message: 'downstream templates are /{all}'
info: Ocelot.RateLimiting.Middleware.RateLimitingMiddleware[0]
      requestId: 0HN50O7EO7FEE:00000001, previousRequestId: No PreviousRequestId, message: 'EndpointRateLimiting is not enabled for /{all}'
info: Ocelot.RateLimiting.Middleware.RateLimitingMiddleware[0]
      requestId: 0HN50O7EO7FEC:00000001, previousRequestId: No PreviousRequestId, message: 'EndpointRateLimiting is not enabled for /{all}'
info: Ocelot.RateLimiting.Middleware.RateLimitingMiddleware[0]
      requestId: 0HN50O7EO7FED:00000001, previousRequestId: No PreviousRequestId, message: 'EndpointRateLimiting is not enabled for /{all}'
info: Microsoft.AspNetCore.Authentication.AuthenticationMiddleware[0]
      requestId: 0HN50O7EO7FED:00000001, previousRequestId: No PreviousRequestId, message: '/customers/api/customers is an authenticated route. AuthenticationMiddleware checking if client is authenticated'
info: Microsoft.AspNetCore.Authentication.AuthenticationMiddleware[0]
      requestId: 0HN50O7EO7FEE:00000001, previousRequestId: No PreviousRequestId, message: '/projects/api/industrialSectors is an authenticated route. AuthenticationMiddleware checking if client is authenticated'
info: Microsoft.AspNetCore.Authentication.AuthenticationMiddleware[0]
      requestId: 0HN50O7EO7FEC:00000001, previousRequestId: No PreviousRequestId, message: '/projects/api/projects is an authenticated route. AuthenticationMiddleware checking if client is authenticated'
info: Microsoft.AspNetCore.Authentication.AuthenticationMiddleware[0]
      requestId: 0HN50O7EO7FED:00000001, previousRequestId: No PreviousRequestId, message: 'Client has been authenticated for /customers/api/customers'
info: Microsoft.AspNetCore.Authentication.AuthenticationMiddleware[0]
      requestId: 0HN50O7EO7FEE:00000001, previousRequestId: No PreviousRequestId, message: 'Client has been authenticated for /projects/api/industrialSectors'
info: Microsoft.AspNetCore.Authentication.AuthenticationMiddleware[0]
      requestId: 0HN50O7EO7FEC:00000001, previousRequestId: No PreviousRequestId, message: 'Client has been authenticated for /projects/api/projects'
info: Ocelot.Authorization.Middleware.AuthorizationMiddleware[0]
      requestId: 0HN50O7EO7FEE:00000001, previousRequestId: No PreviousRequestId, message: 'route is authenticated scopes must be checked'
info: Ocelot.Authorization.Middleware.AuthorizationMiddleware[0]
      requestId: 0HN50O7EO7FEC:00000001, previousRequestId: No PreviousRequestId, message: 'route is authenticated scopes must be checked'
info: Ocelot.Authorization.Middleware.AuthorizationMiddleware[0]
      requestId: 0HN50O7EO7FED:00000001, previousRequestId: No PreviousRequestId, message: 'route is authenticated scopes must be checked'
info: Ocelot.Authorization.Middleware.AuthorizationMiddleware[0]
      requestId: 0HN50O7EO7FED:00000001, previousRequestId: No PreviousRequestId, message: 'user scopes is authorized calling next authorization checks'
info: Ocelot.Authorization.Middleware.AuthorizationMiddleware[0]
      requestId: 0HN50O7EO7FEE:00000001, previousRequestId: No PreviousRequestId, message: 'user scopes is authorized calling next authorization checks'
info: Ocelot.Authorization.Middleware.AuthorizationMiddleware[0]
      requestId: 0HN50O7EO7FEC:00000001, previousRequestId: No PreviousRequestId, message: 'user scopes is authorized calling next authorization checks'
info: Ocelot.Authorization.Middleware.AuthorizationMiddleware[0]
      requestId: 0HN50O7EO7FEE:00000001, previousRequestId: No PreviousRequestId, message: '/{all} route does not require user to be authorized'
info: Ocelot.Authorization.Middleware.AuthorizationMiddleware[0]
      requestId: 0HN50O7EO7FED:00000001, previousRequestId: No PreviousRequestId, message: '/{all} route does not require user to be authorized'
info: Ocelot.Authorization.Middleware.AuthorizationMiddleware[0]
      requestId: 0HN50O7EO7FEC:00000001, previousRequestId: No PreviousRequestId, message: '/{all} route does not require user to be authorized'
info: Ocelot.ServiceDiscovery.ServiceDiscoveryProviderFactory[0]
      requestId: 0HN50O7EO7FEC:00000001, previousRequestId: No PreviousRequestId, message: 'The UseServiceDiscovery mode of the route '^(?i)/projects(?:|/.*)$' is enabled.'
info: Ocelot.ServiceDiscovery.ServiceDiscoveryProviderFactory[0]
      requestId: 0HN50O7EO7FEE:00000001, previousRequestId: No PreviousRequestId, message: 'The UseServiceDiscovery mode of the route '^(?i)/projects(?:|/.*)$' is enabled.'
info: Ocelot.ServiceDiscovery.ServiceDiscoveryProviderFactory[0]
      requestId: 0HN50O7EO7FED:00000001, previousRequestId: No PreviousRequestId, message: 'The UseServiceDiscovery mode of the route '^(?i)/customers(?:|/.*)$' is enabled.'
info: Ocelot.ServiceDiscovery.ServiceDiscoveryProviderFactory[0]
      requestId: 0HN50O7EO7FEE:00000001, previousRequestId: No PreviousRequestId, message: 'Getting service discovery provider of Type 'Consul'...'
info: Ocelot.ServiceDiscovery.ServiceDiscoveryProviderFactory[0]
      requestId: 0HN50O7EO7FEC:00000001, previousRequestId: No PreviousRequestId, message: 'Getting service discovery provider of Type 'Consul'...'
info: Ocelot.ServiceDiscovery.ServiceDiscoveryProviderFactory[0]
      requestId: 0HN50O7EO7FED:00000001, previousRequestId: No PreviousRequestId, message: 'Getting service discovery provider of Type 'Consul'...'
dbug: Ocelot.Provider.Consul.Consul[0]
      requestId: 0HN50O7EO7FEE:00000001, previousRequestId: No PreviousRequestId, message: 'Consul Provider: Found total 1 service entries for 'ProjectsService' service.'
dbug: Ocelot.Provider.Consul.Consul[0]
      requestId: 0HN50O7EO7FEC:00000001, previousRequestId: No PreviousRequestId, message: 'Consul Provider: Found total 1 service entries for 'ProjectsService' service.'
dbug: Ocelot.Provider.Consul.Consul[0]
      requestId: 0HN50O7EO7FED:00000001, previousRequestId: No PreviousRequestId, message: 'Consul Provider: Found total 1 service entries for 'ProjectsService' service.'
dbug: Ocelot.Provider.Consul.Consul[0]
      requestId: 0HN50O7EO7FEC:00000001, previousRequestId: No PreviousRequestId, message: 'Consul Provider: Found total 1 catalog nodes.'
dbug: Ocelot.Provider.Consul.Consul[0]
      requestId: 0HN50O7EO7FED:00000001, previousRequestId: No PreviousRequestId, message: 'Consul Provider: Found total 1 catalog nodes.'
dbug: Ocelot.Provider.Consul.Consul[0]
      requestId: 0HN50O7EO7FEE:00000001, previousRequestId: No PreviousRequestId, message: 'Consul Provider: Found total 1 catalog nodes.'
dbug: Ocelot.DownstreamUrlCreator.Middleware.DownstreamUrlCreatorMiddleware[0]
      requestId: 0HN50O7EO7FEC:00000001, previousRequestId: No PreviousRequestId, message: 'Downstream url is http://localhost:5013/api/projects'
dbug: Ocelot.DownstreamUrlCreator.Middleware.DownstreamUrlCreatorMiddleware[0]
      requestId: 0HN50O7EO7FEE:00000001, previousRequestId: No PreviousRequestId, message: 'Downstream url is http://localhost:5013/api/industrialSectors'
dbug: Ocelot.DownstreamUrlCreator.Middleware.DownstreamUrlCreatorMiddleware[0]
      requestId: 0HN50O7EO7FED:00000001, previousRequestId: No PreviousRequestId, message: 'Downstream url is http://localhost:5013/api/customers'
warn: Ocelot.Requester.Middleware.HttpRequesterMiddleware[0]
      requestId: 0HN50O7EO7FED:00000001, previousRequestId: No PreviousRequestId, message: '404 (Not Found) status code of request URI: http://localhost:5013/api/customers.'
dbug: Ocelot.Requester.Middleware.HttpRequesterMiddleware[0]
      requestId: 0HN50O7EO7FED:00000001, previousRequestId: No PreviousRequestId, message: 'setting http response message'
dbug: Ocelot.Responder.Middleware.ResponderMiddleware[0]
      requestId: 0HN50O7EO7FED:00000001, previousRequestId: No PreviousRequestId, message: 'no pipeline errors, setting and returning completed response'
dbug: Ocelot.Errors.Middleware.ExceptionHandlerMiddleware[0]
      requestId: 0HN50O7EO7FED:00000001, previousRequestId: No PreviousRequestId, message: 'ocelot pipeline finished'
dbug: Microsoft.AspNetCore.Server.Kestrel.Connections[9]
      Connection id "0HN50O7EO7FED" completed keep alive response.
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
      Request finished HTTP/1.1 GET http://localhost:5280/customers/api/customers - 404 0 - 1404.2153ms
info: Microsoft.AspNetCore.Hosting.Diagnostics[16]
      Request reached the end of the middleware pipeline without being handled by application code. Request path: GET http://localhost:5280/customers/api/customers, Response status code: 404
info: Ocelot.Requester.Middleware.HttpRequesterMiddleware[0]
      requestId: 0HN50O7EO7FEC:00000001, previousRequestId: No PreviousRequestId, message: '200 (OK) status code of request URI: http://localhost:5013/api/projects.'
info: Ocelot.Requester.Middleware.HttpRequesterMiddleware[0]
      requestId: 0HN50O7EO7FEE:00000001, previousRequestId: No PreviousRequestId, message: '200 (OK) status code of request URI: http://localhost:5013/api/industrialSectors.'
dbug: Ocelot.Requester.Middleware.HttpRequesterMiddleware[0]
      requestId: 0HN50O7EO7FEC:00000001, previousRequestId: No PreviousRequestId, message: 'setting http response message'
dbug: Ocelot.Requester.Middleware.HttpRequesterMiddleware[0]
      requestId: 0HN50O7EO7FEE:00000001, previousRequestId: No PreviousRequestId, message: 'setting http response message'
dbug: Ocelot.Responder.Middleware.ResponderMiddleware[0]
      requestId: 0HN50O7EO7FEE:00000001, previousRequestId: No PreviousRequestId, message: 'no pipeline errors, setting and returning completed response'
dbug: Ocelot.Responder.Middleware.ResponderMiddleware[0]
      requestId: 0HN50O7EO7FEC:00000001, previousRequestId: No PreviousRequestId, message: 'no pipeline errors, setting and returning completed response'
dbug: Ocelot.Errors.Middleware.ExceptionHandlerMiddleware[0]
      requestId: 0HN50O7EO7FEC:00000001, previousRequestId: No PreviousRequestId, message: 'ocelot pipeline finished'
dbug: Ocelot.Errors.Middleware.ExceptionHandlerMiddleware[0]
      requestId: 0HN50O7EO7FEE:00000001, previousRequestId: No PreviousRequestId, message: 'ocelot pipeline finished'
dbug: Microsoft.AspNetCore.Server.Kestrel.Connections[9]
      Connection id "0HN50O7EO7FEC" completed keep alive response.
dbug: Microsoft.AspNetCore.Server.Kestrel.Connections[9]
      Connection id "0HN50O7EO7FEE" completed keep alive response.
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
      Request finished HTTP/1.1 GET http://localhost:5280/projects/api/projects - 200 2743 application/json;+charset=utf-8 3878.4309ms
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
      Request finished HTTP/1.1 GET http://localhost:5280/projects/api/industrialSectors - 200 339 application/json;+charset=utf-8 3879.4389ms
info: Microsoft.AspNetCore.Hosting.Diagnostics[16]
      Request reached the end of the middleware pipeline without being handled by application code. Request path: GET http://localhost:5280/projects/api/projects, Response status code: 200
info: Microsoft.AspNetCore.Hosting.Diagnostics[16]
      Request reached the end of the middleware pipeline without being handled by application code. Request path: GET http://localhost:5280/projects/api/industrialSectors, Response status code: 200

I'm expecting that for the route /customers it uses CustomersService instead of ProjectsService.

Specifications

Version: Ocelot 23.3.3, Ocelot.Provider.Consul 23.3.3, Consul 1.12.0
Platform: .NET8

The text was updated successfully, but these errors were encountered:

raman-m · 2024-07-10T12:42:25Z

Hello @minnocenti901! What is your full name?
I suspect there is an issue with your project setup, Consul configuration, or networking. Let's work together to identify the root cause.

Ocelot is running on http://localhost:5280

Global configuration:

  "GlobalConfiguration": {
    "RequestIdKey": "ApiGateway",
    "ServiceDiscoveryProvider": {
      "Type": "Consul",
      "Port": "8502",
      "Host": "192.168.0.204"
    }
  }

The BaseUrl property is missing and is required. As outlined in our Getting Started documentation, the BaseUrl must be defined in the Configuration section of the JSON file, as shown below:

  "GlobalConfiguration": {
    "BaseUrl": "http://localhost:5280", // this is crucial!
    "RequestIdKey": "ApiGateway",
    "ServiceDiscoveryProvider": {
      "Type": "Consul",
      "Port": "8502",
      "Host": "192.168.0.204"
    }
  }

Please could you provide the complete ocelot.json file for review?

raman-m · 2024-07-10T13:00:12Z

Expected Behavior

Ocelot should request from Consul the correct service for every route

Steps to Reproduce the Problem

On consul I have 2 services registered:
ProjectsService with url http://localhost:5013
CustomersService with url http://localhost:5004
Ocelot is running on http://localhost:5280

🆗 Could you show us complete Consul configuration please, service mappings, nodes etc.?

Regarding networking... I see you use learning solution at local machine in Docker or without, right?

raman-m · 2024-07-10T13:04:39Z

Specifications

Version: Ocelot 23.3.3, Ocelot.Provider.Consul 23.3.3, Consul 1.12.0

Platform: .NET8

🆗 Please revert the version to 23.2.2 and conduct the same tests.

minnocenti901 · 2024-07-10T13:06:00Z

Adding BaseUrl didn't solve the issue.
This is the complete ocelot.json

{
  "Routes": [
    {
      "DownstreamPathTemplate": "/{all}",
      "DownstreamScheme": "http",
      "ServiceName": "CustomersService",
      "UpstreamPathTemplate": "/customers/{all}",
      "UpstreamHttpMethod": [
        "GET",
        "POST",
        "PUT",
        "DELETE"
      ],
      "ReRouteIsCaseSensitive": false,
      "AuthenticationOptions": {
        "AuthenticationProviderKey": "Bearer"
      },
      "LoadBalancerOptions": {
        "Type": "LeastConnection"
      }
    },
    {
      "DownstreamPathTemplate": "/{all}",
      "DownstreamScheme": "http",
      "ServiceName": "ProjectsService",
      "UpstreamPathTemplate": "/projects/{all}",
      "UpstreamHttpMethod": [
        "GET",
        "POST",
        "PUT",
        "DELETE"
      ],
      "ReRouteIsCaseSensitive": false,
      "AuthenticationOptions": {
        "AuthenticationProviderKey": "Bearer"
      },
      "LoadBalancerOptions": {
        "Type": "LeastConnection"
      }
    }
  ],
  "GlobalConfiguration": {
    "BaseUrl": "http://localhost:5280",
    "RequestIdKey": "ApiGateway",
    "ServiceDiscoveryProvider": {
      "Type": "Consul",
      "Port": "8502",
      "Host": "192.168.0.204"
    }
  }
}

I'm running it inside visual studio 2022 professional on windows 11. The issue is also present if running it in a docker container on Ubuntu Server 22.04

I forgot to mention that I have upgraded from Ocelot 22.0.1 on .NET6 to latest version 23.3.3 on .NET8. Before the upgrade everything was working perfectly.

Consul is running on docker compose with this .yml:

  consul:
    restart: always
    image: consul:1.12.0
    volumes:
      - ./data/consul:/consul/data
    command: agent -server -ui -node=server-1 -client=0.0.0.0 -bootstrap-expect=1
    ports:
      - "8502:8500"
    logging:
      driver: "json-file"
      options:
          max-size: "50m"

minnocenti901 · 2024-07-10T13:08:08Z

Reverting to 23.2.2 fixed the issue

minnocenti901 · 2024-07-10T13:08:37Z

Hello @minnocenti901! What is your full name?

Hello! It's Massimiliano Innocenti

raman-m · 2024-07-10T13:16:07Z

Actual Behavior

If concurrent requests, ocelot uses the same servicename for every concurrent request

Then, at startup, I make 3 different requests at the same time:

GET http://localhost:5280/projects/api/projects

GET http://localhost:5280/projects/api/industrialSectors

GET http://localhost:5280/customers/api/customers

logs...

I'm expecting that for the route /customers it uses CustomersService instead of ProjectsService.

It seems we may have encountered a bug following a significant refactoring of the Consul provider by PR #2067 and commit 34cb3eb

Reverting to 23.2.2 fixed the issue

@minnocenti901 Excellent! Continue using version 23.2.2 until we have resolved or understood the problem.

@ggnaegi, could you please join us and collaborate on this issue? Have you deployed version 23.3.x to Production?

minnocenti901 · 2024-07-10T13:18:11Z

@raman-m Thank you for your time!

raman-m · 2024-07-10T13:26:07Z

@minnocenti901 commented

Hello! It's Massimiliano Innocenti

LoL Massimiliano Innocenti on LinkedIn
Which one could be you? 😄

minnocenti901 · 2024-07-10T13:27:34Z

I prefer not to answer :)

raman-m · 2024-07-10T13:28:30Z

@minnocenti901, are you a C# developer?
If so, would you be willing to collaborate on problem-solving: code researching, debugging, design review and probably open a PR?

minnocenti901 · 2024-07-10T13:30:18Z

@minnocenti901, are you a C# developer?

Yes

If so, would you be willing to collaborate on problem-solving: code researching, debugging, design review and probably open a PR?

No, thank you. I don't have time :\ Im sorry

raman-m · 2024-07-10T13:32:03Z

No, thank you. I don't have time :\ Im sorry

How did you detect the issue? Are you using Ocelot for certain projects?
Regardless, I appreciate you reporting the bug!

minnocenti901 · 2024-07-10T13:40:15Z

I use it as main api gateway for a microservices architecture. I need to upgrade from NET6 to NET8 due to upcoming end of service and with the update I had to recheck that everything still works. Then comes the issue

raman-m · 2024-07-10T13:43:28Z

Understood. For the time being, please proceed with version 23.2.2 for your solution upgrade.

raman-m · 2024-07-19T11:14:36Z

@minnocenti901 commented Jul 10, 2024, 1:08 PM:

Reverting to 23.2.2 fixed the issue

Good. Did you change anything in the Consul services registrations, or was there nothing except a version increase? It seems the issue was introduced by PR #2067. You are correct; the issue might be related to concurrency. Max, could you please share the JSON from the Consul client? I need the complete list of services registered in Consul.

minnocenti901 · 2024-07-19T11:17:00Z

I just increase version of Ocelot. How do I get the JSON you need?

ggnaegi · 2024-08-12T10:34:49Z

@raman-m @minnocenti901 I'm on it, cheers

raman-m · 2024-08-12T12:02:33Z

Let me to proceed with this, @ggnaegi. Please concentrate on issue #2116. It appears that the author is not intending to contribute but rather aims to direct our attention to this bug.

…rviceBuilder` with `ConsulProviderFactory` refactoring to make it thread safe and friendly (#2151) * Review tests * History of Service Discovery testing: add traits * LoadBalancer traits * #2119 Steps to Reproduce * Reuse service handlers of `ConcurrentSteps` * Reuse service counters of `ConcurrentSteps` * Add LoadBalancer namespace and move classes * Move `Lease` * Move `LeaseEventArgs` * Analyze load balancers aka `ILoadBalancerAnalyzer` interface objects * Prefer using named local methods as delegates over anonymous methods for awesome call stack, ensuring the delegate's typed result matches the typed balancer's creator. Additionally, employ an IServiceProvider workaround. * Review load balancing. Assert service & leasing counters as concurrent step. Final version of acceptance test. * Fixed naming violation for asynchronous methods: `Lease` -> `LeaseAsync` * Fix ugly reflection issue of dymanic detection in favor of static type property * Propagate the `ConsulRegistryConfiguration` object through `HttpContext` in the scoped version of the default service builder, utilizing the injected `IHttpContextAccessor` object. Update `ConsulProviderFactory`. Update docs. Update tests. * Add tests from clean experiment * Final review of the tests * Review `IHttpContextAccessor` logic. Convert anonymous delegates to named ones in placeholders processing * Tried to enhance more, but failed

raman-m · 2024-10-03T08:55:57Z

@minnocenti901 Max, are you online?

@raman-m

…Blue Olympic Balumbes release * #2084 Apply default config file paths in `GetMergedOcelotJson` when providing the `folder` argument of `AddOcelot` (#2120) * Adding unit test first * Fixing default global config file not being found in folder * Adding PR trait to test * Backing out whitespace changes * Code review by @raman-m * Create Configuration feature folder and move test classes * Adjust namespace and review what we have * Acceptance tests for #2084 user scenario --------- Co-authored-by: Raman Maksimchuk <dotnet044@gmail.com> * Bump Steeltoe.Discovery.Eureka from 3.2.5 to 3.2.8 in /src/Ocelot.Provider.Eureka (#2122) * Bump Steeltoe.Discovery.Eureka in /src/Ocelot.Provider.Eureka Bumps [Steeltoe.Discovery.Eureka](https://github.com/SteeltoeOSS/Steeltoe) from 3.2.5 to 3.2.8. - [Release notes](https://github.com/SteeltoeOSS/Steeltoe/releases) - [Changelog](https://github.com/SteeltoeOSS/Steeltoe/blob/main/Steeltoe.Release.ruleset) - [Commits](SteeltoeOSS/Steeltoe@3.2.5...3.2.8) --- updated-dependencies: - dependency-name: Steeltoe.Discovery.Eureka dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * Bump Steeltoe.Discovery.ClientCore from 3.2.5 to 3.2.8 --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Raman Maksimchuk <dotnet044@gmail.com> * #2110 Review load balancing and independent fetching the list of services in `Kube` provider (#2111) * Move the creation of the services list from the class field to the method, to prevent modification list from different threads * Early return after data checking * Add unit test for concurrent get list of services * Add logging for invalid service configuration error in RoundRobin load balancer * Code review by @raman-m * Workaround for mistakes made during acceptance testing of load balancing versus service discovery, where tests designed for parallel requests were mistakenly executed sequentially. This resulted in load balancers being loaded by sequential `HttpClient` calls, which was a significant oversight. * Let's DRY StickySessionsTests * Add acceptance tests, but... RoundRobin is not actually RoundRobin 😁 -> 😆 * Independent static indexing iterators per route via service names * Stabilize `CookieStickySessions` load balancer. Review tests after refactoring of `RoundRobin` load balancer * Refactor Lease operation for load balancing. Review LeastConnection load balancer * Leasing mechanism in Round Robin load balancer * Acceptance tests, final version * Apply Retry pattern for K8s endpoint integration * Fix IDE warnings and messages * Follow suggestions and fix issues from code review by @ggnaegi * Bump KubeClient from 2.4.10 to 2.5.8 * Fix warnings * Final version of `Retry` pattern --------- Co-authored-by: Raman Maksimchuk <dotnet044@gmail.com> * Downgrade the Warning to Information on missing `Content-Length` header in `MultiplexingMiddleware` (#2146) * fix: downgrade the warning to information on missing content-length header * chore: add route name to logs * test: fixing multiplexing middleware tests * Code review by @raman-m --------- Co-authored-by: Paul Roy <paul.roy@astriis.com> Co-authored-by: Raman Maksimchuk <dotnet044@gmail.com> * Correct the broken link to the GraphQL sample's `README.md` (#2149) Signed-off-by: Emmanuel Ferdman <emmanuelferdman@gmail.com> Co-authored-by: Raman Maksimchuk <dotnet044@gmail.com> * #2116 Escaping unsafe pattern values of `Regex` constructor derived from URL query parameter values containing special `Regex` chars (#2150) * regex escape handling for url templates * refactored regex method to lamda version * Quick code review by @raman-m * added acceptance test for url regex bug * moved acceptance test to routing tests * Convert to theory: define 2 test cases --------- Co-authored-by: Raman Maksimchuk <dotnet044@gmail.com> * #2119 Review load balancing (2nd round) and redesign `DefaultConsulServiceBuilder` with `ConsulProviderFactory` refactoring to make it thread safe and friendly (#2151) * Review tests * History of Service Discovery testing: add traits * LoadBalancer traits * #2119 Steps to Reproduce * Reuse service handlers of `ConcurrentSteps` * Reuse service counters of `ConcurrentSteps` * Add LoadBalancer namespace and move classes * Move `Lease` * Move `LeaseEventArgs` * Analyze load balancers aka `ILoadBalancerAnalyzer` interface objects * Prefer using named local methods as delegates over anonymous methods for awesome call stack, ensuring the delegate's typed result matches the typed balancer's creator. Additionally, employ an IServiceProvider workaround. * Review load balancing. Assert service & leasing counters as concurrent step. Final version of acceptance test. * Fixed naming violation for asynchronous methods: `Lease` -> `LeaseAsync` * Fix ugly reflection issue of dymanic detection in favor of static type property * Propagate the `ConsulRegistryConfiguration` object through `HttpContext` in the scoped version of the default service builder, utilizing the injected `IHttpContextAccessor` object. Update `ConsulProviderFactory`. Update docs. Update tests. * Add tests from clean experiment * Final review of the tests * Review `IHttpContextAccessor` logic. Convert anonymous delegates to named ones in placeholders processing * Tried to enhance more, but failed --------- Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Emmanuel Ferdman <emmanuelferdman@gmail.com> Co-authored-by: Ben Bartholomew <70723971+ben-bartholomew@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Roman <61905975+antikorol@users.noreply.github.com> Co-authored-by: Paul Roy <paul.achess.roy@gmail.com> Co-authored-by: Paul Roy <paul.roy@astriis.com> Co-authored-by: Emmanuel Ferdman <emmanuelferdman@gmail.com> Co-authored-by: Finn <26823828+int0x81@users.noreply.github.com>

minnocenti901 · 2024-10-04T07:23:46Z

Yes, i've seen you marked as closed. You need some testing?

raman-m · 2024-10-04T10:11:45Z

The hotfix has been released in ver. 23.3.4, so you can download the pack from NuGet.
I've completed all the testing on my side 😸. Now, I would like to ask if you could perform testing in your environment, preferably manual testing in a real setting. What kind of environment do you have? Production, Staging, or Local?

minnocenti901 · 2024-10-04T10:16:44Z

Local and Staging, cannot run tests on production

raman-m · 2024-10-04T10:17:53Z

Can you handle deployment? What is your role in the project, QA?
Reporting back the test results from Staging is sufficient for me.

minnocenti901 · 2024-10-04T10:28:42Z

Yes I can. I have to update to 23.3.4?

raman-m · 2024-10-04T11:03:45Z

Emm... Yes, please, update version, deploy and test.

minnocenti901 · 2024-10-04T11:19:33Z

Updated from 23.2.2 to 23.3.4.
I now receive this error for every request I make:

warn: Ocelot.Responder.Middleware.ResponderMiddleware[0]
      requestId: 0HN74COOPNRLJ:00000001, previousRequestId: No PreviousRequestId, message: 'Error Code: ConnectionToDownstreamServiceError Message: Error connecting to downstream service, exception: System.Net.Http.HttpRequestException: Resource temporarily unavailable (server-1:80)
       ---> System.Net.Sockets.SocketException (11): Resource temporarily unavailable
         at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
         at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
         at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|285_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
         at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
         --- End of inner exception stack trace ---
         at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(QueueItem queueItem)
         at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
         at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
         at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at Ocelot.Requester.TimeoutDelegatingHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
         at Ocelot.Requester.MessageInvokerHttpRequester.GetResponse(HttpContext httpContext) errors found in ResponderMiddleware. Setting error response for request path:/projects/api/projects, request method: GET'

Configuration is not changed and it is the same as the one is posted

EDIT:
server-1 is the node name of consul

raman-m · 2024-10-04T11:51:07Z

Did you read this:

raman-m · 2024-10-04T12:03:47Z

server-1 is the node name of consul

Default service builder uses node name as a host!

Ocelot/src/Ocelot.Provider.Consul/DefaultConsulServiceBuilder.cs

Lines 96 to 97 in 6088515

    
           protected virtual string GetDownstreamHost(ServiceEntry entry, Node node) 
        
               => node != null ? node.Name : entry.Service.Address;

This is current best practice because node may/should contain the same hosts but different ports in load balancing scenarios, when Docker containers are deployed with different ports but same host. In this case it is logical thing to assign real host name to node name.

But I guess if Consul has setup by default, seems node names are abstract.
Please apply quick patch from the docs! Or go to Consul admin tool and assign real host names to node names!

It's curious how your environment functioned properly with version 23.2.2, given that the "node name as host name" feature has been available since version 13.5.2.

minnocenti901 · 2024-10-04T12:10:04Z

After the implementation of a "MyConsulServiceBuilder" using only the service.address it works perfectly. Thanks.

It's curious how your environment functioned properly with version 23.2.2, given that the "node name as host name" feature has been available since version 13.5.2.

I don't know :( it worked.

raman-m · 2024-10-04T12:10:52Z

I'm just curious, could you make node name empty in Consul admin? Seems not, right?

I've asked AI and got this answer :D

In Consul administration, it is not possible to have an empty node name; however, you can change the node name and IP address by following a specific procedure which involves removing the node from the cluster, wiping its local data, and then restarting it as a new node with the updated information. Additionally, ACL rules in Consul allow for read-write access to a node with a specific name, and read-only access to any node name using an empty prefix.

raman-m · 2024-10-04T12:13:32Z

After the implementation of a "MyConsulServiceBuilder" using only the service.address it works perfectly. Thanks.

It's curious how your environment functioned properly with version 23.2.2, given that the "node name as host name" feature has been available since version 13.5.2.

I don't know :( it worked.

Great! Congrats! 🥳
Can you start deployment and testing please?
I am expecting your feedback by the end of the day...

minnocenti901 · 2024-10-04T13:02:41Z

Local and Staging works perfectly. Every concurrent request receives the correct route. Can't test in production, sorry

raman-m · 2024-10-04T13:57:44Z

It appears we've officially resolved the issue. 😃

Can't test in production, sorry

No worries! Are other team members responsible for Production?
It would be beneficial to inform them about the successful tests in Staging so they can initiate a new Prod deployment.

Thank you for your contribution!

raman-m added bug Identified as a potential bug needs validation Issue has not been replicated or verified yet Service Discovery Ocelot feature: Service Discovery Consul Service discovery by Consul highest Highest priority labels Jul 10, 2024

raman-m self-assigned this Jul 10, 2024

raman-m added this to the Summer'24 milestone Jul 10, 2024

raman-m added the Oct'24 October 2024 release label Jul 10, 2024

raman-m modified the milestones: Summer'24, v23.3.x Hotfixes Jul 10, 2024

raman-m mentioned this issue Aug 2, 2024

23.3.0 broken #2133

Closed

ggnaegi unassigned raman-m Aug 12, 2024

ggnaegi assigned raman-m Aug 12, 2024

raman-m added in progress Someone is working on the issue. Could be someone on the team or off. and removed needs validation Issue has not been replicated or verified yet Oct'24 October 2024 release labels Aug 12, 2024

raman-m added a commit that referenced this issue Sep 2, 2024

#2119 Steps to Reproduce

9537e43

raman-m mentioned this issue Sep 13, 2024

#2119 Review load balancing (2nd round) and redesign DefaultConsulServiceBuilder with ConsulProviderFactory refactoring to make it thread safe and friendly #2151

Merged

raman-m added the Load Balancer Ocelot feature: Load Balancer label Sep 13, 2024

raman-m mentioned this issue Oct 1, 2024

Bump Steeltoe.Discovery.Eureka from 3.2.5 to 3.2.8 in /src/Ocelot.Provider.Eureka #2122

Merged

raman-m closed this as completed in #2151 Oct 3, 2024

raman-m added merged Issue has been merged to dev and is waiting for the next release and removed in progress Someone is working on the issue. Could be someone on the team or off. labels Oct 3, 2024

LeonSchourek mentioned this issue Oct 24, 2024

consul config option resolving address #2184

Closed

Concurrent requests with different services, using consul, returns the same service and then 404 #2119

Concurrent requests with different services, using consul, returns the same service and then 404 #2119

Comments

minnocenti901 commented Jul 10, 2024 • edited by raman-m Loading

Expected Behavior

Actual Behavior

Steps to Reproduce the Problem

Specifications

raman-m commented Jul 10, 2024

raman-m commented Jul 10, 2024

Expected Behavior

Steps to Reproduce the Problem

raman-m commented Jul 10, 2024 • edited Loading

Specifications

minnocenti901 commented Jul 10, 2024 • edited Loading

minnocenti901 commented Jul 10, 2024

minnocenti901 commented Jul 10, 2024

raman-m commented Jul 10, 2024

Actual Behavior

minnocenti901 commented Jul 10, 2024

raman-m commented Jul 10, 2024

minnocenti901 commented Jul 10, 2024

raman-m commented Jul 10, 2024 • edited Loading

minnocenti901 commented Jul 10, 2024 • edited Loading

raman-m commented Jul 10, 2024 • edited Loading

minnocenti901 commented Jul 10, 2024

raman-m commented Jul 10, 2024

raman-m commented Jul 19, 2024

minnocenti901 commented Jul 19, 2024

ggnaegi commented Aug 12, 2024

raman-m commented Aug 12, 2024 • edited Loading

raman-m commented Oct 3, 2024

minnocenti901 commented Oct 4, 2024

raman-m commented Oct 4, 2024 • edited Loading

minnocenti901 commented Oct 4, 2024

raman-m commented Oct 4, 2024 • edited Loading

minnocenti901 commented Oct 4, 2024

raman-m commented Oct 4, 2024

minnocenti901 commented Oct 4, 2024 • edited Loading

raman-m commented Oct 4, 2024

raman-m commented Oct 4, 2024 • edited Loading

minnocenti901 commented Oct 4, 2024 • edited Loading

raman-m commented Oct 4, 2024

raman-m commented Oct 4, 2024

minnocenti901 commented Oct 4, 2024

raman-m commented Oct 4, 2024

minnocenti901 commented Jul 10, 2024 •

edited by raman-m

Loading

raman-m commented Jul 10, 2024 •

edited

Loading

minnocenti901 commented Jul 10, 2024 •

edited

Loading

raman-m commented Jul 10, 2024 •

edited

Loading

minnocenti901 commented Jul 10, 2024 •

edited

Loading

raman-m commented Jul 10, 2024 •

edited

Loading

raman-m commented Aug 12, 2024 •

edited

Loading

raman-m commented Oct 4, 2024 •

edited

Loading

raman-m commented Oct 4, 2024 •

edited

Loading

minnocenti901 commented Oct 4, 2024 •

edited

Loading

raman-m commented Oct 4, 2024 •

edited

Loading

minnocenti901 commented Oct 4, 2024 •

edited

Loading