Cannot resolve scoped service KubeClient.IKubeApiClient from root provider

[Marusyk]

Expected Behavior / New Feature

redirect to service in k8s cluster

Actual Behavior / Motivation for New Feature

[19-08-01 11:02:11.11Z INF] requestId: 0HLOMDUGQ7EF3:00000004, previousRequestId: no previous request id, message: EndpointRateLimiting is not enabled for /{everything} <s:Development/Ocelot.RateLimit.Middleware.ClientRateLimitMiddleware>
[19-08-01 11:02:11.11Z INF] requestId: 0HLOMDUGQ7EF3:00000004, previousRequestId: no previous request id, message: No authentication needed for /devices/v1/devices <s:Development/Ocelot.Authentication.Middleware.AuthenticationMiddleware>
[19-08-01 11:02:11.11Z INF] requestId: 0HLOMDUGQ7EF3:00000004, previousRequestId: no previous request id, message: /{everything} route does not require user to be authorised <s:Development/Ocelot.Authorisation.Middleware.AuthorisationMiddleware>
[19-08-01 11:02:11.11Z WRN] requestId: 0HLOMDUGQ7EF3:00000004, previousRequestId: no previous request id, message: unabe to find load balancer for /devices/{everything}|Get| exception is System.InvalidOperationException: Cannot resolve scoped service 'KubeClient.IKubeApiClient' from root provider.
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteValidator.ValidateResolution(Type serviceType, IServiceScope scope, IServiceScope rootScope)
at Microsoft.Extensions.DependencyInjection.ServiceProvider.Microsoft.Extensions.DependencyInjection.ServiceLookup.IServiceProviderEngineCallback.OnResolve(Type serviceType, IServiceScope scope)
at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngine.GetService(Type serviceType, ServiceProviderEngineScope serviceProviderEngineScope)
at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.GetService(Type serviceType)
at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetService[T](IServiceProvider provider)
at Ocelot.Provider.Kubernetes.KubernetesProviderFactory.GetkubeProvider(IServiceProvider provider, ServiceProviderConfiguration config, String name, IOcelotLoggerFactory factory)
at Ocelot.ServiceDiscovery.ServiceDiscoveryProviderFactory.GetServiceDiscoveryProvider(ServiceProviderConfiguration config, String key)
at Ocelot.ServiceDiscovery.ServiceDiscoveryProviderFactory.Get(ServiceProviderConfiguration serviceConfig, DownstreamReRoute reRoute)
at Ocelot.LoadBalancer.LoadBalancers.LoadBalancerFactory.Get(DownstreamReRoute reRoute, ServiceProviderConfiguration config)
at Ocelot.LoadBalancer.LoadBalancers.LoadBalancerHouse.Get(DownstreamReRoute reRoute, ServiceProviderConfiguration config) <s:Development/Ocelot.LoadBalancer.Middleware.LoadBalancingMiddleware>
[19-08-01 11:02:11.11Z WRN] requestId: 0HLOMDUGQ7EF3:00000004, previousRequestId: no previous request id, message: Error Code: UnableToFindLoadBalancerError Message: unabe to find load balancer for /devices/{everything}|Get| exception is System.InvalidOperationException: Cannot resolve scoped service 'KubeClient.IKubeApiClient' from root provider.
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteValidator.ValidateResolution(Type serviceType, IServiceScope scope, IServiceScope rootScope)
at Microsoft.Extensions.DependencyInjection.ServiceProvider.Microsoft.Extensions.DependencyInjection.ServiceLookup.IServiceProviderEngineCallback.OnResolve(Type serviceType, IServiceScope scope)
at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngine.GetService(Type serviceType, ServiceProviderEngineScope serviceProviderEngineScope)
at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.GetService(Type serviceType)
at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetService[T](IServiceProvider provider)
at Ocelot.Provider.Kubernetes.KubernetesProviderFactory.GetkubeProvider(IServiceProvider provider, ServiceProviderConfiguration config, String name, IOcelotLoggerFactory factory)
at Ocelot.ServiceDiscovery.ServiceDiscoveryProviderFactory.GetServiceDiscoveryProvider(ServiceProviderConfiguration config, String key)
at Ocelot.ServiceDiscovery.ServiceDiscoveryProviderFactory.Get(ServiceProviderConfiguration serviceConfig, DownstreamReRoute reRoute)
at Ocelot.LoadBalancer.LoadBalancers.LoadBalancerFactory.Get(DownstreamReRoute reRoute, ServiceProviderConfiguration config)
at Ocelot.LoadBalancer.LoadBalancers.LoadBalancerHouse.Get(DownstreamReRoute reRoute, ServiceProviderConfiguration config) errors found in ResponderMiddleware. Setting error response for request path:/devices/v1/devices, request method: GET <s:Development/Ocelot.Responder.Middleware.ResponderMiddleware>

Steps to Reproduce the Problem

Follow this docs https://ocelot.readthedocs.io/en/latest/features/kubernetes.html

services.AddOcelot(_configuration).AddKubernetes(false); // or true

My service in k8s: devices-api.dev:10010

Config:

{
    "ReRoutes": [
        {
            "DownstreamPathTemplate": "/{everything}",
            "DownstreamScheme": "http",
            "UpstreamPathTemplate": "/devices/{everything}",
            "ServiceName": "devices-api.dev",
            "UpstreamHttpMethod": [ "Get" ]
        }
    ],
    "GlobalConfiguration": {
        "ServiceDiscoveryProvider": {
            "Namespace": "dev",
            "Type": "kube"
        }
    }
}

Specifications

• Version: Ocelot 13.5.2, Ocelot.Provider.Kubernetes 13.5.2
• Platform: Docker
• Subsystem: ASP.NET Core that runs in docker container in k8s cluster

[AmadorSV]

Have you managed to solve this? I'm having the same issue.

[JakubSlonka]

Any news regarding this issue? I am also having this problem.

[ussamoo]

This happen when the KubeClient is not able to load the KUBECONFIG from the cluster during his instantiation (Injection), for example when you are running on a local machine.

[sergiomcalzada]

That happens because ServiceDiscoveryFinderDelegate is registered as a singleton and the IKubeApiClient is registered as a scoped.
That error happens on AspNetCore 3.x

[neetra]

@Marusyk did you found any solution for this, I am facing this same issue

[Marusyk]

@neetra
no, I just don't use it. I run Ocelot in kubernetes but without AddKubernetes

[sergiomcalzada]

Hello, you cant register your own IServiceDiscoveryProvider that initiates a scope before resolving the IKubeApiClient.

Not so elegant because a scope is created every time but It works :)

Another option is to register manually the IKubeApiClient as a singleton, but it was a difficult task :/

public class KubeKubernetesServiceDiscoveryProvider : IServiceDiscoveryProvider
    {
        private readonly KubeRegistryConfiguration kubeRegistryConfiguration;
        private readonly IOcelotLogger logger;
        private readonly IServiceProvider provider;

        public KubeKubernetesServiceDiscoveryProvider(KubeRegistryConfiguration kubeRegistryConfiguration,
            IOcelotLoggerFactory factory, IServiceProvider provider)
        {
            this.kubeRegistryConfiguration = kubeRegistryConfiguration;
            this.provider = provider;
            this.logger = factory.CreateLogger<KubeKubernetesServiceDiscoveryProvider>();
        }

        public async Task<List<Service>> Get()
        {
            using (var scope = this.provider.CreateScope())
            {
                var kubeClient = scope.ServiceProvider.GetService<IKubeApiClient>();

                this.logger.LogDebug($"call service namespace:{this.kubeRegistryConfiguration.KubeNamespace} service:{this.kubeRegistryConfiguration.KeyOfServiceInK8s}");

                var service = await kubeClient.GetAsync(
                    this.kubeRegistryConfiguration.KeyOfServiceInK8s,
                        this.kubeRegistryConfiguration.KubeNamespace);


                var services = new List<Service>();
                if (service != null)
                {
                    services.Add(new Service(
                        service.Name,
                        new ServiceHostAndPort(service.Host, service.Port),
                        service.Id,
                        service.Version,
                        Enumerable.Empty<string>()));
                }
                else
                {
                    this.logger.LogWarning($"namespace:{this.kubeRegistryConfiguration.KubeNamespace} service:{this.kubeRegistryConfiguration.KeyOfServiceInK8s} not found");
                }

                return services;
            }
        }
    }

[neetra]

@sergiomcalzada

I created a file named KubeKubernetesServiceDiscoveryProvider and pasted above code, but not able to compile, can you please help me with the which version you are using (I am using Version=2.3.11.0)

_

'IKubeApiClient' does not contain a definition for 'GetAsync' and no accessible extension method 'GetAsync' accepting a first argument of type 'IKubeApiClient' could be found (are you missing a using directive or an assembly reference?)

_

[neetra]

This happen when the KubeClient is not able to load the KUBECONFIG from the cluster during his instantiation (Injection), for example when you are running on a local machine.

@ussamoo how do you come to know, how it is loaded?

[neetra]

@neetra
no, I just don't use it. I run Ocelot in kubernetes but without AddKubernetes

@Marusyk what about service discovery?

[sergiomcalzada]

Sorry, I have a custom "kube-api-envelop" and the code was copied from it and modified. But it is based in the original file. Instead injecting the "IKubeApiClient", inject the "IServiceProvider" and resolve in a new scope before using it. Something like this

 public class KubernetesServiceDiscoveryProvider : IServiceDiscoveryProvider
    {
        private readonly KubeRegistryConfiguration _kubeRegistryConfiguration;
        private readonly IOcelotLogger _logger;
        private readonly IServiceProvider provider;

        public KubernetesServiceDiscoveryProvider(KubeRegistryConfiguration kubeRegistryConfiguration, IOcelotLoggerFactory factory, IServiceProvider provider)
        {
            _kubeRegistryConfiguration = kubeRegistryConfiguration;
            _logger = factory.CreateLogger<KubernetesServiceDiscoveryProvider>();
            this.provider = provider;
        }

        public async Task<List<Service>> Get()
        {
			using(var scope = this.provider.CreateScope()) {
				var _kubeApi = scope.ServiceProvider.GetService<IKubeApiClient>();
				var endpoint = await _kubeApi
					.ResourceClient(client => new EndPointClientV1(client))
					.Get(_kubeRegistryConfiguration.KeyOfServiceInK8s, _kubeRegistryConfiguration.KubeNamespace);

				var services = new List<Service>();
				if (endpoint != null && endpoint.Subsets.Any())
				{
					services.AddRange(BuildServices(endpoint));
				}
				else
				{
					_logger.LogWarning($"namespace:{_kubeRegistryConfiguration.KubeNamespace }service:{_kubeRegistryConfiguration.KeyOfServiceInK8s} Unable to use ,it is invalid. Address must contain host only e.g. localhost and port must be greater than 0");
				}
				return services;
			}
            
        }

        private List<Service> BuildServices(EndpointsV1 endpoint)
        {
            var services = new List<Service>();

            foreach (var subset in endpoint.Subsets)
            {
                services.AddRange(subset.Addresses.Select(address => new Service(endpoint.Metadata.Name,
                    new ServiceHostAndPort(address.Ip, subset.Ports.First().Port),
                    endpoint.Metadata.Uid, string.Empty, Enumerable.Empty<string>())));
            }
            return services;
        }
    }

[sergiomcalzada]

The kube-client library is dotnet-kube-client and the file that register the IKubeClient in the injector is that extension

Here you can check from where it reads the configuration and that is registered a scoped services. Maybe check there if the default scope should be scoped or if they want to include a method to change scope.

[mot256]

We found that the scope error only occurres when environment variable ASPNETCORE_ENVIRONMENT = Development
I dug a bit deeper and by far the easiest workaround for the issue is by adding this to the host builder:

webBuilder.UseDefaultServiceProvider(options => options.ValidateScopes = false);

[sergiomcalzada]

Hi @mot256

If you set this setting to false you are going to have an exception at runtime ;)

[mot256]

Hi @mot256

If you set this setting to false you are going to have an exception at runtime ;)

Nope. Works just fine. Its without setting it to false in runtime that we get an issue...

[raman-m]

Reopened due #2178 (comment)

[raman-m]

@kick2nick, welcome!
Would you mind being assigned? How quickly can you deliver the fix?

[raman-m]

Discussed in #1948

• See Got error when using k8s service discovery #1948 (comment)

[raman-m]

Duplicate of #1470

[kick2nick]

Hi @raman-m, I'll take that. Need one day if we stick to solution proposed in #2178 (comment) - register IKubeApiClient as singleton as far as it already used like singleton.
What about Acceptance Tests? I need to enable scopes validation(hopes it won't break something else), but it also required in #2178 . Should I wait it?

[raman-m]

@mot256 commented on Feb 1, 2021

Validating scopes is essential, and DI-scopes should also be verified within your solutions.
Sergio has provided a suitable solution with provider.CreateScope()

[raman-m]

@kick2nick commented on Oct 21
Hi @raman-m, I'll take that.

Thank you for your intention to contribute!

Need one day if we stick to solution proposed in #2178 (comment) - register IKubeApiClient as singleton as far as it already used like singleton.

However, this is not the core issue. The problem lies in the incorrect implementation within the WithKubernetes and WithKubernetesAndRoundRobin helpers. It should be restructured using ClientRegistrationExtensions, recycling the options from the constructor as parameters for ClientRegistrationExtensions. Is the concept clear?

What about Acceptance Tests? I need to enable scopes validation (hopes it won't break something else)

Yes, you do! Please enable scope validation.

but it also required in #2178 . Should I wait it?

There's no need to wait for the issue mentioned in #2178. Your focus should be on Kube (not Consul) and conducting related tests with scope validation disabled. Here's how I see the development process:

• Write a test that reproduces the issue.
• Enable scope validation in the tests, which will likely cause them to fail.
• Implement a fix in the provider and/or the factory.

Again, waiting for anything related to #2178 is unnecessary.

[kick2nick]

Sergio has provided a #977 (comment) with provider.CreateScope()

I have some doubts about this approach:

• HttpClient MessageHandlers won't be reused - KubeClient creates new one for each KubeApiClient instance, that can cause socket exhaustion issues on heavy loads.
• This scope is separate from request HttpContext scope and no services would be potentially reused.
• There is no option to use long polling with this approach, HttpClient will be disposed with scope
  So my opinion is better leave KubeApiClient lifetime as is (it's now promoted to singleton), but make it explicit by registering it as singleton. But if you insist on Sergio's approach than no problem, I'll implement it :)