Unable to skip SSL authentication when connecting to Emulator running in Linux Docker Container

[davidguidali]

Hi there
I am developing a C# .NET Core app (on windows) and want to connect to the cosmos emulator running in a linux docker container on docker desktop for windows. I don't want to use SSL authentication so I try to make use of the most recent CosmosClient of Microsoft.Azure.Cosmos package. However, this doesn't seem to work.

Steps to reproduce:

On a windows machine, pull the most recent image of the cosmos emulator

docker pull mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator

Start up a new emulator container

docker run -p 8081:8081 -p 10251:10251 -p 10252:10252 -p 10253:10253 -p 10254:10254 -m 3g --cpus=2.0 --name=test-linux-emulator -e AZURE_COSMOS_EMULATOR_PARTITION_COUNT=10 -e AZURE_COSMOS_EMULATOR_ENABLE_DATA_PERSISTENCE=true -it mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator

Navigate to https://localhost:8081/_explorer/index.html and download the sample app for C# .NET Core.

Open the sample app in Visual Studio and update the Nuget package Microsoft.Azure.Cosmos to Version 3.20.1.

Open the class Program.cs and change the method GetStartedDemoAsync to skip SSL authentication:

public async Task GetStartedDemoAsync()
{
   CosmosClientOptions cosmosClientOptions = new CosmosClientOptions()
   {
       HttpClientFactory = () =>
       {
           HttpMessageHandler httpMessageHandler = new HttpClientHandler()
           {
               ServerCertificateCustomValidationCallback = HttpClientHandler.DangerousAcceptAnyServerCertificateValidator
           };
           return new HttpClient(httpMessageHandler);
       },
       ConnectionMode = ConnectionMode.Gateway
   };

   // Create a new instance of the Cosmos Client
   this.cosmosClient = new CosmosClient(EndpointUri, PrimaryKey, cosmosClientOptions);
   await this.CreateDatabaseAsync();
   await this.CreateContainerAsync();
   await this.ScaleContainerAsync();
   await this.AddItemsToContainerAsync();
   await this.QueryItemsAsync();
   await this.ReplaceFamilyItemAsync();
   await this.DeleteFamilyItemAsync();
   await this.DeleteDatabaseAndCleanupAsync();
}

Then hit F5 to run the program.

Expected result
The program runs successfully.

Actual result
The program hangs in this.CreateDatabaseAsync();, never returning and never creating a database.

PS: I posted this also on the emulator github project but I think here its the better place..

[j82w]

What is likely happening is the SDK is not able to connect to the emulator. Please take a look at this issue: #1551

[davidguidali]

What is likely happening is the SDK is not able to connect to the emulator. Please take a look at this issue: #1551

This does not solve my issue. I already used the code mentioned in #1551. I can browse the cosmos db in the browser, however I can't connect via the C# client.
Also my case is slightly different: I try to connect from an app (on my host) to the linux emulator (in docker), not vice versa.
Container to container connection works, and as far as the topic goes, container to host connection should also be possible. However, I'm struggling with the host to container connection.

Fun fact: It works when running the emulator as a Windows container, but not when running as a Linux container. I want to use the linux container.

[ealsur]

Can you share the actual error you are getting? The program cannot just hang, can you leave it running for some minutes?

This might not be an SSL issue, but rather, the application not able to contact the endpoint, which has nothing to do with the SDK or SSL configuration.

[davidguidali]

Well the endpoint is there, i can browse it via

https://localhost:8081/_explorer/index.html

and also via

https://myip:8081/_explorer/index.html

There is no error message, even though I let it run for multiple minutes.

If i stop the linux emulator, switch to windows containers and run the windows emulator, it works w/o issues.

[ealsur]

The only thing I can think of is some issue on the Linux Emulator (which is still in preview?)

If this was an issue on the SDK, it would repro on a real account and any of the emulators.

Have you tried running the app outside of VS? Maybe VS is breaking on an exception and it feels like it's doing nothing?

[davidguidali]

You're right, I also think now it might be an issue with the emulator, since the official page only describes how to use the linux emulator on a linux host but not on a windows host. I will raise an issue to the emulator team. However, it still is strange that theres no error coming, even if I run it outside of VS as you suggested.

[ealsur]

@davidguidali Was there any progress with the Linux emulator?

[ealsur]

Closing due to inactivity, please reopen if there is any new finding

[alexanderbikk]

hi @ealsur I got exactly the same issue and behavior. I'm trying to connect to cosmosdb linux emulator(docker) from host (.net 6 app) suing SDK v3. No exceptions at all.
I see that someone else also faced this issue.
I think this issue should be linked.

Probably you are right and this is problems related to cosmosdb linux emulator since it's still in preview.

[ealsur]

Assuming you are also setting the ServerCertificateCustomValidationCallback, this would make the client HttpClient not check for SSL validation, could be that the problem is not really the SSL certificate, but something networking related and the HttpClient simply cannot connect to the docker Linux emulator

[alexanderbikk]

@ealsur exactly. What I can see in debug that time to time my break point triggered here

So the program hangs on cosmos client call. Looks like endless retry it's hard to say since I see no errors.

UPD: is it hard to use generated certificate(in linux container) for windows? For my case I should use it in CI with linux build agent as well.
So it will be much easier to skip ssl validation but seems like it's not possible for now. Maybe windows image could be an option.

[lstorka]

You're right, I also think now it might be an issue with the emulator, since the official page only describes how to use the linux emulator on a linux host but not on a windows host. I will raise an issue to the emulator team. However, it still is strange that theres no error coming, even if I run it outside of VS as you suggested.

@davidguidali I am also struggling with this issue, but my setup is a bit different as I have Docker on Ubuntu on WSL2 and started the emulator according to the documentation link
Now, when I try to open https://localhost:8081/_explorer/index.html on Windows - I get the Cosmos DB explorer and I am able to create db, container etc.
When I try to run my test app on Windows and create some db or container, then I get following errors (after some time):

DocDBTrace Information: 0 : Current WriteEndpoints = (https://172.17.0.1:8081/) ReadEndpoints = (https://172.17.0.1:8081/)
DocDBTrace Information: 0 : GlobalEndpointManager: StartLocationBackgroundRefreshWithTimer() refreshing locations
DocDBTrace Warning: 0 : ClientRetryPolicy: Gateway HttpRequestException Endpoint not reachable. Failed Location: https://172.17.0.1:8081/; ResourceAddress: dbs/twix
DocDBTrace Information: 0 : GlobalEndpointManager: Marking endpoint https://172.17.0.1:8081/ unavailable for read
DocDBTrace Information: 0 : Current WriteEndpoints = (https://172.17.0.1:8081/) ReadEndpoints = (https://172.17.0.1:8081/)
DocDBTrace Information: 0 : Endpoint https://172.17.0.1:8081/ unavailable for Read added/updated to unavailableEndpoints with timestamp 08/18/2022 07:57:04
DocDBTrace Information: 0 : GlobalEndpointManager: Marking endpoint https://172.17.0.1:8081/ unavailable for Write
DocDBTrace Information: 0 : Current WriteEndpoints = (https://172.17.0.1:8081/) ReadEndpoints = (https://172.17.0.1:8081/)
DocDBTrace Information: 0 : Endpoint https://172.17.0.1:8081/ unavailable for Write added/updated to unavailableEndpoints with timestamp 08/18/2022 07:57:04
DocDBTrace Information: 0 : Current WriteEndpoints = (https://172.17.0.1:8081/) ReadEndpoints = (https://172.17.0.1:8081/)

One interesting thing is that the localhost was transformed into the ip of the Gateway of the Cosmos DB container.

Third scenario is when I run the exact same app on Ubuntu/WSL2 - everything works and the application is creating the db and containers without any problems. So I think that confirms your assumptions, that Cosmos DB Emulator for Linux is limited to use only on that OS...

[worldspawn]

I have the exact same problem. Emulator running in wsl (latest tag) and client 3.32.1. Client code is running on windows.

My debug output is looping this:

DocDBTrace Warning: 0 : ClientRetryPolicy: Gateway HttpRequestException Endpoint not reachable. Failed Location: https://172.17.0.6:8081/; ResourceAddress: 
DocDBTrace Information: 0 : GlobalEndpointManager: Marking endpoint https://172.17.0.6:8081/ unavailable for read
DocDBTrace Information: 0 : Current WriteEndpoints = (https://172.17.0.6:8081/) ReadEndpoints = (https://172.17.0.6:8081/)
DocDBTrace Information: 0 : Endpoint https://172.17.0.6:8081/ unavailable for Read added/updated to unavailableEndpoints with timestamp 03/03/2023 00:02:55
DocDBTrace Information: 0 : GlobalEndpointManager: Marking endpoint https://172.17.0.6:8081/ unavailable for Write
DocDBTrace Information: 0 : Current WriteEndpoints = (https://172.17.0.6:8081/) ReadEndpoints = (https://172.17.0.6:8081/)
DocDBTrace Information: 0 : Endpoint https://172.17.0.6:8081/ unavailable for Write added/updated to unavailableEndpoints with timestamp 03/03/2023 00:02:55
DocDBTrace Information: 0 : ClientRetryPolicy: Failover happening. retryCount 9
DocDBTrace Information: 0 : Current WriteEndpoints = (https://172.17.0.6:8081/) ReadEndpoints = (https://172.17.0.6:8081/)

172.17.0.6 is the ip of the container the emulator is running in.

Client code:

var cosmosOptions = new CosmosClientOptions();
        var httpMessageHandler = new HttpClientHandler()
        {
            ServerCertificateCustomValidationCallback = (req, cert, chain, errors) => true
        };

        cosmosOptions.HttpClientFactory = () => new HttpClient(httpMessageHandler);
        cosmosOptions.ConnectionMode = ConnectionMode.Gateway;
        
        var client = new CosmosClient("https://127.0.0.1:8081/", "C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw==", cosmosOptions);
        var database = await client.CreateDatabaseAsync("accounts");

The awaited call to client.CreateDatabaseAsync never returns. There is no output in the emulators console, it just says:

This is an evaluation version.  There are [72] days left in the evaluation period.

Starting
Started 1/11 partitions
Started 2/11 partitions
Started 3/11 partitions
Started 4/11 partitions
Started 5/11 partitions
Started 6/11 partitions
Started 7/11 partitions
Started 8/11 partitions
Started 9/11 partitions
Started 10/11 partitions
Started 11/11 partitions
Started

Everything works fine making direct REST calls.

[worldspawn]

@ealsur is there anyway to resolve this?

[worldspawn]

var addresses = Dns.GetHostAddresses("host.docker.internal");

.WithEnvironment("AZURE_COSMOS_EMULATOR_IP_ADDRESS_OVERRIDE", addresses[0].ToString())

Low and behold the solution has been available this whole time. But because there is no kind of reference documentation for the docker image (just a bunch of learn articles) no one knows wtf all the env variables are or what they do. Waste of a morning.

[rapontual]

@worldspawn I'm running on the same issue
I read your answer but I didn't manage where to use the .WithEnvironment extension method. I'm using the project I've downloaded from the emulator ("CosmosGettingStarted"). Since it starts the Program I don't know exactly where to apply your piece of code, could you give me more details pls?
Tks

[worldspawn]

Those are testcontainers method. So it starts cosmos in Docker. If you are running as a process on your host machine you just need to set the same environment variable

[andrewtek]

I was experiencing the same issue. I worked around the issue with this:

	CosmosClientOptions cosmosClientOptions = new()
		{
			HttpClientFactory = () => new HttpClient(new HttpClientHandler()
			{
				ServerCertificateCustomValidationCallback = HttpClientHandler.DangerousAcceptAnyServerCertificateValidator
			}),
			ConnectionMode = ConnectionMode.Gateway,
			LimitToEndpoint = true
		};

I noticed that what was causing the error was that the CosmosDB emulator was giving out endpoints of its own for reading and writing. Unfortunately, those endpoints are not accessible on the host machine as the IPs are on the private Docker network.

By adding LimitToEndpoint = true, those inaccessible endpoints are ignored. I am guessing it is possible to start the CosmosDB emulator in a way to emit ports on the host.docker.internal IP, but this flag got me up and running.