IMPORT PGDUMP cannot import from Azure Blob Storage in Government

[nlowe]

Describe the problem

It would appear that the Account Base URL for Azure Storage is hard-coded to the Azure Public Cloud end-point:

cockroach/pkg/ccl/storageccl/export_storage.go

Line 767 in f0dd1dc

u, err := url.Parse(fmt.Sprintf("https://%s.blob.core.windows.net", conf.AccountName))

(more recently:

cockroach/pkg/storage/cloud/azure_storage.go

Line 59 in c257a8e

u, err := url.Parse(fmt.Sprintf("https://%s.blob.core.windows.net", conf.AccountName))

)).

This prevents consumers with blob storage in Azure Government Cloud from restoring backups from their blob storage accounts.

To Reproduce

NOTE: All Azure Services in the following steps were created in the Azure US Government Cloud

1. Setup a Secure CockroachDB Cluster in Azure AKS in following https://github.com/cockroachdb/cockroach/tree/master/cloud/kubernetes

2. Create an Azure Storage Blob Account

3. Create an Azure Storage Blob Container in the account created in step 2

4. Upload backup.sql taken from cockroach dump to the storage container

5. Create the database via cockroach sql:

    CREATE DATABASE test;

6. Attempt to restore the database:

   IMPORT PGDUMP 'azure://mycontainer/test.sql?AZURE_ACCOUNT_NAME=myaccount&AZURE_ACCOUNT_KEY=***' WITH max_row_size='100MB';
   
   Error: pq: failed to create azure reader: -> github.com/cockroachdb/cockroach/vendor/github.com/Azure/azure-pipeline-go/pipeline.newDefaultHTTPClientFactory.func1.1, /go/src/github.com/cockroachdb/cockroach/vendor/github.com/Azure/azure-pipeline-go/pipeline/core.go:234
   HTTP request failed
   
   Get https://myaccount.blob.core.windows.net/mycontainer/test.sql?timeout=61: dial tcp: lookup myaccount.blob.core.windows.net on 192.168.0.10:53: no such host
   
   Failed running "sql"

Expected behavior

A user is either able to specify the full path to their storage account or force CockroachDB to look for storage accounts in different azure environments like US Government Cloud.

Additional data / screenshots

I also tried specifying the full account URL:

IMPORT PGDUMP 'azure://myaccount.blob.core.usgovcloudapi.net/mycontainer/test.sql?AZURE_ACCOUNT_NAME=myaccount&AZURE_ACCOUNT_KEY=***' WITH max_row_size='100MB';

but that clearly wasn't interpreted correctly:

Get https://myaccount.blob.core.windows.net/mycontainer.blob.core.usgovcloudapi.net/test.sql?timeout=61: dial tcp: lookup myaccount.blob.core.windows.net on 192.168.0.10:53: no such host

After finding this Stack Overflow Post about using the Azure GO SDKs with Azure Government, I even tried setting AZURE_ENVIRONMENT to AZUREUSGOVERNMENTCLOUD on all of my nodes including the client I was using to execute the IMPORT PGDUMP. This had no effect.

Environment:

• CockroachDB version: v2.1.0
• Server OS: Official Docker Containers in Kubernetes 1.16
• Client app: cockroach sql

Additional context

Another method like HTTP or Node Local files will have to be used for backup/restore.

Jira issue: CRDB-5033