Better error message when private link enabled workspaces reject requests #290

mgyucht · 2024-05-17T12:44:44Z

Changes

Port of databricks/databricks-sdk-go#924 to the Java SDK.

When a user tries to access a Private Link-enabled workspace configured with no public internet access from a different network than the VPC endpoint belongs to, the Private Link backend redirects the user to the login page, rather than outright rejecting the request. The login page, however, is not a JSON document and cannot be parsed by the SDK, resulting in this error message:

$ databricks current-user me
Error: unexpected error handling request: invalid character '<' looking for beginning of value. This is likely a bug in the Databricks SDK for Go or the underlying REST API. Please report this issue with the following debugging information to the SDK issue tracker at https://github.com/databricks/databricks-sdk-go/issues. Request log:
GET /login.html?error=private-link-validation-error:<WSID>
> * Host: 
> * Accept: application/json
> * Authorization: REDACTED
> * Referer: https://adb-<WSID>.azuredatabricks.net/api/2.0/preview/scim/v2/Me
> * User-Agent: cli/0.0.0-dev+5ed10bb8ccc1 databricks-sdk-go/0.39.0 go/1.22.2 os/darwin cmd/current-user_me auth/pat
< HTTP/2.0 200 OK
< * Cache-Control: no-cache, no-store, must-revalidate
< * Content-Security-Policy: default-src *; font-src * data:; frame-src * blob:; img-src * blob: data:; media-src * data:; object-src 'none'; style-src * 'unsafe-inline'; worker-src * blob:; script-src 'self' 'unsafe-eval' 'unsafe-hashes' 'report-sample' https://*.databricks.com https://databricks.github.io/debug-bookmarklet/ https://widget.intercom.io https://js.intercomcdn.com https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js https://databricks-ui-assets.azureedge.net https://ui-serving-cdn-testing.azureedge.net https://uiserviceprodwestus-cdn-endpoint.azureedge.net https://databricks-ui-infra.s3.us-west-2.amazonaws.com 'sha256-47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=' 'sha256-YOlue469P2BtTMZYUFLupA2aOUsgc6B/TDewH7/Qz7s=' 'sha256-Lh4yp7cr3YOJ3MOn6erNz3E3WI0JA20mWV+0RuuviFM=' 'sha256-0jMhpY6PB/BTRDLWtfcjdwiHOpE+6rFk3ohvY6fbuHU='; report-uri /ui-csp-reports; frame-ancestors *.vocareum.com *.docebosaas.com *.edx.org *.deloitte.com *.cloudlabs.ai *.databricks.com *.myteksi.net
< * Content-Type: text/html; charset=utf-8
< * Date: Fri, 17 May 2024 07:47:38 GMT
< * Server: databricks
< * Set-Cookie: enable-armeria-workspace-server-for-ui-flags=false; Max-Age=1800; Expires=Fri, 17 May 2024 08:17:38 GMT; Secure; HTTPOnly; SameSite=Strict
< * Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
< * X-Content-Type-Options: nosniff
< * X-Ui-Svc: true
< * X-Xss-Protection: 1; mode=block
< <!doctype html>
< <html>
<  <head>
<   <meta charset="utf-8">
<   <meta http-equiv="Content-Language" content="en">
<   <title>Databricks - Sign In</title>
<   <meta name="viewport" content="width=960">
<   <link rel="icon" type="image/png" href="https://databricks-ui-assets.azureedge.net/favicon.ico">
<   <meta http-equiv="content-type" content="text/html; charset=UTF8">
<   <script id="__databricks_react_script"></script>
<   <script>window.__DATABRICKS_SAFE_FLAGS__={"databricks.infra.showErrorModalOnFetchError":true,"databricks.fe.infra.useReact18":true,"databricks.fe.infra.useReact18NewAPI":false,"databricks.fe.infra.fixConfigPrefetch":true},window.__DATABRICKS_CONFIG__={"publicPath":{"mlflow":"https://databricks-ui-assets.azureedge.net/","dbsql":"https://databricks-ui-assets.azureedge.net/","feature-store":"https://databricks-ui-assets.azureedge.net/","monolith":"https://databricks-ui-assets.azureedge.net/","jaws":"https://databricks-ui-assets.azureedge.net/"}}</script>
<   <link rel="icon" href="https://databricks-ui-assets.azureedge.net/favicon.ico">
<   <script>
<   function setNoCdnAndReload() {
<       document.cookie = `x-databricks-cdn-inaccessible=true; path=/; max-age=86400`;
<       const metric = 'cdnFallbackOccurred';
<       const browserUserAgent = navigator.userAgent;
<       const browserTabId = window.browserTabId;
<       const performanceEntry = performance.getEntriesByType('resource').filter(e => e.initiatorType === 'script').slice(-1)[0]
<       sessionStorage.setItem('databricks-cdn-fallback-telemetry-key', JSON.stringify({ tags: { browserUserAgent, browserTabId }, performanceEntry}));
<       window.location.reload();
<   }
< </script>
<   <script>
<   // Set a manual timeout for dropped packets to CDN
<   function loadScriptWithTimeout(src, timeout) {
<      return new Promise((resolve, reject) => {
<         const script = document.createElement('script');
<           script.defer = true;
<           script.src = src;
<           script.onload = resolve;
<           script.onerror = reject;
<           document.head.appendChild(script);
<           setTimeout(() => {
<               reject(new Error('Script load timeout'));
<           }, timeout);
<       });
<   }
<   loadScriptWithTimeout('https://databricks-ui-assets.azureedge.net/static/js/login/login.8a983ca2.js', 10000).catch(setNoCdnAndReload);
< </script>
<  </head>
<  <body class="light-mode">
<   <uses-legacy-bootstrap>
<    <div id="login-page"></div>
<   </uses-legacy-bootstrap>
<  </body>
< </html>

To address this, I add one additional check in the error mapper logic to inspect whether the user was redirected to the login page with the private link validation error response code. If so, we return a synthetic error with error code PRIVATE_LINK_VALIDATION_ERROR that inherits from PermissionDenied and has a mock 403 status code.

After this change, users will see an error message like this:

Exception in thread "main" com.databricks.sdk.core.error.PrivateLinkValidationError: The requested workspace has Azure Private Link enabled and is not accessible from the current network. Ensure that Azure Private Link is properly configured and that your device has access to the Azure Private Link endpoint. For more information, see https://learn.microsoft.com/en-us/azure/databricks/security/network/classic/private-link-standard#authentication-troubleshooting.
	at com.databricks.sdk.core.error.PrivateLinkInfo.createPrivateLinkValidationError(PrivateLinkInfo.java:63)
	at com.databricks.sdk.core.error.AbstractErrorMapper.apply(AbstractErrorMapper.java:46)
	at com.databricks.sdk.core.error.ApiErrors.getDatabricksError(ApiErrors.java:29)
	at com.databricks.sdk.core.ApiClient.executeInner(ApiClient.java:276)
	at com.databricks.sdk.core.ApiClient.getResponse(ApiClient.java:235)
	at com.databricks.sdk.core.ApiClient.execute(ApiClient.java:227)
	at com.databricks.sdk.core.ApiClient.GET(ApiClient.java:148)
	at com.databricks.sdk.service.compute.ClustersImpl.list(ClustersImpl.java:94)
	at com.databricks.sdk.support.Paginator.flipNextPage(Paginator.java:58)
	at com.databricks.sdk.support.Paginator.<init>(Paginator.java:51)
	at com.databricks.sdk.service.compute.ClustersAPI.list(ClustersAPI.java:295)
	at com.databricks.sdk.examples.ListClustersExample.main(ListClustersExample.java:11)

The error message is tuned to the specific cloud so that we can redirect users to the appropriate documentation, the cloud being inferred from the request URI.

Tests

Unit tests cover the private link error message mapping.
To manually test this, I created a private link workspace in Azure, created an access token, restricted access to the workspace, then ran the ListClustersExample example using the host & token:

<SNIP>
14:41 [DEBUG] > GET /api/2.0/clusters/list
< 200 OK
< <!doctype html>
< <html>
<  <head>
<   <meta charset="utf-8">
<   <meta http-equiv="Content-Language" content="en">
<   <title>Databricks - Sign In</title>
<   <meta name="viewport" content="width=960">
<   <link rel="icon" type="image/png" href="https://databricks-ui-assets.azureedge.net/favicon.ico">
<   <meta http-equiv="content-type" content="text/html; charset=UTF8">
<   <script id="__databricks_react_script"></script>
<   <script>window.__DATABRICKS_SAFE_FLAGS__={"databricks.infra.showErrorModalOnFetchError":true,"databricks.fe.infra.useReact18":true,"databricks.fe.infra.useReact18NewAPI":false,"databricks.fe.infra.fixConfigPrefetch":true},window.__DATABRICKS_CONFIG__={"publicPath":{"mlflow":"https://databricks-ui-assets.azureedge.net/","dbsql":"https://databricks-ui-assets.azureedge.net/","feature-store":"https://databricks-ui-assets.azureedge.net/","monolith":"https://databricks-ui-assets.azureedge.net/","jaws":"https://databricks-ui-assets.azureedge.net/"}}</script>
<   <link rel="icon" href="https://databricks-ui-assets.... (1420 more bytes)
Exception in thread "main" com.databricks.sdk.core.error.PrivateLinkValidationError: The requested workspace has Azure Private Link enabled and is not accessible from the current network. Ensure that Azure Private Link is properly configured and that your device has access to the Azure Private Link endpoint. For more information, see https://learn.microsoft.com/en-us/azure/databricks/security/network/classic/private-link-standard#authentication-troubleshooting.
	at com.databricks.sdk.core.error.PrivateLinkInfo.createPrivateLinkValidationError(PrivateLinkInfo.java:63)
	at com.databricks.sdk.core.error.AbstractErrorMapper.apply(AbstractErrorMapper.java:46)
	at com.databricks.sdk.core.error.ApiErrors.getDatabricksError(ApiErrors.java:29)
	at com.databricks.sdk.core.ApiClient.executeInner(ApiClient.java:276)
	at com.databricks.sdk.core.ApiClient.getResponse(ApiClient.java:235)
	at com.databricks.sdk.core.ApiClient.execute(ApiClient.java:227)
	at com.databricks.sdk.core.ApiClient.GET(ApiClient.java:148)
	at com.databricks.sdk.service.compute.ClustersImpl.list(ClustersImpl.java:94)
	at com.databricks.sdk.support.Paginator.flipNextPage(Paginator.java:58)
	at com.databricks.sdk.support.Paginator.<init>(Paginator.java:51)
	at com.databricks.sdk.service.compute.ClustersAPI.list(ClustersAPI.java:295)
	at com.databricks.sdk.examples.ListClustersExample.main(ListClustersExample.java:11)
Disconnected from the target VM, address: '127.0.0.1:55559', transport: 'socket'

Process finished with exit code 1
exit status 2

gp

hectorcast-db

We should add some unit tests here too.

mgyucht · 2024-05-17T14:22:19Z

Added unit tests

### Improvements * Better error message when private link enabled workspaces reject requests ([#290](#290)). ### API Changes: * Changed `list()` method for `workspaceClient.connections()` service to require request of `com.databricks.sdk.service.catalog.ListConnectionsRequest` class. * Renamed `workspaceClient.lakehouseMonitors()` service to `workspaceClient.qualityMonitors()`. * Renamed `com.databricks.sdk.service.catalog.DeleteLakehouseMonitorRequest` class to `com.databricks.sdk.service.catalog.DeleteQualityMonitorRequest`. * Changed `schemaName` field for `com.databricks.sdk.service.catalog.DisableRequest` to `String` class. * Removed `com.databricks.sdk.service.catalog.DisableSchemaName` class. * Changed `schemaName` field for `com.databricks.sdk.service.catalog.EnableRequest` to `String` class. * Removed `com.databricks.sdk.service.catalog.EnableSchemaName` class. * Renamed `com.databricks.sdk.service.catalog.GetLakehouseMonitorRequest` class to `com.databricks.sdk.service.catalog.GetQualityMonitorRequest`. * Added `nextPageToken` field for `com.databricks.sdk.service.catalog.ListConnectionsResponse`. * Added `dashboardId` field for `com.databricks.sdk.service.catalog.UpdateMonitor`. * Added `com.databricks.sdk.service.catalog.ListConnectionsRequest` class. * Added `com.databricks.sdk.service.catalog.MonitorRefreshListResponse` class. * Changed `clusterStatus()` method for `workspaceClient.libraries()` service to return `com.databricks.sdk.service.compute.ClusterLibraryStatuses` class. * Removed `clusterSource` field for `com.databricks.sdk.service.compute.ClusterAttributes`. * Changed `spec` field for `com.databricks.sdk.service.compute.ClusterDetails` to `com.databricks.sdk.service.compute.ClusterSpec` class. * Removed `cloneFrom` and `clusterSource` fields for `com.databricks.sdk.service.compute.ClusterSpec`. * Removed `com.databricks.sdk.service.compute.ClusterStatusResponse` class. * Removed `clusterSource` field for `com.databricks.sdk.service.compute.CreateCluster`. * Removed `cloneFrom` and `clusterSource` fields for `com.databricks.sdk.service.compute.EditCluster`. * Removed `sortBySpec` field for `com.databricks.sdk.service.marketplace.ListListingsRequest`. * Added `isAscending` field for `com.databricks.sdk.service.marketplace.ListListingsRequest`. * Added `sortBy` field for `com.databricks.sdk.service.marketplace.ListListingsRequest`. * Added `isAscending` field for `com.databricks.sdk.service.marketplace.SearchListingsRequest`. * Removed `com.databricks.sdk.service.marketplace.SortBySpec` and `com.databricks.sdk.service.marketplace.SortOrder` classes. * Added `gatewayDefinition` field for `com.databricks.sdk.service.pipelines.CreatePipeline`. * Added `gatewayDefinition` field for `com.databricks.sdk.service.pipelines.EditPipeline`. * Added `tableConfiguration` field for `com.databricks.sdk.service.pipelines.ManagedIngestionPipelineDefinition`. * Added `gatewayDefinition` field for `com.databricks.sdk.service.pipelines.PipelineSpec`. * Added `tableConfiguration` field for `com.databricks.sdk.service.pipelines.SchemaSpec`. * Added `tableConfiguration` field for `com.databricks.sdk.service.pipelines.TableSpec`. * Added `com.databricks.sdk.service.pipelines.IngestionGatewayPipelineDefinition` class. * Added `com.databricks.sdk.service.pipelines.TableSpecificConfig` class. * Added `com.databricks.sdk.service.pipelines.TableSpecificConfigScdType` class. * Added `deploymentArtifacts` field for `com.databricks.sdk.service.serving.AppDeployment`. * Added `contents` field for `com.databricks.sdk.service.serving.ExportMetricsResponse`. * Changed `openaiApiKey` field for `com.databricks.sdk.service.serving.OpenAiConfig` to no longer be required. * Added `microsoftEntraClientId`, `microsoftEntraClientSecret` and `microsoftEntraTenantId` fields for `com.databricks.sdk.service.serving.OpenAiConfig`. * Added `com.databricks.sdk.service.serving.AppDeploymentArtifacts` class. * Added `storageRoot` field for `com.databricks.sdk.service.sharing.CreateShare`. * Added `storageLocation` and `storageRoot` fields for `com.databricks.sdk.service.sharing.ShareInfo`. * Added `storageRoot` field for `com.databricks.sdk.service.sharing.UpdateShare`. * Added `scanIndex()` method for `workspaceClient.vectorSearchIndexes()` service. * Added `embeddingWritebackTable` field for `com.databricks.sdk.service.vectorsearch.DeltaSyncVectorIndexSpecRequest`. * Added `embeddingWritebackTable` field for `com.databricks.sdk.service.vectorsearch.DeltaSyncVectorIndexSpecResponse`. * Added `com.databricks.sdk.service.vectorsearch.ListValue`, `com.databricks.sdk.service.vectorsearch.MapStringValueEntry`, `com.databricks.sdk.service.vectorsearch.ScanVectorIndexRequest`, `com.databricks.sdk.service.vectorsearch.ScanVectorIndexResponse`, `com.databricks.sdk.service.vectorsearch.Struct`and `com.databricks.sdk.service.vectorsearch.Value` classes. OpenAPI SHA: 7eb5ad9a2ed3e3f1055968a2d1014ac92c06fe92, Date: 2024-05-21

…ient` (#305) ## Changes This PR addresses a regression reportedly introduced in PR #290 when connecting via a proxy that requires authentication (see context). The problem comes from a casting exception in `CommonHttpClient` which attempts to cast a `BasicHttpRequest` (which represents the next request after receiving the auth required response) into a `HttpUriRequest`. This PR solves the casting issue by processing the ancestor of both classes, `HttpRequest`. ### Context The regression was detected when trying to authenticate via a Proxy that requires authentication. In particular, the request to respond to the proxy's authentication request is automatically created as a `BasicHttpRequest` by the apache `HttpClient`. ## Tests Unit tests and integration tests are passing. The fix was also verified by the user who uncovered the issue. Note: we should add a regression test that simulate the proxy setting. However, I'd like to take the time to experiment with different configurations. Given that the change is relatively simple, I'd recommend to proceed with this PR and add the test in a follow-up PR.

mgyucht added 3 commits May 17, 2024 13:21

private link

f65191d

fixes

6585e17

work

47648eb

gp

mgyucht requested a review from hectorcast-db May 17, 2024 13:44

fmt

05c2f42

hectorcast-db approved these changes May 17, 2024

View reviewed changes

mgyucht added 2 commits May 17, 2024 16:11

add unit test for ApiClient behavior

8efa2b0

unit tests

20e640a

fmt

a0e7d5d

mgyucht enabled auto-merge May 17, 2024 14:22

mgyucht added this pull request to the merge queue May 17, 2024

Merged via the queue into main with commit e03c05e May 17, 2024
9 checks passed

mgyucht deleted the private-link branch May 17, 2024 14:27

hectorcast-db mentioned this pull request May 22, 2024

Release v0.26.0 #294

Merged

renaudhartert-db mentioned this pull request Jul 5, 2024

Accept any HttpRequest instead of HttpUriRequest in CommonHttpClient #305

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Better error message when private link enabled workspaces reject requests #290

Better error message when private link enabled workspaces reject requests #290

mgyucht commented May 17, 2024

hectorcast-db left a comment

mgyucht commented May 17, 2024

Better error message when private link enabled workspaces reject requests #290

Better error message when private link enabled workspaces reject requests #290

Conversation

mgyucht commented May 17, 2024

Changes

Tests

hectorcast-db left a comment

Choose a reason for hiding this comment

mgyucht commented May 17, 2024