cannot read storage credential: unexpected error handling request: invalid character '<' looking for beginning of value. #982

kusum91 · 2024-07-12T15:17:45Z

Description
I'm trying to create storage credential, external location and catalog for unity enabled workspace using Terraform, the storage credential is created however the plan fails in the next run when it has to fetch the credential details for creating
external location.

Error message
│ Error: cannot read storage credential: unexpected error handling request: invalid character '<' looking for beginning of value. This is likely a bug in the Databricks SDK for Go or the underlying REST API. Please report this issue with the following debugging information to the SDK issue tracker at https://github.com/databricks/databricks-sdk-go/issues. Request log:
│ GET /login?next_url=/api/2.1/unity-catalog/storage-credentials/dev-unity
│ > * Host:
│ > * Accept: application/json
│ > * Authorization: REDACTED
│ > * Referer: https://accounts.azuredatabricks.net/api/2.1/unity-catalog/storage-credentials/dev-unity?
│ > * Traceparent: 00-e8da3f6ff962525a056d5e9d28f8843a-e3a2f94912ea5300-01
│ > * User-Agent: databricks-tf-provider/1.48.1 databricks-sdk-go/0.43.0 go/1.22.4 os/linux terraform/1.5.7 resource/storage_credential auth/azure-cli cicd/azure-devops
│ > * X-Databricks-Azure-Sp-Management-Token: eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6Ik1HTHFqOThWTkxvWGFGZnBKQ0JwZ0I0SmFLcyIsImtpZCI6Ik1HTHFqOThWTkxvWGFGZnBKQ0JwZ0I0SmFLcyJ9.eyJhdWQiOiJodHRwczovL21hbmFnZW1lbnQuY29yZS53aW5kb3dzLm5ldC8iLCJpc3MiOiJodHRwczovL3N0cy53aW5kb3dzLm5ldC82ZTkzYTYyNi04YWNhLTRkYzEtOTE5MS1jZTI5MWI0Yjc1YTEvIiwiaWF0IjoxNzIwNzk2MzE4LCJuYmYiOjE3MjA3OTYzMTgsImV4cCI6MTcyMDgwMDIxOCwiYWlvIjoiQVNRQTIvOFhBQUFBa2w2ZHBKRVV3VVd2QnAvK3NqK09Pbll4azFnRGxUQjJ2Tm5VRGw4K1RMOD0iLCJhcHBpZCI6IjZjNTYxODZkLTIyOTItNDcxOC1hMWQxLWU0MjI1YTYwMmNhNSIsImFwcGlkYWNyIjoiMiIsImdyb3VwcyI6WyI4ZTA0NmEwMi0yYTk5LTQ3ZTQtODg1NC0zZGU0MTJlNzhmODYiLCJhOWQ5NTkxMC03YTAyLTRiZDUtYTQ1NC0wYjE2ZmYzNWQxNGEiLCJkYzdjN2JhMi1mYmUwLTQxNjgtYjBjYy03Y2QwZGY1MmY1NzAiLCI2MzU2ODRhZS04MzY3LTQ0MGMtYWU1OS0yOTg2NWVkZTYzNmMiXSwiaWRwIjoiaHR0cHM6Ly9zdHMud2luZG93cy5uZXQvNmU5M2E2MjYtOGFjYS00ZGMxLTkxOTEtY2UyOTFiNGI3NWExLyIsImlkdHlwIjoiYXBwIiwib2lkIjoiYjZkY2M5MzctN2YwYy00NmNlLTkwMDYtN2U2ODAzMjI1Y2NmIiwicmgiOiIwLkFSQUFKcWFUYnNxS3dVMlJrYzRwRzB0MW9VWklmM2tBdXRkUHVrUGF3ZmoyTUJNUUFBQS4iLCJzdWIiOiJiNmRjYzkzNy03ZjBjLTQ2Y2UtOTAwNi03ZTY4MDMyMjVjY2YiLCJ0aWQiOiI2ZTkzYTYyNi04YWNhLTRkYzEtOTE5MS1jZTI5MWI0Yjc1YTEiLCJ1dGkiOiJUQTRLYlZXXzNrU080dzNORnU4dkFBIiwidmVyIjoiMS4wIiwieG1zX2NhZSI6IjEiLCJ4bXNfY2MiOlsiQ1AxIl0sInhtc19pZHJlbCI6IjIgNyIsInhtc19yZCI6IjAuNDJMbFlCSmlGQUFBIiwieG1zX3NzbSI6IjEiLCJ4bXNfdGNkdCI6MTM5NjYxNDA0Mn0.XVhMrA7MuWwijMH8morT2N2ws4bbtEH4npsPF8Lrec7msnEfM00PRlXBA-b8hc_6O5ooG3fxYpw4PvzI4JU-MfmI0jTgdWoNv0eqb2MGTqoptzo5A56XJjL9PMAKrNag-mroxgi_mbWUhegUF9jbZvaNvB7LbWwiN78RbPrhtQXH1cZ9pVy3I9meSZMHLBq_CqKrcDfQiVvmWXGW594M-EGlP9yE8Ayjjmvx_Dow0f9u85s00Dyx3skqMWpwfdmgS3s1OIc_V3egTIQ9Ldop0UWt1blhHYXE7KkM0NxwJJLqJa4P89_Wm36zGgjYAsGPg63-kpapfGQb96WQ5i9rYg
│ < HTTP/2.0 200 OK
│ < * Cache-Control: no-cache, no-store, must-revalidate
│ < * Content-Type: text/html; charset=utf-8
│ < * Date: Fri, 12 Jul 2024 15:03:46 GMT
│ < * Server: databricks
│ < * Set-Cookie: enable-armeria-server-for-ui-flags=false; Max-Age=1800; Expires=Fri, 12 Jul 2024 15:33:46 GMT; Secure; HTTPOnly; SameSite=Strictenable-armeria-workspace-server-for-ui-flags=false; Max-Age=1800; Expires=Fri, 12 Jul 2024 15:33:46 GMT; Secure; HTTPOnly; SameSite=Strict
│ < * Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
│ < * X-Content-Type-Options: nosniff
│ < * X-Ui-Svc: true
│ < <!doctype html>
│ <
│ <
│ <
│ <
│ <
│ <
│ < <title>Databricks - Sign in</title>
│ < <script>window.DATABRICKS_CONFIG={"isCuttingEdge":false,"publicPath":{"accounts-console":"https://databricks-ui-assets.azureedge.net/"}}</script>
│ < ](https://databricks-ui-assets.azureedge.net/favicon.ico%22%3E)
│ < <script defer src="[https://databricks-ui-assets.azureedge.net/static/js/1793.d7fb1564.js"></script>](https://databricks-ui-assets.azureedge.net/static/js/1793.d7fb1564.js%22%3E%3C/script%3E)
│ < <script defer src="[https://databricks-ui-assets.azureedge.net/static/js/6001.e71693bb.js"></script>](https://databricks-ui-assets.azureedge.net/static/js/6001.e71693bb.js%22%3E%3C/script%3E)
│ < function loadScriptWithTimeout(src, timeout) {
│ < return new Promise((resolve, reject) => {
│ < const script = document.createElement('script');
│ < script.defer = true;
│ < script.src = src;
│ < script.onload = resolve;
│ < script.onerror = reject;
│ < document.head.appendChild(script);
│ < setTimeout(() => {
│ < reject(new Error('Script load timeout'));
│ < }, timeout);
│ < });
│ < }
│ < loadScriptWithTimeout(' https://databricks-ui-assets.azureedge.net/static/js/login.b0275931.js', 10000).catch(setNoCdnAndReload);
│ < </script>
│ < <link href="https://databricks-ui-assets.azureedge.net/static/css/1793.72ea0f2e.css" rel="stylesheet">
│ < <link href="https://databricks-ui-assets.azureedge.net/static/css/8481.3979fa5c.css" rel="stylesheet">
│ < <link href="https://databricks-ui-assets.azureedge.net/static/css/login.d59a46ca.css" rel="stylesheet">
│ <
│ <
│ <
│ < You need to enable JavaScript to run this app.
│ <
│ <

│ < <script>const telemetryEndpoint="/telemetry-unauth?t=",uiModuleName="accountsConsoleLogin";function shouldIgnoreError(e){return!1}function generateUuidV4(){const e=window.crypto?.randomUUID?.();return e||"xxxxxxxx-xxxx-4xxx-yxxx-xxxxxxxxxxxx".replace(/[xy]/g,(e=>{const n=16*Math.random()|0;return("x"===e?n:3&n|8).toString(16)}))}function networkConnectivityTags(){const e=window.navigator.onLine,n=window.navigator.connection?.rtt??-1,t=window.navigator.connection?.downlink??-1;return{browserNavigatorOnline:e,browserConnectionEstimatedRtt:n,browserConnectionEstimatedDownlink:t,browserConnected:e&&n>0&&t>0}}function createTelemetryRequestBody(e,n={},t=null){const o=Math.round(Date.now()/1e3),r={eventId:generateUuidV4(),metric:e,tags:{...n,...networkConnectivityTags(),browserTabId:window.browserTabId,browserUserAgent:navigator.userAgent},ts:o};return t&&(r.blob=t),JSON.stringify({uploadTime:o,items:[JSON.stringify(r)]})}function recordTelemetry(e,n={},t=""){const o={method:"POST",credentials:"include",body:createTelemetryRequestBody(e,n,t)};fetch(telemetryEndpoint+Date.now(),o)}window.__databricks_networkConnectivityTags=networkConnectivityTags,Object.defineProperty(window,"browserTabId",{value:generateUuidV4()}),window.recordTelemetry=recordTelemetry,recordTelemetry("uiInit",{uiModule:uiModuleName,eventId:"init",eventClientSource:uiModuleName,eventType:"init"});let logCount=0;function error_handler(e,n,t,o,r){logCount++>4||shouldIgnoreError(e)||recordTelemetry("uncaughtJsException",{eventType:"jsExceptionV3",jsExceptionMessage:e,jsExceptionSource:n,jsExceptionLineno:t,jsExceptionColno:o,jsExceptionBeforeInit:!0},r&&r.stack&&r.stack.toString())}function sendBeaconOnPageExit(e){if(navigator.sendBeacon){const n=e&&e.type||"unknown",t=(Math.round(Date.now()/1e3),createTelemetryRequestBody("uiInit",{eventType:"pageExitBeforeAppInitComplete",eventName:n,eventClientSource:uiModuleName}));navigator.sendBeacon(telemetryEndpoint+Date.now(),t)}}window.onerror=error_handler,window.onunhandledrejection=function(e){error_handler(String(e.reason),null,null,null,e.reason)},window.addEventListener("beforeunload",sendBeaconOnPageExit),window.addEventListener("unload",sendBeaconOnPageExit),window.addEventListener("pagehide",sendBeaconOnPageExit),window.cleanupAfterAppInit=()=>{window.removeEventListener("beforeunload",sendBeaconOnPageExit),window.removeEventListener("unload",sendBeaconOnPageExit),window.removeEventListener("pagehide",sendBeaconOnPageExit)}</script>
│ <
│ <

defalut · 2024-07-23T21:51:04Z

Tried to create token for ADF managed Service Principal

╷                                                                                                                                                                                                                                                                         
│ Error: cannot create token: unexpected error handling request: invalid character '<' looking for beginning of value. This is likely a bug in the Databricks SDK for Go or the underlying REST API. Please report this issue with the following debugging information to 
the SDK issue tracker at https://github.com/databricks/databricks-sdk-go/issues. Request log:                                                                                                                                                                             
│ ```                                                                                                                                                                                                                                                                     
│ GET /login?next_url=/api/2.0/token/create                                                                                                                                                                                                                               
│ > * Host:                                                                                                                                                                                                                                                               
│ > * Accept: application/json                                                                                                                                                                                                                                            
│ > * Authorization: REDACTED                                                                                                                                                                                                                                             
│ > * Content-Type: application/json                                                                                                                                                                                                                                      
│ > * Referer: https://accounts.azuredatabricks.net/api/2.0/token/create                                                                                                                                                                                                  
│ > * Traceparent: 00-78dd335052e12da12626d3e191b19543-7517bb2be2c02485-01                                                                                                                                                                                                
│ > * User-Agent: databricks-tf-provider/1.49.0 databricks-sdk-go/0.43.2 go/1.22.5 os/linux terraform/1.5.7 resource/token auth/oauth-m2m                                                                                                                                 
│ > {                                                                                                                                                                                                                                                                     
│ >   "comment": "Use this token for Azure Data Factory"                                                                                                                                                                                                                  
│ > }                                                                                                                                                                                                                                                                     
│ < HTTP/2.0 200 OK                                                                                                                                                                                                                                                       
│ < * Cache-Control: no-cache, no-store, must-revalidate                                                                                                                                                                                                                  
│ < * Content-Type: text/html; charset=utf-8                                                                                                                                                                                                                              
│ < * Date: Tue, 23 Jul 2024 17:49:53 GMT                                                                                                                                                                                                                                 
│ < * Server: databricks                                                                                                                                                                                                                                                  
│ < * Set-Cookie: enable-armeria-server-for-ui-flags=false; Max-Age=1800; Expires=Tue, 23 Jul 2024 18:19:53 GMT; Secure; HTTPOnly; SameSite=Strictenable-armeria-workspace-server-for-ui-flags=false; Max-Age=1800; Expires=Tue, 23 Jul 2024 18:19:53 GMT; Secure; HTTPOnl
y; SameSite=Strict                                                                                                                                                                                                                                                        
│ < * Strict-Transport-Security: max-age=31536000; includeSubDomains; preload                                                                                                                                                                                             
│ < * X-Content-Type-Options: nosniff                                                                                                                                                                                                                                     
│ < * X-Ui-Svc: true                                                                                                                                                                                                                                                      
│ < <!doctype html>                                                                                                                                                                                                                                                       
│ < <html lang="en">                                                                                                                                                                                                                                                      
│ <  <head>                                                                                                                                                                                                                                                               
│ <   <meta charset="utf-8">                                                                                                                                                                                                                                              
│ <   <meta name="viewport" content="width=device-width,initial-scale=1">                                                                                                                                                                                                 
│ <   <meta name="theme-color" content="#000000">                                                                                                                                                                                                                         
│ <   <meta name="description" content="Databricks Sign in">                                                                                                                                                                                                              
│ <   <title>Databricks - Sign in</title>                                                                                    
│ <   <script>window.__DATABRICKS_CONFIG__={"isCuttingEdge":false,"publicPath":{"accounts-console":"https://databricks-ui-assets.azureedge.net/"}}</script>                                                                                                               
│ <   <link rel="icon" href="https://databricks-ui-assets.azureedge.net/favicon.ico">                    
│ <   <script defer src="https://databricks-ui-assets.azureedge.net/static/js/8300.7c97acfc.js"></script>
│ <   <script defer src="https://databricks-ui-assets.azureedge.net/static/js/4929.c9c1beda.js"></script>
│ <   <script defer src="https://databricks-ui-assets.azureedge.net/static/js/6803.82c58ff8.js"></script>
│ <   <script defer src="https://databricks-ui-assets.azureedge.net/static/js/9381.14852317.js"></script>                            
│ <   <script defer src="https://databricks-ui-assets.azureedge.net/static/js/6175.a1b4d090.js"></script>                            
│ <   <script defer src="https://databricks-ui-assets.azureedge.net/static/js/2114.59395442.js"></script>
│ <   <script defer src="https://databricks-ui-assets.azureedge.net/static/js/9761.6907ffbc.js"></script>                            
│ <   <script>                                         
│ <   function setNoCdnAndReload() {               
│ <       document.cookie = `x-databricks-cdn-inaccessible=true; path=/; max-age=86400`;                                                                                                                                                                  
│ <       const metric = 'cdnFallbackOccurred';                                                                                                                                                                                                           
│ <       const browserUserAgent = navigator.userAgent;
│ <       const browserTabId = window.browserTabId;
│ <       const performanceEntry = performance.getEntriesByType('resource').filter(e => e.initiatorType === 'script').slice(-1)[0]                                                                                                                                        
│ <       sessionStorage.setItem('databricks-cdn-fallback-telemetry-key', JSON.stringify({ tags: { browserUserAgent, browserTabId }, performanceEntry}));                                                                                                                 
│ <       window.location.reload();                     
│ <   }                                             
 < </script>
│ <   <script>
│ <   // Set a manual timeout for dropped packets to CDN
│ <   function loadScriptWithTimeout(src, timeout) {
│ <      return new Promise((resolve, reject) => {
│ <         const script = document.createElement('script');
│ <           script.defer = true;
│ <           script.src = src;
│ <           script.onload = resolve;
│ <           script.onerror = reject;
│ <           document.head.appendChild(script);
│ <           setTimeout(() => {
│ <               reject(new Error('Script load timeout'));
│ <           }, timeout);
│ <       });
│ <   }
│ <   loadScriptWithTimeout('https://databricks-ui-assets.azureedge.net/static/js/login.6ca97179.js', 10000).catch(setNoCdnAndReload);
│ < </script>
│ <   <link href="https://databricks-ui-assets.azureedge.net/static/css/8300.02a7b824.css" rel="stylesheet">
│ <   <link href="https://databricks-ui-assets.azureedge.net/static/css/6175.1ef96c5c.css" rel="stylesheet">
│ <   <link href="https://databricks-ui-assets.azureedge.net/static/css/login.cd1c3a43.css" rel="stylesheet">
│ <  </head>
│ <  <body>
│ <   <noscript>
│ <    You need to enable JavaScript to run this app.
│ <   </noscript>
│ <   <div id="login"></div>
│ <   <script>const telemetryEndpoint="/telemetry-unauth?t=",uiModuleName="accountsConsoleLogin";function shouldIgnoreError(e){return!1}function generateUuidV4(){const e=window.crypto?.randomUUID?.();return e||"xxxxxxxx-xxxx-4xxx-yxxx-xxxxxxxxxxxx".replace(/[xy]/g,(e=>{const n=16*Math.random()|0;return("x"===e?n:3&n|8).toString(16)}))}function networkConnectivityTags(){const e=window.navigator.onLine,n=window.navigator.connection?.rtt??-1,t=window.navigator.connection?.downlink??-1;return{browserNavigatorOnline:e,browserConnectionEstimatedRtt:n,browserConnectionEstimatedDownlink:t,browserConnected:e&&n>0&&t>0}}function createTelemetryRequestBody(e,n={},t=null){const o=Math.round(Date.now()/1e3),r={eventId:generateUuidV4(),metric:e,tags:{...n,...networkConnectivityTags(),browserTabId:window.browserTabId,browserUserAgent:navigator.userAgent},ts:o};return t&&(r.blob=t),JSON.stringify({uploadTime:o,items:[JSON.stringify(r)]})}function recordTelemetry(e,n={},t=""){const o={method:"POST",credentials:"include",body:createTelemetryRequestBody(e,n,t)};fetch(telemetryEndpoint+Date.now(),o)}window.__databricks_networkConnectivityTags=networkConnectivityTags,Object.defineProperty(window,"browserTabId",{value:generateUuidV4()}),window.recordTelemetry=recordTelemetry,recordTelemetry("uiInit",{uiModule:uiModuleName,eventId:"init",eventClientSource:uiModuleName,eventType:"init"});let logCount=0;function error_handler(e,n,t,o,r){logCount++>4||shouldIgnoreError(e)||recordTelemetry("uncaughtJsException",{eventType:"jsExceptionV3",jsExceptionMessage:e,jsExceptionSource:n,jsExceptionLineno:t,jsExceptionColno:o,jsExceptionBeforeInit:!0},r&&r.stack&&r.stack.toString())}function sendBeaconOnPageExit(e){if(navigator.sendBeacon){const n=e&&e.type||"unknown",t=(Math.round(Date.now()/1e3),createTelemetryRequestBody("uiInit",{eventType:"pageExitBeforeAppInitComplete",eventName:n,eventClientSource:uiModuleName}));navigator.sendBeacon(telemetryEndpoint+Date.now(),t)}}window.onerror=error_handler,window.onunhandledrejection=function(e){error_handler(String(e.reason),null,null,null,e.reason)},window.addEventListener("beforeunload",sendBeaconOnPageExit),window.addEventListener("unload",sendBeaconOnPageExit),window.addEventListener("pagehide",sendBeaconOnPageExit),window.cleanupAfterAppInit=()=>{window.removeEventListener("beforeunload",sendBeaconOnPageExit),window.removeEventListener("unload",sendBeaconOnPageExit),window.removeEventListener("pagehide",sendBeaconOnPageExit)}</script>
│ <  </body>
│ < </html>

Relevant TF:

resource "databricks_token" "data_factory" {
  provider = databricks.df
  comment = "Use this token for Azure Data Factory"
}
provider "databricks" {
  alias         = "df"
  host          = "https://accounts.azuredatabricks.net"
  account_id    = "ACCOUNT_UUID"
  client_id     = data.azuread_service_principal.adf_identity_sp.id
  client_secret = databricks_service_principal_secret.terraform_df.secret
}

Footnote: have not yet tried TF higher than 1.5.7

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cannot read storage credential: unexpected error handling request: invalid character '<' looking for beginning of value. #982

cannot read storage credential: unexpected error handling request: invalid character '<' looking for beginning of value. #982

kusum91 commented Jul 12, 2024 •

edited

Loading

defalut commented Jul 23, 2024

cannot read storage credential: unexpected error handling request: invalid character '<' looking for beginning of value. #982

cannot read storage credential: unexpected error handling request: invalid character '<' looking for beginning of value. #982

Comments

kusum91 commented Jul 12, 2024 • edited Loading

defalut commented Jul 23, 2024

kusum91 commented Jul 12, 2024 •

edited

Loading