staging operation over HTTP was unsuccessful: <HeaderName>Transfer-Encoding</HeaderName>

[elise-prequel]

Hi! This might be an issue with https://github.com/databricks/databricks-sdk-go; opening here because we saw some changes to staging operations in the latest version release notes on this package, and nothing particularly suspicious in the other.

Description
We upgraded our versions from

github.com/databricks/databricks-sdk-go v0.37.0 // indirect
github.com/databricks/databricks-sql-go v1.5.3 // indirect

to the latest version of both packages.

Our exact query is:

PUT '{{.LocalFilePath}}' INTO '{{.UnityVolumePath}}' OVERWRITE;

and we get an error about the Transfer-Encoding header not being accepted, full error is

databricks: driver error: staging operation over HTTP was unsuccessful: 400-<?xml version="1.0" encoding="utf-8"?><Error><Code>UnsupportedHeader</Code><Message>One of the HTTP headers specified in the request is not supported.
RequestId:cac6b3df-d01e-0000-1e66-9abf07000000
Time:2024-04-29T18:50:37.3231103Z</Message><HeaderName>Transfer-Encoding</HeaderName></Error>

Reproduction
We think it is any PUT with the golang setup. We tested on our regular instance (SQL endpoint on the latest release channel) and on Azure.

Expected behavior
Normal PUT.

Is it a regression?
Yes, downgrading to 1.5.3 fixed our issue.

[kravets-levko]

Hi @elise-prequel! The fix for this issue will be available in upcoming v1.5.5 (#219) very soon

[elise-prequel]

Thank you so much @kravets-levko !!

[niger-prequel]

@kravets-levko would you mind explaining which change in v1.5.5. addresses this? based on the change log it would appear to be #217 but that seems to be focusing on the Content-Size header rather than Transfer-Encoding. Is it just the case that not having the correct Content-Size header ends up resulting in an error about Transfer-Encoding?

[kravets-levko]

@niger-prequel Yes, I think that it's related to #217 - there are no much changes in 1.5.4 comparing to 1.5.3, so it should be the one

[niger-prequel]

Ok @kravets-levko , I just want to ensure that this is not related to github.com/databricks/databricks-sdk-go jumping from v0.37.0 to v0.39.0. Because if it is and we upgrade, we'll encounter the same issue.

[kravets-levko]

@niger-prequel @elise-prequel v1.5.5 is out - please give it a try and let me know if you still see this error

[niger-prequel]

@kravets-levko it upgrading to v1.5.5 passed our test suite

[kravets-levko]

Great, I'm glad to hear that. I'm closing the issue, but feel free to re-open if needed, or open a new one