Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support text/plain response as streaming request body #638

Merged
merged 2 commits into from
Oct 9, 2023

Conversation

mgyucht
Copy link
Contributor

@mgyucht mgyucht commented Oct 5, 2023

Changes

For APIs using the text/plain content type (today, only usage download), support these by simply streaming the response body back to the end user. This way, they can read the response and save it to a file or post-process it further.

Tests

Made a simple app that downloads billable usage to a temporary file:

package main

import (
	"context"
	"fmt"
	"io"
	"io/ioutil"
	"os"

	databricks "github.com/databricks/databricks-sdk-go"
	"github.com/databricks/databricks-sdk-go/logger"
	"github.com/databricks/databricks-sdk-go/service/billing"
)

func writeToTempFile(r io.ReadCloser) (*os.File, error) {
	// Ensure r is closed after reading
	defer r.Close()

	// Create a temporary file
	tmpFile, err := ioutil.TempFile("", "prefix-")
	if err != nil {
		return nil, err
	}

	// Copy data from io.ReadCloser to the temporary file
	_, err = io.Copy(tmpFile, r)
	if err != nil {
		tmpFile.Close() // Close and remove temporary file in case of errors
		os.Remove(tmpFile.Name())
		return nil, err
	}

	// Rewind the temporary file for further reading
	tmpFile.Seek(0, io.SeekStart)

	return tmpFile, nil
}

func main() {
	logger.DefaultLogger = &logger.SimpleLogger{Level: logger.LevelTrace}
	a, err := databricks.NewAccountClient(&databricks.Config{
		Profile:      "...",
		DebugHeaders: true,
	})
	if err != nil {
		panic(err)
	}
	ctx := context.Background()
	res, err := a.BillableUsage.Download(ctx, billing.DownloadRequest{
		StartMonth:   "2023-08",
		EndMonth:     "2023-09",
		PersonalData: true,
	})
	if err != nil {
		panic(err)
	}
	tmp, err := writeToTempFile(res.Contents)
	if err != nil {
		panic(err)
	}
	fmt.Printf("tmp: %v\n", tmp.Name())
}

This works.

Note that this test depends on a change to the content type for the OpenAPI spec for usage download to match the true content type.

@codecov-commenter
Copy link

codecov-commenter commented Oct 5, 2023

Codecov Report

All modified lines are covered by tests ✅

Files Coverage Δ
openapi/model.go 5.19% <ø> (ø)

📢 Thoughts on this report? Let us know!.

@mgyucht mgyucht requested a review from tanmay-db October 6, 2023 08:57
@mgyucht mgyucht added this pull request to the merge queue Oct 9, 2023
Merged via the queue into main with commit d821075 Oct 9, 2023
@mgyucht mgyucht deleted the support-text/plain-response branch October 9, 2023 13:31
pietern added a commit that referenced this pull request Oct 12, 2023
Note: this release includes custom JSON marshalling that captures whether or
not a value is set and should be included in the marshalled output, if its
value is equal to Go's zero value (i.e. `0` for an int field).

* Add ForceSendFields and a custom marshaller ([#615](#615)).
* Support text/plain response as streaming request body ([#638](#638)).
* Added log statement if OIDC response is non 200 ([#644](#644)).

API Changes:

 * Changed `Download` method for [a.BillableUsage](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/billing#BillableUsageAPI) account-level service to start returning [billing.DownloadResponse](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/billing#DownloadResponse).
 * Added [billing.DownloadResponse](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/billing#DownloadResponse).
 * Changed `Delete` method for [a.AccountStorageCredentials](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#AccountStorageCredentialsAPI) account-level service with new required argument order.
 * Changed `Get` method for [a.AccountStorageCredentials](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#AccountStorageCredentialsAPI) account-level service with new required argument order.
 * Changed `Update` method for [a.AccountStorageCredentials](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#AccountStorageCredentialsAPI) account-level service with new required argument order.
 * Added `GetBindings` method for [w.WorkspaceBindings](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#WorkspaceBindingsAPI) workspace-level service.
 * Added `UpdateBindings` method for [w.WorkspaceBindings](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#WorkspaceBindingsAPI) workspace-level service.
 * Removed `Name` field for [catalog.AccountsUpdateStorageCredential](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#AccountsUpdateStorageCredential).
 * Added `StorageCredentialName` field for [catalog.AccountsUpdateStorageCredential](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#AccountsUpdateStorageCredential).
 * Removed `Name` field for [catalog.DeleteAccountStorageCredentialRequest](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#DeleteAccountStorageCredentialRequest).
 * Added `StorageCredentialName` field for [catalog.DeleteAccountStorageCredentialRequest](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#DeleteAccountStorageCredentialRequest).
 * Removed `Name` field for [catalog.GetAccountStorageCredentialRequest](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#GetAccountStorageCredentialRequest).
 * Added `StorageCredentialName` field for [catalog.GetAccountStorageCredentialRequest](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#GetAccountStorageCredentialRequest).
 * Added `Owner` field for [catalog.UpdateConnection](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#UpdateConnection).
 * Added [catalog.GetBindingsRequest](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#GetBindingsRequest).
 * Added [catalog.UpdateWorkspaceBindingsParameters](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#UpdateWorkspaceBindingsParameters).
 * Added [catalog.WorkspaceBinding](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#WorkspaceBinding).
 * Added [catalog.WorkspaceBindingBindingType](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#WorkspaceBindingBindingType).
 * Added [catalog.WorkspaceBindingsResponse](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#WorkspaceBindingsResponse).
 * Added `Spec` field for [compute.ClusterDetails](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/compute#ClusterDetails).
 * Added `ApplyPolicyDefaultValues` field for [compute.ClusterSpec](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/compute#ClusterSpec).
 * Removed `AwsAttributes` field for [compute.EditInstancePool](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/compute#EditInstancePool).
 * Removed `AzureAttributes` field for [compute.EditInstancePool](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/compute#EditInstancePool).
 * Removed `DiskSpec` field for [compute.EditInstancePool](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/compute#EditInstancePool).
 * Removed `EnableElasticDisk` field for [compute.EditInstancePool](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/compute#EditInstancePool).
 * Removed `GcpAttributes` field for [compute.EditInstancePool](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/compute#EditInstancePool).
 * Removed `PreloadedDockerImages` field for [compute.EditInstancePool](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/compute#EditInstancePool).
 * Removed `PreloadedSparkVersions` field for [compute.EditInstancePool](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/compute#EditInstancePool).
 * Added `Deployment` field for [jobs.CreateJob](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#CreateJob).
 * Added `UiState` field for [jobs.CreateJob](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#CreateJob).
 * Added `Deployment` field for [jobs.JobSettings](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#JobSettings).
 * Added `UiState` field for [jobs.JobSettings](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#JobSettings).
 * Removed `ConditionTask` field for [jobs.RunOutput](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#RunOutput).
 * Added `WebhookNotifications` field for [jobs.Task](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#Task).
 * Added [jobs.CreateJobUiState](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#CreateJobUiState).
 * Added [jobs.JobDeployment](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#JobDeployment).
 * Added [jobs.JobDeploymentKind](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#JobDeploymentKind).
 * Added [jobs.JobSettingsUiState](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#JobSettingsUiState).
 * Added `WorkloadType` field for [serving.ServedModelInput](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/serving#ServedModelInput).
 * Added `WorkloadType` field for [serving.ServedModelOutput](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/serving#ServedModelOutput).
 * Removed [a.AccountNetworkPolicy](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/settings#AccountNetworkPolicyAPI) account-level service.
 * Changed `List` method for [w.IpAccessLists](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/settings#IpAccessListsAPI) workspace-level service to return [settings.ListIpAccessListResponse](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/settings#ListIpAccessListResponse).
 * Removed [settings.AccountNetworkPolicyMessage](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/settings#AccountNetworkPolicyMessage).
 * Removed [settings.DeleteAccountNetworkPolicyRequest](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/settings#DeleteAccountNetworkPolicyRequest).
 * Removed [settings.DeleteAccountNetworkPolicyResponse](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/settings#DeleteAccountNetworkPolicyResponse).
 * Removed `IpAccessLists` field for [settings.GetIpAccessListResponse](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/settings#GetIpAccessListResponse).
 * Added `IpAccessList` field for [settings.GetIpAccessListResponse](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/settings#GetIpAccessListResponse).
 * Removed [settings.ReadAccountNetworkPolicyRequest](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/settings#ReadAccountNetworkPolicyRequest).
 * Removed [settings.UpdateAccountNetworkPolicyRequest](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/settings#UpdateAccountNetworkPolicyRequest).
 * Added [settings.ListIpAccessListResponse](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/settings#ListIpAccessListResponse).

OpenAPI SHA: 493a76554afd3afdd15dc858773d01643f80352a, Date: 2023-10-12

Dependency updates:

 * Bump golang.org/x/mod from 0.12.0 to 0.13.0 ([#639](#639)).
 * Bump google.golang.org/api from 0.140.0 to 0.146.0 ([#643](#643)).
 * Bump golang.org/x/net from 0.15.0 to 0.17.0 ([#646](#646)).
 * Bump golang.org/x/oauth2 from 0.12.0 to 0.13.0 ([#642](#642)).
 * Bump golang.org/x/net from 0.8.0 to 0.17.0 in /examples/zerolog ([#647](#647)).
 * Bump golang.org/x/net from 0.8.0 to 0.17.0 in /examples/slog ([#648](#648)).
@pietern pietern mentioned this pull request Oct 12, 2023
github-merge-queue bot pushed a commit that referenced this pull request Oct 12, 2023
Note: this release includes custom JSON marshalling that captures
whether or
not a value is set and should be included in the marshalled output, if
its
value is equal to Go's zero value (i.e. `0` for an int field).

* Add ForceSendFields and a custom marshaller
([#615](#615)).
* Support text/plain response as streaming request body
([#638](#638)).
* Added log statement if OIDC response is non 200
([#644](#644)).

API Changes:

* Changed `Download` method for
[a.BillableUsage](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/billing#BillableUsageAPI)
account-level service to start returning
[billing.DownloadResponse](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/billing#DownloadResponse).
* Added
[billing.DownloadResponse](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/billing#DownloadResponse).
* Changed `Delete` method for
[a.AccountStorageCredentials](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#AccountStorageCredentialsAPI)
account-level service with new required argument order.
* Changed `Get` method for
[a.AccountStorageCredentials](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#AccountStorageCredentialsAPI)
account-level service with new required argument order.
* Changed `Update` method for
[a.AccountStorageCredentials](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#AccountStorageCredentialsAPI)
account-level service with new required argument order.
* Added `GetBindings` method for
[w.WorkspaceBindings](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#WorkspaceBindingsAPI)
workspace-level service.
* Added `UpdateBindings` method for
[w.WorkspaceBindings](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#WorkspaceBindingsAPI)
workspace-level service.
* Removed `Name` field for
[catalog.AccountsUpdateStorageCredential](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#AccountsUpdateStorageCredential).
* Added `StorageCredentialName` field for
[catalog.AccountsUpdateStorageCredential](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#AccountsUpdateStorageCredential).
* Removed `Name` field for
[catalog.DeleteAccountStorageCredentialRequest](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#DeleteAccountStorageCredentialRequest).
* Added `StorageCredentialName` field for
[catalog.DeleteAccountStorageCredentialRequest](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#DeleteAccountStorageCredentialRequest).
* Removed `Name` field for
[catalog.GetAccountStorageCredentialRequest](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#GetAccountStorageCredentialRequest).
* Added `StorageCredentialName` field for
[catalog.GetAccountStorageCredentialRequest](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#GetAccountStorageCredentialRequest).
* Added `Owner` field for
[catalog.UpdateConnection](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#UpdateConnection).
* Added
[catalog.GetBindingsRequest](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#GetBindingsRequest).
* Added
[catalog.UpdateWorkspaceBindingsParameters](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#UpdateWorkspaceBindingsParameters).
* Added
[catalog.WorkspaceBinding](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#WorkspaceBinding).
* Added
[catalog.WorkspaceBindingBindingType](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#WorkspaceBindingBindingType).
* Added
[catalog.WorkspaceBindingsResponse](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#WorkspaceBindingsResponse).
* Added `Spec` field for
[compute.ClusterDetails](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/compute#ClusterDetails).
* Added `ApplyPolicyDefaultValues` field for
[compute.ClusterSpec](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/compute#ClusterSpec).
* Removed `AwsAttributes` field for
[compute.EditInstancePool](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/compute#EditInstancePool).
* Removed `AzureAttributes` field for
[compute.EditInstancePool](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/compute#EditInstancePool).
* Removed `DiskSpec` field for
[compute.EditInstancePool](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/compute#EditInstancePool).
* Removed `EnableElasticDisk` field for
[compute.EditInstancePool](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/compute#EditInstancePool).
* Removed `GcpAttributes` field for
[compute.EditInstancePool](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/compute#EditInstancePool).
* Removed `PreloadedDockerImages` field for
[compute.EditInstancePool](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/compute#EditInstancePool).
* Removed `PreloadedSparkVersions` field for
[compute.EditInstancePool](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/compute#EditInstancePool).
* Added `Deployment` field for
[jobs.CreateJob](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#CreateJob).
* Added `UiState` field for
[jobs.CreateJob](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#CreateJob).
* Added `Deployment` field for
[jobs.JobSettings](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#JobSettings).
* Added `UiState` field for
[jobs.JobSettings](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#JobSettings).
* Removed `ConditionTask` field for
[jobs.RunOutput](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#RunOutput).
* Added `WebhookNotifications` field for
[jobs.Task](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#Task).
* Added
[jobs.CreateJobUiState](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#CreateJobUiState).
* Added
[jobs.JobDeployment](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#JobDeployment).
* Added
[jobs.JobDeploymentKind](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#JobDeploymentKind).
* Added
[jobs.JobSettingsUiState](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#JobSettingsUiState).
* Added `WorkloadType` field for
[serving.ServedModelInput](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/serving#ServedModelInput).
* Added `WorkloadType` field for
[serving.ServedModelOutput](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/serving#ServedModelOutput).
* Removed
[a.AccountNetworkPolicy](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/settings#AccountNetworkPolicyAPI)
account-level service.
* Changed `List` method for
[w.IpAccessLists](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/settings#IpAccessListsAPI)
workspace-level service to return
[settings.ListIpAccessListResponse](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/settings#ListIpAccessListResponse).
* Removed
[settings.AccountNetworkPolicyMessage](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/settings#AccountNetworkPolicyMessage).
* Removed
[settings.DeleteAccountNetworkPolicyRequest](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/settings#DeleteAccountNetworkPolicyRequest).
* Removed
[settings.DeleteAccountNetworkPolicyResponse](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/settings#DeleteAccountNetworkPolicyResponse).
* Removed `IpAccessLists` field for
[settings.GetIpAccessListResponse](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/settings#GetIpAccessListResponse).
* Added `IpAccessList` field for
[settings.GetIpAccessListResponse](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/settings#GetIpAccessListResponse).
* Removed
[settings.ReadAccountNetworkPolicyRequest](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/settings#ReadAccountNetworkPolicyRequest).
* Removed
[settings.UpdateAccountNetworkPolicyRequest](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/settings#UpdateAccountNetworkPolicyRequest).
* Added
[settings.ListIpAccessListResponse](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/settings#ListIpAccessListResponse).

OpenAPI SHA: 493a76554afd3afdd15dc858773d01643f80352a, Date: 2023-10-12

Dependency updates:

* Bump golang.org/x/mod from 0.12.0 to 0.13.0
([#639](#639)).
* Bump google.golang.org/api from 0.140.0 to 0.146.0
([#643](#643)).
* Bump golang.org/x/net from 0.15.0 to 0.17.0
([#646](#646)).
* Bump golang.org/x/oauth2 from 0.12.0 to 0.13.0
([#642](#642)).
* Bump golang.org/x/net from 0.8.0 to 0.17.0 in /examples/zerolog
([#647](#647)).
* Bump golang.org/x/net from 0.8.0 to 0.17.0 in /examples/slog
([#648](#648)).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants