Skip to content

Commit

Permalink
databricks: lower file upload concurrency
Browse files Browse the repository at this point in the history
databricks driver loads files to memory in full, causing OOM issues.
This is a temporary measure until we fix  this
  • Loading branch information
mdibaiee committed Mar 22, 2024
1 parent 6470e9d commit 6f6b4f3
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion materialize-databricks/staged_file.go
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ import (
)

const fileSizeLimit = 128 * 1024 * 1024
const uploadConcurrency = 5 // that means we may use up to 1.28GB disk space
const uploadConcurrency = 3

// fileBuffer provides Close() for a *bufio.Writer writing to an *os.File. Close() will flush the
// buffer and close the underlying file.
Expand Down

0 comments on commit 6f6b4f3

Please sign in to comment.