Convenience layer for uploading file to ADLS

[fantyz]

Hi,

We're having issues uploading files to ADLS using the SDK.

We're using:

• github.com/Azure/azure-sdk-for-go/services/datalake/store/2016-11-01/filesystem (v21.1.0, 6d20bdbae88c06c36d72eb512295417693bfdf4e)
• github.com/Azure/go-autorest (v10.15.5, 9bc4033dd347c7f416fca46b2f42a043dc1fbdf6)

When doing client.Create we get the following error consistently:

filesystem.Client#Create: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: error response cannot be parsed: "" error: EOF

It only happens with larger files (I do not know what the cut-off point is, but a 500kb file uploads fine and a 37mb file consistently fails.

It would also be great if you could return informative errors instead of this. I mean it might as well have returned a plain: "Some error happened. Too bad."

[fantyz]

A quick test of only creating an empty file with Create and write the data with Append resulted in the same error:

filesystem.Client#Append: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: error response cannot be parsed: "" error: EOF

[jhendrixMSFT]

I don't think this is specific to the Go SDK as I can repro this same behavior using powershell. @joshgav can you please loop in somebody from the data lake store team to help?

[fantyz]

It would be great if someone could take a look at this?
We have a release blocked by this issue making it somewhat urgent for us.

[jhendrixMSFT]

@fantyz I'm trying to find somebody from the service team to take a look.

[jhendrixMSFT]

OK, the issue here is that the service only supports uploads in 4MB chunks. This is documented here however is easy to overlook; see the buffersize parameter. I have asked the service team to update the service to return a meaningful error message.
As for code changes to make it work, I will investigate now that I understand the root cause.

[jhendrixMSFT]

Here's a working example.

package main

import (
	"bytes"
	"context"
	"fmt"
	"io"
	"io/ioutil"
	"os"

	"github.com/Azure/azure-sdk-for-go/services/datalake/store/2016-11-01/filesystem"
	"github.com/Azure/go-autorest/autorest/azure/auth"
)

const (
	accountName = "<account name>"
	destination = "<destination>"
	inputFile   = `<big file for upload>`
	bufferSize  = 4 * 1024 * 1024
)

func main() {
	a, err := auth.NewAuthorizerFromEnvironment()
	if err != nil {
		panic(err)
	}
	client := filesystem.NewClient()
	client.Authorizer = a

	f, err := os.Open(inputFile)
	if err != nil {
		panic(err)
	}
	defer f.Close()

	resp, err := client.Create(context.Background(), accountName, destination, nil, nil, filesystem.DATA, nil, nil)
	if err != nil {
		panic(err)
	}
	fmt.Println(resp.Status)

	buffer := make([]byte, bufferSize, bufferSize)
	for {
		n, err := f.Read(buffer)
		if err == io.EOF {
			break
		}
		flag := filesystem.DATA
		if n < bufferSize {
			// last chunk
			flag = filesystem.CLOSE
		}
		chunk := ioutil.NopCloser(bytes.NewReader(buffer))
		fmt.Printf("uploading chunk %d...", n)
		resp, err = client.Append(context.Background(), accountName, destination, chunk, nil, flag, nil, nil)
		if err != nil {
			panic(err)
		}
		fmt.Printf("%d\n", resp.StatusCode)
	}
	fmt.Println("done!")
}

[jhendrixMSFT]

Opened Azure/azure-rest-api-specs#4464 to update the SDK docs.

[fantyz]

Thanks - we'll do the suggested work-around.

I was kinda hoping for such low level code as calls to Read and juggling byte buffers would be handled for me. You do not see this logic as something that should be part of the SDK and not the end user?

I would avoid doing the flag = filesystem.CLOSE as port of the loop though and handle it afterwards (or in the io.EOF case). Read does not provide any guarantee that n == len(buffer) (in fact quite the opposite) filesystem.CLOSE might be set prematurely.

From https://golang.org/pkg/io/#Reader

Read reads up to len(p) bytes into p. It returns the number of bytes read (0 <= n <= len(p)) and any error encountered. Even if Read returns n < len(p), it may use all of p as scratch space during the call. If some data is available but not len(p) bytes, Read conventionally returns what is available instead of waiting for more.

You also have the case where the last call to Read before io.EOF returns a full buffer where filesystem.CLOSE never happens.

[jhendrixMSFT]

Sorry I should have prefaced my sample as non-production code.
Agreed that this functionality belongs in the SDK. For some context, the packages we have are generated from our OpenAPI specs which is what we refer to as the "protocol layer". Functionality like this would go into a hand-written "convenience layer" built on top of the protocol layer as there's no way we'd know how to generate this functionality.
We have discussed internally how to package convenience layer code, I will use this issue to track it for DLS but given other priorities at present (e.g. modules) I suspect a solution is a ways out.

[RickWinter]

Track1 will not provide a GA library with datalake support.