[azblob] file upload validation

[vijayrajah]

Bug Report

I'm trying to upload a file to azure storage (v2) using
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob

it is not clear (at least to me ) how to do upload validation using MD5.

I have added this in code

opts := azblob.UploadFileOptions{
		TransactionalContentMD5: md5sum,
		AccessTier:              to.Ptr(blob.AccessTierCool),
	}

I'm computing MD5 using

h := md5.New()
	if _, err := io.Copy(h, fil); err != nil {
		return errors.Wrap(err, "unable to compute MD5 for the file "+f)
	}
md5sum := h.Sum(nil)

The Md5 from service and my locally computed MD5 do not match, but the upload still proceeds, when it should fail.

Am I missing something? is there a code sample that I can refer to?

[ghost]

Hi @vijayrajah. Thank you for your feedback and we will look into it soon. Meanwhile, feel free to share your experience using the Azure SDK in this survey.

[siminsavani-msft]

Hi @vijayrajah ! If it is possible, could you please provide your code from start to finish?

[vijayrajah]

Here is the function that does the upload

import (
	"context"
	"crypto/md5"
	"encoding/hex"
	"fmt"
	"github.com/Azure/azure-sdk-for-go/sdk/azcore"
	"github.com/Azure/azure-sdk-for-go/sdk/azcore/policy"
	"github.com/Azure/azure-sdk-for-go/sdk/azcore/to"
	"github.com/Azure/azure-sdk-for-go/sdk/azidentity"
	"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob"
	"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/blob"
	"github.com/pkg/errors"
	"io"
	"os"
	"strings"
	"time"
)

func uploadFileToStorage(ctx context.Context, f string) error {

	fil, err := os.Open(f)
	if err != nil {
		log.Error().Err(err).Msgf("unable to open file %s", f)
		return err
	}

	fn := strings.Split(f, "/")
	blobname := fn[len(fn)-1]

	// calculate md5 sum
	h := md5.New()
	if _, err := io.Copy(h, fil); err != nil {
		return errors.Wrap(err, "unable to compute MD5 for the file "+f)
	}
	md5sum := h.Sum(nil)

	opts := azblob.UploadFileOptions{
		TransactionalContentMD5: md5sum,
		AccessTier:              to.Ptr(blob.AccessTierCool),
	}

	resp, err := client.UploadFile(ctx, contName, blobname, fil, &opts)
	if err != nil {
		log.Error().Err(err).Msgf("Error while uploading file %s to azure storage", f)
		return err
	}

	md5sumStr := hex.EncodeToString(h.Sum(nil))
	respMd5 := hex.EncodeToString(resp.ContentMD5)

	log.Info().Msgf("Uploaded file %s with server md5 sum %s & local md5 %s", f, respMd5, md5sumStr)

	return nil
}

There are other funcs in the code. This is the one that does the upload

client is an package level variable of type *azblob.Client. We use Azure SP for auth & this auth process sets this client

log is zerolog.Logger

Let me know if you need any additional info.

[siminsavani-msft]

Thanks for sharing this! What is the size of your file that is being uploaded?

[vijayrajah]

@siminsavani-msft -- it varies from few MB's to few 10's of GB (depending on the app ). At this time I'm doing a POC for upload validation. It is important for us to validate the upload

[siminsavani-msft]

I believe I was able to catch the bug. I will update this thread with a PR soon. Thanks!

[siminsavani-msft]

Hi @vijayrajah ! Sorry for the late update. As I was working on the checksums in azblob package, I realized that UploadFile is a multipart upload and the way that UploadFile works is by uploading files in blocks to a block blob and checksums cannot be validated at a block level.

We are planning on deprecating the options TransactionalContentCRC64 and TransactionalContentMD5 in UploadFileOptions.

Thank you for reporting this issue!

[vijayrajah]

@siminsavani-msft -- how do we validate the file upload completed successfully? How do we validate the file checksum?

[siminsavani-msft]

@vijayrajah , I would recommend replacing UploadFile with Upload and UploadFileOptions with UploadOptions as seen in your original code.

Just as an FYI, this will likely not work after our next release as I am still working on cleaning up the Transactional checksums and we will be deprecating TransactionalContentMD5 in UploadOptions and replacing it with a different name and set up as seen in AppendBlob: https://github.com/Azure/azure-sdk-for-go/blob/36055639a4d318aa8aa0e4d9995a2eee60ced75e/sdk/storage/azblob/appendblob/models.go#LL75C1-L80C36

It won't be too much of a breaking change, just cleaner way to use MD5 and CRC64 checksums. I will update this thread once that happens!