Replace MD5 calculations with use of S3 hashes

[lladdy]

No description provided.

[lladdy]

Double check that the S3 files always have MD5 as etags.

[lladdy]

Need to address multipart uploads as detailed here: https://stackoverflow.com/questions/12186993/what-is-the-algorithm-to-compute-the-amazon-s3-etag-for-a-file-larger-than-5gb

[lladdy]

Items of note:
Multipart chunk size depends on upload client.
Chunk size for tested bot files appears to be 8 mebibytes.
AWS users have report that the chunk size is not necessarily consistent for every chunk (though, potentially not of concern for us).

Reference python implementation:

def calculate_s3_etag(file_path, chunk_size=8 * 1024 * 1024):
    md5s = []

    with open(file_path, 'rb') as fp:
        while True:
            data = fp.read(chunk_size)
            if not data:
                break
            md5s.append(hashlib.md5(data))

    if len(md5s) < 1:
        return hashlib.md5().hexdigest()

    if len(md5s) == 1:
        return md5s[0].hexdigest()

    digests = b''.join(m.digest() for m in md5s)
    digests_md5 = hashlib.md5(digests)
    return '"{}-{}"'.format(digests_md5.hexdigest(), len(md5s))

md5 = calculate_s3_etag("/path/to/bot.zip")
print(md5)

[lladdy]

I've deployed a new AC API version which uses S3 etags: #747

[lladdy]

After the ACs are updated, we can hopefully remove the manual hash calc code.

[lladdy]

Dan might have time to update ACs the week starting the 17th March.