Add an option for using the Microsoft VSO-Hash digest function

[erikmav]

This allows Remote Exec implementations for Microsoft Build Accelerator (https://github.com/Microsoft/BuildXL) to exchange its standard hashes

[googlebot]

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here (e.g. I signed it!) and we'll verify it.

---

What to do if you already signed the CLA

Individual signers

• It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

• Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
• The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
• The email used to register you as an authorized contributor must also be attached to your GitHub account.

ℹ️ Googlers: Go here for more info.

[buchgr]

LGTM. Can you please sign the CLA?

[erikmav]

Signed CLA

[buchgr]

@Erikma I believe it didn't work. Did you sign with the same e-mail address as is used in your git commit?

[erikmav]

CLA problems, working on finding the right MS-internal CLA owner. ETA probably a couple of days.

[erikmav]

Retry on CLA

[googlebot]

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

[johnterickson]

Hey @EdSchouten - You asked about this PR at BazelCon and I thought I'd add to @Erikma's answer :)

The pages in the VSO0 hash are used for performance as they are hashed concurrently on a single processor using SIMD operations (e.g. as provided by bcrypt).

The blocks can be hashed concurrently on multiple threads or systems. This is super helpful when performing large uploads to a CAS. If you have an X GB file, at no point does any HTTP handler have to hash all X bytes. Each call is only responsible for hashing 2MB.

We also have intra-file deduplication. We don't talk about it much (yet!) but it powers Universal Packages and Pipeline Artifacts in Azure DevOps. It's based on https://docs.microsoft.com/en-us/windows-server/storage/data-deduplication/overview so it uses variable-sized blocks.

Best way to play with it is to fire up a free DevOps account and tinker with Pipeline Artifacts. You can see the efficacy in the output spew:

Total Content: 21,232.0 MB
Physical Content Uploaded: 105.7 MB
Logical Content Uploaded: 220.6 MB
Compression Saved: 114.9 MB
Deduplication Saved: 21,011.4 MB
Number of Chunks Uploaded: 3,655
Total Number of Chunks: 927,490