FastRsyncNet - C# delta syncing library

The Fast Rsync .NET library is Rsync implementation derived from Octodiff tool.

Unlike the Octodiff which is based on SHA1 algorithm, the FastRsyncNet allows a variety of hashing algorithms to choose from. The default one, that is xxHash64, offers significantly faster calculations and smaller signature size than the SHA1, while still providing sufficient quality of hash results.

FastRsyncNet supports also SHA1 and is 100% compatible with signatures and deltas produced by Octodiff.

Since version 2.0.0 the signature and delta format has changed. FastRsyncNet 2.x is still able to work with signatures and deltas from FastRsync 1.x and Octodiff. However, files made with FastRsyncNet 2.x are not going to be recognized by FastRsyncNet 1.x.

Install

Add To project via NuGet:

1. Right click on a project and click 'Manage NuGet Packages'.
2. Search for 'FastRsyncNet' and click 'Install'.

Examples

Calculating signature

using FastRsync.Signature;

...

var signatureBuilder = new SignatureBuilder();
using (var basisStream = new FileStream(basisFilePath, FileMode.Open, FileAccess.Read, FileShare.Read))
using (var signatureStream = new FileStream(signatureFilePath, FileMode.Create, FileAccess.Write, FileShare.Read))
{
    signatureBuilder.Build(basisStream, new SignatureWriter(signatureStream));
}

Calculating delta

using FastRsync.Delta;

...

var delta = new DeltaBuilder();
builder.ProgressReport = new ConsoleProgressReporter();
using (var newFileStream = new FileStream(newFilePath, FileMode.Open, FileAccess.Read, FileShare.Read))
using (var signatureStream = new FileStream(signatureFilePath, FileMode.Open, FileAccess.Read, FileShare.Read))
using (var deltaStream = new FileStream(deltaFilePath, FileMode.Create, FileAccess.Write, FileShare.Read))
{
    delta.BuildDelta(newFileStream, new SignatureReader(signatureStream, delta.ProgressReporter), new AggregateCopyOperationsDecorator(new BinaryDeltaWriter(deltaStream)));
}

Patching (applying delta)

using FastRsync.Delta;

...

var delta = new DeltaApplier
        {
            SkipHashCheck = true
        };
using (var basisStream = new FileStream(basisFilePath, FileMode.Open, FileAccess.Read, FileShare.Read))
using (var deltaStream = new FileStream(deltaFilePath, FileMode.Open, FileAccess.Read, FileShare.Read))
using (var newFileStream = new FileStream(newFilePath, FileMode.Create, FileAccess.ReadWrite, FileShare.Read))
{
    delta.Apply(basisStream, new BinaryDeltaReader(deltaStream, progressReporter), newFileStream);
}

Calculating signature on Azure blobs

FastRsyncNet might not work on Azure Storage emulator due to issues with stream seeking.

using FastRsync.Signature;

...

var storageAccount = CloudStorageAccount.Parse("azure storage connectionstring");
var blobClient = storageAccount.CreateCloudBlobClient();
var blobsContainer = blobClient.GetContainerReference("containerName");
var basisBlob = blobsContainer.GetBlockBlobReference("blobName");

var signatureBlob = container.GetBlockBlobReference("blob_signature");

var signatureBuilder = new SignatureBuilder();
using (var signatureStream = await signatureBlob.OpenWriteAsync())
using (var basisStream = await basisBlob.OpenReadAsync())
{
    await signatureBuilder.BuildAsync(basisStream, new SignatureWriter(signatureStream));
}

Available algorithms and relative performance

Following signature hashing algorithms are available:

• XxHash64 - default algorithm, signature size 6.96 MB, signature calculation time 5209 ms
• SHA1 - signature size 12.9 MB, signature calculation time 6519 ms
• XxHash3 - signature size 6.96 MB, signature calculation time 5024 ms
• MD5 - originally used in Rsync program, signature size 10.9 MB, signature calculation time 6767 ms

The signature sizes and calculation times are to provide some insights on relative perfomance. The real perfomance on your system will vary greatly. The benchmark had been run against 0.99 GB file.

Following rolling checksum algorithms are available:

• Adler32RollingChecksum - default algorithm, it uses low level optimization that makes it faster but provides worse quality of checksum
• Adler32RollingChecksumV2 - the original Adler32 algorithm implementation, slower but better quality of checksum

GZip compression that is rsync compatible

If you synchronize a compressed file, a small change in a compressed file may force rsync algorithm to synchronize whole compressed file, instead of just the changed blocks. To fix this, a custom GZip compression method may be used that periodically reset the compressor state to make it block-sync friendly. Install FastRsyncNet.Compression package and use following method:

FastRsync.Compression.GZip.Compress(Stream sourceStream, Stream destStream)

To uncompress you may use any GZip method (e.g. System.IO.Compression.GZipStream).