AdoNetGrainStorage HashPicker customization support

[vladislav-prishchepa]

Hi,

I have several clusters powered by Orleans v3.x with ADO.NET grain persistence (MySQL/PostgreSQL, payload in json/jsonb column) and want to upgrade them to v8.2+ with reasonable downtime (shutdown cluster -> deploy new version) and without persistent state loss. To achieve this goal I've tried to:

• switch to new packages, annotate DTO and handle another changes described in migration guide
• set correct state names in PersistentStateAttribute
• slightly change Read/Write/Clear queries as I need to keep storing grain state payload in json column (for better readability and indexing support)

And it's almost here but with one problem: GrainIdHash and GrainTypeHash mismatch.

As I can see hashing algorithm was changed from JenkinsHash to xxHash32 in v7 (#7949). So, to complete the upgrade we need to handle this hashing change. Changing values in database seems to be too hard (big table without primary key, unable to compute new hashes just using SQL). But we have another way to do it: customizing HashPicker in AdoNetGrainStorage. Unfortunately, this property can't be simply configured via dependency injection and the only way to do it is something like that:

siloBuilder.AddAdoNetGrainStorageAsDefault();

var storageDescriptor = siloBuilder.Services.LastOrDefault(static d
    => d.IsKeyedService && d.ServiceType == typeof(IGrainStorage));
    
if (storageDescriptor is null)
    throw new InvalidOperationException("Unable to find IGrainStorage service descriptor.");
if (storageDescriptor.KeyedImplementationFactory is null)
    throw new InvalidOperationException("Unexpected IGrainStorage service descriptor content.");

siloBuilder.Services.Remove(storageDescriptor);
siloBuilder.Services.Add(ServiceDescriptor.KeyedSingleton(
    storageDescriptor.ServiceType,
    storageDescriptor.ServiceKey,
    (provider, key) =>
    {
        var storage = storageDescriptor.KeyedImplementationFactory.Invoke(provider, key);
        if (storage is not AdoNetGrainStorage adoNetGrainStorage)
            throw new InvalidOperationException("Unexpected IGrainStorage service implementation type.");

        adoNetGrainStorage.HashPicker = new StorageHasherPicker([new Orleans3Hasher()]);
        return adoNetGrainStorage;
    }));

Although it works (upgraded cluster behaves as expected, existing grain state is available), this workaround is not reliable as it depends on internal Orleans implementation and could stop working after future updates.

Taking into account the above, I propose the following changes:

public class AdoNetGrainStorageOptions
{
    // <existing properties>

    public IStorageHasherPicker HashPicker { get; set; }
}

public class AdoNetGrainStorage
{
    public AdoNetGrainStorage(
        // other parameters
        IOptions<AdoNetGrainStorageOptions> options)
    {
        //other assignments

        this.HashPicker = options.Value.HashPicker;
    }
}

This new property should have the same behavior as AdoNetGrainStorageOptions.GrainStorageSerializer with setting default value via PostConfigure if it's null.

Also it would be nice to add public Orleans v3-compatible IHasher implementation to keep customers from copy-paste JenkinsHash-based hasher.

In addition, I suppose this v3->v7 hashing change should be described in migration guide as its handling is a required step to migrate existing cluster without data loss.

[jbusuttil83]

I agree 100%, this is keeping a lot of people from actually migrating from what I see. @ReubenBond what do you think about this?

[AndyDreistadtTR]

Agree with this analysis and it is preventing us from migrating to Orleans 8 as well. @ReubenBond do you feel that the workaround provided would be an appropriate solution or is there an alternative you could recommend?

I have to imagine this is a common issue for people migrating off of 3.x who would like to keep their data - wondering how others have dealt with this.

[ReubenBond]

Great idea! If you are happy to open a PR, we will review & merge it. If you're willing to share the code you used to accomplish this, that would also be very useful to others.

[vladislav-prishchepa]

Ok, I'll prepare a PR in the next few days

[jbusuttil83]

Ok, I'll prepare a PR in the next few days

I can test it as well.

[vladislav-prishchepa]

I also published 3.7.2->8.2.0 migration sample against MySQL/PostgreSQL: https://github.com/vladislav-prishchepa/orleans_3_to_8_migration_poc

The implementation turned out to be a little more tricky because I've found another breaking change: in v3 IGrainWithStringKey grain IDs was treated as integer + extensions, so they produced 8 more zero bytes to hash source which are gone in v8.