Improve our SQL Server client-generated GUIDs

[roji]

There's a report that our current client-side sequential GUIDs generator for SQL Server doesn't work well for concurrent environments; #30753 was opened to consider switching to server-generated GUIDs instead, as a way of mitigating that.

However, @JackTrapper posted an analysis in #30753 (comment), according to which the issue could be mitigated by improving the client-side generation logic instead. If this indeed makes our client-generated GUIDs as indexing-efficient as server-side ones generated via NEWSEQUENTIALID, that would be a good improvement that should also maintain EF backwards compatibility.

[roji]

@JackTrapper would you be interested in submitting a PR for this? EF's client-side GUID generator for SQL Server is in https://github.com/dotnet/efcore/blob/main/src/EFCore/ValueGeneration/SequentialGuidValueGenerator.cs.

[cremor]

dotnet/runtime#103658 was completed recently and .NET 9 will contain Guid.CreateVersion7().
Will this help? So will this new method solve the problems mentioned by @JackTrapper in #30753 (comment)?

[roji]

@cremor I'm not sure, this will require some careful investigation.

[cremor]

I searched a bit more and it looks like UUID v7 (which is used by the new Guid.CreateVersion7()) is not matching the SQL Server sorting behavior.

See:
https://github.com/mareek/UUIDNext/blob/main/Src/UUIDNext/Uuid.cs#L28-L35
https://github.com/mareek/UUIDNext/blob/main/Src/UUIDNext/Generator/UuidV8SqlServerGenerator.cs
mareek/UUIDNext#2

[JackTrapper]

dotnet/runtime#103658 was completed recently and .NET 9 will contain Guid.CreateVersion7(). Will this help? So will this new method solve the problems mentioned by @JackTrapper in #30753 (comment)?

I'm certain it will not help for SQL Server because it does not tailor itself to SQL Server's sort order.

Which is appropriate; it is correct for C# to be agnostic to SQL Server, Oracle, DB2, Postgres.

No, this feature needs to be a database client feature. And it's not like one-size-fits all, because different databases use different sort rules.

No, what you need is:

public Guid CreateSQLServerSortableGuid(); // uniqueidentifier data type

public Guid CreatePostgresSortableGuid(); // uuid data type

I don't believe any other database engines have a uniqueidentifier/uuid data type; but just binary(16), char(36), or varchar(36).

So there is no such thing as a one-size-fits-all solution. You have to know you are generating a UUID to be fed to Microsoft SQL Sever. I don't know where/if in EF Core you tell it the flavor of database engine you're using-so that way it can adjust the bits of a standard type-1 uuid to make is collation compatible with the database back-end engine.

No matter what you want to start with UuidCreateSequential (or the local OS's equivalent) becuase there are rules about how to handle two UUIDs being generated within the same 100 ns tick. And it's not an issue you can ignore; modern CPUs are fast enough you can generate 200,000 UUIDs in the same 100 ns tick. The solution is the system-wide collision counter documented in the RFC, and already implemented on Windows by UuidCreateSequential.

Once you have your type-1 uuid created by UuidCreateSequential, you then have to move the bytes around so that the 64-bit (8-byte) timestamp is sorted first.

So, no, any implemenation of Guid.CreateVersion7() will not be useful here, since it was not designed with my specific favorite database engine in mind.

[osexpert]

Uuidnext has a variant of uuid v8 that is sortable in ms sql server : https://github.com/mareek/UUIDNext/blob/main/Src/UUIDNext/Generator/UuidV8SqlServerGenerator.cs

[ConnerPhillis]

@JackTrapper

I ran your code for sortable guids in (https://github.com/dotnet/efcore/issues/30753#issuecomment-2027429203)[his comment] on #30753. I found that it does not significantly differ from the earlier mentioned GUID generation strategies. The GUIDs generated used an average page space of 67%, and fragmentation was 41%. I have upgraded hardware at this point, so there could be some skew in there.

I'm wondering if maybe there's something wrong with my testing methodology now, though, as this seems really bad for something that seems so well documented and thought out.

Is there any chance that you can explain why I got such a high fragmentation %?

[JackTrapper]

Is there any chance that you can explain why I got such a high fragmentation %?

I'd have to say:

• SQL Server doesn't sort UUIDs in the order i thought
• SQL Server changed the way it sorts UUIDs to something different than i originally thought
• there's a bug in how i shuffle around the bytes of the type-1 UUID to match SQL Server's sort order