Introduce packaging, distribution and documentation for the C# driver #623

farost · 2024-04-03T16:42:48Z

Usage and product changes

We introduce packaging, distribution and documentation C# driver for TypeDB (the original driver PR). It is built using the cross-platform .NET 6 framework.

Usage:

The driver is distributed as a series of Nuget packages. To use the driver, import the latest versions of the driver (TypeDB.Driver) and its Pinvoke runtime (TypeDB.Driver.Pinvoke) suitable for your platform.

CS project:
Here is an example from a .csproj for MacOS x86-64:

<PackageReference Include="TypeDB.Driver" Version={VERSION} />
<PackageReference Include="TypeDB.Driver.Pinvoke.osx-x64" Version={VERSION} />

If you aim to build a platform-independent package, reference all the needed runtimes (it will affect the size of your application by downloading a respective set of platform-specific dynamic libraries):

<PackageReference Include="TypeDB.Driver.Pinvoke.osx-x64" Version={VERSION} />
<PackageReference Include="TypeDB.Driver.Pinvoke.linux-x64" Version={VERSION} />
<PackageReference Include="TypeDB.Driver.Pinvoke.win-x64" Version={VERSION} />
<PackageReference Include="TypeDB.Driver.Pinvoke.osx-arm64" Version={VERSION} />
...

Bazel:

Import both the TypeDB.Driver and TypeDB.Driver.Pinvoke nuget packages as dependencies to your build rule.
For development, you can also use a csharp bazel rule, passing targets //csharp:driver-csharp (the driver itself), //csharp/Api:api (exposed Api namespace), //csharp/Common:common (exposed Common namespace) as dependencies.

A simple usage example (see csharp/Test/Integration/Examples for more):

using TypeDB.Driver.Api;
using TypeDB.Driver.Common;

public class TypeDBExample
{
        public void SetupTypeDB()
        {
            string dbName = "access-management-db";
            string serverAddr = "127.0.0.1:1729";

            try
            {
                using (ITypeDBDriver driver = Drivers.CoreDriver(serverAddr))
                {
                    driver.Databases.Create(dbName);
                    IDatabase database = driver.Databases.Get(dbName);

                    using (ITypeDBSession session = driver.Session(dbName, SessionType.Schema))
                    {
                        using (ITypeDBTransaction transaction = session.Transaction(TransactionType.Write))
                        {
                            transaction.Query.Define("define person sub entity;").Resolve();

                            string longQuery = "define name sub attribute, value string; person owns name;";
                            transaction.Query.Define(longQuery).Resolve();

                            transaction.Commit();
                        }
                    }

                    database.Delete();
                }
            }
            catch (TypeDBDriverException e)
            {
                Console.WriteLine($"Caught TypeDB Driver Exception: {e}");
            }
        }
}

Implementation

Documentation:
Documentation for C# is generated in a similar way as the C/C++ ones: doxygen -> adoc. This way, the C# parser has some extra logic of parsing the intermediate doxygen files but follows a similar (copypasted) structure. Additionally, some small bugs for both old parsers have been fixed.

Distribution:
We decided to distribute the C# driver as a series of packages to allow the end user to choose between multi-platform and compactness. This way, we have:
TypeDB.Driver.{X-X-X-version}.nupkg - the main package with the driver, which fails in runtime if no runtime is additionally provided.
TypeDB.Driver.Pinvoke.{X-X-X-version}.{platform}.nupkg - platform-specific runtimes (osx-x64, osx-arm64, linux-x64, linux-arm64, win-x64).

The distribution is implemented via several bazel rules: nuget_pack (with a wrapper for native packages in this PR) and nuget_push and their respective calls for both runtime and main packages.

Tests:
Integration tests use NUnit framework as a built-in framework for C# bazel test rules. This PR presents both test-core and test-cloud as small usage examples within the integration tests. These tests are also triggered by Factory.

There is also a new Deployment test, which creates a .csproj project and tries to fetch all the needed (and all the available runtime) packages to use them for another small C# program. This test is triggered by CircleCI.

typedb-bot · 2024-04-03T16:42:51Z

farost · 2024-04-03T16:44:44Z

.circleci/config.yml

It doesn't fully work on CircleCI, but it has some successful passes for some platforms (the whole workflow), so you can have a look. I'll put another comment once it's in the final state.

farost · 2024-04-03T16:46:20Z

.circleci/windows/clib/deploy_release.bat

@@ -25,5 +25,5 @@ SET DEPLOY_ARTIFACT_USERNAME=%REPO_TYPEDB_USERNAME%
 SET DEPLOY_ARTIFACT_PASSWORD=%REPO_TYPEDB_PASSWORD%

 SET /p VER=<VERSION
-bazel --output_user_root=C:\bazel run --verbose_failures --define version=%VER% //c:deploy-windows-x86_64-zip --compilation_mode=opt -- release
+bazel --output_user_root=C:\b run --verbose_failures --define version=%VER% //c:deploy-windows-x86_64-zip --compilation_mode=opt -- release


We lack symbols for Windows path. It started failing even within the Rust build steps, so we can't even change these names. Right now it has just several free symbols until 260...

farost · 2024-04-03T16:47:10Z

.circleci/windows/csharp/deploy_release.bat

+
+ECHO Building and deploying windows package...
+SET DEPLOY_NUGET_API_KEY=%REPO_TYPEDB_PASSWORD%
+@REM TODO: Add new key for nuget as a new variable!


Only for release, we'll configure it with @flyingsilverfin once the nuget bot account with the key is created.

farost · 2024-04-03T16:50:40Z

.gitignore

@@ -97,4 +97,3 @@ Cargo.toml

 # Temporary paket files for C#
 paket-files/
-paket.lock


https://fsprojects.github.io/Paket/faq.html#Why-should-I-commit-the-lock-file

Yeah we do normally commit the lock files for various languages' package dependencies!

farost · 2024-04-03T16:51:41Z

csharp/Api/BUILD

@@ -31,13 +31,21 @@ csharp_library(
        "//csharp:typedb_driver_pinvoke",
        "@paket.csharp_deps//newtonsoft.json",
    ],
+    out = "TypeDB.Driver.Api",


That's how I name the output dlls! All following their namespaces.

farost · 2024-04-03T16:52:49Z

csharp/Api/Concept/Type/IThingType.cs

@@ -335,7 +335,7 @@ IThingType IConcept.AsThingType()
        public class Annotation : NativeObjectWrapper<Pinvoke.Annotation>
        {
            /**
-             * @hidden
+             * @private


The equivalent of @hidden for the java documentation. It causes warnings that a member lacks documentation while the doxygen build (which is dumb), but I find it okay.

farost · 2024-04-03T16:53:50Z

csharp/BUILD

+)
+
+
+swig_native_nuget_pack(


Produces a set of native libs, picks one for our current platform to push further.

farost · 2024-04-03T16:54:55Z

csharp/BUILD

+)
+
+
+nuget_push(


Pushes the generated .nuspec and .snuspec files based on the snapshot / release cmd flag.

farost · 2024-04-03T16:57:04Z

csharp/BUILD

+nuget_pack(
+    name = "driver-csharp-pack",
+    id = "TypeDB.Driver",
+    libs = {


We include all the libs, so their namespaces are available for users. However, only Api and Common are documented, so we consider it as something okay.

farost · 2024-04-03T16:57:28Z

csharp/BUILD

+)
+
+
+# TODO: Not hermetic - requires doxygen on the host


Inherit this TODO from C++, not touching it now.

farost · 2024-04-03T17:00:03Z

csharp/Drivers.cs

@@ -25,7 +25,7 @@

 namespace TypeDB.Driver
 {
-    public static class TypeDB
+    public static class Drivers


I did not know how to call this entry class, and TypeDB was okay while I was using libs as Bazel dependencies. But once I started calling TypeDB.CoreDriver in the outer CS project, it started conflicting with the namespace name (TypeDB.Driver), which was expected... to fire earlier.

I have no idea how to call it, so it is Drivers. Feel free to suggest better options.

It'll be ok for now!

farost · 2024-04-03T17:01:18Z

csharp/Test/Integration/Examples/BUILD

Connection tests are now explicit Examples.

farost · 2024-04-03T17:02:15Z

csharp/Test/Integration/Examples/CoreExamplesTest.cs

+        }
+
+        [Test]
+        public void DocExample()


This test is for copypasting into README. The upper ones show the same workflow with different ways of doing things.

farost · 2024-04-03T17:03:59Z

dependencies/vaticle/repositories.bzl

+#        name = "vaticle_dependencies",
+#        remote = "https://github.com/vaticle/dependencies",
+#        commit = "7518f8adaf5714512a8af96128b15cca7dedd475", # sync-marker: do not remove this comment, this is used for sync-dependencies by @vaticle_dependencies
+#    ) # TODO: return afther the PR is merged!


I'll fix it once the other pull requests are merged.

farost · 2024-04-03T17:05:44Z

tool/docs/adoc/AsciiDocTableBuilder.kt

@@ -32,7 +32,7 @@ class AsciiDocTableBuilder(private val headers: List<String>) {
    }

    private fun header(): String {
-        return "[cols=\"~" + ",~".repeat(this.headers.size - 1) +
+        return "[cols=\"" + ",".repeat(this.headers.size - 1) +


Vladimir asked me to remove these symbols as we can't understand what they do and they just distract us while debugging tables' behavior.

farost · 2024-04-03T17:07:35Z

tool/docs/c/DoxygenParserC.kt

-        val argsMap = argsList.toMap()
+        val argsMap = argsList.associate { (first, second) ->
+            Pair(
+                addZeroWidthWhitespaces(first),


Zero-width whitespaces help tables to squeeze gently and look fresh. Needed for special cases when we store Some.Namespaces.Inside.The.Tables.Without.Spaces.Which.Freaks.Browsers.Out.

farost · 2024-04-03T17:11:29Z

There is a series of connected PRs:

farost requested a review from krishnangovindraj April 3, 2024 16:42

farost requested review from dmitrii-ubskii and flyingsilverfin as code owners April 3, 2024 16:42

typedb-bot assigned krishnangovindraj, dmitrii-ubskii and flyingsilverfin Apr 3, 2024

farost commented Apr 3, 2024

View reviewed changes

csharp/BUILD

)

swig_native_nuget_pack(

Copy link

Member Author

farost Apr 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Produces a set of native libs, picks one for our current platform to push further.

farost commented Apr 3, 2024

View reviewed changes

csharp/BUILD

)

nuget_push(

Copy link

Member Author

farost Apr 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pushes the generated .nuspec and .snuspec files based on the snapshot / release cmd flag.

farost commented Apr 3, 2024

View reviewed changes

csharp/Test/Integration/Examples/BUILD Outdated

Copy link

Member Author

farost Apr 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Connection tests are now explicit Examples.

farost commented Apr 3, 2024

View reviewed changes

This was referenced Apr 3, 2024

Dependencies and rules refactoring for the C# driver deployment typedb/typedb-dependencies#521

Merged

Add Bazel rules for C# packaging, distribution and documentation typedb/bazel-distribution#406

Merged

farost added type: build type: feature priority: high type: documentation labels Apr 3, 2024

farost added 27 commits April 4, 2024 19:01

Try fixing macos and windows circleci

1f8e677

Add a series of test changes for circleci

221935c

Fix circleci

4b94322

Add another try for mac and linux

af1d63d

Fix config.yml

42c2f01

Add install script for mac dotnet for tests

f4d6466

Remove comments from config.yml for build fix

fc29f90

Remove linux from builds, experiment with mac

fba3e50

Turn off everything except of dotnet mac

5be8318

Test dotnet install for mac

ab1a2ca

Turn test snapshot any off

a552b3a

Fix build

fe39c41

Add find dotnet for mac builds

fd2eed5

Add ln for mac installation

80c72f6

Add additional logs, fix versioning for tests

5f4581e

Remove requires for windows

d207070

Return windows snapshot deploy

6e6f13a

Comment out excessive rules

a54dfc4

Update ref with more runtimes

343f8f7

Return all the new build steps, update refs with fixes

5b386b4

Fix config style

92784e3

Fix windows runtime deployment scripts

d6cdd18

Remove excessive spaces

c433ef6

Add sleeps for the mac tests

3bc11e0

Fix release config + fix issue typedb#613

6e64940

Fix docs for the updated method in python

b40940d

Return builds, fix comments and refs

035c7c3

farost force-pushed the csharp-docs-deployment branch from 7c60c10 to 035c7c3 Compare April 4, 2024 18:03

farost merged commit 3c70b84 into typedb:development Apr 4, 2024
1 of 4 checks passed

farost deleted the csharp-docs-deployment branch October 4, 2024 11:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce packaging, distribution and documentation for the C# driver #623

Introduce packaging, distribution and documentation for the C# driver #623

farost commented Apr 3, 2024 •

edited

Loading

typedb-bot commented Apr 3, 2024 •

edited by flyingsilverfin

Loading

farost Apr 3, 2024

farost Apr 3, 2024

farost Apr 3, 2024

farost Apr 3, 2024

flyingsilverfin Apr 4, 2024

farost Apr 3, 2024

farost Apr 3, 2024

farost Apr 3, 2024

farost Apr 3, 2024

farost Apr 3, 2024

farost Apr 3, 2024

farost Apr 3, 2024

flyingsilverfin Apr 4, 2024

farost Apr 3, 2024

farost Apr 3, 2024

farost Apr 3, 2024

farost Apr 3, 2024

flyingsilverfin Apr 4, 2024

farost Apr 3, 2024 •

edited

Loading

farost commented Apr 3, 2024

Introduce packaging, distribution and documentation for the C# driver #623

Introduce packaging, distribution and documentation for the C# driver #623

Conversation

farost commented Apr 3, 2024 • edited Loading

Usage and product changes

Usage:

Implementation

typedb-bot commented Apr 3, 2024 • edited by flyingsilverfin Loading

PR Review Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

farost Apr 3, 2024 • edited Loading

Choose a reason for hiding this comment

farost commented Apr 3, 2024

farost commented Apr 3, 2024 •

edited

Loading

typedb-bot commented Apr 3, 2024 •

edited by flyingsilverfin

Loading

farost Apr 3, 2024 •

edited

Loading