Vendor integration

[leifurhauks]

I'm thinking it might be useful to have an internal API that lets goblin know about the graph vendor it's running against. Sort of like vendor-level config.

For example, when running against DSE, get_hashable_id should default to our dict hasher function unless the user explicitly sets it to something else.

Also, if we're using DSE it's not really necessary to query the database to discover whether transactions are supported; they're known to be supported for DSE.

(Incidentally, not querying the database for this would also fix another issue, which is that if an alias hasn't been set, the query in Goblin.supports_transactions fails. It should be possible to get an app with Goblin.open without setting an alias, because the alias could be set on the session instead.)

Maybe the way to do this would be to have an optional kwarg on Goblin.open / Goblin.__init__ that accepts a "vendor information" object, which might be a class or mapping. The default value for that kwarg could be the vendor information object for TinkerGraph.

If this sounds reasonable, @davebshow , @jsenecal , I'll get a PR ready for this.

[davebshow]

So, has this idea evolved into what you were saying yesterday about a plugin? Can we enable this sort of configuration just by installing goblin-dse?

[leifurhauks]

That's what I was thinking -- by creating an app with goblin-dse, goblin.app.Goblin gets a configuration object that holds the vendor-specific information.

[davebshow]

So, just curious-how does it work? This is what I imagine:

goblin.App gets an optional kwarg - vendor_config

Ok, then when the end user uses goblin-dse it is something like:

from goblin_dse import App

Which provides a subclass of App with DSE config..?

Is this what you were thinking?

[leifurhauks]

That's what I was thinking, yeah

[aolieman]

Hi,
I'm wondering if there is any interest to work on schema/migration utilities as was previously discussed in goblin-legacy#3.
Asking it here because this would need to be (at least partially) vendor-specific of course.

I will likely have some time in the coming months to work on basic schema generation for DSE based on goblin elements and properties. It seems that DSE and Janusgraph have much conceptual overlap in their schema definitions, but some difference in syntax. So perhaps the vendor-specific aspects of schema definitions could fit into Provider instances.

[davebshow]

That sounds great. In an ideal world where I had more time (or was getting paid to work on Goblin), I would like to maintain a series of provider specific plugins (Python packages) in separate repos (goblin-dse, goblin-janus, etc. etc.) I am happy to have community members contribute on any aspect of this. Btw, I wrote a super simple script to generate Janus schema for an ETL project I was working on. It isn't full featured by any means, and doesn't handle meta property schema creation nor edge properties, but if you are interested, here it is: https://gist.github.com/davebshow/e788e3c477bea8464864b2ca2be5100f

[aolieman]

Thanks for sharing that schema script @davebshow !
I may be able to produce a DSE version of such a script soon, and will then look into the extensions needed to support meta & edge properties, vertex & edge indexes, etc.

To me, generating the full schema specification from Goblin element classes seems a good initial goal. I can see myself needing to support schema migrations in the future, but realistically this will be more complex than schema migrations in SQL (e.g. Django migrations).

For now I'm stuck on an unrelated vendor-specific issue. Somehow the Gremlin Server that is bundled with DSE 5.1.3 (TinkerPop 3.2.6) is not able to deserialize/parse all bytecode that is generated by Goblin 2.1.0rc3 (via aiogremlin/gremlinpython 3.2.6). I am under the impression that GraphSon 2.0 is used on both sides, so I'm not sure how to proceed here.

@leifurhauks have you come across this issue before? If there is more to it than a simple config or package version mismatch, I'll open a new issue.

Edit: long story short - I had gotten the type of a kwarg of a traversal strategy wrong (str instead of List[str]).