Support meta-attribute queries

[levand]

Unlike Datomic, DataScript does not store attributes as entities. Instead, it uses the actual attribute keyword in tuples. This makes sense from an engineering and performance point of view, since tuples are guaranteed to exist in memory.

However, many applications do wish to refer to attributes explicitly, for a variety of reasons. Fortunately, it is possible to manually transact entities for each attribute, as you would in Datomic, and this works for most use cases.

However, there is one thing that remains simply impossible: writing a datalog query with a logic variable that unifies in both attribute and entity position.

For example:

 ;; find entities that have an identity attribute
  (ds/q '[:find ?e
          :where
          [?attr :db/unqiue :db.unique/identity]
          [?e ?attr _]]
    db)

This unfortunately cannot be made to work in DataScript, as it stands. There are two possible ways that this could be fixed:

1. Add :db/ident support, so keywords can be used in "entity position" in queries.
2. Use a fully reified schema where attributes are actually entities (the way Datomic does)

[tonsky]

Wait, but will that work?

  (ds/q '[:find ?e
          :where
          [?a :db/unqiue :db.unique/identity]
          [?a :db/ident ?attr]
          [?e ?attr _]]
    db)

[levand]

That should work... unfortunately not for me though, since I'd like the same query to work with both Datomic and DataScript.

So I probably overstated the problem. The primary issue isn't that it can't be done... there are definitely ways to code around it it. The actual problem is the fundamental incompatibility with Datomic caused by the lack of idents and reified schema.

And maybe that's on me, for trying to treat them as more similar then they are. Perhaps sticking with one and dropping support for the other is the best. But I suspect lots of people are in more or less the same boat, too, since the API similarity is one of DataScript's attractive features to start with.

[tonsky]

Yes, there are differences, all documented and discussed:

• I have nothing against reified schema & schema migrations, it just seemed we can get away without them so far. Here are my thoughts on schema migrations Editable schema #174
• Integer ids for attributes, a billion of special cases for idents, and that abstraction being leaky and leading to lots of mistakes all lead to a very conscious decision not to repeat that mistake in DS (my thoughts :db/ident support #162)
• There’re also transaction differences you probably should be aware of Transaction data with ref attributes is order dependent (differs from Datomic) #172

Whether or not coping with all this worth it is your call. But I’m curious, why did you wanted to use both Datomic and DataScript in the first place? And why interoperability? What code did you planned to share between the two?

[levand]

Well it's a long story, but to put it briefly:

I'm working on Arachne, a modular application development framework for Clojure (https://www.kickstarter.com/projects/1346708779/arachne-rapid-web-development-for-clojure). The premise of Arachne's architecture is to build a rich model of your entire application, as much as possible, in a "config database." I want to use Datomic (and/or DataScript) for many reasons, but mostly: (1) I want a database-as-a-value, (2) it's easy to work with from Clojure (3) it has a powerful and extensible schema, so each module can

The current implementation is actually an abstraction layer that works with both DataScript and Datomic, using the "greatest common denominator" of functionality and papering over some simple differences (like tempids.) For testing and validation, I actually have a multiplexing implementation that runs every operation against both Datomic and DataScript and asserts that all results are equivalent before returning.

It's essential that the code for all modules work with both Datomic and DataScript, because you want modules to be usable no matter what implementation the user chooses; I don't want to split into two separate sets of "Datomic modules" and "DataScript modules."

So far, this has worked fine; I am restricted to the common denominator of functionality between the two, but that is sufficiently large that I haven't had many serious problems until now. But as I'm building a type/ontology system on top of the config model, to allow modules to create richer schemas, I find myself really needing to incorporate attributes into queries.

So I'm at a point where I could support either Datomic or DataScript pretty easily, but both only with some effort (at this point, it looks like that would involve walking query data to "fix it up" before passing to the actual implementation (gross)).

The problem is that if I pin on just one implementation, I lose one of two groups of people:

• Those who will not write software that has a closed-source dependency.
• Those who have an existing investment in Datomic and want Cognitect support and backing

On a technical level, too, I want to preserve Datomic compatibility since it's only using Datomic that we can actually persist configurations, and track them over time in a durable database. Not everyone needs that, but those who do really want it.

So I'm not quite sure what to do. I realize this probably doesn't help, but that's the outline of my current situation. (and why I'm making all these weird requests ;)

[levand]

Actually, I just thought of one small-ish change that would also allow me to paper over this difference relatively easily.

If DataScript added support for symbol-based query functions (even only when running in the JVM), I could write an (attr-eid) function that did the right thing depending on the platform, and use that pervasively.

[levand]

PR #178 enables a workaround, if it looks ok to you.

[tonsky]

Just released 0.15.3 with your PR, I assume I can close this

[levand]

Yep, workaround is in place. Thanks!