Cascading Restrictions

[CBroz1]

I have a couple shelved ideas that I never put into DataJoint issues, but instead implemented some fix for in my own work

Problem

For long pipelines, my team has hit some issues with key length and the need to version pipelines. We solved each of these by 'burying' foreign keys into a single primary key alias, usually a UUID.

class MyParent(dj.Manual):
    definition = """
    my_pk1 : varchar(32)
    my_pk2 : varchar(32)
    -> ManyOtherPks
    ---
    my_sk1 : varchar(32)
    """

class MyChild(dj.Manual):
    definition = """
    this_id : UUID
    ---
    -> MyParent
    -> SomeOtherPks
    """

It then became difficult for our users to figure out the right joins so they could restrict these aliasing tables based on some upstream field. The following was not intuitive for them, and even less so if the field they wanted to restrect was 2 or 3 joins upstream.

((MyParent & {'my_sk1': 'some_val'}) * MyChild).fetch('this_id')

Ideas

I think we could address this difficulty by extracting the cascade functionality from delete here and finding a way to apply the same idea to graph functions.

MyChild.ancestors(as_objects=True, restriction={'my_sk1': 'some_val'})

I took the idea a step further and introduced 'long distance' restrictions to our codebase (doc, implementation) allowing users to do the following...

restricted_child = MyChild() << 'my_sk1 = "some_val"'
restricted_parent = MyParent() >> f'this_id = {some_UUID}'

See also our demo in pytests: tables, and tests.

It's slower than a join, but it's seen wide adoption among our users, and makes it much easier to explore how a single subject, for example, is populated across many different downstream tables. It's not something I would suggest for use in a production fetch, but it's been more intuitive for sorting through the table graph.

To this end, it would be helpful if assert_join_compatibility here could be split up into a is_join_compatible func that returned a boolean, and another that raised the error