Document Entities and Profiles

[mynameisfiber]

This issue proposes a solution for two very linked feature requests in aleph: document entities and entity profiles.

Document entities encompasses the desire to have any outputs of the document NER analysis create real Aleph entities. Currently, the tokens extracted from the document are stored as text attached to the Document entity and are second-class citizens. By turning them into true entities, they can be x-ref'd, used in diagrams and more easily accessed by users. In order to accomplish this, a new type of entity, the DocumentExtract entity, is going to be created that can be xref'd on but otherwise will be invisible to the system. This means that the entity could be linked with another entity and shown in the profile view, described below, but otherwise won't clutter the system. The reason for doing this is that any NER extraction is bound to be noisy and create many pathological entities. By making them invisible unless explicitly linked to an existing entity, we allow users to control when these extracted entities are considered "relevant" to their Aleph use-case.

Entity profiles is a way of clustering entities which have been linked together through the xref system (and subsequently merged through the Linkage system by a user). By having clusters of entities we have confirmation represent the same physical thing, we have the ability to show in the UI a unified profile representing all the information we know. This is useful in the case that we have multiple entities for, say, Putin, but each entity contains different information. The merged profile will show all of the information in one place and make available a single entity for easier analysis.

An additional requirement of the profile system is that profiles are namespaced to a dataset. A dataset is considered a "workspace" for a particular project and thus can have it's own requirements of the considerations when determining whether two entities are "the same". For example, one project may think that every branch of a bank should be merged into the same profile while another project wants to consider each branch separately. By namespacing the profiles to a dataset, we can still reference any entity (from all of Aleph) without potentially stepping on other projects notion of similarity.

To implement this, we've identified the following steps:

• Merge Linkage data model into EntitySet and EntitySetItem models, adding a score and added_by field to the EntitySetItem
• Update Linkage API to use the new models
• Add profile membership information into Aleph entity
• Update FTM to add DocumentExcerpt family of entities to represent extracted tokens from a document
• Create DocumentExcerpt entities from tokens extracted from a document in ingest-file
• Ensure DocumentExcerpt entities are xref'd on
• Update xref UI to display profile data when the xref target entity is already part of a profile

In the near future, we can also add the following features using the above system:

• Use profile data in the xref calculation
• Use profile data to boost search relevancy
• Create profile view to replace or enhance the single-entity view

[pudo]

Thanks for summarising this, @mynameisfiber. I'm totally with you on the proposed abolition of Linkage in favor of EntitySet, and I'd propose that this become PR #1. PR #2 would then be invisible entities, which should be easy but we need to consider a bit the API changes it implies.

To me, the most controversial part in this is PR #3, the proposed DocumentExcerpt. I would somwhat prefer a solution in which the analyze stage would simply emit Person/Company entities (invisible, keyed on a slug of their name) and a Documentation entity that creates a link between a document and the person/company. The Documentation could at most mention the number of occurrences (rather than the offsets), but I have yet to recognise the added value of doing the offsets with regards to the two objectives stated in this ticket.

Another thing I'd been pondering over the weekend was the notion of temporarily introducing two xref systems in Aleph: the existing one (which in production use), and a derived version that also generates scored entity sets. That would help us play with this result type a bit, perhaps behind a feature flag :)

[pudo]

Ok, one more thought: should we also take this refactor of the Linkage as an occasion to begin to differentiate between no judgment and unsure on the decision field of entity set items?

[kjacks]

Thanks for writing this up! :)

Agreed - unless there is a strong benefit to having a separate DocumentExtract type, that it makes sense to utilize the existing schemas we have available to store extracted entities.

I was a bit unclear on what "Add profile membership information into Aleph entity" means in the initial proposed implementation steps - I'm assuming membership in a profile would be indicated solely by being a part of a profile EntitySet and wouldn't be stored in the entity itself?

Definitely agree with wanting a distinct no judgement from unsure if that's possible to add while we're updating the Linkages anyways. :)

From a UI standpoint my biggest open questions are:

• how we differentiate between extracted entities and confirmed entities in the xref screen
• how to display a profile in the xref results such that it's clear that what's being matched is a cluster of entities

[mynameisfiber]

In my thoughts, the main benefit to having a separate entity type is to not clutter the main entity indexes with tokens that may be incorrectly identified. Also, given that the NER system doesn't specifically categorize things into the schemas we have, it would be nice to have more generalized schemas to store their outputs so the decision of whether a thing is a Person or Organization or LegalEntity is punted into the linkage step.

I definitely agree with the splitting of this issue into multiple PR's and the additional judgement types. I was thinking about having this be a integer field in EntitySetItem which, in the case of Linkage, codes to the various decisions. For other EntitySet usecases, it can serve in the same way to code items or as a sort priority.

@kjacks My thought was to attach profile membership to the entity as a cache essentially so that whenever we retrieve an entity from the index we can make the decision to show it's profile information instead of having to do separate lookups for each entity. In the future this'll also make it easier to aggregate search results on the profile information in order to dedupe entities within the same profile and display that instead. In general, I think this would make it easier for the UI to be given an entity and make the decision client-side whether to display the profile view of it or the entity itself.

For the UI elements, maybe we can sketch some things up next week? I'm also unsure the best ways to do this but with the additional information we'll have access to there's probably some interesting things we can do.

[kjacks]

@mynameisfiber
Very cool - the option to have both the entity and it's profile cluster cached on the client would give us so much flexibility, and would definitely avoid a lot of unnecessary requests - so if that's not too much to implement, then that sounds great

Yes! Sounds good - would love to talk through some options for UI directions we could take with this :)

[simonwoerpel]

Hey, just for the record my 2 cents as I sometimes was part of the discussions around these things...

I prefer the approach having abstract DocumentExcerpt entities (as @mynameisfiber proposes) instead of converting NER results directly to Person / Organization simply because for me it feels like a more straightforward approach with two seperate clear "stages" that these entities have in their lifetime:

1. being abstract, waiting for user judgement about schema & relevance
2. turning into a real Person, Company, whatever

instead of this not so straightforward lifecycle path:

1. automatically becoming a Person
2. turning into an Organization by user judgement

totally looking forward to this feature! if there is anything i can assist in implementing, let me know.