Indexing PR prep

[emetsger]

PR to get Travis' feedback on changes. :)

[karenhanson]

This looks OK except for the comments around how status changes are handled. I think as it stands it might be possible to delete active discos, if I'm understanding correctly. If so, we can either put in an issue, or fix before merging.

[karenhanson]

some of these no longer exist

[karenhanson]

why fix? there can only ever be one active disco in a lineage, and if you are able to inactivate then you are inactivating that document. So, if this is done it wouldn't harm anything, but also wouldn't do anything.

[karenhanson]

Also not something to fix - you can tombstone or delete one version of a DiSCO without it affecting other versions. Say you create a DiSCO and share that DiSCO's PID. Then you notice you accidentally put a private email address in the disco, You can create a new version and tombstone the previous version to hide the private information. The PID would allow people to find the latest version without seeing the deleted data.

[emetsger]

Ok, so just making sure I understand:

1. Create Disco A (implies a lineage with progenitor A is created), with the "bad" data
2. Create Disco A' with "clean" data, a DERIVATION of A, and implies a lineage with progenitor A' is created
3. Then, TOMBSTONE Disco A, which implies that it (or any previous versions, if they existed), should not be viewable.

Is my interpretation above correct?

If so, what if step 3 read:
3. Then, TOMBSTONE Disco A, which implies that its lineage should not be viewable.

[karenhanson]

Hopefully this clarifies things. These would all be on the same lineage progenitor (discoV1). A DERIVATION is a whole new lineage, lineages do not affect each other.

//create discoV1
discoV1 (active)
//update discoV1
discoV1 (inactive)
discoV2 (active)
//update discoV2
discoV1 (inactive)
discoV2 (inactive)
discoV3 (active)
//update discoV2 
Cant update, the most recent version is discoV3
//tombstone discoV2
discoV1 (inactive)
discoV2 (tombstoned - but content in triplestore)
discoV3 (active)
//hard delete discoV3
discoV1 (inactive)
discoV2 (tombstoned - but content in triplestore)
discoV3 (deleted - content not in triplestore)

[karenhanson]

see comment above. you can delete a version without deleting the entire lineage. status is per disco version not per lineage

[emetsger]

@karenhanson all decisions made in DiscosIndexer except for DELETION event types were made prior to lineage being available, so they definitely deserve to be revisited, and can be re-cast in terms of operations on the lineage.

Because there is no concept of a Solr transaction at the moment, it is possible to add a Solr document representing an ACTIVE DiSCO, and have the system crash before the previously ACTIVE DiSCO is inactivated. I think we would need some kind of transaction/two-phase commit implementation to insure this doesn't happen. Since we don't have that, the post processing doesn't necessarily make assumptions about how many ACTIVE documents to expect when deactivating or tombstoning.

If/when these operations are recast in terms of lineage (e.g. "tombstone this lineage"), probably not much will change, but at least the mental model ("operations on lineages") will be a little cleaner.

[karenhanson]

After a conversation, the reason for going through the lineage and setting ACTIVE->INACTIVE on an update makes good sense. I've logged issue #176 for work on the other comment.