Consolidation of indexing in Kitodo.Production

[andre-hohmann]

Description

The Indexing in Kitodo.Production can lead to long downtimes if there are many processes - for example 500,000.

One reasons is that only the entire index (operations, tasks, ...) can be created - even if only one or a few processes or tasks need to be indexed. This can take approximately eight hours for 500,000 operations. Second, the indexing can be easily interrupted, which leads to a prolongation of the indexing process. Even if complete indexing does not occur often - if one occurs, it leads to severe consequences, as the system is not available.

Furthermore, there are many issues regarding indexing and it seems, that these should be examined in a concerted action.

Related Issues

• Indexing does not work #3999
• Inconsistent creation of processes in the event of errors #4154 (example corrupted index)
• Re-Indexing fails on big data #4258
• Indexing is failing on many structure and meta data element #4724
• Indexing generate a lot of requests on indexing all #4726
• Reindexing: no information #4939
• Remove time limit to start new indexing threads #4947
• Left over property index #5014
• Indexing of type TASK is extremely slow #5131
• indexing - static initialisation and waste of ressources #5174
• indexing - no tolerance for errors #5175
• indexing - not successful index remaining not indexed objects #5176
• indexing - wrong calculation on indexLimit and batch size #5177
• indexing - missuse of configuration parameter #5178
• indexing - high rate of index access while indexing #5179
• indexing - indexing all only successful on open browser tab #5180
• indexing - no possibility to cancel running indexing #5181
• indexing - issues on indexing on es search limit #5182
• Not all indexed objects are checked against the database in case index/database are out of sync #5379

Expected Benefits of this Development

The expected benefit is mainly the secure operation of Kitodo.Production - especially installations with many processes.
This is an advantage for administrators and for users.

Estimated Costs and Complexity

The costs are probably high, because a complex system and many issues need to be analyzed and adapted.