Corpora - store and retrieve cachedAnalysis

[johnml1135]

To support punctuation de-normalization, we need to store data with the corpus.

• We can create a dict of string: strings called cachedAnalysis that can be read/writeable from the rest of Serval.
• With this update, other code in Serval can create, cache, and read analysis data as stored in the corpus.
• We will clear (delete) all cachedAnalysis when any file is updated on the corpus.

[ddaspit]

When will the punctuation analysis be performed? During the build?

[johnml1135]

It will bee performed when retrieving the USFM. It will take a few seconds at best (and then be cached for future use).

[ddaspit]

I would like to be careful about how long a synchronous request might take to respond.

@benjaminking How fast is the punctuation analysis?

[benjaminking]

The punctuation analysis usually takes about a second to run over a large USFM file like Genesis. I don't have great estimates on how long it takes for a whole project, because SILNLP (which is providing the capability to load a Paratext project) adds 1.5-2 minutes of overhead. I'll have to do some profiling eventually, to figure out where the time is being spent, but I would expect that the time grows roughly linearly with the number of chapters.

[ddaspit]

I assume that we will need to perform the analysis over the whole translation. This sounds like it could add a significant amount of time to the request. We should probably consider running the analysis asynchronously. Maybe as part of the preprocessing stage of a build.

[johnml1135]

@benjaminking - if you can make the process be 5 seconds or less (when it is ported to C# it may be 100 times as fast), I would recommend this approach. If not, we will need to tie it to the updating of a corpus, which will create a few more consumers, and other nice things to maintain :-).

If you find that most of the time is spent in looping, then it would be a good candidate for vectorization (list comprehension or numpy) and will likely speed up massively later. I wonder if Copilot would be able to find ways to run the algorithm faster, I haven't asked it for things like that yet (maybe try Claude 3.7 thinking)?

[benjaminking]

I have done some profiling on the quote convention detector, as well as comparing its runtime against an empty USFMParserHandler that does nothing. Both methods are pointing to the actual quote convention detection being a relatively small portion of the running time, with USFM parsing taking up most of it.

The results of the timing indicate that quote convention detection on a full Bible takes under 1.5 seconds in Python. Here are the results for the ESV and the NIV:

15.8 seconds on ESV with quote convention detection
14.8 seconds on ESV for USFM parsing alone

7.9 seconds on NIV with quote convention detection
6.6 seconds on NIV for USFM parsing alone

These numbers represent an over-estimate for our use case, since our typical case will involve parsing many fewer books. (These versions also have significantly more USFM markup than a typical translation project) And as John has stated, USFM parsing should be faster in C#, and I expect that quote convention detection will also experience a speed-up. All this leads me to believe that quote convention detection should be able to run synchronously once we implement it in Serval.

[johnml1135]

Based upon a +50x parsing speedup in C#, let's stay with the "cachedAnalysis" approach.