VerseRef parsing error

[Enkidu93]

We recently saw this error on production:

SIL.Scripture.VerseRefException: Invalid reference : LUK 12:
   at SIL.Scripture.VerseRef.Parse(string verseStr)
   at SIL.Machine.Corpora.ScriptureRef.Parse(string str, ScrVers versification)
   at System.Linq.Enumerable+SelectListIterator<TSource, TResult>.ToArray()
   at System.Linq.Enumerable+SelectListIterator<TSource, TResult>.ToArray()
   at System.Linq.Enumerable+SelectIPartitionIterator<TSource, TResult>.ToList()
   at Serval.Translation.Services.PretranslationService+<GetUsfmAsync>d__6.MoveNext() in /app/src/Serval/src/Serval.Translation/Services/PretranslationService.cs:line 248

[Enkidu93]

I can't find much context from the logs.

[Enkidu93]

Going to query the pretranslation collection and see if any refs look invalid.

[Enkidu93]

I didn't find this exact reference, but there are lots of incorrect references in the database. Most are of the form:

MAT 1:1RandomWordOrPunctuation

I'm not sure how much we can do about these. This may be something we could detect and flag. As long as the refs behave correctly during reinsertion, the translation should get added, but it will be missing a word or punctuation potentially.

There also quite a few with the form:

MAT 1:/1:s1

These are not parsable since ScriptureRef.Parse() tries to parse the VerseRef. Nothing seems unusual with the USFM at a glance. All of these are from fairly old builds though, so maybe this is no longer an issue. I will confirm.

[Enkidu93]

Alright, I tracked down the second issue - which is actually related to the first. If you have USFM like

\id LUK
\c 1
\v 1 verse one
\v 2verse, two
\p
\s section header
\p
\v 3 verse three

NOTE: The comma is important!
For the verse ref created (i.e., LUK 1:2verse,), HasMultiple is true, but AllVerses() gives you LUK 1:2verse and LUK 1: - the second of which is not valid. In other words, it understands this as a range like \v1,2-3 without its final member. (This happens in ScriptureRefUsfmParserHandler.CreateVerseRefs()). Unfortunately, there are verse refs which can exist as VerseRef objects that fail to be parsed from their own ToString().

Given their prevalence, we should definitely 1) flag these for the EITL and for ourselves through a warning and 2) we should consider ToString()ing and TryParseing the refs in the USFM handler and skip unparseable references. We could also use TryParse on the Serval side when we parse the references.

[Enkidu93]

@ddaspit, what do you think?

[Enkidu93]

I've confirmed that the current USFM/versification mismatch detector will flag the verse-ref-smushed-with-word issue, but it actually crashes on the verse-ref-word-comma case. Maybe we could add a category of error there?

[ddaspit]

This seems like a bug in the VerseRef class. We should probably skip these verses. I like the idea of flagging them. We should consider submitting a PR to libpalaso.