Build fails when sending message with a large number of versification warnings

[pmachapman]

I am documenting this, not sure whether it should be fixed or not.

I had accidentally set the wrong versification for project, and started a build, which crashed with the following error:

info: Serval.Machine.Shared.Services.NmtPreprocessBuildJob[0]
      {"Event":"BuildPreprocess","EngineId":"68a67829fc48dd84c2668600","BuildId":"6938a86931469496ecee9cd2","NumTrainRows":34248,"NumPretranslateRows":2508,"EngineSourceLanguageTag":"la","EngineTargetLanguageTag":"en","SourceLanguageResolved":"lat_Latn","TargetLanguageResolved":"eng_Latn","Warnings":["USFM does not match project versification for parallel corpus 68a6782bfc48dd84c2668609, monolingual corpus 68a6782afc48dd84c2668603: Expected verse , Actual verse GEN 5:31, Mismatch type MissingVerse", ...]}
fail: Serval.Machine.Shared.Services.NmtPreprocessBuildJob[0]
      Build faulted (6938a86931469496ecee9cd2)
      System.ArgumentException: The content is too large for request UpdateBuildExecutionData with group ID 68a67829fc48dd84c2668600. It is 1058919 bytes, but the maximum is 1000000 bytes.
         at SIL.ServiceToolkit.Services.OutboxService.EnqueueMessageAsync(String outboxId, String method, String groupId, Object content, Stream stream, CancellationToken cancellationToken) in /app/src/ServiceToolkit/src/SIL.ServiceToolkit/Services/OutboxService.cs:line 51
         at Serval.Machine.Shared.Services.ServalTranslationPlatformService.UpdateBuildExecutionDataAsync(String engineId, String buildId, BuildExecutionData executionData, CancellationToken cancellationToken) in /app/src/Machine/src/Serval.Machine.Shared/Services/ServalTranslationPlatformService.cs:line 172
         at Serval.Machine.Shared.Services.NmtPreprocessBuildJob.UpdateBuildExecutionData(String engineId, String buildId, Int32 trainCount, Int32 pretranslateCount, String sourceLanguageTag, String targetLanguageTag, IReadOnlyList`1 corpora, CancellationToken cancellationToken) in /app/src/Machine/src/Serval.Machine.Shared/Services/NmtPreprocessBuildJob.cs:line 116
         at Serval.Machine.Shared.Services.PreprocessBuildJob`1.DoWorkAsync(String engineId, String buildId, IReadOnlyList`1 data, String buildOptions, CancellationToken cancellationToken) in /app/src/Machine/src/Serval.Machine.Shared/Services/PreprocessBuildJob.cs:line 51
         at Serval.Machine.Shared.Services.HangfireBuildJob`2.RunAsync(String engineId, String buildId, TData data, String buildOptions, CancellationToken cancellationToken) in /app/src/Machine/src/Serval.Machine.Shared/Services/HangfireBuildJob.cs:line 56
fail: Hangfire.AutomaticRetryAttribute[0]
      Failed to process the job '6938a869f97b795d1c2e0199': an exception occurred.
      System.ArgumentException: The content is too large for request UpdateBuildExecutionData with group ID 68a67829fc48dd84c2668600. It is 1058919 bytes, but the maximum is 1000000 bytes.
         at SIL.ServiceToolkit.Services.OutboxService.EnqueueMessageAsync(String outboxId, String method, String groupId, Object content, Stream stream, CancellationToken cancellationToken) in /app/src/ServiceToolkit/src/SIL.ServiceToolkit/Services/OutboxService.cs:line 51
         at Serval.Machine.Shared.Services.ServalTranslationPlatformService.UpdateBuildExecutionDataAsync(String engineId, String buildId, BuildExecutionData executionData, CancellationToken cancellationToken) in /app/src/Machine/src/Serval.Machine.Shared/Services/ServalTranslationPlatformService.cs:line 172
         at Serval.Machine.Shared.Services.NmtPreprocessBuildJob.UpdateBuildExecutionData(String engineId, String buildId, Int32 trainCount, Int32 pretranslateCount, String sourceLanguageTag, String targetLanguageTag, IReadOnlyList`1 corpora, CancellationToken cancellationToken) in /app/src/Machine/src/Serval.Machine.Shared/Services/NmtPreprocessBuildJob.cs:line 116
         at Serval.Machine.Shared.Services.PreprocessBuildJob`1.DoWorkAsync(String engineId, String buildId, IReadOnlyList`1 data, String buildOptions, CancellationToken cancellationToken) in /app/src/Machine/src/Serval.Machine.Shared/Services/PreprocessBuildJob.cs:line 51
         at Serval.Machine.Shared.Services.HangfireBuildJob`2.RunAsync(String engineId, String buildId, TData data, String buildOptions, CancellationToken cancellationToken) in /app/src/Machine/src/Serval.Machine.Shared/Services/HangfireBuildJob.cs:line 56
         at Serval.Machine.Shared.Services.HangfireBuildJob`2.RunAsync(String engineId, String buildId, TData data, String buildOptions, CancellationToken cancellationToken) in /app/src/Machine/src/Serval.Machine.Shared/Services/HangfireBuildJob.cs:line 120
         at Serval.Machine.Shared.Services.HangfireBuildJob`2.RunAsync(String engineId, String buildId, TData data, String buildOptions, CancellationToken cancellationToken) in /app/src/Machine/src/Serval.Machine.Shared/Services/HangfireBuildJob.cs:line 124
         at InvokeStub_TaskAwaiter.GetResult(Object, Object, IntPtr*)
         at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)

Should we limit the number of these versification warnings (5274 were present in the execution data), or do you think that having the build fail (as it did in this case) is sufficient?

[ddaspit]

Yes, we should definitely limit the number of versification warnings.