[ML] Allow a certain number of ill-formatted rows when delimited format is specified

[benwtrent]

While it is good to not be lenient when attempting to guess the file format, it is frustrating to users when they KNOW it is CSV but there are a few ill-formatted rows in the file (via some entry error, etc.).

This commit allows for up to 10% of sample rows to be considered "bad". These rows are effectively ignored while guessing the format.

This percentage of "allows bad rows" is only applied when the user has specified delimited formatting options. As the structure finder needs some guidance on what a "bad row" actually means.

related to #38890

[elasticmachine]

Pinging @elastic/ml-core (:ml)

[droberts195]

I think you should try out the end-to-end UI experience with this change. Try:

1. The CSV file with different numbers of fields from [ML] Data Visualizer to accept data without timestamp kibana#60196 (comment) with no overrides
2. The CSV file with different numbers of fields from [ML] Data Visualizer to accept data without timestamp kibana#60196 (comment) with the delimiter overridden to ,
3. The CSV file with different numbers of fields from [ML] Data Visualizer to accept data without timestamp kibana#60196 (comment) with the structure overridden to delimited, but no explicit delimiter specified
4. A semi-structured application log that is not CSV with the number of lines to analyse overridden to 100000

[benwtrent]

@droberts195 to test the end-to-end UI experience, the uploader needs to allow me to supply overrides after the initial parsing failed. When I last tried, this is not possible.

I will do some patching locally to work around this.

[droberts195]

to test the end-to-end UI experience, the uploader needs to allow me to supply overrides after the initial parsing failed. When I last tried, this is not possible.

Good point.

You could still test the case of a semi-structured log file though. Check that the explanations of why it wasn't delimited are formatted correctly.

I will do some patching locally to work around this.

Yes, you could just hardcode the overrides onto every call to find_file_structure in a local Kibana build. Obviously that local build will only be any good for the single test case, but it's a way to escape the chicken-and-egg situation of needing both this backend change and a UI change to see the full benefit.

[benwtrent]

Scenarios tested:

1. It fails. Saying the row it failed on and why (number of fields differing than the first row).
2. It succeeds and data was able to be imported without issue
3. It succeeds and data was able to be imported without issue
4. I did not experience any timeouts. Without overrides, it behaves much like it has before as the very first line that has a mismatch causes the failure.

[benwtrent]

@elasticmachine update branch

[droberts195]

The more I look at this the more complicated a seemingly simple idea becomes 😢

[benwtrent]

run elasticsearch-ci/packaging-sample-unix-docker

[droberts195]

LGTM if you could just adjust a comment to make clear that totalNumberOfRows is an approximation.

Thanks for all the iterations on this and end-to-end testing with the UI.

The bug with lineMergeSizeLimit not being considered for CSV can be left to another PR or fixed in this one if you prefer.

[benwtrent]

@elasticmachine update branch

[przemekwitek]

LGTM