Records are dropped when merging view parquet files.

[bashir2]

During the incremental step of the pipeline, the merger process has a bug which results in dropping some of the records when merging incremental and old records. This was discovered while I was debugging e2e flakiness after fixing #1295 in #1312. To be specific, this number 108 is incorrect. There should be at least 2*106+1 records (i.e., twice first full-run here).

[bashir2]

For an example of an e2e run that failed because of this bug, see this run; note the discrepancy between 109 vs 108 patient_flat records (both of which are wrong).

Correction: After debugging further and fixing this in #1314, I think the root cause of the particular e2e failure linked above, is not this bug. The numbers 108 and 109 are both wrong and are fixed now but the merging bug seemed to be deterministic and cannot explain that particular discrepancy.