S3 csv source lost data

[huangjw806]

Describe the bug

S3 csv source lost data

To Reproduce

s3 csv file:

id,name,idNum
11,xx1,10341
12,xx2,20231

1. lost data：

dev=> create materialized source s1(
    id int,
    name varchar,
    idNum varchar,
    primary key(id)
) with (
    connector = 's3',
    s3.region_name = 'ap-southeast-1',
    s3.bucket_name = 'jianwei-test',
    s3.credentials.access = 'xxxx',
    s3.credentials.secret = 'xxxx'
) row format csv delimited by ',';

dev=> select * from s1;
 id | name | idnum
----+------+-------
 11 | xx1  | 10341
(1 row)

[waruto210]

I guess it's because there is no line break at the end of the last line.

A record ends at a line terminator

Please add and retry, if the error still occurs, please attach the log of the compute node.

[fuyufjh]

A record ends at a line terminator

I don't think that is necessary 🤔

[waruto210]

A record ends at a line terminator

I don't think that is necessary 🤔

The csv definition I found requires this. And the csv lib we used in the csv parser also requires this.

[huangjw806]

It does work fine after adding a line break at the end of the last line.

If it can't be avoided maybe we should inform users in the documentation that s3 files must obey csv rules.

[fuyufjh]

@waruto210 I just tried the official example provided by rust-csv (which is based on csv_core, developed by the same author) and it turns out the last row without new-line is correctly parsed.

[waruto210]

@waruto210 I just tried the official example provided by rust-csv (which is based on csv_core, developed by the same author) and it turns out the last row without new-line is correctly parsed.

Thank you，I'll fix that.