Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

lightning: specify collation when parquet value to string datum #38391

Merged
merged 3 commits into from
Oct 17, 2022
Merged

lightning: specify collation when parquet value to string datum #38391

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
9d2448a
lightning: specify collation when parquet value to string datum
dsdashun Oct 11, 2022
62d9688
adapt the unit tests
dsdashun Oct 11, 2022
51cd5da
Merge branch 'master' into fix-38351
ti-chi-bot Oct 17, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions br/pkg/lightning/mydump/parquet_parser.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -458,7 +458,7 @@ func setDatumByString(d *types.Datum, v string, meta *parquet.SchemaElement) {
ts = ts.UTC()
v = ts.Format(utcTimeLayout)
}
d.SetString(v, "")
d.SetString(v, "utf8mb4_bin")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are many places need to consider string encodings, one is string data in parquet file, the other one is string variables in the memory of lightning process which read by parquet reader. Since golang string is always assumed utf8-encoded I think this PR is OK. But I'm not sure if parquet file has another encoding for string data and go-parquet reader wrongly cast it to golang string without encode/decode.

}

func binaryToDecimalStr(rawBytes []byte, scale int) string {
Expand Down Expand Up @@ -515,20 +515,20 @@ func setDatumByInt(d *types.Datum, v int64, meta *parquet.SchemaElement) error {
}
val := fmt.Sprintf("%0*d", minLen, v)
dotIndex := len(val) - int(*meta.Scale)
d.SetString(val[:dotIndex]+"."+val[dotIndex:], "")
d.SetString(val[:dotIndex]+"."+val[dotIndex:], "utf8mb4_bin")
case logicalType.DATE != nil:
dateStr := time.Unix(v*86400, 0).Format("2006-01-02")
d.SetString(dateStr, "")
d.SetString(dateStr, "utf8mb4_bin")
case logicalType.TIMESTAMP != nil:
// convert all timestamp types (datetime/timestamp) to string
timeStr := formatTime(v, logicalType.TIMESTAMP.Unit, timeLayout,
utcTimeLayout, logicalType.TIMESTAMP.IsAdjustedToUTC)
d.SetString(timeStr, "")
d.SetString(timeStr, "utf8mb4_bin")
case logicalType.TIME != nil:
// convert all timestamp types (datetime/timestamp) to string
timeStr := formatTime(v, logicalType.TIME.Unit, "15:04:05.999999", "15:04:05.999999Z",
logicalType.TIME.IsAdjustedToUTC)
d.SetString(timeStr, "")
d.SetString(timeStr, "utf8mb4_bin")
default:
d.SetInt64(v)
}
Expand Down
2 changes: 1 addition & 1 deletion br/pkg/lightning/mydump/parquet_parser_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ func TestParquetParser(t *testing.T) {
verifyRow := func(i int) {
require.Equal(t, int64(i+1), reader.lastRow.RowID)
require.Len(t, reader.lastRow.Row, 2)
require.Equal(t, types.NewCollationStringDatum(strconv.Itoa(i), ""), reader.lastRow.Row[0])
require.Equal(t, types.NewCollationStringDatum(strconv.Itoa(i), "utf8mb4_bin"), reader.lastRow.Row[0])
require.Equal(t, types.NewIntDatum(int64(i)), reader.lastRow.Row[1])
}

Expand Down
6 changes: 3 additions & 3 deletions br/pkg/lightning/restore/get_pre_info_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -349,15 +349,15 @@ INSERT INTO db01.tbl01 (ival, sval) VALUES (444, 'ddd');`
ExpectFirstRowDatums: [][]types.Datum{
{
types.NewIntDatum(1),
types.NewCollationStringDatum("name_1", ""),
types.NewCollationStringDatum("name_1", "utf8mb4_bin"),
},
{
types.NewIntDatum(2),
types.NewCollationStringDatum("name_2", ""),
types.NewCollationStringDatum("name_2", "utf8mb4_bin"),
},
{
types.NewIntDatum(3),
types.NewCollationStringDatum("name_3", ""),
types.NewCollationStringDatum("name_3", "utf8mb4_bin"),
},
},
ExpectColumns: []string{"id", "name"},
Expand Down