spark SQL在yarn上获取Doris数据中文乱码

[demon-gu]

val dorisSparkDF = sqlContext.read.format("doris")
.option("doris.table.identifier", "rms.ods_rms_device")
.option("doris.fenodes", "aidata-node003:8035")
.option("user", "root")
.option("password", "")
.load()

dorisSparkDF.show()
代码如上，本地跑中文不乱码，提交到yarn上跑中文就乱码

[HappenLee]

This maybe the problem of encoding. Doris only support utf8 now, please check the encoding of spark and yarm.

[demon-gu]

This maybe the problem of encoding. Doris only support utf8 now, please check the encoding of spark and yarm.

ok，thanks

[vagetablechicken]

I've found a similar error. The reason is:

1. be side: use the utf8 charset to encode
   https://github.com/apache/incubator-doris/blob/65d33cf43c837e56a2a36e78b358bfc0a9d1916b/be/src/util/arrow/row_batch.cpp#L80
2. spark-doris-connector side: use the default charset
   https://github.com/apache/incubator-doris/blob/65d33cf43c837e56a2a36e78b358bfc0a9d1916b/extension/spark-doris-connector/src/main/java/org/apache/doris/spark/serialization/RowBatch.java#L271

In my environment, the default charset is US-ASCII, so the Chinese characters become messy.
It's better to specify charset UTF_8 in serialization/RowBatch.