How to configure Strimzi Kafka to support Japanese fonts?

[leehung-net]

Hello Strimzi team and all,

I am currently using Strimzi Kafka on AWS EKS, and I am facing an issue with displaying Japanese characters properly. Some Japanese text appears corrupted when processed or stored.

I would like to know the proper way to configure Strimzi Kafka to fully support Japanese fonts and character encoding. Specifically:

Should I install additional fonts (e.g., Noto CJK) in the Kafka container?
How can I set the appropriate locale (e.g., ja_JP.UTF-8) for the Kafka brokers within the Strimzi configuration?
Are there any specific configurations required for Kafka topics or client connections to handle Japanese text properly?
Below is a snippet of my current Kafka custom resource configuration:

apiVersion: kafka.strimzi.io/v1beta2 kind: Kafka metadata: name: my-cluster namespace: kafka annotations: strimzi.io/node-pools: enabled strimzi.io/kraft: enabled spec: kafka: version: 3.7.1 replicas: 1 listeners: - name: plain port: 9092 type: internal tls: false - name: tls port: 9093 type: internal tls: true template: pod: securityContext: fsGroup: 0 runAsUser: 0 runAsGroup: 0 kafkaContainer: securityContext: runAsUser: 0 runAsGroup: 0 env: - name: KAFKA_OPTS value: "-Dfile.encoding=UTF-8" - name: LANG value: "ja_JP.UTF-8" - name: LC_ALL value: "ja_JP.UTF-8" config: offsets.topic.replication.factor: 1 transaction.state.log.replication.factor: 1 transaction.state.log.min.isr: 1 default.replication.factor: 1 min.insync.replicas: 1 metricsConfig: type: jmxPrometheusExporter valueFrom: configMapKeyRef: name: kafka-metrics key: kafka-metrics-config.yml entityOperator: topicOperator: {} userOperator: {} kafkaExporter: topicRegex: ".*" groupRegex: ".*"

If there's an existing guide or best practice for configuring Strimzi Kafka to handle Japanese characters, could you please point me to it?

Thank you for your support!

[scholzj]

What exactly do you want to use the Japanese characters for? Kafka source code is not translated into Japanese. So, any log messages etc. will still be in English. And when it comes to the messages, for Kafka brokers, they are all just byte arrays. So, it does not need any special support for various languages.

[leehung-net]

Hi @scholzj,
Thanks for your answer.

I am using Kafka Connect and Kafka. Currently, I am using https://github.com/jcustenborder/kafka-connect-spooldir to read CSV files.
For CSV files with Latin characters, everything works fine, but when the files contain Japanese characters, they cannot be read.
I believe this issue is related to fonts, so I would like to know if there is a way to configure Kafka and Kafka Connect to support the Japanese language?

Message in Topic: