Table column type bigint is converted to AVRO decimal logical type but should be converted to long type

[keweishang]

What did you do?

1. Create a table called employees. Its id column is bigint.

CREATE TABLE employees (
  id bigint(20) unsigned NOT NULL AUTO_INCREMENT,
  employee_name char(50) NOT NULL,
  PRIMARY KEY (id)
);

2. Create TiCDC changefeed.

tiup cdc cli changefeed create \
--pd="http://localhost:2379" \
--sink-uri="kafka://localhost:9092/test?protocol=avro&partition-num=1&replication-factor=3" \
--config=./configs/test.toml \
--opts "registry=http://localhost:8081" \
--changefeed-id=test

3. Insert a row to the table.

INSERT INTO employees values (1, 'test1');

4. Check Kafka Schedule Registry's schema.

...
{
      "name" : "id",
      "type" : {
        "type" : "bytes",
        "logicalType" : "decimal",
        "precision" : 8,
        "scale" : 0
      }
}
...

What did you expect to see?

The AVRO type in Kafka Schema Registry for the id column is decimal AVRO logical type.

What did you see instead?

The AVRO type in Kafka Schema Registry for the id column should be long AVRO type instead.

Versions of the cluster

Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

Release Version: v5.3.0
Edition: Community
Git Commit Hash: 4a1b2e9fe5b5afb1068c56de47adb07098d768d6
Git Branch: heads/refs/tags/v5.3.0
UTC Build Time: 2021-11-24 13:31:41
GoVersion: go1.16.4
Race Enabled: false
TiKV Min Version: v3.0.0-60965b006877ca7234adaced7890d7b029ed1306
Check Table Before Drop: false

TiCDC version (execute cdc version):

5.3.0

[dveeden]

Hi, according to https://avro.apache.org/docs/1.11.0/spec.html#schema_primitive avro's long is a 64-bit signed integer. So I don't think this works with TiDB's BIGINT UNSIGNED. Looking at the code this should work ok for BIGINT which maps to long.

In the code there is this:

var unsignedLongAvroType = avroLogicalType{
        Type:        "bytes",
        LogicalType: decimalType,
        Precision:   8,
        Scale:       0,
}

and

        case mysql.TypeLonglong:
                if col.Flag.IsUnsigned() {
                        return unsignedLongAvroType, nil
                }
                return "long", nil

This is in cdc/sink/codec/avro.go.

[keweishang]

Hi @dveeden, thanks for the update. Indeed, I didn't notice that it is unsigned bigint. In such a case, I think using Avro Decimal is a legit option.

May I ask if TiCDC has a similar configuration as Debezium where the user can configure whether Avro long or Decimal should be used?

[keweishang]

Shall I close this ticket?

[lonng]

@keweishang I think it's OK to keep this issue open status because we haven't the decision about configuration.
cc @leoppro

[keweishang]

@lonng thanks for the update. For the default behavior of using Avro Decimal type. I was also wondering if the precision should be 20 instead of 8, because the max signed bigint has 19 digits (max unsigned bigint has 20 digits): 9,223,372,036,854,775,807.