Lookup vindex with lookup key type 'unicode_loose_md5' seems to be broken in 8.0.0

[bezmax]

Overview of the Issue

While trying to upgrade to v8.0.0 we have noticed that case-insensitive vindexes seem to not be working anymore.

Reproduction Steps

From the directory with sources of v8.0.0 of Vitess:

make docker_local
./docker/local/run.sh

In docker shell:

source ./env.sh
CELL=zone1 TABLET_UID=300 ./scripts/mysqlctl-up.sh
CELL=zone1 TABLET_UID=301 ./scripts/mysqlctl-up.sh
CELL=zone1 TABLET_UID=400 ./scripts/mysqlctl-up.sh
CELL=zone1 TABLET_UID=401 ./scripts/mysqlctl-up.sh
CELL=zone1 TABLET_UID=500 ./scripts/mysqlctl-up.sh
CELL=zone1 TABLET_UID=501 ./scripts/mysqlctl-up.sh
CELL=zone1 TABLET_UID=600 ./scripts/mysqlctl-up.sh
CELL=zone1 TABLET_UID=601 ./scripts/mysqlctl-up.sh
SHARD=-80 CELL=zone1 KEYSPACE=customer TABLET_UID=300 ./scripts/vttablet-up.sh
SHARD=-80 CELL=zone1 KEYSPACE=customer TABLET_UID=301 ./scripts/vttablet-up.sh
SHARD=80- CELL=zone1 KEYSPACE=customer TABLET_UID=400 ./scripts/vttablet-up.sh
SHARD=80- CELL=zone1 KEYSPACE=customer TABLET_UID=401 ./scripts/vttablet-up.sh
SHARD=-80 CELL=zone1 KEYSPACE=lookup TABLET_UID=500 ./scripts/vttablet-up.sh
SHARD=-80 CELL=zone1 KEYSPACE=lookup TABLET_UID=501 ./scripts/vttablet-up.sh
SHARD=80- CELL=zone1 KEYSPACE=lookup TABLET_UID=600 ./scripts/vttablet-up.sh
SHARD=80- CELL=zone1 KEYSPACE=lookup TABLET_UID=601 ./scripts/vttablet-up.sh
vtctlclient InitShardMaster -force customer/-80 zone1-300
vtctlclient InitShardMaster -force customer/80- zone1-400
vtctlclient InitShardMaster -force lookup/-80 zone1-500
vtctlclient InitShardMaster -force lookup/80- zone1-600
vtctlclient ApplySchema -sql-file lookup_schema.sql lookup
vtctlclient ApplyVSchema -vschema_file lookup_vschema.json lookup
vtctlclient ApplySchema -sql-file customer_schema.sql customer
vtctlclient ApplyVSchema -vschema_file customer_vschema.json customer

After connecting to DB with mysql client:

USE customer;
INSERT INTO aliases (id, alias, customer_id) VALUES (1, 'test', 1);
SELECT * FROM aliases WHERE alias = 'Test';

Expected result: 1 row
Actual result: 0 rows

Files required to reproduce the issue:

lookup_vschema.json

{
  "sharded": true,
  "vindexes": {
    "varchar_hash": {
      "type": "unicode_loose_md5"
    }
  },
  "tables": {
    "aliases_alias_lookup": {
      "columnVindexes": [
        {
          "column": "alias",
          "name": "varchar_hash"
        }
      ]
    }
  }
}

lookup_schema.sql

 CREATE TABLE `aliases_alias_lookup` (
  `id` bigint(20) NOT NULL AUTO_INCREMENT,
  `alias` varchar(191) NOT NULL,
  `customer_id` bigint(20) NOT NULL,
  PRIMARY KEY (`id`),
  KEY `idx_alias` (`alias`)
) ENGINE=InnoDB AUTO_INCREMENT=1237907392 DEFAULT charset=utf8mb4 COLLATE=utf8mb4_general_ci;

customer_vschema.json

{
  "sharded": true,
  "vindexes": {
    "hash": {
      "type": "hash"
    },
    "aliases_alias_lookup": {
      "type": "lookup_hash",
      "params": {
        "table": "lookup.aliases_alias_lookup",
        "from": "alias",
        "to": "customer_id",

        "ignore_nulls": "true",
        "autocommit_on_insert": "true",
        "upsert_on_update": "false",
        "upsert_on_insert": "true",
        "scatter_if_absent": "true",

        "autocommit": "true"
      },
      "owner": "aliases"
    }
  },
  "tables": {
    "aliases": {
      "columnVindexes": [
        {
          "column": "customer_id",
          "name": "hash"
        },
        {
          "column": "alias",
          "name": "aliases_alias_lookup"
        }
      ]
    }
  }
}

customer_schema.sql

CREATE TABLE `aliases` (  
    `id` bigint(20) NOT NULL AUTO_INCREMENT,  
    `alias` varchar(191) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NOT NULL,  
    `customer_id` bigint(20) NOT NULL,  
    `hashed_alias_id` bigint(20) DEFAULT NULL,  
    PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=440699859 DEFAULT charset=utf8 COLLATE=utf8_general_ci;

Binary version

Git tag v8.0.0, commit 7e09d0c.

Version: 7e09d0c20 (Git branch 'HEAD') built on Tue Jan  5 17:54:43 UTC 2021 by vitess@504c9a859469 using go1.13.15 linux/amd64

Operating system and Environment details

Bug is reproducible in docker_local environment.

Log Fragments

/debug/querylog output when running the problematic query:

On master tablet in customer namespace:

 SelectImpossible        "select * from aliases where 1 != 1"

On master tablet in lookup namespace:

Select  "select alias, customer_id from aliases_alias_lookup where alias in ::__vals"   map[#maxLimit:type:INT64 value:"10001"  __vals:type:VARBINARY value:"1 items"  alias:type:VARBINARY value:"1 items" ]    1       "select alias, customer_id from aliases_alias_lookup where alias in ('Test') limit 10001"

[bezmax]

I believe the problem is in this piece of code being case-sensitive:

vitess/go/vt/vtgate/vindexes/lookup_internal.go

Lines 120 to 128 in 88671b0

	for _, row := range result.Rows {
	resultMap[row[0].ToString()] = append(resultMap[row[0].ToString()], []sqltypes.Value{row[1]})
	}
	for _, id := range ids {
	results = append(results, &sqltypes.Result{
	Rows: resultMap[id.ToString()],
	})
	}