Bug Report: No schema found for table in vstreamer for tables with TABLESPACE innodb_system

[kovyrin]

Overview of the Issue

When a table is created in Vitess with a TABLESPACE innodb_system option (as a result of https://bugs.mysql.com/bug.php?id=110383 or for some other reason) and Vitess is running on a MySQL 8.0 cluster, it leads to vstreamer failing with an unexpected "No schema found for table" error.

This is caused by the fact, that the schema engine code relies on a query to information schema that joins tables with innodb_tablespaces using a RIGHT JOIN to get a list of tables in a given database, while mysql does not provide tablespace information for tables within the innodb_system tablespace. This leads to all tables with the TABLESPACE innodb_system option being filtered out from the result set. Later, when vstreamer uses the schema engine to get table information, it blows up failing to find the table.

Interestingly enough, the 5.7 query uses a LEFT JOIN, so it does not suffer from the same issue.

Reproduction Steps

1. Start Vitess on MySQL 8.0
2. Create a test table:

CREATE TABLE `test1` (
  `id` bigint NOT NULL AUTO_INCREMENT,
  PRIMARY KEY (`id`)
) /*!50100 TABLESPACE `innodb_system` */ ENGINE=InnoDB;

4. Start a VStream that would include events from the new test table.
5. Create a record in the test table:

insert into test1 set id=1;

6. VStream client will blow up with an exception like this:

java.lang.RuntimeException: io.grpc.StatusException: UNKNOWN: vttablet: rpc error: code = Unknown desc = stream (at source tablet) error @ 920df59c-c72c-11ed-913e-fef65b3db030:1-405: unknown table test1 in schema

7. VTtablet will have the following log messages:

I0320 15:14:24.082870     128 tablet_picker.go:157] tablet picker found tablet alias:{cell:"test" uid:7} port_map:{key:"grpc" value:9007} port_map:{key:"vt" value:8007} keyspace:"storefront_renderer_dev_shard" shard:"0" type:PRIMARY db_name_override:"vt_storefront_renderer_dev_shard_0" mysql_port:6114 primary_term_start_time:{seconds:1679322981 nanoseconds:12757099}
I0320 15:14:24.083063     128 vstream_manager.go:461] Picked tablet cell:"test" uid:7 for for test/storefront_renderer_dev_shard/0/PRIMARY
I0320 15:14:24.083090     128 vstream_manager.go:498] Starting to vstream from cell:"test" uid:7
I0320 15:14:24.083166     128 uvstreamer.go:367] Stream() called
I0320 15:14:24.084761     128 uvstreamer.go:302] sendEventsForCurrentPos
I0320 15:14:24.084832     128 vstreamer.go:155] Starting Stream() with startPos MySQL56/920df59c-c72c-11ed-913e-fef65b3db030:1-403
I0320 15:14:24.086258     128 binlog_connection.go:79] new binlog connection: serverID=1951836689
I0320 15:14:24.086343     128 binlog_connection.go:125] sending binlog dump command: startPos=920df59c-c72c-11ed-913e-fef65b3db030:1-403, serverID=1951836689
I0320 15:14:35.900463     128 engine.go:490] table test1 not found in vttablet schema, current tables: map[.... a list of tables that do not include test1...]
I0320 15:14:35.904363     128 vstreamer.go:724] No schema found for table test1
E0320 15:14:35.904439     128 vstreamer.go:938] stream (at source tablet) error @ 920df59c-c72c-11ed-913e-fef65b3db030:1-405: unknown table test1 in schema
I0320 15:14:35.904497     128 binlog_connection.go:279] closing binlog socket to unblock reads
I0320 15:14:35.904589     128 binlog_connection.go:287] waiting for binlog dump thread to end
I0320 15:14:35.904607     128 binlog_connection.go:293] closing binlog MySQL client with serverID 1951836689. Will recycle ID.
E0320 15:14:35.912928     128 vstream_manager.go:600] vstream for storefront_renderer_dev_shard/0 error: Code: UNKNOWN
vttablet: rpc error: code = Unknown desc = stream (at source tablet) error @ 920df59c-c72c-11ed-913e-fef65b3db030:1-405: unknown table test1 in schema
E0320 15:14:35.913513     128 vstream_manager.go:316] Error in vstream for keyspace:"storefront_renderer_dev_shard" shard:"0" gtid:"MySQL56/920df59c-c72c-11ed-913e-fef65b3db030:1-404": vttablet: rpc error: code = Unknown desc = stream (at source tablet) error @ 920df59c-c72c-11ed-913e-fef65b3db030:1-405: unknown table test1 in schema

Binary Version

Tried it on a custom-built 14.0.0, but the issue exists in main as well.

Operating System and Environment details

My vttestserver is running on an OCI image, but I do not believe that is relevant:


NAME="Oracle Linux Server"
VERSION="8.7"
ID="ol"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="8.7"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Oracle Linux Server 8.7"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:oracle:linux:8:7:server"
HOME_URL="https://linux.oracle.com/"
BUG_REPORT_URL="https://bugzilla.oracle.com/"

ORACLE_BUGZILLA_PRODUCT="Oracle Linux 8"
ORACLE_BUGZILLA_PRODUCT_VERSION=8.7
ORACLE_SUPPORT_PRODUCT="Oracle Linux"
ORACLE_SUPPORT_PRODUCT_VERSION=8.7

[kovyrin]

I believe #12672 should fix the issue.