More than 160 DDLs in transaction block crashing the server

[nmantrav2]

Jira Link: DB-4864
On a 3RF universe with c5.large type VMs, tried running a test written in Python with psycopg2.

Test does following steps:

• starts a transaction block, using cur.execute("""BEGIN""") statement
• Executes 500 create table statements
• cur.execute("""COMMIT""")

Test throws 'Timed out: Read RPC (request call id 25658) to 172.151.59.101:9100 timed out after 14.274s" error after about 160 tables are created.

Creation of 500 tables succeeds if we add "SPLIT INTO 1 TABLETS" option to create table statement.

[nmantrav2]

Test code from itest-system test suite. Please edit it for running against local server.

import psycopg2
import time
import random
import logging
from universe_tests.test_universe_base import TestUniverseBase
from multiprocessing.dummy import Pool as ThreadPool
from yw_proxy import YugaWareProxy
class TestDDLsInTxnBlk(TestUniverseBase):
    def __init__(self, host, port, cloud_config):
        super(TestDDLsInTxnBlk, self).__init__(host, port, cloud_config)
        self.num_ddls = 500

    # This test executed DDLs in a transaction block
    def tables_test(self, universe, split_option=True):
        server_ip = \
                self.yw_proxy.get_ysql_servers(universe.universe_uuid).split(',')[0].split(':')[0]
        server_port = \
            self.yw_proxy.get_ysql_servers(universe.universe_uuid).split(',')[0].split(':')[1]
        connect_string = \
            "host={} dbname=yugabyte user=yugabyte port={}".format(server_ip, server_port)
        logging.info("Connection String {}.".format(connect_string))

        conn = psycopg2.connect(connect_string)
        conn.set_session(autocommit=True)
        cur = conn.cursor()

        cur.execute("""BEGIN""")

        for counter in range(self.num_ddls):
            cur.execute("""DROP TABLE IF EXISTS users_{}""".format(counter))
            if split_option:
                cur.execute("""CREATE TABLE users_{}(
                                  id     text,
                                  ename  text,
                                  age    int,
                                  city   text,
                                  about_me text,
                                  PRIMARY KEY(id, ename)) SPLIT INTO 1 TABLETS
                          """.format(counter))
                logging.info("Created table users_{} with split option".format(counter))
            else:
                cur.execute("""CREATE TABLE users_{}(
                                  id     text,
                                  ename  text,
                                  age    int,
                                  city   text,
                                  about_me text,
                                  PRIMARY KEY(id, ename))
                          """.format(counter))

                logging.info("Created table users_{}".format(counter))

        cur.execute("""COMMIT""")

    def do_custom_work(self):
        universe = self.create_and_verify_universe()

        self.tables_test(universe)

        self.tables_test(universe, split_option=False)`

[nmantrav2]

Closed by mistake.

[ddorian]

Yes creating a lot of tablets has overhead especially in small machines.
This is related to https://github.com/yugabyte/yugabyte-db/projects/28.
I think this will be fixed by Creating colocated tables by default: #5239

[nmantrav2]

In a recent run, I got "InternalError_('Timed out: Re ad(tablet: c963dbca7de74163b5c0cfeb092e186e, num_ops: 1, num_attempts: 201, txn: 04cdc811-5135-4d8b-a745-4f7782e63774) passed its deadline 79 6.311s (passed: 60.489s): Leader does not have a valid lease. (yb/consensus/consensus.cc:164): This leader has not yet acquired a lease. (tab let server error 15)\n')" error.

[ddorian]

Leader does not have a valid lease. (yb/consensus/consensus.cc:164): This leader has not yet acquired a lease. (tab let server error 15)\n')" error.

Probably the master crashed here. Can you retry the same experiment but in a colocated database ?