Hive: Lock Issues with multithreaded commits #2540
Description
When trying to commit to the same Iceberg table from multiple threads in the same application, the meta-store ends up building expensive and large lock chains on the HMS side. This can end up overloading the underlying RDBMS and we have done several fixes (#2263, #1873) to try to reduce this thrashing but I would like to have a better solution if possible.
Since the HiveTableOperations knows that only a single commit operation will ever be able to lock uniquely, it probably makes sense for us to also lock at the JVM level for all incoming HiveTableOperation commit operations. There is never a benefit to allowing a second thread to attempt to acquire a lock while we know a first thread has already acquired it. In fact, we know that the opposite is true. The more simultaneous lock attempts and checks, the greater the pressure on the HMS and worse performance of ALL behaviors.
While we can't stop multiple processes from simultaneously attempting lock (which is the point of the hms lock in the first place), we can do a much cheaper JVM level lock for a single process. For example if we had an application that previously had dozens of threads which all attempt to simultaneously commit, say the SparkThriftServer or something like that. We could add a locking mechanism such that only allow a single thread would be able to attempt to get the hive lock, while all others would queue up behind it. If we don't care about order within threads we could use a synchronize and if we do we could use a concurrent queue or alike.