Deadlock during index creation when parallel workers are used

[ilya071294]

Steps to reproduce:

1. Set parameters in firebird.conf:

MaxParallelWorkers = 8
ParallelWorkers = 8

2. Execute the script in isql:
   index_create_deadlock.txt

As a result 7 out of 8 threads are waiting for a SR lock on the t1_idx1 index. Besides that, another interesting thing is that the main thread (which coordinates worker threads) is waiting for this lock too, while one of the worker threads has already done its job. I guess there is a problem in getWorkItem(). The worker thread sometimes is fast enough to get an item with the main attachment (where m_ownAttach is false) before the main thread. I'm not sure that's the way it's supposed to work.

[hvlad]

The problem is not directly related with parallel index creation, AFAIU.

The main (and only so far) thread run commit and it have two deferred tasks:

• delete_index, work_id = 0 (this is index ID), and
• create index, work_id = 1.

At the DFW phase 3 it first run delete_index task and acquires EX lock on index with ID = 0.
Then it run create index(), phase 3, and going to build new index with ID = 1.
It creates additional worker attachments and each run IndexCreateTask::handler() in new thread.
Every new attachment have to load metadata for table.
There is computed field min_id that depends on procedure p1, that uses index with ID = 0 (now dropping).
Thus when worker attachment acquires SH lock for index with ID = 0 it waits (for main attachment).
The main thread, that owns EX lock for index with ID = 0, runs its own worker successfully, as it don't have to wait for lock on index with ID = 0.
The main thread wait for workers to finish its jobs.
The workers waits for main thread to release lock of index with ID = 0.
Deadlock.

At first look, optimizer should not use suspicious (dropping now) index when compile procedure p1.
For example, it could try to acquire SH lock on candidate index and don't use if lock not acquired.

[ilya071294]

What about the worker thread which can use the main attachment instead of its own?

The main thread, that owns EX lock for index with ID = 0, runs its own worker successfully, as it don't have to wait for lock on index with ID = 0.

As I mentioned before, this is not always the case. When I run the test script, the main thread is waiting for the SR lock. Also, when I try to recreate the index this way in a separate isql session and then use fb_lock_print to determine who owns the EX lock, it shows that the owner has a thread id of the worker thread which is confusing and not actually true.

[hvlad]

What about the worker thread which can use the main attachment instead of its own?

Not sure I understand what you are going to ask here.

With current code the only worker that uses main attachment runs in main thread, this is syncWorker, see Coordinator::runSync() and IndexCreateTask() ctor.

The main thread, that owns EX lock for index with ID = 0, runs its own worker successfully, as it don't have to wait for lock on index with ID = 0.

As I mentioned before, this is not always the case. When I run the test script, the main thread is waiting for the SR lock

I explain what I seen, not what you saw. Anyway, it makes no difference even if worker that uses main attachment runs not in the main thread (looks impossible for me, see above why, but anyway). In any case, the worker that uses main attachment (which owns EX lock) is the only worker that could run its part of task. All other workers waits for the SR lock on dropping index, which can't be released by the main attachment.
I.e. the main thread waits for SR lock, the all other worker threads, except one, also waits for SR lock, the one worker thread is idle. This is what you saw ?

If you want to say that there is some kind of races - please, explain it in more details. So far I not see it.

I try to recreate the index this way in a separate isql session and then use fb_lock_print to determine who owns the EX lock, it shows that the owner has a thread id of the worker thread which is confusing and not actually true.

This could happen if worker thread delivers AST into another owner (marking it with current TID) and thread of that owner didn't enter LM since that.

[hvlad]

With current code the only worker that uses main attachment runs in main thread, this is syncWorker, see Coordinator::runSync() and IndexCreateTask() ctor.

Should correct myself - the sync worker not uses main attachment.
But, as I already said above, it doesn't matters.

[ilya071294]

Not sure I understand what you are going to ask here.

I just wanted to know it's allowed by design or not.

If you want to say that there is some kind of races - please, explain it in more details. So far I not see it.

When I set a breakpoint in IndexCreateTask::getWorkItem, the worker thread gets there first, and it takes the first Item from m_items which has m_ownAttach = false. syncWorker->work(NULL) is called later.
I'm not sure it doesn't matter. Such attachment exchange between the main thread and the worker thread doesn't seem to give any benefits and just makes it more complicated. I gave an example with fb_lock_print.

[hvlad]

I just wanted to know it's allowed by design or not.

Design makes no assumptions on how workers, threads and attachments is related to each other.
Thus I said it doesn't matters.

I'm not sure it doesn't matter.

I can't and don't want to convince you.

Such attachment exchange between the main thread and the worker thread doesn't seem to give any benefits and just makes it more complicated. I gave an example with fb_lock_print.

It is not about benefits or complicate things. Imagine single coordinator working with many tasks simultaneously, when workers and worker threads are acquired and released by different tasks asyncronously.

As for the fb_lock_print - I already explained it. It happens often with any kind of locks and have no direct relation with current subject.
I added TID in owner to make debugging a bit more easy and in most cases it suits its role well, but there is no promise it is absolutely correct at all moments.

[ilya071294]

I added TID in owner to make debugging a bit more easy and in most cases it suits its role well, but there is no promise it is absolutely correct at all moments.

Is there any reliable (and preferably simple, not requiring knowledge of implementation details) way to figure out the exact thread and attachment id which holds the EX lock when such deadlocks happen?

[hvlad]

Attachment ID could be found via corresponding owner id.
Thread ID of attachment can't be known exactly as attachment could be idle and have no running threads at the moment.
Or attachment did not enters LM during current API call and runs not in the same thread as it was when attachment enters LM last time. Note, attachment could use different thread to handle every incoming packet (or API call).
All we have is last known thread ID which was run when attachment last time enters the LM.