deadlock detected

[yzj19870824]

I'm using Fonduer to construct knowledge base from Chinese corpus,

Spacy doesn't support Chinese, So I train a Chinese model by myself , instead of downloading model from Spacy. And my Chinses model cannot split sentences, so the files in the directory which is assigned in TextDocProcess are already splitted by my script.

But when I run featurizer or labeler applys, it's too slow to bear,
Something wrong is recorded in logfile of PostgreSQL:

Not only applying Labeler, but also applying featurizer, deadlock is detected

ERROR: deadlock detected
DETAIL: Process 31694 waits for Sharelock on transaction 71392072; blocked by process 31697.
        Process 31697 waits for Sharelock on transaction 71392071; blocked by process 31694.
        Process 31694: INSERT INTO Label_key(name, candidate_classes) VALUES('subAction_5', ARRAY['sub_action']) ON CONFLICT(name) DO UPDATE SET name=exclued.name, candidate_classes=exclued.candidate_classes;INSERT INTO Label_key(name, candidate_classes) VALUES('subAction_2', ARRAY['sub_action']) ON CONFLICT(name) DO UPDATE SET name=exclued.name, candidate_classes=exclued.candidate_classes;INSERT INTO Label_key(name, candidate_classes) VALUES('subAction_3', ARRAY['sub_action']) ON CONFLICT(name) DO UPDATE SET name=exclued.name, candidate_classes=exclued.candidate_classes;
        Process 31697: INSERT INTO Label_key(name, candidate_classes) VALUES('subAction_5', ARRAY['sub_action']) ON CONFLICT(name) DO UPDATE SET name=exclued.name, candidate_classes=exclued.candidate_classes;INSERT INTO Label_key(name, candidate_classes) VALUES('subAction_2', ARRAY['sub_action']) ON CONFLICT(name) DO UPDATE SET name=exclued.name, candidate_classes=exclued.candidate_classes

HINT: see server log for query details
CONTEXT: while inserting index tuple(1,66) in relation "label_key"
STATEMENT:INSERT INTO Label_key(name, candidate_classes) VALUES('subAction_5', ARRAY['sub_action']) ON CONFLICT(name) DO UPDATE SET name=exclued.name, candidate_classes=exclued.candidate_classes;INSERT INTO Label_key(name, candidate_classes) VALUES('subAction_2', ARRAY['sub_action']) ON CONFLICT(name) DO UPDATE SET name=exclued.name, candidate_classes=exclued.candidate_classes;INSERT INTO Label_key(name, candidate_classes) VALUES('subAction_3', ARRAY['sub_action']) ON CONFLICT(name) DO UPDATE SET name=exclued.name, candidate_classes=exclued.candidate_classes;

I don't know why that happen.

[yzj19870824]

I set parallelism one when I apply featurizer or labeler. And I can bear that speed.
But another problem happens:
The speed of training discriminative model, such as LR, or LSTM, is too slow.
the only reason is DataLoader which fetches the next step data costs almost 7s .
And when I wanna set num_workers 4 or other number, error occurs like this:

THCudaCheck FAIL THCCachingAllocator.cpp line=507 error=3: initialization error.
THCudaCheck FAIL THCCachingAllocator.cpp line=507 error=3: initialization error.
THCudaCheck FAIL THCCachingAllocator.cpp line=507 error=3: initialization error.
THCudaCheck FAIL THCCachingAllocator.cpp line=507 error=3: initialization error.
THCudaCheck FAIL THCCachingAllocator.cpp line=507 error=3: initialization error.
THCudaCheck FAIL THCCachingAllocator.cpp line=507 error=3: initialization error.

Runtime Error: Cuda runtime error(3): initialization error at THCCachingAllocator.cpp:507

the last sentence of error stack is:
X.cuda() classifier.py

there is the same question in pytorch 0.4 or 1.0
I don't know how to speed up the train

[HiromuHota]

@senwu @lukehsiao assign this deadlock issue to me. I have an idea and am working on it.

[HiromuHota]

@yzj19870824 THCCachingAllocator.cpp error is different from the dead lock. I'd suggest to create another issue.