Multithreaded reduce

[windj007]

Hi Snorkel folks!

I'm trying to use Snorkel v0.5 with Postgres 9.6 and I'm currently struggling with slow feature extraction. Profiling showed that the bottleneck (>90% of time) is session.execute in AnnotatorUDF.reduce. Currently it takes about 300 seconds to extract features from ~300 candidates. Reduce is called from a single thread (the main one). Why not making reduce multithreaded/multiprocessed? Current interface and usage of reduce gives no idea about why it must be single-threaded. Will it improve performance? Maybe there are other possible causes of such slow performance?

[chrismre]

couldn't agree more! The plan is to eventually have some nice scale-out layer here... That said, no one really listens to me :)

I believe here is already a branch that has the multiprocessing functionality on it...bosses, @ajratner @stephenbach ?

[windj007]

Currently, I partially resolved my issue with bulk insert: windj007@f20ccde

But for really large datasets we will need a bit more complicated batching strategy...

[ajratner]

Hi @windj007,

First of all, just to confirm- you are running with parallelism > 1 per the tutorial, correct?

Either way you are right in identifying a current bottleneck in the simple way we implemented parallel execution of Annotator classes (like feature extraction). Basically, extracting the features is done in parallel; however, since we don't know ahead of time which features will be extracted, we need to coordinate somehow on inserting them into the database, hence the single reduce thread.

Bulk insert here was one quick fix that was on our list- excited to look over your code for this, and feel free to submit a PR! As for going beyond this, as Chris said, we do have plans to implement a more high-performance scale out layer at some point.

Thanks,
Alex

[lorenzoranucci]

Hi guys,
First of all, thank you for the beautiful project.

I'm having the same issue because I'm using a large set of documents. As reported in SQLAlchemy FAQ, the solution seems to use bulk inserts and, above all, to chunk inserts. As an example, I noticed a good speed up simply adding a flushing operation in every step of candidate extraction. Now I'm facing a very slow insert when I call save_marginals, here the bulk insert is used, but the insertion is still very slow. Do you think it's safe to persist data in chunks?

[ajratner]

Closing for now, happy to re-open!