Stop using @threads?

[andreasnoack]

In isolation @threads works okay but Julia doesn't handle nested threaded loops. I work on a project where I'd like to use threads on larger code chunks but I believe it is clashing with GLM's @threads right now. I'll suggest that we postpone using @threads here until Julia has a (more mature) threading model.

[nalimilan]

It's too bad we need to lose an optimization for most users to support an advanced case, but if there's no other solution... I guess passing an argument to disable threading isn't possible?

[andreasnoack]

The problem is that it can be really tricky to detect the issues with nested threaded loops. The feature is really experimental. We could provide a flag for turning it on and off but for the previous reason, I think it should be off by default. From my own experience, such a flag can also be extremely tricky, see JuliaLang/julia#14948.

[nalimilan]

OK, just disable it if you want then.

[dmbates]

Could you create a branch before disabling the threads in the master branch? I have been working with a company in Chicago that is fitting a lot of GLMMs to large data sets and they do see a noticeable difference with multiple threads. I would like to leave them the option of installing such a version of GLM (the MixedModels package uses functions types and methods from GLM).

[ararslan]

@dmbates I realize it hasn't been updated in a while, but would your ParallelGLM package suit their needs?

[dmbates]

In the end I found that the multiple threads was a better option than trying to work with DistributedArrays or SharedArrays. The parallelism is very simple, a lot of the execution time is spent in one loop to update several vectors. Doing this with threads avoids all the setup of exotic array types.

[andreasnoack]

I'll try to make prepare a branch where threading can be enabled/disabled with some kind of flag.

[dmbates]

@andreasnoack I didn't mean that you should need to make a lot of effort to control threads enabling with a flag. I would just prefer if you could create a branch with threading, preferably making the other changes in master first then creating a branch then turning threads off. However, if you think it would work best with a flag that would be great.

[andreasnoack]

@yuyichao has fixed this so I'll close and celebrate.

[Balinus]

How was it fixed? I face the same potential problem with my package (using @threads for climate indices calculations) and having a flag (or information about the fix) would be awesome.

Thanks!
@andreasnoack

[yuyichao]

JuliaLang/julia#21452

[Balinus]

Nice work! Looking at the test, it now looks like nested threaded loop are working. That is a major hurdle going away.

I need to test that on my 96 core server! :)