[NO-ISSUE] feat(import): use poolboy to have bounded concurrency

[blackheaven]

Proposed changes

Contributor checklist

• My PR is related to <insert ticket number>
• I have read and understood the CONTRIBUTING guide
• I have inserted my change and a link to this PR in the CHANGELOG

Change

I finally found the time to have a shot at a simple bounded concurrency library.

# Before (Streamly)
real    17m31.962s
user    97m15.617s
sys     26m13.162s

# After (Poolboy)
real    10m40.314s
user    92m33.556s
sys     22m42.311s

On my Intel(R) Core(TM) i9-10850K CPU @ 3.60GHz (10 Cores / 20 Threads (HT)).

Regarding max metrics:

• load avg: 32 / 19 / 9
• RAM: 3 GiB

[tchoutri]

Ah great I can now ask the questions I had about poolboy! :D

1. What is the difference with resource-pool?
2. You say it is "Simple". This means you've decided to cut some corners or that you consciously don't support some things. What are they?

[blackheaven]

Ah great I can now ask the questions I had about poolboy! :D

1. What is the difference with resource-pool?

resource-pool is "passive" in the sense, you have a function withResource, which checks the Pool, if there's an element available in it, it takes it, otherwise it creates in (going beyond the limit). Once done, when they are with the item, it tries to put it back in the pool, if they is already as many items, a it is configured in the pool, it frees it. In a sense, it's an unbounded cache pool.

poolboy is quite simple, it spawns N threads polling a queue of computations (it's a work queue).

2. You say it is "Simple". This means you've decided to cut some corners or that you consciously don't support some things. What are they?

It's simple by design, no fancy stuff in, I may have miss some things inherent to work queues (not willingly though).

[tchoutri]

no fancy stuff in

Could you expand a bit on that? I'm not a pool expert in any way. :)

[blackheaven]

no fancy stuff in

Could you expand a bit on that? I'm not a pool expert in any way. :)

We could imagine an option which changes the number of thread depending on the CPU load, it's actually possible to dynamically change the number of thread, but you'll have to do it yourself.

[tchoutri]

Ah perfect, thank you!

[blackheaven]

I think hackage is not up-to-date on the CI

[tchoutri]

Yes you should update the timestamp in the freeze file I believe

[blackheaven]

Weirdly cabal freeze didn't update the snapshot :/

I did some tests regarding Pools' size, it seems that smaller pools are faster:

# Before (Streamly)
real    17m31.962s
user    97m15.617s
sys     26m13.162s

# Poolboy (100)
real    10m40.314s
user    92m33.556s
sys     22m42.311s

# Poolboy (80)
real    8m36.634s
user    93m2.330s
sys     23m12.678s

# Poolboy (60)
real    7m44.719s
user    91m14.997s
sys     22m57.244s

# Poolboy (50)
real    7m46.905s
user    91m17.109s
sys     22m56.540s

# Poolboy (40)
real    7m45.533s
user    90m50.408s
sys     22m51.399s

# Poolboy (30)
real    7m41.438s
user    90m35.791s
sys     22m53.644s

# Poolboy (20)
real    8m8.061s
user    90m22.348s
sys     22m25.652s

[blackheaven]

Seems all right, should I squash?

[tchoutri]

Yep', go for it!

[tchoutri]

@blackheaven I'll let you correct the commit name / PR title and then MergifyBot will merge :)