CPU overhead for submitting tasks

[simon-tarr]

Hello,

Not an issue but more a question;

What's the expected (local) CPU overhead for submitting tasks when the data being uploaded to batch is minimal (say, a dataframe <5MB)? I have three concurrent R sessions running jobs on Azure and it can quite comfortably suck all my clock cycles if all three sessions happen to be submitting tasks at the same time.

I'm running Pop_OS! (Ubuntu 18.10) on a Dell XPS 15 9550. This laptop is a few years old now but it's no slouch by any means...I'd have expected substantially less resource overhead given that the processing is occuring in the cloud.

I notice a similarly large increase in CPU usage on my Win10 workstation at uni but, as it's a far more powerful machine, it's always usable if many tasks are being submitted concurrently across R sessions.

Perhaps this ties into issue #300 and #217?

Cheers,
SImon

[brnleehng]

Hi Simon,

I'll do some benchmarking on CPU overhead on task submission. How many tasks is each R session submitting?

Thanks,
Brian

[simon-tarr]

Thanks Brian.

I'm submitting 511 tasks within each job (to a pool of 512 cores).

EDIT - I've noticed that the problem is excercerbated if one (or more) pools is downloading a merged result. When returning a result, this will also cause high CPU load and really slow down the time it takes to then submit tasks in other sessions.

I'm on a gigabit internet connection at my university so I don't think it's a bandwidth issue causing slow uploads/downloads. The .csv returned from each run is only 8.4MB in size so I'd expect it to download pretty rapidly over both my work and home connection.