Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Odilia Pins CPU at 100% #19

Closed
TTWNO opened this issue Aug 29, 2022 · 3 comments
Closed

Odilia Pins CPU at 100% #19

TTWNO opened this issue Aug 29, 2022 · 3 comments
Labels
bug Something isn't working Core Implementation of core functionality help wanted Extra attention is needed

Comments

@TTWNO
Copy link
Member

TTWNO commented Aug 29, 2022

Try running Odilia either in debug or release mode and notice that it will pin one of your cores at 100%.

Can @mcb2003 or @albertotirla confirm this?

Either it's a bug I'm doing something weird to trigger, or this is a serious problem probably based around polling constantly for events, etc.
I know this is sometimes a problem for applications which constantly poll for updates, perhaps some kind of pause is in order, but I'm not certain where.

I have tested this behavior on both stable (1.63) and nightly (1.65) with the same result.

This is a critical bug, if confirmed.

@albertotirla
Copy link
Member

yes, it's a critical bug indeed, however it's not entirely consistent. Take this output, for example

$ ps -eo pcpu,pid,user,args | sort -r -k1 |grep odilia
 0.8    2241 owner    target/debug/odilia
$ 

it's not exactly 100, but that's got me woried worse than any other bugs this may have. I wonder what's causing it to spike like that? perhaps a digging with tokio-console would be required, since gdb isn't helping at all, I get lost in the nested types.
yes, perhaps it's because we continuously pole for events, but that might also be because either we didn't enable a zbus mechanism, or zbus doesn't just wake our task when new events appear, in stead it continuously poles the queue, waisting system resources. Also, since the executor doesn't have many other tasks to run in parallel and there aren't wakers associated with our event handler task if my hypothesis about zbus is correct, tokio would continuously pole our task in a busy loop, which totally explains the 100% cpu usage.
the solution would be to drastically increase the number of tasks handling events, perhaps many short lived tasks handling the event stream, one per event, may sometimes be better than a huge one which basically does everything, not giving the executor the chance to switch between other tasks because there aren't many others. I think using a single threaded executor won't help matters here, actually it would put even more stress on that one core, eventually being unable to keep up with zbus events and throttling the channel, deadlocking it, as we've seen before.
any ideas? terminal outputs? solutions? this is a critical bug, we can't let this go in release.

@albertotirla albertotirla added bug Something isn't working help wanted Extra attention is needed Core Implementation of core functionality labels Aug 29, 2022
@mcb2003
Copy link
Contributor

mcb2003 commented Aug 30, 2022

Doesn't do it fore me:

$ ps -eo pcpu,pid,user,args | sort -r -k1 |grep '[o]dilia'
 0.4   46931 mikey    ./target/release/odilia
$ ps -eo pcpu,pid,user,args | sort -r -k1 |grep '[o]dilia'
 0.2   46931 mikey    ./target/release/odilia
$ ps -eo pcpu,pid,user,args | sort -r -k1 |grep '[o]dilia'
 0.1   46931 mikey    ./target/release/odilia
$ ps -eo pcpu,pid,user,args | sort -r -k1 |grep '[o]dilia'
 0.1   46931 mikey    ./target/release/odilia

Consumes some CPU at the start before settling down as you'd expect.

That also pretty much disproves my theory that this is causing #20. :-(

@albertotirla
Copy link
Member

hmm, this is very, very interesting, works on my machine symdrome all over again lol. It seemns we desperately need those testers, we can't reliably test the software we programmed because we already know the happy path so to speak, so our minds aren't detached enough to test the...hmm...less happy path.

@TTWNO TTWNO closed this as completed in b3afb89 Aug 30, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working Core Implementation of core functionality help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

3 participants