[Bug] Very high CPU and memory usage

[010011x]

Describe the bug
I install openbb via conda. Immediately after starting openbb, memory and CPU usage of the openbb process jump to very high numbers.
Unfortunately, this makes this amazing software unusable for me.

To Reproduce

conda env create -n obb --file https://raw.githubusercontent.com/OpenBB-finance/OpenBBTerminal/main/build/conda/conda-3-9-env.yaml
conda activate obb
pip install openbb --no-cache-dir
openbb

Screenshots
top

perf

Desktop (please complete the following information):
LXQT

• OS: EndeavourOS
• Python version: 3.9.13

[010011x]

I see a looooooooot of these in strace output

recvmsg(10, {msg_namelen=0}, 0)         = -1 EAGAIN (Resource temporarily unavailable)
clock_gettime(CLOCK_MONOTONIC, {tv_sec=13223, tv_nsec=41713144}) = 0
clock_gettime(CLOCK_MONOTONIC, {tv_sec=13223, tv_nsec=41760500}) = 0
clock_gettime(CLOCK_MONOTONIC, {tv_sec=13223, tv_nsec=41806390}) = 0
clock_gettime(CLOCK_MONOTONIC, {tv_sec=13223, tv_nsec=41852140}) = 0
clock_gettime(CLOCK_MONOTONIC, {tv_sec=13223, tv_nsec=41899915}) = 0

[deeleeramone]

Curious if your installation is getting configured as Linux/AMD instead AARCH. It might be a good idea to try using Python 3.10, just replace the 9 in the URL.

[010011x]

Curious if your installation is getting configured as Linux/AMD instead AARCH. It might be a good idea to try using Python 3.10, just replace the 9 in the URL.

can you please elaborate ? I'm on AMD

uname -m
x86_64

Tried doing Python 3.10, same issue.
Thanks a lot !

[deeleeramone]

I see, so much for that theory! The reason I was curious architecture emulation can be a real drag sometimes, but that does not fit with your symptoms. It is extremely difficult to support all Linux configurations, because there are many variables and variations.

Some distributions work better with Miniforge: https://github.com/conda-forge/miniforge#miniforge

What is your CPU generation, make, and model? Please be as specific as possible with your hardware and software configurations. Otherwise, we don't really have a starting point. :)

[010011x]

I see, so much for that theory! The reason I was curious architecture emulation can be a real drag sometimes, but that does not fit with your symptoms. It is extremely difficult to support all Linux configurations, because there are many variables and variations.

Some distributions work better with Miniforge: https://github.com/conda-forge/miniforge#miniforge

What is your CPU generation, make, and model? Please be as specific as possible with your hardware and software configurations. Otherwise, we don't really have a starting point. :)

Does this help ?

Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         48 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  16
  On-line CPU(s) list:   0-15
Vendor ID:               AuthenticAMD
  Model name:            AMD Ryzen 7 PRO 4750U with Radeon Graphics
    CPU family:          23
    Model:               96
    Thread(s) per core:  2
    Core(s) per socket:  8
    Socket(s):           1
    Stepping:            1
    Frequency boost:     enabled
    CPU(s) scaling MHz:  68%
    CPU max MHz:         1700.0000
    CPU min MHz:         1400.0000
    BogoMIPS:            3394.01
    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm c
                         onstant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave a
                         vx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_n
                         b bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clfl
                         ushopt clwb sha_ni xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd cppc arat npt 
                         lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid ove
                         rflow_recov succor smca
Virtualization features: 
  Virtualization:        AMD-V

Is there any background process that does I/O and might be stuck in the loop ? Tried disabling posthog, didn't helped.

Also tried installing from source, didn't helped.

it's actually gotten worse.

[deeleeramone]

Are you running 32-bit processes? What is your exact Linux distribution, build, and configuration?

Can this be a factor?

https://endeavouros.com/latest-release/

[010011x]

Are you running 32-bit processes? What is your exact Linux distribution, build, and configuration?

Can this be a factor?

https://endeavouros.com/latest-release/

doesn't seems like any of this is related. I'm on AMD.
I'm just using stock endeavouros, 64bit, amd.

so there is no way I can debug what thread is using all memory ? I've tried cProfile and stuff with no luck. There is nothing that comes up to your mind in regards to cpu consumption ?

Also tried installing poetry install without poetry.lock.

I don't need any forecasting libs, I just want to see couple of charts.

[010011x]

As a Python developer myself, I have a bunch of python projects and libs, browsers, editors etc etc.

none of the other Python processing are using anywhere close of this amount of CPU.
I have other conda environments and all is good.

I've been going thought openbb source code, from entry point and to input loops trying to find where bottleneck might be. I'm very sure it's something in the code, but you never know.

[010011x]

I'm sure others might have similar issue but just didn't noticed. It's more noticeable on my machine.

Docker way works fine, but it's too much overhead for me.

Thanks a lot for your help! ❤️

[deeleeramone]

Love the idea of this project, but will revisit in next year when it's more stable / usable. This is clearly a no-go.

You are currently the only user reporting this issue, so it is likely something related to your local system configuration. As you showed, you are running Intel, x86/64, processes instead of ARM/AMD, which specifically supports AMD processors. I would look into that.

[010011x]

Love the idea of this project, but will revisit in next year when it's more stable / usable. This is clearly a no-go.

You are currently the only user reporting this issue, so it is likely something related to your local system configuration. As you showed, you are running Intel, x86/64, processes instead of ARM/AMD, which specifically supports AMD processors. I would look into that.

My processor is neither Intel nor ARM.

https://en.wikichip.org/wiki/amd/ryzen_7/pro_4750u

ISA | x86-64 (x86)

[deeleeramone]

@hjoaquim, have you ran into this before?

[010011x]

another experiment.
just starting openbb enables AMD CPU boosting, jumping that CPU clock from 1.7 to 4.1Ghz, consistently staying at 100% CPU. But only only one thread. But it kinda kills entire performance, causing heating on the system.
In parallel I'm running a lot of other python processes, browsers, Spotify, etc. Everything else takes less than 5%.

I will go though all the Threads / Multiprocessing / etc parts of the openbb code.

                     ./o.                  bruce@bruce-amd 
                   ./sssso-                ---------------                                                                                                                  
                 `:osssssss+-              OS: EndeavourOS Linux x86_64                                                                                                     
               `:+sssssssssso/.            Host: 20UF000RRT ThinkPad X13 Gen 1                                                                                              
             `-/ossssssssssssso/.          Kernel: 6.3.5-arch1-1                                                                                                            
           `-/+sssssssssssssssso+:`        Uptime: 13 hours, 42 mins                                                                                                        
         `-:/+sssssssssssssssssso+/.       Packages: 1094 (pacman)                                                                                                          
       `.://osssssssssssssssssssso++-      Shell: zsh 5.9                                                                                                                   
      .://+ssssssssssssssssssssssso++:     Resolution: 1920x1080                                                                                                            
    .:///ossssssssssssssssssssssssso++:    DE: LXQt 1.3.0                                                                                                                   
  `:////ssssssssssssssssssssssssssso+++.   WM: Openbox                                                                                                                      
`-////+ssssssssssssssssssssssssssso++++-   Theme: Adwaita [GTK2/3]                                                                                                          
 `..-+oosssssssssssssssssssssssso+++++/`   Icons: Adwaita [GTK2/3]                                                                                                          
   ./++++++++++++++++++++++++++++++/:.     Terminal: qterminal                                                                                                              
  `:::::::::::::::::::::::::------``       Terminal Font: Bitstream Vera Sans Mono 14                                                                                       
                                           CPU: AMD Ryzen 7 PRO 4750U with Radeon Graphics (16) @ 1.700GHz 
                                           GPU: AMD ATI 06:00.0 Renoir 
                                           Memory: 5908MiB / 15212MiB

[hjoaquim]

It looks like I have the same issue (if I'm interpreting the htop correctly):

Although the CPU goes to 100% I don't really experience any performance issues on my machine.

[010011x]

It looks like I have the same issue (if I'm interpreting the htop correctly):

Although the CPU goes to 100% I don't really experience any performance issues on my machine.

Thank you so much for reproducing and confirming. I think the more you use it - the more performance degrades.

I will try to debug this. It's a big deal for me and a blocker to start using it daily.

Also, please note that not a lot of users have a bunch of CPUs, so burning one for hours might not be the best way forward.

[010011x]

@hjoaquim @deeleeramone

it happens here https://github.com/OpenBB-finance/OpenBBTerminal/blob/316e83dc003f613c51a6ba2c8fedfc3e480a0236/openbb_terminal/terminal_controller.py#L1082

haven't found exact reason, but this is the line where separate process starts and CPU jumps. will solve it later today.

is it check_backend ? 🤔

[010011x]

Created follow up OpenBB-finance/pywry#77

I think we are close

[tehcoderer]

Replied in PyWry issue thread! 😊

Fixed in #78 v0.5.12 #5122 !