Optimizing parsing .bru files while program starts

[SPIRIT-00]

I have checked the following:

• I've searched existing issues and found nothing related to my issue.

Describe the feature you want to add

I've large collections, about 3,8k files. And each start of program be like: "launch Bruno and go away...". In many cases i need just a few specific reqs, not all collection.

I suggest:

1. Do not block all interface while files in load, just block top level folders* inside collection if not all inner files (recursively) was loaded.
2. Unblock top level folders after it was fully loaded, one by one.
3. Load folder in priority if user clicked on it, then continue load by sequence from top to bottom of folder list.
4. It may help mitigate this bug 'Requests' freezing while moving it between folders by drags #2613 if do update states of folders (not recursively) when transfer requests from folder to folder.

*Top level folders means one step after collection level:

• Collection
  • folder
  • folder
  • folder

Probably be solved by this pr #3618

Mockups or Images of the feature

[sreelakshmi-bruno]

Thanks for reporting, adding it to our development pipeline.

[sreelakshmi-bruno]

Hi @SPIRIT-00 , we have a beta build for this issue. Please check it out and let me know how it goes. Thanks!

[SPIRIT-00]

Hi @SPIRIT-00 , we have a beta build for this issue. Please check it out and let me know how it goes. Thanks!

Hi there, i downloaded portable version. Loading became much slowly.
First 20-30 secs:

2 min:

after 10 min still loading:

Fully loaded after ~20mins.

On start loading interface still freezes but fast unfreeze.

Requests loading fully randomly? Mean many folders have several unloaded reqs.

[tlaloc911]

I had a similar experience, my biggest collection has around 1K requests

Before

• Application freezes for 1 min and a half to load all my collections: 1K request plus 5 small collections.
• Everything is ready to be used

Now

• Application starts without freezing
• Full loading of the 1K requests collection takes around 10 mins.

[sreelakshmi-bruno]

Hi, thanks for checking it out and providing feedback. We're working on reducing time to full load, will post updates when a test build is available.

[lohxt1]

@SPIRIT-00 @tlaloc911 can you try running this pr on your systems and let me know if you notice any difference in perf ?

[tlaloc911]

Hi @lohxt1

It worked well with a 200 requests collection

It didnt work with the biggest one. Starting the app and waiting without doing anything else

Electron's memory starts growing until 4 GB and then it crashes without a trace. Just this:

npm error Lifecycle script `dev` failed with error:
npm error code 3221225477
npm error path d:\dev\bruno-github-main\packages\bruno-electron
npm error workspace bruno@v1.38.1
npm error location d:\dev\bruno-github-main\packages\bruno-electron
npm error command failed
npm error command C:\WINDOWS\system32\cmd.exe /d /s /c electron .

I notice that at some moment request count grew until 1143

but there are only 986 bru files in folder

(Get-ChildItem -Path "." -Filter "*.bru" -Recurse | Measure-Object).Count
986

I also tried moving around the app while the loading happens, and it also crashes, with some more info

<--- Last few GCs --->

[8388:00002D343F878000]     1101 ms: Scavenge 18.1 (27.0) -> 14.3 (29.5) MB, pooled: 0 MB, 38.64 / 0.00 ms  (average mu = 1.000, current mu = 1.000) allocation failure;
[8388:00002D343F878000]     1248 ms: Scavenge (interleaved) 21.2 (30.2) -> 17.1 (32.0) MB, pooled: 0 MB, 45.58 / 0.00 ms  (average mu = 1.000, current mu = 1.000) allocation failure;


<--- JS stacktrace --->


<--- Last few GCs --->

[8388:00002D3440D30000]      126 ms: Scavenge 2.2 (3.8) -> 1.9 (3.8) MB, pooled: 0 MB, 1.92 / 0.00 ms  (average mu = 1.000, current mu = 1.000) allocation failure;
[8388:00002D3440D30000]      327 ms: Scavenge 3.1 (5.3) -> 2.8 (5.3) MB, pooled: 0 MB, 6.80 / 0.01 ms  (average mu = 1.000, current mu = 1.000) allocation failure;


<--- JS stacktrace --->

FATAL ERROR: NewSpace::EnsureCurrentCapacity Allocation failed - JavaScript heap out of memory
----- Native stack trace -----

FATAL ERROR: NewSpace::EnsureCurrentCapacity Allocation failed - JavaScript heap out of memory
----- Native stack trace -----

 1: 00007FF72459166A
 2: 00007FF72450A43B
 3: 00007FF7277B83A4
 4: 00007FF7277B8313
 5: 00007FF7278973C7
 6: 00007FF7231A776E
 7: 00007FF723192828
 8: 00007FF7231920AD
 9: 00007FF72319A4D6
10: 00007FF72319A36B
11: 00007FF726AFF4FD
12: 00007FF725D00F17
13: 00007FF725CFB11F
14: 00007FF725CEF2E7
15: 00007FF725F57F32
16: 00007FF725F56B84
17: 00007FF7263E3F3A
18: 00007FF7264DCCDA
19: 00007FF72633D3DE
20: 00007FF72633D3DE
21: 00007FF78CD9291E

<--- Last few GCs --->

[8388:00002D343FAEC000]      620 ms: Scavenge 11.1 (16.0) -> 8.5 (25.0) MB, pooled: 0 MB, 5.87 / 0.00 ms  (average mu = 1.000, current mu = 1.000) allocation failure;
[8388:00002D343FAEC000]      882 ms: Scavenge 16.8 (26.2) -> 9.3 (26.2) MB, pooled: 0 MB, 21.75 / 0.00 ms  (average mu = 1.000, current mu = 1.000) allocation failure;
[8388:00002D343FAEC000]      993 ms: Scavenge 17.7 (26.4) -> 12.3 (26.7) MB, pooled: 0 MB, 16.73 / 0.00 ms  (average mu = 1.000, current mu = 1.000) allocation failure;


<--- JS stacktrace --->

FATAL ERROR: NewSpace::EnsureCurrentCapacity Allocation failed - JavaScript heap out of memory
----- Native stack trace -----

 1: 00007FF72459166A
 2: 00007FF72450A43B
 3: 00007FF7277B83A4
 4: 00007FF7277B8313
 5: 00007FF7278973C7
 6: 00007FF7231A776E
 7: 00007FF723192828
 8: 00007FF7231920AD
 9: 00007FF72319A4D6
10: 00007FF72319A36B
11: 00007FF726AFF4FD
12: 00007FF725D00F17
13: 00007FF725CFB11F
14: 00007FF725CEAE9C
15: 00007FF725F5626E
16: 00007FF7263E3F3A
17: 00007FF78CAFBC5B
npm error Lifecycle script `dev` failed with error:
npm error code 134
npm error path d:\dev\bruno-github-main\packages\bruno-electron
npm error workspace bruno@v1.38.1
npm error location d:\dev\bruno-github-main\packages\bruno-electron
npm error command failed
npm error command C:\WINDOWS\system32\cmd.exe /d /s /c electron .

[lohxt1]

thanks for testing the pr @tlaloc911

in the below code

this.WORKER_QUEUE_MAX_LENGTH  = process.env.WORKER_QUEUE_MAX_LENGTH || 100;

located at bruno-electron/src/bru/workers/index.js

could you try setting a higher number ~500 ? instead of 100

[lohxt1]

and would it be possible to share the collection if you're okay with that ? here, or send it to my email lohit@usebruno.com

[tlaloc911]

thanks for testing the pr @tlaloc911

in the below code

this.WORKER_QUEUE_MAX_LENGTH  = process.env.WORKER_QUEUE_MAX_LENGTH || 100;

located at bruno-electron/src/bru/workers/index.js

could you try setting a higher number ~500 ? instead of 100

It loaded setting it to 500 👍 :

still it showed a total of 1143 requests, and the loaded requests count also reached that 1143

but when loading finished it adjusted to the correct 965

Total time of 3 mins an a half was better than the previous version but still slower with no worker's version. Ill test with other WORKER_QUEUE_MAX_LENGTH values

EDIT:

350

250

200

150

[tlaloc911]

and would it be possible to share the collection if you're okay with that ? here, or send it to my email lohit@usebruno.com

I'm afraid I can't share the original collection, but can prepare a dummy one with the same characteristics. Ill send it during the weekend.

[lohxt1]

I'm afraid I can't share the original collection, but can prepare a dummy one with the same characteristics. Ill send it during the weekend.

yeah sure, i just wanted to get an idea of the request file size distribution in your collection, i have tested the pr with collections containing 300, 1000 and 3000 requests with the file sizes varying between 1KB and 10KB each

Total time of 3 mins an a half was better than the previous version but still slower with no worker's version. Ill test with other WORKER_QUEUE_MAX_LENGTH values

were you able to interact with the requests that were fully loaded while the others were still loading ?

and did you notice any sort of improvement in the app's responsiveness with the worker's version ?

[tlaloc911]

I'm afraid I can't share the original collection, but can prepare a dummy one with the same characteristics. Ill send it during the weekend.

yeah sure, i just wanted to get an idea of the request file size distribution in your collection, i have tested the pr with collections containing 300, 1000 and 3000 requests with the file sizes varying between 1KB and 10KB each

Total time of 3 mins an a half was better than the previous version but still slower with no worker's version. Ill test with other WORKER_QUEUE_MAX_LENGTH values

were you able to interact with the requests that were fully loaded while the others were still loading ?

and did you notice any sort of improvement in the app's responsiveness with the worker's version ?

This is the distribution of bru files in the collection

Size in bytes	Bru files
< 1 000 000	2
< 500 000	29
< 50 000	127
< 100 000	58
< 10000	76
< 5000	675
<1000	17

The general feeling is good, now starting is fast and it was possible to interact with loaded request.

I perceive slowness / unresponsiveness while loading two collections at the same time.

Also at executing runner while loading, may be it is convenient to lock runner while loading is happening.

Thanks @lohxt1 for your efforts, this change is possitive for UX

[lohxt1]

This is the distribution of bru files in the collection

Size in bytes	Bru files
< 1 000 000	2
< 500 000	29
< 50 000	127
< 100 000	58
< 10000	76
< 5000	675
<1000	17

thanks for the stats, i will try creating a collection based on this and perform more tests.

Also at executing runner while loading, may be it is convenient to lock runner while loading is happening.

currently, there is only a warning text indicating that the collection requests are loading. disabling the runner based on the collection/folder request load status could be a valid option
cc @helloanoop

[GeekyMonkey]

I'm hoping this will prioritize the bru files that I want to open first. If I select a folder, will it jump to the top of the loading queue?

[SPIRIT-00]

@SPIRIT-00 @tlaloc911 can you try running this pr on your systems and let me know if you notice any difference in perf ?

Nope, still not loading faster then it's was before. And it have infinity load animation on top level folder. Same with 1.39 version. I may send my reqs if needed, just need to delete some sensitive info.

[sreelakshmi-bruno]

Hi, we've a build out for the slow load issue. Please check it out and let me know how it goes. Thanks!

[SPIRIT-00]

Interface freezes a bit longer and totally stuck, but after unfreeze it loaded all folders faster.