Loading home page seems very slow

[h3llrais3r]

Describe the bug
I can be wrong, but since the application becomes more and more vueified, I have the impression that page loading is becoming slower.
For example: loading my home page takes more than 15s while in the past this was a lot faster.
Is this normal?

If you need more info/logs, please let me know and I'll provide it.

Screenshots

Medusa (please complete the following information):

• OS: Windows-8-6.2.9200
• Branch: master
• Commit: 4614efc
• Version: v0.2.10
• Database: 44.12

[OmgImAlexis]

@p0psicles this is part of the reason I was suggesting vuex-persistedstate. I think we should look at caching the shows array and having it update on load but in the background. Since new shows are just shown with a loading... right now anyways I don't see this being a huge issue.

@h3llrais3r would you mind checking the network tab, the rendering of vue isn't the issue it's the network requests, we currently block the whole UI from rendering while we fetch a large resource. I have a feeling if we moved this to a vuex-persistedstate and just loaded a cached list from the first load we'd have a lot better results.

For example try running a new copy of medusa seperate from your main install and look at how quick the pages load without any shows.

[p0psicles]

15s is a long time. Lets first analyze what does this.
@h3llrais3r can you give some more info?
System your running on. Number of shows. Any debug logs at time of load?
Maybe check your systems resource usage at time of load. Cpu, mem, swapping?

[h3llrais3r]

@OmgImAlexis, @p0psicles
Running it on my dev machine (latest develop branch) with only 2 anime shows and 1 normal show, it loads really fast.

I notice the slow loading on my server installation:

@OmgImAlexis, these are the details of the network tab during a refresh of the page:

[p0psicles]

That's like 12s only loading the html? There isn't any api call in there right? @OmgImAlexis ?

[h3llrais3r]

I just checked the debug logs, but no logging at the time of requesting the /home path.

[p0psicles]

Are you getting the same loading times when you do a hard reload?

[h3llrais3r]

Yes, boh F5 and ctrl + F5 give the same 12s wait time...

[p0psicles]

I also have like an initial wait time of 12s. But a hard reload shortens it to about 2.5

[p0psicles]

You know, the thing is. We're still in a transition fase of implementing a new frontend. It's going slow but steady. Currently we're getting the shows when calling the /home index.html. But after that we're also getting all the shows through the apiv2.

When everything has moved to apiv2, we can remove the calls in /home index.html. And that loading time will drop significally.

I would say, bare with us a little. In the mean time, we could ask @OmgImAlexis and @sharkykh to free up some time for vuefiying /home further.

[h3llrais3r]

No need to rush, I just noticed that it was a lot slower than before and wanted to share it. 😉

[OmgImAlexis]

@h3llrais3r would you mind making these changes? I'd love to see the page loading times after. This SHOULD remove that 12s on the mako file.

diff --git a/medusa/server/web/home/handler.py b/medusa/server/web/home/handler.py
index ac5e07436..01b23b4c7 100644
--- a/medusa/server/web/home/handler.py
+++ b/medusa/server/web/home/handler.py
@@ -134,40 +134,40 @@ class Home(WebRoot):
 
     def index(self):
         t = PageTemplate(rh=self, filename='home.mako')
-        selected_root = int(app.SELECTED_ROOT)
-        shows_dir = None
-        if selected_root is not None and app.ROOT_DIRS:
-            backend_pieces = app.ROOT_DIRS
-            backend_dirs = backend_pieces[1:]
-            try:
-                shows_dir = backend_dirs[selected_root] if selected_root != -1 else None
-            except IndexError:
-                # If user have a root selected in /home and remove the root folder a IndexError is raised
-                shows_dir = None
-                app.SELECTED_ROOT = -1
-
-        series = []
-        if app.ANIME_SPLIT_HOME:
-            anime = []
-            for show in app.showList:
-                if shows_dir and not show._location.startswith(shows_dir):
-                    continue
-                if show.is_anime:
-                    anime.append(show)
-                else:
-                    series.append(show)
-
-            show_lists = [[order, {'Series': series, 'Anime': anime}[order]] for order in app.SHOW_LIST_ORDER]
-        else:
-            for show in app.showList:
-                if shows_dir and not show._location.startswith(shows_dir):
-                    continue
-                series.append(show)
-            show_lists = [['Series', series]]
-
-        stats = self.show_statistics()
-        return t.render(show_lists=show_lists, show_stat=stats[0],
-                        max_download_count=stats[1], controller='home', action='index')
+        # selected_root = int(app.SELECTED_ROOT)
+        # shows_dir = None
+        # if selected_root is not None and app.ROOT_DIRS:
+        #     backend_pieces = app.ROOT_DIRS
+        #     backend_dirs = backend_pieces[1:]
+        #     try:
+        #         shows_dir = backend_dirs[selected_root] if selected_root != -1 else None
+        #     except IndexError:
+        #         # If user have a root selected in /home and remove the root folder a IndexError is raised
+        #         shows_dir = None
+        #         app.SELECTED_ROOT = -1
+
+        # series = []
+        # if app.ANIME_SPLIT_HOME:
+        #     anime = []
+        #     for show in app.showList:
+        #         if shows_dir and not show._location.startswith(shows_dir):
+        #             continue
+        #         if show.is_anime:
+        #             anime.append(show)
+        #         else:
+        #             series.append(show)
+
+        #     show_lists = [[order, {'Series': series, 'Anime': anime}[order]] for order in app.SHOW_LIST_ORDER]
+        # else:
+        #     for show in app.showList:
+        #         if shows_dir and not show._location.startswith(shows_dir):
+        #             continue
+        #         series.append(show)
+        #     show_lists = [['Series', series]]
+
+        # stats = self.show_statistics()
+        return t.render(show_lists=[], show_stat=[],
+                        max_download_count=0, controller='home', action='index')
 
     @staticmethod
     def show_statistics():

[h3llrais3r]

@OmgImAlexis Will try this evening (or this weekend) and let you know.

[h3llrais3r]

@OmgImAlexis Running it with your changes results indeed in fast load.

[p0psicles]

The more we move away from the mako templates and implement the new UI the faster everything will become. Tnx adressing btw.

[h3llrais3r]

@p0psicles It loads fast, but it doesn't show any shows in the list.

So isn't it normal that it loads fast due to the shows not being loaded?
When I run it on my dev setup with only 3 shows, it also loads fast.

[p0psicles]

Its being worked on. Np prob

[OmgImAlexis]

@h3llrais3r thank you for that. Yes that's expected as you commented out the whole section that's rendered there. That's why it's empty but it does confirm that if we can remove the mako file it'll be quicker than before.

I'm working on a Vueifying the homepage right now but I can't promise anything. Is there a layout you'd want me to do first? I'm more than happy to do one layout at a time so at least people with lots of shows can use it without lag.

[OmgImAlexis]

@h3llrais3r would you mind checking out the vueify-home branch and just try the banner layout. It should load quite a bit faster but there are a few columns still unfinished.

[h3llrais3r]

@OmgImAlexis Thanks for that. I'm using the simple table layout.
But like I already said, you don't need to rush with it, I was just mentioning it that it's slow with a lot of shows.
I can checkout the vueify-home branch, but only on my dev setup. (I don't want to switch my normal setup to the develop branch)
What exactly do you want me to test on that branch?

[OmgImAlexis]

Just switch to banner layout and check the loading times. It should load almost instantly.

[h3llrais3r]

@OmgImAlexis Both simple and banner layout are loading fast in the branch, but I think that's normal since I don't have a lot of shows on my dev setup...

Left: loading with 3 shows on simple layout
Right: loading with 3 shows on banner layout

[OmgImAlexis]

@h3llrais3r would you mind loading your production database in a test install and trying again? I'd like to see if it fixes it with all your shows or if the speed up is really only going to stay with small libraries.

[h3llrais3r]

I'll see what I can do this week (I don't have the time today, but hopefully somewhere this week). I'll keep you posted.

[OmgImAlexis]

@h3llrais3r no worries. The simple layout should also now be working on that branch.

I know the XEM and next/prev date columns are still broken, working on that.

[h3llrais3r]

@OmgImAlexis Performance is a bit better, but still a lot of time needed for /home (in banner mode on vueify-home branch)

Perhaps a stupid remark: I don't see any changes yet in the branch in the handler.py for the /home path.... so isn't it logic that it's still slow because the delay was coming from here?

[h3llrais3r]

@OmgImAlexis, @p0psicles
I did some debugging and it turns out that almost all the lost time is within the show_statistics() function.
I debugged on my dev setup with my database of my server and almost the complete time is required to get those statistics...
I've added some logging and this is the result:

Code changes:

Log output:

2018-10-02 20:57:14 INFO     Thread_0 :: [a2cce69] before show_statistics()
2018-10-02 20:57:33 INFO     Thread_0 :: [a2cce69] after show_statistics()

Performance & network:

As you can see, the almost 20s needed to get the /home loaded is due to the show_statistics().
So I think the db might need some performance tuning. 😉

[OmgImAlexis]

@h3llrais3r try the newest commit on that branch. show_statistics actually isn't needed for the banner/simple layout anymore.

[h3llrais3r]

@OmgImAlexis
/home is now loading really fast, but now the rest calls are taking a lot of time...

It's still the stats call that takes a lot of time, so imho the query behind it will need some tuning for large collections.

[OmgImAlexis]

/home is now loading really fast, but now the rest calls are taking a lot of time...

Yay. 🎉

It's still the stats call that takes a lot of time, so imho the query behind it will need some tuning for large collections.

That's to be expected at least for now. The UI SHOULD load while that happening though as we don't actually wait for the call to finish, we just dispatch it and the layout will update after the stats are pulled in.

[h3llrais3r]

Indeed the layout updates after the stats... but that takes 20s 😉
So basically it still takes 20+ seconds to load the home page completely. 😢

[p0psicles]

What omg is saying, is that the page is already usable before it loads the stats. Previously that was not the case. Or you only load the page for the stats?

[h3llrais3r]

no, the problem is that the page show 'loading' very quickly, but the show list is only shown at the end (after the 20s of the stats api call)

after /home is loaded and as long as stats api call is pending:

after stats api call:

[p0psicles]

Ooh thats not good. I think we can fix that.

[OmgImAlexis]

That's not an issue, I expected that. Already fixed more or less locally.

[p0psicles]

@OmgImAlexis can you share your branch where you 'more or less' fixed this? We could continue from there.

[sharkykh]

I was writing the comment below, and as I was gathering the numbers, I realized they didn't add up:
~50 lines per show, times 48 shows is only ~2,400 lines, not where close to the 5,000 I was seeing.
Then I found out that everything (the layout code) was imported twice.
#6754 fixes this bug, and hopefully restore at least 50% of the speed.

---

I'll just put this here, as I think it's relevant:

sharkykh [11:38]

From the little testing I did right now by mimicking a show list 10 times larger than what I have (so... 480 shows), I found out that the problem we have is that the HTML is so big, that Vue is working VERY hard to parse the HTML it's getting on the home page, and that seems to be the root of the issue.

Adding to that - an explanation:

Python generates (using Mako) the complete show list, ALL the shows in the database, into one HTML document and serves that up to the client.

Taking the banner layout as an example, I simple calculation:
My show list contains 48 shows at the moment of writing.
Viewing the source of the home page results in 5,313 lines.
Of these lines, Vue needs to parse, compile and render (well, re-render most of it) around 5,140‬ lines.
That is a lot, and remember, we're talking about "just" 48 shows.
Excluding some HTML elements that are only used once, or already converted to components (like the header), that means each show in your databases adds, roughly, 50 lines per show.

So as quoted above, I tested the home page with a show list of
🛑 🛑 🛑
And this is when I found the bug 🤣

[rafi-d]

Just an observation - with ~50 shows now, there is still not much of a difference in load time (still ~10 sec on old C2D). Maybe with a larger number there is .

One other thing - there is home.mako under ...\dark and ...\light (in 0.29) . Shouldn't those be modified too (I did modify them for testing..) ?

[sharkykh]

For one, 10 seconds is half of the 20 you showed on the video from #6740.
And on that same comment you also said you were running a modified version of v0.2.9.
You're behind by 7 versions so we can't really support that... Just keep that in mind.

And second, this isn't a permanent solution to the problem.
The problem is still there, and it will stay there probably until the home page transition is complete.
In the meantime I think the Mass Update page is the easiest workaround if the home page is too slow to bear.

One other thing - there is home.mako under ...\dark and ...\light (in 0.29) . Shouldn't those be modified too (I did modify them for testing..) ?

Not sure what you're seeing, because I did change them in my PR.

P.S.
I think 10-year-old processors are perhaps struggling with the required processing power for what counts as mainstream web apps nowadays. There's a not-so-small amount of JavaScript code to run, and that's all done by the clients/web-browsers. So yeah, maybe it does have an effect over your performance.

[rafi-d]

Right, my mistake, I've missed "dark/light" in your PR.

For one, 10 seconds is half of the 20 you showed on the video from #6740.

You've missed my update:
#6740 (comment)
^ 10 sec was right after I reduced from 70 to 50 shows, removed the 20 "ended" shows.

Agreed, 10 years old PC is old, but good enough to compare before and after (even easier...) .
What is the refresh time for you for your 50 shows?

Rafi

[sharkykh]

Oh, oops. Sorry.

For me it's around 1.5-2 seconds. But my processor is an i7-8700K.
There's an option on the Performance tab of Chrome DevTools to throttle the CPU.
If I throttle it by "x4 slowdown", I get the the same 10 seconds you're reporting, so it makes sense.
BTW, that's 6-7 seconds of JavaScript execution (= Vue)

[rafi-d]

So, on my i7 laptop as client over LAN (also "old" 2720QM @2.2GHz) , it is ~3 second before and after.
For some reason, it feels like minimal improvement. Strange.

Did you see significant improvement in time with 500 shows?

I think the show page is even more problematic, especially with series like The Simpsons...

[rafi-d]

PS:
Maybe it is worth while to investigate why reducing only 20 "ended" show - had such big effect (50% here...) . Maybe some not so optimized handling of "ended" shows? Eventually every-user is ending with such "ended" shows.

[sharkykh]

Did you see significant improvement in time with 500 shows?

Yes.

I think the show page is even more problematic, especially with series like The Simpsons...

It is as problematic. The severity really depends on the show in question.
The home page is more consistent - tied directly to the amount of shows it's trying to render.
I don't have shows as big as The Simpsons or The Daily Show but I have Family Guy which is 17 seasons. So yeah, I know.

PS:
Maybe it is worth while to investigate why reducing only 20 "ended" show - had such big effect (50% here...) . Maybe some not so optimized handling of "ended" shows? Eventually every-user is ending with such "ended" shows.

Because those 20 were actually 40 shows, as I explained above.
So before the HTML had 140 shows in it, and after you removed the ended shows it jumped down to a 100.
And with the latest change it's back down to the real 50 shows, so it should be faster.
I guess there's a tipping point of some sort... I don't know for sure.

[medariox]

This discussion is 1 year old, much has changed since then. Please open a new discussion if needed.