website: improve cheat-sheet page performance #1252

rszyma · 2023-05-20T09:25:31Z

Description

Update cheat-sheet page for better performance

Requirements / Checklist

Read the Contributing Guidelines
Verified the license of any newly added font, glyph, or glyph set

What does this Pull Request (PR) do?

Modifications to cheat-sheet page:

Improved page load time
Greatly improved search time
- Use MiniSearch lib as search engine
- Lazy render search results. Currently up to 250 entries at a time. More entries loaded if scrolled to bottom.
Show all icons on page load.
Empty search field shows all icons.
Removed "Search" and "Show All Icons" buttons, since they are would be useless anyway.

Other

Remove "sidecar", because it's not working anymore.

How should this be manually tested?

manually

Any background context you can provide?

Main motivation for these changes is terrible UX/performance of https://www.nerdfonts.com/cheat-sheet

What are the relevant tickets (if any)?

There are no tickets for neither bad cheat-sheet performance, nor sidecar not working.

Tandem PR: #1254

rszyma · 2023-05-20T09:33:50Z

One thing though: "removed" icons are now all over the search results, and not at the end as before. However, IMHO it's nothing compared to improvements this PR brings (and it's definitely fixable, but I don't have that much time to think of the best way to do this right now).

Finii · 2023-05-20T10:46:39Z

Thanks for the PR!

Camping right now, so I can't check it out right now. But wanted to comment that 'removed comes last' was not intended :-D

Finii · 2023-05-20T10:59:59Z

Further note: _posts/2017-01-04*. md is autogenerated. I guess you replaced it by glyphs.html. Can not see that on the phone, but propably it needs to be regererated on glyph set changes. Or you read the data or css file.
If it happens to need updates itself, do you have a generator script?

See

rszyma · 2023-05-20T13:17:49Z

Yeah, I didn't know it was autogenerated. Maybe we should add a disclaimer like "This file was autogenerated by . Do not edit." at the beginning of the autogenerated files so it's more obvious?

Your guess about glyphs.html was right. I moved the cheatsheet data to glyphs.html. However I converted it to json for smaller size and faster loading time. Also MiniSearch library I've used as search engine needs json data, so converting to json was obvious choice.

Edit:

I've moved the glyphs data back to 2017-01-04-icon-cheat-sheet.md in e132662)

From my understanding it should be safe to update the part of the generate-css.sh script that generates the 2017-01-04-icon-cheat-sheet.md file, because it's not used anywhere else besides the cheat-sheet on the website, right?

rszyma · 2023-05-21T08:02:46Z

Further note: _posts/2017-01-04*. md is autogenerated. I guess you replaced it by glyphs.html. Can not see that on the phone, but propably it needs to be regererated on glyph set changes. Or you read the data or css file. If it happens to need updates itself, do you have a generator script?

See

https://github.com/ryanoasis/nerd-fonts/tree/master/bin/scripts#readme

https://github.com/ryanoasis/nerd-fonts/blob/master/bin/scripts/generate-css.sh

I've updated generate-css.sh accordingly in #1254

Finii · 2023-05-27T06:43:42Z

Still struggling to iron out bugs in 3.0.1 and prepare 3.0.2, please do not think this is forgotten.

A casual glance left the impression that you pull in the (in)famous NPM? How deep is that dependency well?
In other places we go to some length to reduce the dependencies; can we not improve it without opening another 100 dependencies? 😬

[why] Disfunctional See PR #1252

Finii · 2023-05-27T08:15:51Z

Cheery picked already

beb808e Remove gitter-sidecar as 34cddad

Finii · 2023-05-27T08:26:10Z

Started your branch locally, the search behavior is a bit strange? Is that full word search?

alt

alti

altimete

altimeter

rszyma · 2023-05-27T09:07:12Z

It is full word search. While it works differently than before I'd argue it's better.
Reasons:

a lot faster
don't show unrelated glyphs for short search terms like "a"

If you don't like the full-word search, there are some options to tweak:
https://lucaong.github.io/minisearch/#search-options

prefix: true - searching for "alti" would show results for "altimeter". But the search is a lot a lot slower.
fuzzy - searches for misspelled words

There is also an Auto Suggestion feature, you can see it in action here

rszyma · 2023-05-27T09:13:55Z

A casual glance left the impression that you pull in the (in)famous NPM? How deep is that dependency well? In other places we go to some length to reduce the dependencies; can we not improve it without opening another 100 dependencies? 😬

If you're talking about MiniSearch, it's downloaded from CDN. See here:
https://github.com/ryanoasis/nerd-fonts/pull/1252/files#diff-022702df0bec434e85269514ea8900f6f54e698486b484ad91aede3837cfb788R21
And it's only 29 KB in size,
with no external dependencies (or at least they claim that on their site).

rszyma · 2023-05-27T09:20:41Z

I'll comment further on what is going on in this PR to make the review easier.

Moved the code specific to cheat sheet from site.js to cheat-sheet.js.
I don't know anything about the gtag functions that are used throughout the code, so I just ignored them. You might want to add/remove/readjust some of them.
I'm sorry about so many changes in _includes/css/nerd-font-tweaks.scss.
Seems like my autoformatter acted on it's own. All I intended to change there is to remove sidecar-related css and adjust (mostly remove) css to work well will changes made to search.

[why] Disfunctional See PR #1252

Finii · 2023-05-27T13:32:46Z

@allcontributors please add @SilverMira for bug

allcontributors · 2023-05-27T13:32:55Z

@Finii

I've put up a pull request to add @SilverMira! 🎉

Finii · 2023-05-30T11:47:49Z

First of all, I finally spend some time really looking into the code here, and it is excellent! Thank you.

The following comments are out of date. Instead I pushed some commits. Please review them.

Performance is not the main goal for me, it is still far better than before. I believe the prefix search is needed for usability and the sorting is at least a nice-to-have. I have no means to benchmark that stuff apart from gut-value ;-)

One thing though: "removed" icons are now all over the search results, and not at the end as before. However, IMHO it's nothing compared to improvements this PR brings (and it's definitely fixable, but I don't have that much time to think of the best way to do this right now).

Maybe you have time for a quick look at the following patch?
Surprisingly it does not always work as expected.

From ee38d0ac6869c8a28be07724675a2b7716072da4 Mon Sep 17 00:00:00 2001
From: Fini Jastrow <ulf.fini.jastrow@desy.de>
Date: Tue, 30 May 2023 11:45:01 +0200
Subject: [PATCH] cheat_sheet: Sort removed icons last

Signed-off-by: Fini Jastrow <ulf.fini.jastrow@desy.de>
---
 cheat-sheet.js | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/cheat-sheet.js b/cheat-sheet.js
index 144987e23..6b9561104 100644
--- a/cheat-sheet.js
+++ b/cheat-sheet.js
@@ -9,7 +9,7 @@ document.addEventListener('DOMContentLoaded', function () {
 
     // Index all glyphs/icons
     let miniSearch = new MiniSearch({
-        fields: ['id', 'code'], // fields to index for full-text search
+        fields: ['id', 'code', 'isNew'], // fields to index for full-text search
         storeFields: ['id', 'code', 'isRemoved'], // fields to return with search results
     })
     miniSearch.addAll(Object.entries(glyphs).map(
@@ -17,6 +17,7 @@ document.addEventListener('DOMContentLoaded', function () {
             return {
                 id: key,
                 code: value,
+                isNew: key.startsWith('nfold') ? false : key,
             }
         }
     ));
@@ -136,6 +137,7 @@ document.addEventListener('DOMContentLoaded', function () {
             {
                 prefix: false,
                 combineWith: "AND",
+                boost: { isNew: 2 },
             }
         );
 
-- 
2.39.2

prefix: true - searching for "alti" would show results for "altimeter". But the search is a lot a lot slower.

I think this is needed at least. I also can not really see any speed impact?

The isRemoved is never filled?
I added this code:

     miniSearch.addAll(Object.entries(glyphs).map(
         ([key, value]) => {
             return {
                 id: key,
                 code: value,
+                isRemoved: key.startsWith('nfold'),
             }
         }
     ));

But then, I guess it can be removed as it is never used.

Finii · 2023-05-30T11:56:35Z

I don't know anything about the gtag functions that are used throughout the code, so I just ignored them. You might want to add/remove/readjust some of them.

Well, I guess Ryan set up google analytics to have some insight in usage / usage-patterns.
I have no access to the results, so 'I dont care'.
Would not remove it either, googletagmanager is anyhow ~~disabled~~ blocked in my browser ;-)

Finii · 2023-05-30T12:59:37Z

cheat-sheet.js

+ {
+ prefix: true,
+ combineWith: "AND",
+ boostDocument: ((documentId, term, storedFields) => {


The downside of the boost based sorting is ... that it follows documents text searcher rules.

Take for example this...

Because eae9 has cod in the name twice it is listed too early (if you expect a sort-by-name list).

(file_code in front of account)

This could be resolved by boosting down 'key' field and adding another field with just the part of the id after the last - part of icon class e.g. for nf-md-bluetooth_audio it would be bluetooth_audio

The downside of the boost based sorting is ... that it follows documents text searcher rules.

It's not a downside to boost based sorting, but the consequence of setting prefix: true

Finii · 2023-05-30T13:39:34Z

I also tried a lot variants with boost, it did never work out. What is wrong with just sorting? Adding all the strings also costs memory, so it is hard to tell if sticking to boost has any upside.

Example with your latest changes:

No real sort on startup:

Because we search for nf which is in various nfc icons

Sorting still not according to id ;-)

nf-cod-code is definitively first for this search, but it's faaar down

rszyma · 2023-05-30T14:37:51Z

I changed boost for id field from 0.001 to 0.00001, because if you search for fa and scroll down you can see that the removed icons come before some of the not-removed icons. Changing the boost to even lower fixes that.

I also tried a lot variants with boost, it did never work out.

The variant you wanted is here: 09ad62b
It can't be done with boost option. boostDocument is the option you wanted.

What is wrong with just sorting?

Probably nothing. I did the changes in 09ad62b because I found that the sorting code didn't really work. Also there's a chance that using boostDocument is more optimized than just sorting.

Adding all the strings also costs memory, so it is hard to tell if sticking to boost has any upside.

I've just checked how much memory the loaded page takes and it's about 20 MB, so I'd say it's completely acceptable. The additional "name" field is used so to fix the issue related to setting prefix: true (on the screenshot) #1252 (comment)

No real sort on startup

resolved with 61a9c4d

rszyma · 2023-05-30T14:42:14Z

Sorting still not according to id

The results look correct to me.

All of the below names contain prefix "cod", so they are ranked equally:

coda
codrops
codiepie
codeigniter
code

Finii · 2023-05-30T14:58:33Z

I also tried a lot variants with boost, it did never work out.

The variant you wanted is here: 09ad62b It can't be done with boost option. boostDocument is the option you wanted.

Of course it can be done with just boost, see

ad4c9bc cheat_sheet: Sort removed icons last

but because the minisearch sort algo takes into account to number of occurrences in a document (what we not only do not care for but actually do not want), I dropped that for a regular custom search with

9d09f38 cheat-sheet: Sort the results

Anyhow, resorting an almost sorted array should be really cheap. I do not know which sort is used by JS, but all usual candidates do that in linear time, right?

I'm not sure that adding all that "complexity" with boost and (admittedly nice prefixSearchEnabled) is worth it, just to avoids the one sort() call?

Finii · 2023-05-30T15:01:32Z

Sorting still not according to id

The results look correct to me.

All of the below names contain prefix "cod", so they are ranked equally:

coda codrops codiepie codeigniter code

Right, they have the same RANK, but they are not sorted for their ID within the rank.
Obviously codiepie (with an i after cod) needs to come before codrops (which has an r after cod).

rszyma · 2023-05-30T15:33:57Z

Of course it can be done with just boost

I see! I thought it didn't work, so you changed it. My bad.

I dropped that for a regular custom search with
9d09f38 cheat-sheet: Sort the results

I see now why you changed to manual sort. As I said before, I changed the code to use boostDocument because the manual sorting code didn't work as expected:

Anyhow, resorting an almost sorted array should be really cheap. I do not know which sort is used by JS, but all usual candidates do that in linear time, right

I don't know much about sorting in JS, so I'd rather believe minisort than myself to sort it for me (by using boost and boostDocument) 😄

is worth it, just to avoids the one sort() call?

I'm not necessarily trying to avoid manual sort(), but rather prefer the build-in features minisort already offers. (also I bet it's faster that way)

Right, they have the same RANK, but they are not sorted for their ID within the rank.
Obviously codiepie (with an i after cod) needs to come before codrops (which has an r after cod).

TBH, sounds like a non-issue to me. Maybe that could be changed with use of some of advanced parameters in minisearch, but I think it's not worth the trouble for now.

Finii · 2023-05-30T16:00:34Z

Two questions

Is this ready for merging from your perspective?
Do you have an opinion/solution for Update search page style #1265?
The coding style is nice
Hope you do not mind this discussion
I probably would like to add some of the comments in this PR to the commit messages if that is ok for you.
(I like to have explanations and reasons in git rather than github)

Finii · 2023-05-30T16:03:42Z

because the manual sorting code didn't work as expected

I need to check that out :-D
Somewhere I have a machine with jekyll, ... *noise of searching through a bunch of laptops*

Finii · 2023-05-30T16:17:30Z

Hmm, at current commit...

Why is nf-fa-fa first? This minisearch's sort is rubbish :-D

Finii · 2023-05-30T16:22:33Z

I see now why you changed to manual sort. As I said before, I changed the code to use boostDocument because the manual sorting code didn't work as expected:

At 9d09f38 (i.e. "manual search") I get this result...

... scrolled down ...

Edit: Your image seems to be not from 'my' manual search that takes isRemoved into account?

        remainingSearchResults.sort((a, b) => {
            if (a.isRemoved != b.isRemoved) {
                return a.isRemoved > b.isRemoved
            }
            return a.id > b.id
        })

rszyma · 2023-05-30T16:42:04Z

Is this ready for merging from your perspective?

I don't see any major issues with this PR right now, so yes.

Do you have an opinion/solution for #1265?

I'd suggest working on #1265 after this PR is merged. Worth noting, that #1265 breaks lazy-loading implemented in this PR.

The coding style is nice

Thanks!

Hope you do not mind this discussion

Not at all

I probably would like to add some of the comments in this PR to the commit messages if that is ok for you.
(I like to have explanations and reasons in git rather than github)

Sure, go ahead.

Why is nf-fa-fa first? This minisearch's sort is rubbish :-D

Honestly, no clue. Maybe it has something to do with the id having double 'f'?
There's is this issue discussing possible workarounds if one wanted to "fix" it. One suggestion is to have an another dummy field that will search for all results, but I'm sure you won't like it 😄

Your image seems to be not from 'my' manual search that takes isRemoved into account?

I ran 9d09f38 again just now and it still doesn't work. Idk.

Finii · 2023-05-30T17:11:45Z

Resolving some conflicts manually, force push

Signed-off-by: Fini Jastrow <ulf.fini.jastrow@desy.de>

[why] Sometimes one does not know what the exact search term is. For example if you want to find a 'homefolder' but the name is 'homedirectory' it is impossible to find. [how] Allow prefix search, in this case at least 'home' will find both variants. I do believe a full substring search would be even better, but that is not supported by minisearch. Signed-off-by: Fini Jastrow <ulf.fini.jastrow@desy.de>

[why] Often it is easier to find what one wants if the search result is sorted. [how] Do a full result sort. * Sort by id (class name) * Put removed icons last We do not need the boost function anymore, so that pre-sorting is removed. Signed-off-by: Fini Jastrow <ulf.fini.jastrow@desy.de>

Finii · 2023-05-30T17:20:16Z

🙄

Manual merge via commit to get PR number into git

Finii · 2023-05-30T17:21:42Z

@allcontributors add @rszyma for code

allcontributors · 2023-05-30T17:21:51Z

@Finii

I've put up a pull request to add @rszyma! 🎉

Finii · 2023-05-30T17:53:40Z

Changed/added some small things.
Recreated the cheat-sheat file via script to get it 100% as generated.

✔️ Works ;-)

Thanks for the great contribution!

Finii · 2023-05-31T05:54:39Z

I don't know much about sorting in JS, so I'd rather believe minisort than myself to sort it for me

minisort just uses ordinary sort, just on the weights [1] with a sort lambda [2] like this:

const byScore = ({ score: a }: Scored, { score: b }: Scored) => b - a

...

search (query: Query, searchOptions: SearchOptions = {}): SearchResult[] {

    ...

    results.sort(byScore)
    return results
  }

Javascript seems to use [3].
minisearch does not have any magic here, this is 100% standard stuff.
If we assign no weights the sort is kind of a noop and we can later on sort in a manner that makes sense for the cheat-sheet.

[1] https://github.com/lucaong/minisearch/blob/93bb0bc1878a48fcf2cf64e17099ae5a7042f9bd/src/MiniSearch.ts#LL1216C9-L1216C9
[2] https://github.com/lucaong/minisearch/blob/93bb0bc1878a48fcf2cf64e17099ae5a7042f9bd/src/MiniSearch.ts#LL1934C1-L1934C1
[3] https://stackoverflow.com/questions/234683/javascript-array-sort-implementation/236534#236534

[why] PR ryanoasis#1252 and this ran slightly out of sync. Signed-off-by: Fini Jastrow <ulf.fini.jastrow@desy.de>

rszyma changed the title ~~website: cheat-sheet page performance~~ website: improve cheat-sheet page performance May 20, 2023

rszyma mentioned this pull request May 21, 2023

modify generate-css.sh for changes in #1252 #1254

Merged

Finii pushed a commit that referenced this pull request May 27, 2023

Remove gitter-sidecar

34cddad

[why] Disfunctional See PR #1252

Finii pushed a commit that referenced this pull request May 27, 2023

Remove gitter-sidecar

2d96b7f

[why] Disfunctional See PR #1252

allcontributors bot mentioned this pull request May 27, 2023

docs: add SilverMira as a contributor for bug #1264

Merged

Finii mentioned this pull request May 30, 2023

Update search page style #1265

Closed

2 tasks

Finii reviewed May 30, 2023

View reviewed changes

rszyma and others added 8 commits May 30, 2023 19:13

fix: cheatsheet page performance

156981d

cheat_sheet: Sort removed icons last

c06a149

Signed-off-by: Fini Jastrow <ulf.fini.jastrow@desy.de>

cheat-sheet: fix negative boost for removed icons

d2503ca

cheat-sheet: boost the part after last -

2360483

cheat-sheet: add more negative boost for old icons

5d1ccc6

cheat-sheet: don't show nfc icons on empty search

270a604

Finii merged commit 3d817cf into ryanoasis:gh-pages May 30, 2023

allcontributors bot mentioned this pull request May 30, 2023

docs: add rszyma as a contributor for code #1269

Merged

rszyma deleted the fix/gh-pages-cheatsheet-performance branch May 30, 2023 17:54

LNKLEO pushed a commit to LNKLEO/Nerd that referenced this pull request Nov 24, 2023

modify generate-css.sh for changes in ryanoasis#1252

22b0101

LNKLEO pushed a commit to LNKLEO/Nerd that referenced this pull request Nov 24, 2023

generate-css: Adapt text

ecd026b

[why] PR ryanoasis#1252 and this ran slightly out of sync. Signed-off-by: Fini Jastrow <ulf.fini.jastrow@desy.de>

website: improve cheat-sheet page performance #1252

website: improve cheat-sheet page performance #1252

Conversation

rszyma commented May 20, 2023 • edited by Finii Loading

Description

Requirements / Checklist

What does this Pull Request (PR) do?

Modifications to cheat-sheet page:

Other

How should this be manually tested?

Any background context you can provide?

What are the relevant tickets (if any)?

rszyma commented May 20, 2023 • edited Loading

Finii commented May 20, 2023

Finii commented May 20, 2023

rszyma commented May 20, 2023 • edited Loading

rszyma commented May 21, 2023

Finii commented May 27, 2023

Finii commented May 27, 2023

Finii commented May 27, 2023

alt

alti

altimete

altimeter

rszyma commented May 27, 2023 • edited Loading

rszyma commented May 27, 2023

rszyma commented May 27, 2023

Finii commented May 27, 2023

allcontributors bot commented May 27, 2023

Finii commented May 30, 2023

Finii commented May 30, 2023 • edited Loading

Finii May 30, 2023 • edited Loading

Choose a reason for hiding this comment

rszyma May 30, 2023

Choose a reason for hiding this comment

rszyma May 30, 2023

Choose a reason for hiding this comment

Finii commented May 30, 2023

No real sort on startup:

Sorting still not according to id ;-)

rszyma commented May 30, 2023 • edited Loading

rszyma commented May 30, 2023 • edited Loading

Finii commented May 30, 2023

Finii commented May 30, 2023 • edited Loading

rszyma commented May 30, 2023 • edited Loading

Finii commented May 30, 2023

Finii commented May 30, 2023

Finii commented May 30, 2023

Finii commented May 30, 2023 • edited Loading

rszyma commented May 30, 2023

Finii commented May 30, 2023

Finii commented May 30, 2023 • edited Loading

Finii commented May 30, 2023

allcontributors bot commented May 30, 2023

Finii commented May 30, 2023

Finii commented May 31, 2023

rszyma commented May 20, 2023 •

edited by Finii

Loading

rszyma commented May 20, 2023 •

edited

Loading

rszyma commented May 20, 2023 •

edited

Loading

rszyma commented May 27, 2023 •

edited

Loading

Finii commented May 30, 2023 •

edited

Loading

Finii May 30, 2023 •

edited

Loading

rszyma commented May 30, 2023 •

edited

Loading

rszyma commented May 30, 2023 •

edited

Loading

Finii commented May 30, 2023 •

edited

Loading

rszyma commented May 30, 2023 •

edited

Loading

Finii commented May 30, 2023 •

edited

Loading

Finii commented May 30, 2023 •

edited

Loading