full-text search using Lunr.js

[ZoeLeBlanc]

As an alternative to List.js, we could also use Lunr.js which has more robust search capacity.

This branch has my current attempt to get the two of these working together. It requires pulling down the pre-built indices and corpora, then filtering them for a search term, trying to reuse some of the existing UX by having the search results appear where the lesson abstract usually does.

One issue I'm having is that I'm not getting the right position for the search terms in the documents. @mdlincoln I'm using your code from the gist so if you wouldn't mind taking a look I would really appreciate it!

closes #1018

Checklist

• Assign yourself in the "Assignees" menu
• Assign at least one individual or team to "Reviewers"
  • if the text needs to be translated, assign the relevant language team(s) as "Reviewers" and tag both the team as well as the managing edtor in your PR. Please follow the translation request guidelines when writing your PR description
• Add the appropriate "Label"
• Ensure the Travis CI checks pass
• Check the live preview of your PR on Netlify
• If this PR closes an open issue, add the phrase Closes #ISSUENUMBER to the description above

If you are having difficulty fixing Travis errors, first consult https://github.com/programminghistorian/jekyll/wiki/Making-Technical-Contributions carefully, especially "Common Travis Errors". Then contact the technical team if you need further help.

[mdlincoln]

I think you're getting the right search position, but applying it to the wrong documents! E.g. if I search georeferencing it doesn't bring up the georeferencing lesson at all, but three other ones that don't even mention the term.

There's an indexing issue somewhere in your code, where you're picking the wrong lessons from the search results and then trying to applying the highlighting to the text bodies and thus getting bizarre results - haven't pinpointed it yet, but that's where I'd look.

[ZoeLeBlanc]

Thanks for figuring this out @mdlincoln !!! Definitely appreciate the help!! I wonder if it the search corpus and index are mismatched somehow 🤔

Otherwise it would likely be the results from the idx.search or where I'm creating the var docs 🕵️‍♀️

[mdlincoln]

When I inspect the results from idx.search(searchString); and compare to the ids in search.json they are definitely correct! So at least that part is working.

Where is the portion of the code where you try to match that id number to the entries in the <ul> on the lessons page? Because those integer IDs only exist rn when creating the search.json file 🤔

[mdlincoln]

huh, so in the corpus object you create, the id value is totally wrong, but the index location in the corpus object is correct?

e.g. at position 34 in corpus, the item has id: 84. A search for georeferencing returns the id 34, and the doc in position 34 in corpus is the right doc, it's just that the id doesn't correspond.

[ZoeLeBlanc]

Thanks for the feedback @mdlincoln !! I updated the reference to the search index and I think fixed the issue. You were totally right about the doc.id being the wrong thing. Turns out our Lunr ref was looking for url, so I updated the search-index repo and now everything seems to be working. Gonna add in the logic for multiple languages and then I think we're ready for testing 🎉

[ZoeLeBlanc]

So a weird bug that I could use your help with @mdlincoln. If you've noticed sometimes it gives a blank answer or doesn't show the updated search results or doesn't erase them when you empty out the input.

After lots of testing turns out it does all of that if you click enter at the end of the search query, which I'm pretty sure is an issue with Chrome not updating the new html, but could have to do with List.JS as well...

I've tried using jquery trigger, as well as resizing the window, and using setTimeout to hide and show the search results. So far nothing has worked... any thoughts or ideas of what we should try???

[ZoeLeBlanc]

Sorry for the very similar commits here, for some reason I wasn't seeing my pushes in the Github UI so overdid it a bit.

Anyways, pretty much have search fully working before tomorrow's meeting 🎉

You can now click on a topic or activity, search within those lessons, undo your search, and still have those lessons selected.

I also added logic for a button or enter press which is pretty much fixing the display bug 98% of the time (sometimes you need a second enter press).

Looking forward to hearing how it works for others 😅

[mdlincoln]

• Instead of "enable search lessons?" I might write "Start searching" in the initial search button.

• I type in a search term and click Search Lessons. I then edit the search text and click Search Lessons again, but nothing happens. I would expect that it would execute a new search, but instead I first need to click the "Reset" button

• I type in a search term and click Search Lessons. I then click on one of the faceting buttons above (e.g. Digital Publishing), and the list expands to include all the digital publishing lessons plus the text search lessons. I would expect that it would show the intersect of the two queries.

  • HOWEVER If I do the above and then click Search Lessons again, we get the intersect.

• I love that the search query text entry is automatically capitalized, which is a useful way to suggest that its text insensitive. It IS case-insensitive, right?

• Is the index creation and index search language-sensitive when it does word stemming? Can we set that up if it's not already? (e.g. in English search, "lesson" will also match "lessons" but in Spanish, "lección" will not match "lecciones"
  Questions:

• You'd mentioned we could enable more advanced wildcard searching. Do we think this is really necessary, or overkill? I lean towards thinking it'd be overkill - we don't even have 100 lessons right now.

[jenniferisasi]

• 'Enable search lessons?' to 'Start searching' or 'Search in lessons'
• I looked for text*, then I do text (thinking from a general to specific search) and the search is correct or accurate. Now, upon deleting the work, in this case text the result don't clear out. When I click the return key or the 'search lessons' tab it takes back to the previous search results, here text*, then another return or search lessons and it clears out to the lesson index.
• In Spanish I realize it shows some singular/plurals and some verb conjugations in the search but not always (not using the * here). Examples:
  -- valor valores
  -- texto textos
  -- dato datos
  -- escribe escribí
  -- cerrar cerrará
  -- guardar guardará (so this indicates I think it takes also whatever follows the characters in the search
• In the same way, it didn't do all the plurals or follow the characters...
  -- red, didn't show 'redes'
• After a few searches for the above experiment, it "run out of searches" in other words, I had to reset to see all lessons (had to click a couple of times)
• It is accent mark sensitive - if missing it won't find the word
• The * search retrieves "all" possible results for the use of a root, ie. 'utiliza*' (verbs + nouns)

I did some searches in French and this is what I found there:

• 'avancée' retrieves avancée, avancées, avancés.

• 'caractéristique' retrieves the plural 'caractéristiques'

• l'etu(missing the

• It doesn't retrieve a string of characters that doesn't show the word completely? I do 'orig' thinking of original.
  -- origin , retrieves original; original retrieves origin

• It looks to me that after a few searches the search lesson stops working until you reset.

[mdlincoln]

Thanks Zoe I see your commits - hoping to peer through them this weekend and then we can launch 🎷

[mdlincoln]

Could we move the lunr stemming code into a js/vendor subdirectory just to make clear that it's drop-in code that didn't need to be customized for our site?

Thinking ahead, how difficult will it e to add in support for Portuguese stemming? Can we make sure that process is documented in the code comments or elsewhere?

[mdlincoln]

Remove the console log functions now that we're ready to push this

[ZoeLeBlanc]

These were there originally (i.e. before I started working on search) but happy to remove them.

[ZoeLeBlanc]

Should I also remove where we console.log the new URI?

[ZoeLeBlanc]

@mdlincoln I like the idea of the js/vendor and happy to document (though it is almost entirely just following the lunr docs). One question I've had is where to document everything? Do you have any suggestions?

Also for the search-info pop out. Are we planning to keep it? If yes, I can add it to our snippets. Finally should I take out all the console.logs or just the one you highlighted. There's quite a few of them so let me know. Thanks!

[mdlincoln]

Ditch all the console.log()

A high-level overview page similar to the wiki I wrote for the translation links would be very helpful. You don't need to explain every single line of code bc the comments are quite good, but mainly to document the architecture.

And that mini-help text is great. Let's add it to snippets

[ZoeLeBlanc]

Alrighty! All console.logs are gone and snippet added 🎉

The example of Twitter and Network in the search info works for Spanish but not French, based on the lessons available. I would recommend for French maybe doing +Python -Ruby but it raises the question of whether we may want to standardize examples across languages or just let each team decide which example search terms would be most useful to readers.

I'll start working on the documentation tutorial this week but seems like we're in the final mile 🙌

[mdlincoln]

@rivaquiroga @programminghistorian/spanish-team
@spapastamkou @programminghistorian/french-team

If you can please help translate the final snippet that provides search documentation for users: https://github.com/programminghistorian/jekyll/pull/1726/files#diff-a407e023d60dfefccac66d5a7959003cR375-R384

[mdlincoln]

@ZoeLeBlanc please put the boilerplate JS code into a /vendor directory as discussed above

[ZoeLeBlanc]

@mdlincoln thanks for catching that! Just moved the files to vendor directory and updated the lesson index to only load appropriate stemmer for each language (rather than loading them all on each page).

[mdlincoln]

ohhh good catch, I didn't even think to check it was loading all that extra JS!

[ZoeLeBlanc]

hmm weirdly isn't showing that I created vendor and moved the files 🤔 Should I try pushing up again??

[mdlincoln]

oh shoot - at one point I think vendor/ was added to our gitignore b/c of the way that ruby gems are installed locally at /vendor/bundle/ruby

It ought to be /vendor in gitignore so that it doesn't ignore all directories that have the name vendor in them, just the one at the root

[ZoeLeBlanc]

ahh that makes more sense! I'll go ahead and fix that in gitignore, and then push up again