Implement cheaper search

[cjyabraham]

CNCF pays for the current search on this site. It uses both Google Custom Search (for requests outside of China) and Bing Search API (for requests in China). Due to recent changes in Bing Search pricing, this is now costing CNCF too much money so we should implement a different search solution such as those suggested here.

These other search-related issues could be addressed at the same time:

• Improvements for search bar #44416
• Limit Search Results to current language #43817
• Replace "search" with "search this site" #43291

[sftim]

What prioritization would you put on this, @cjyabraham ?

/area web-development

[cjyabraham]

I would say important-soon

[dims]

@cjyabraham How much is it costing us? (may be drop an email to steering-private if not public?)

[cjyabraham]

November cost $2400 just for the Bing API costs. I don't know Google off-hand.

[natalisucks]

@cjyabraham Thanks for opening this issue. With my SIG Docs co-chair hat on, I'd like to know what extra resources the CNCF plan to allocate to this work, given that SIG Docs as a whole at the moment doesn't have the people to work on this. Let's chat about this in the new year at one of our bi-weekly community meetings so that we can understand the scope and give a better estimate of effort in partnership with you.

[sftim]

/priority important-soon

[cjyabraham]

(@natalisucks I don't know about official allocations of resources from CNCF but @caniszczyk may be able to help...)

As it is, I did a bit of research and got quite excited by this relatively new Hugo search option: Pagefind. I've tried it out on the kubernetes.io site on this PR with preview here: https://deploy-preview-44530--kubernetes-io-main-staging.netlify.app/

You can play with it there on the homepage just as a POC. LMK how that performs from your POV.

FYI you can learn more about Pagefind by watching this vid presenting how it works in general and what features are in the v1.0 release.

[sftim]

The performance (latency) for #44530 looks fine to me. We'd want to tweak the UI. Also, the relevance of results is a lot worse than the Google search.

For example, try searching for “feature gates”.

[sftim]

I suggest we focus on getting to a UI that works and won't provoke issues (right now, if we merged #44530, people would file bug reports). Then we can merge that and iterate on the quality of the search results.

Maybe there is a search API service that is more affordable and that we can use in combination with PageFind (or equivalent). That would be more work, but this is open source. Someone might actually relish the challenge.

[cjyabraham]

A few thoughts:

• I suspect Google Custom Search, which powers all non-China search queries on kubernetes.io, is free, so we could keep that in place if it's better than pagefind. What motivated this issue initially is that the cost of the Bing custom search (used for China users) has gone from $40/month to $2400/month since we've had to switch on to their new API and pricing model.
• Pagefind does have a number of ways it can be fine-tuned which we could experiment with. That said, it may never be as good as Google.

Open questions:

1. Can someone verify if Google Custom Search is, in fact, free, for kubernetes.io? I don't have access to its control panel, however, I run it for several other CNCF sites and it's free for all of them since we're nonprofit etc.
2. If Pagefind results are "good enough", should we use it for China requests instead of Bing? If so, I can refine the PR to improve the UI before submitting it for review.
3. Would it be worth investigating if Bing would be willing to donate credits for this since it's Kubernetes? Not sure if anyone has contacts there...

[sftim]

If Pagefind results are "good enough", should we use it for China requests instead of Bing? If so, I can refine the PR to improve the UI before submitting it for review.

Seems legit. Help is welcome from people whose access to Google Search is blocked through national policy - or from anyone else who'd like to help.

Most people (≅ 67%) in China speak mandarin Chinese as their main language, so we should check more than superficially around at the search results in that language. I know Russia has state censorship too; their main language is Russian; again, the quality of indexing for English is not much of a guide.

[sftim]

On whether Kubernetes pays for the Google Search: I suspect it's easy to check, but hard to find contributors who know who to ask.

[cjyabraham]

Here's the status for support for Chinese language in pagefind.

[sftim]

• maybe we can rally some contributions into PageFind
• maybe there is an alternative tool that is particularly good at indexing ideographic languages?

Help is most definitely welcome.

/area localization

[sftim]

/triage accepted

[cjyabraham]

Algolia DocSearch is another option we could try. It's free, however, it would add a dependency on the Algolia crawler and search engine, and we'd need to display their branding.

[natalisucks]

@cjyabraham Our next SIG Docs community meeting is on January 9th at 18:30 UTC and I'll be adding this to our agenda for discussion. It would be great to see you there, and joining the SIG Docs Google Group will give you access to the calendar invitation for that call. We'll also post the link to our Zoom meeting in the #sig-docs channel on Kubernetes Slack on the same day

[cjyabraham]

Hi @natalisucks , unfortunately that meeting would be at 1:30AM my time so I don't think I'll be able to make it. If you have any questions for me in the run up to the meeting please let me know.

[natalisucks]

@cjyabraham Per my comment here, I'd really like a rep from the CNCF to come and chat with us so we can understand how we can work in tandem with you, once again, given resources. Let me know who else I should reach out to. If @kubernetes/steering-committee members also wish to attend and discuss, alongside anyone else involved in the thread on Slack, that would be great.

[idvoretskyi]

Looping @jeefy @castrojo in here :)

[jeefy]

I unfortunately have a conflict during the next SIG-Docs meeting, hopefully @onlydole might be able to join.

I think in terms of timing, we should try to target no later than KubeCon EU, but if we can all rally together sooner, dope.

Another thing to consider is potentially using a more local search engine. Example: Baidu's a member and also services the region. So... :) Those are my immediate two cents.

[caniszczyk]

You may want to chat with @nate-double-u about experience using https://lunrjs.com on CNCF projects

[natalisucks]

@onlydole Let's chat at the meeting next Tuesday then so we can do some planning and resource work. Thanks!

[onlydole]

You betcha! I'll read up on some of these options and will be prepared to chat with our group next Tuesday.

[dipesh-rawat]

There are few open-source and commercial search alternatives listed in the Hugo documentation (https://gohugo.io/tools/search).

[nate-double-u]

You may want to chat with @nate-double-u about experience using https://lunrjs.com on CNCF projects

I'm just back from vacation now and am catching up on things, always happy to help tho, just need a bit to read the various discussions that have already happened.

[pacoxu]

In Slack thread https://kubernetes.slack.com/archives/CPNFRNLTS/p1703270890381669, we have some inputs.

Some possible choices:

1. https://docsearch.algolia.com/ is another choice. This is free to opensource. I am not sure if CNCF projects can use it.

• Kevin: For https://docsearch.algolia.com/, there are some CNCF projects integrated already on their website, but we may need more research on the search quality part.

2. Another choice is using a redirect to bing.com directly for free.
3. Currently, can we just support local search for kubernetes.io? I am not sure if Hugo can support the local search like VitePress, which supports fuzzy full-text search using a in-browser index.

A question is there "Are we okay with ads on the chinese side of things? ".

[dylantientcheu]

Hey there!

At Algolia we love K8s. Indeed DocSearch is open and we'll be happy to provide an awesome (and free) experience your multi language documentation website.

Here's a demo we've set up - https://7cnxmj.csb.app/

[sftim]

What's the next step for this issue? I can't tell.

[natalisucks]

@sftim Hi Tim – as per our SIG Docs meeting on January 9th and the feedback shared there, @onlydole attended and has been tasked with the next step of the CNCF to come up with a plan and further research to improve search – due to the nature of this being a CNCF cost challenge, and not a strictly user/contributor-facing issue that SIG Docs leads would prioritize and lead work on. Taylor will hopefully be updating us on this issue when that plan and/or research is ready to share

[onlydole]

Howdy, folks - we will be prioritizing this on the CNCF side after KubeCon + CloudNativeCon EU in Paris wraps up, and @nate-double-u will lead on that effort.

[nate-double-u]

Hi @dylantientcheu, Thanks for building that demo. Could you contact me (natew@cncf.io)? I'd like to chat with you about it.

[nate-double-u]

Sorry for the late update here. I updated the sig-docs meeting notes, but not this issue.

I've done some research and think that PageFind is the best candidate to replace Bing. Algolia was in the running, but between their fee structure and the fact that going with them will add another dependency, we're best off going with PageFind. PageFind is well used around the CNCF and we have some experience supporting it. It's designed to build its index as a part of the regular site build process, making the search a part of the site, sidestepping any issue we may run into with firewalled locations.

• Regarding Algolia: I spoke with Algolia; pricing depends on traffic. We'd probably need the Grow plan (production) level, which offers 10,000 free calls (50¢/1000 calls after).

We should change as little as possible here, so we should only replace Bing, affecting only the localizations with Bing as their search provider (i.e., I don't think we should remove or update any other search providers).

@cjyabraham has made progress based on @sftim's work to implement PageFind for Chinese users and will soon open a PR for discussion/review.

[cjyabraham]

PageFind solution has been deployed.