Docusaurus sends incorrect Algolia DocSearch request under /search

[fnune]

🐛 Bug Report

Prerequisites

• I'm using the latest version of Docusaurus.
• I have tried the npm run clear or yarn clear command.
• I have tried rm -rf node_modules yarn.lock package-lock.json and re-installing packages.
• I have tried creating a repro with https://new.docusaurus.io
• I have read the console error message carefully (if applicable)

Description

Docusaurus' Algolia DocSearch integration, added following Docusaurus docs, results in a problem where searches using the widget return correct results. However, searching under website.com/search always produces zero hits. The bug becomes more apparent when searching through the widget: a link "See all 43 results" appears, but when clicking it, the resulting /search?q=query page displays "No results were found".

Have you read the Contributing Guidelines on issues?

Yes

Steps to reproduce

1. Add Algolia DocSearch following this documentation section.
2. Access /search.
3. See that no results are returned.

Expected behavior

The search behaves the same, or similarly to the results in the search widget.

Actual behavior

See the bug description.

Your environment

• Docusaurus version used: 2.0.0-beta.2
• Environment name and version: Mozilla Firefox 89.0.2, Node.js 12.22.1
• Operating system and version: Ubuntu 21.04

Reproducible demo

https://codesandbox.io/s/hardcore-lehmann-pwvxd?file=/docusaurus.config.js

• In the app, open the widget and search for nrf
• Notice the valid results, and how there's a "See all ## results" at the bottom
• Click "See all ## results"
• See that the search actually contains no results
• Search for anything else, confirm that the search never returns any results

Additional context

I've diffed the requests from the widget and the requests from /search. Apparently, only the parameters are different:

--- request-bad.json	2021-06-29 16:14:22.446046969 +0200
+++ request-correct.json	2021-06-29 16:32:30.988285483 +0200
@@ -2,15 +2,8 @@
     "requests": [
         {
             "indexName": "memfault",
-            "params": "hitsPerPage=15&advancedSyntax=true&query=nrf&page=0&facets=%5B%22language%22%2C%22docusaurus_tag%22%5D&tagFilters=&facetFilters=%5B%5B%22language%3Aen%22%5D%2C%5B%22docusaurus_tag%3Adefault%22%2C%22docusaurus_tag%3Adocs-default-current%22%5D%5D"
-        },
-        {
-            "indexName": "memfault",
-            "params": "hitsPerPage=1&advancedSyntax=true&query=nrf&page=0&attributesToRetrieve=%5B%5D&attributesToHighlight=%5B%5D&attributesToSnippet=%5B%5D&tagFilters=&analytics=false&clickAnalytics=false&facets=language&facetFilters=%5B%5B%22docusaurus_tag%3Adefault%22%2C%22docusaurus_tag%3Adocs-default-current%22%5D%5D"
-        },
-        {
-            "indexName": "memfault",
-            "params": "hitsPerPage=1&advancedSyntax=true&query=nrf&page=0&attributesToRetrieve=%5B%5D&attributesToHighlight=%5B%5D&attributesToSnippet=%5B%5D&tagFilters=&analytics=false&clickAnalytics=false&facets=docusaurus_tag&facetFilters=%5B%5B%22language%3Aen%22%5D%5D"
+            "query": "nrf",
+            "params": "hitsPerPage=20&highlightPreTag=%3Cmark%3E&highlightPostTag=%3C%2Fmark%3E&attributesToRetrieve=%5B%22hierarchy.lvl0%22%2C%22hierarchy.lvl1%22%2C%22hierarchy.lvl2%22%2C%22hierarchy.lvl3%22%2C%22hierarchy.lvl4%22%2C%22hierarchy.lvl5%22%2C%22hierarchy.lvl6%22%2C%22content%22%2C%22type%22%2C%22url%22%5D&attributesToSnippet=%5B%22hierarchy.lvl1%3A10%22%2C%22hierarchy.lvl2%3A10%22%2C%22hierarchy.lvl3%3A10%22%2C%22hierarchy.lvl4%3A10%22%2C%22hierarchy.lvl5%3A10%22%2C%22hierarchy.lvl6%3A10%22%2C%22content%3A10%22%5D&snippetEllipsisText=%E2%80%A6&facetFilters=%5B%5D"
         }
     ]
 }

...where request-correct.json contains the params of the request that produced the expected results, and request-bad.json the params of the request that produced no results.

[fnune]

I've parsed the params above for easier comparison.

Here's the bad request:

{
    "requests": [
        {
            "indexName": "memfault",
            "params": {
                "hitsPerPage": "15",
                "advancedSyntax": "true",
                "query": "nrf",
                "page": "0",
                "facets": "[\"language\",\"docusaurus_tag\"]",
                "tagFilters": "",
                "facetFilters": "[[\"language:en\"],[\"docusaurus_tag:default\",\"docusaurus_tag:docs-default-current\"]]"
            }
        },
        {
            "indexName": "memfault",
            "params": {
                "hitsPerPage": "1",
                "advancedSyntax": "true",
                "query": "nrf",
                "page": "0",
                "attributesToRetrieve": "[]",
                "attributesToHighlight": "[]",
                "attributesToSnippet": "[]",
                "tagFilters": "",
                "analytics": "false",
                "clickAnalytics": "false",
                "facets": "language",
                "facetFilters": "[[\"docusaurus_tag:default\",\"docusaurus_tag:docs-default-current\"]]"
            }
        },
        {
            "indexName": "memfault",
            "params": {
                "hitsPerPage": "1",
                "advancedSyntax": "true",
                "query": "nrf",
                "page": "0",
                "attributesToRetrieve": "[]",
                "attributesToHighlight": "[]",
                "attributesToSnippet": "[]",
                "tagFilters": "",
                "analytics": "false",
                "clickAnalytics": "false",
                "facets": "docusaurus_tag",
                "facetFilters": "[[\"language:en\"]]"
            }
        }
    ]
}

And here's the correct request:

{
    "requests": [
        {
            "indexName": "memfault",
            "query": "nrf",
            "params": {
                "hitsPerPage": "20",
                "highlightPreTag": "<mark>",
                "highlightPostTag": "</mark>",
                "attributesToRetrieve": "[\"hierarchy.lvl0\",\"hierarchy.lvl1\",\"hierarchy.lvl2\",\"hierarchy.lvl3\",\"hierarchy.lvl4\",\"hierarchy.lvl5\",\"hierarchy.lvl6\",\"content\",\"type\",\"url\"]",
                "attributesToSnippet": "[\"hierarchy.lvl1:10\",\"hierarchy.lvl2:10\",\"hierarchy.lvl3:10\",\"hierarchy.lvl4:10\",\"hierarchy.lvl5:10\",\"hierarchy.lvl6:10\",\"content:10\"]",
                "snippetEllipsisText": "…",
                "facetFilters": "[]"
            }
        }
    ]
}

[fnune]

I've managed to confirm that changing themeConfig.algolia.searchParameters does not change the params passed in the /search request.

[slorber]

FYI the search page is a bit out-of-date and using an older/different API from Algolia.
We know that it does not emit exactly the same query, and particularly it does not respect your searchParameters in the same way the modal do (see #3805), and the search page always apply at least some facet filtering on tags the algolia theme put in your pages (while it's only there when you enable contextualSearch for the modal)

What I see:

• Your live site does not contain the correct metadata tags, because you didn't enable the algolia theme on production
• Thus the Algolia crawler/index does not contain the facet filters we declare and query on

This is a bit confusing but you have to deploy first a site with the algolia theme somewhere, get your index populated correctly, to only then be able to query it appropriately.

You can either run the Algolia crawler locally, or wait 24h after the prod deployment (or ask DocSearch team for a crawl), or use a staging deployment as start_url temporarily.

If you deploy this to production, it should start working within 24h

A good way to see if your index contains the correct metadata is to use contextualSearch: true: if the modal keeps working (using facet filters in the query) it's a sign that your index contains appropriate data.

[fnune]

I see!

I suppose I could think up a documentation patch for Docusaurus where I try to clarify this? I think it's easy for future developers to fall into the same trap. Developers may not feel confident enough to deploy the service to production if it's not working fully.

Thanks a lot for your clarification.

[slorber]

Yes, it's not the first time this comes up.

Actually we decouple Docusaurus from Algolia, and the search tags we put by default in pages are generic and overrideable by any search plugin (ie you can build one for Elastic if you want).

// Note: we don't couple this to Algolia/DocSearch on purpose
// We may want to support other search engine plugins too
// Search plugins should swizzle/override this comp to add their behavior
export default function SearchMetadatas({
  locale,
  version,
  tag,
}: Props): JSX.Element {
  return (
    <Head>
      {locale && <meta name="docusaurus_locale" content={locale} />}
      {version && <meta name="docusaurus_version" content={version} />}
      {tag && <meta name="docusaurus_tag" content={tag} />}
    </Head>
  );
}

// Override default/agnostic SearchMetas to use Algolia-specific metadatas
export default function AlgoliaSearchMetadatas({locale, version, tag}) {
  const language = locale;
  return (
    <Head>
      {language && <meta name="docsearch:language" content={language} />}
      {version && <meta name="docsearch:version" content={version} />}
      {tag && <meta name="docsearch:docusaurus_tag" content={tag} />}
    </Head>
  );
}

---

In theory, it is a good idea but in practice most of our users use DocSearch. Previously we added the DocSearch metadatas even when the plugin is not used, and we didn't have this problem.

I'm thinking of re-biasing Docusaurus toward DocSearch usage, this looks more pragmatic

[slorber]

Closing because we already re-biased search metadata to Algolia by default (more convenient, less confusing for users), and we have an issue to track the differences between search page + search modal (#3805)