Improve search index generation for PHP.net

[lhsazevedo]

Note

This is a companion PR to php/web-php#1084, but it is not dependent on it and can be merged independently.

Intro

PHD index entries can have both short (sdesc) and long (ldesc) descriptions. Often, the short description is empty, leading to the current fallback mechanism in getShortDescription and getLongDescription:

// phpdotnet/phd/Format.php
final public function getLongDescription($id, &$isLDesc = null) {
    if ($this->indexes[$id]["ldesc"]) {
        $isLDesc = true;
        return $this->indexes[$id]["ldesc"];
    } else {
        $isLDesc = false;
        return $this->indexes[$id]["sdesc"];
    }
}
final public function getShortDescription($id, &$isSDesc = null) {
    if ($this->indexes[$id]["sdesc"]) {
        $isSDesc = true;
        return $this->indexes[$id]["sdesc"];
    } else {
        $isSDesc = false;
        return $this->indexes[$id]["ldesc"];
    }
}

Problem

The current search index JSON generation doesn't utilize this fallback
mechanism, resulting in missing entries in PHP.net search results.

Solution

This PR addresses the issue by:

• Using the long description as the title when the short description is empty.
• Using the parent book title as the long description in such cases.
• Ignoring non-page entries (non-chunk entries) when generating the search index JSON.

Impact

• Improves search result completeness and accuracy.
• Reduces search index size by excluding irrelevant entries.

Examples

Currently, "PHP Manual > Language Reference > Types > String" is missing from search results due to an empty short description. This change will use "Strings" (the long description) as the title and "Language Reference" (the parent book title) as the long description.

search-index.json

 [
-  "",
+  "Strings",
   "language.types.string",
   "sect1"
 ],
-[
-  "",
-  "language.types.string",
-  "sect2"
-],

...
  [
-   "",
+   "Enumerations",
    "language.types.enumerations",
    "sect1"
  ],

search-description.json

-  "language.types.string": "Strings",
+  "language.types.string": "Language Reference",
...
-  "language.types.enumerations": "Enumerations",
+  "language.types.enumerations": "Language Reference",

Statistics

stat	before	after	change
Entries count	29,765	11,256	-62%
Entries lacking title (sdesc)	19,955	0	-100%
Search index size	1,383KB	693KB	-50%

Generated Search Index Diff Preview

Only the first 100 lines are shown for brevity.

 [
   [
-    "",
+    "Copyright",
     "copyright",
     "legalnotice"
   ],
   [
-    "",
-    "index",
-    "info"
-  ],
-  [
-    "",
-    "index",
-    "book"
-  ],
-  [
-    "",
-    "preface",
-    "section"
-  ],
-  [
-    "",
+    "Preface",
     "preface",
     "preface"
   ],
   [
-    "",
-    "intro-whatis",
-    "example"
-  ],
-  [
-    "",
+    "What is PHP?",
     "intro-whatis",
     "section"
   ],
   [
-    "",
+    "What can PHP do?",
     "intro-whatcando",
     "section"
   ],
   [
-    "",
+    "Introduction",
     "introduction",
     "chapter"
   ],
   [
-    "",
+    "What do I need?",
     "tutorial.requirements",
     "section"
   ],
   [
-    "",
-    "tutorial.firstpage",
-    "example"
-  ],
-  [
-    "",
-    "tutorial.firstpage",
-    "example"
-  ],
-  [
-    "",
+    "Your first PHP-enabled page",
     "tutorial.firstpage",
     "section"
   ],
   [
-    "",
-    "tutorial.useful",
-    "example"
-  ],
-  [
-    "",
-    "tutorial.useful",
-    "example"
-  ],
-  [
-    "",
-    "tutorial.useful",
-    "example"
-  ],
-  [
-    "",
+    "Something Useful",
     "tutorial.useful",
     "section"
   ],
   [
-    "",
-    "tutorial.forms",
-    "example"
-  ],

[kamil-tekiela]

Will this break the search if we merge it now?

[lhsazevedo]

No. It is safe to merge as it doesn't alter the JSON structure.

[lhsazevedo]

Thank you!

[kamil-tekiela]

Just curious. You said it would not affect the current functionality, right? Does that mean you changed something in the other PR that will avail of this change?

[lhsazevedo]

I said that it wouldn't break the current search, but you can already see the improved results on php.net.

Unfortunately, the current UI hides them under the last result group ("Other Matches"). You'll probably need to scroll down the menu to see it. You may also need to clear you local storage because the search index is cached for two weeks.

Try searching for "syntax", "types" or "operators". You should see some language reference results under "Other Matches". Those pages were missing from the index before because they don't have an sdesc.

[haszi]

Problem

The current search index JSON generation doesn't utilize this fallback mechanism, resulting in missing entries in PHP.net search results.

Solution

This PR addresses the issue by:

...

• Ignoring non-page entries (non-chunk entries) when generating the search index JSON.

One question: how does removing non-chunk entries address the issue of empty long/short descriptions not using their short/long counterpart? Beside that this change not related to the problem, it also removes the possibility of searching for any useful sections that don't necessarily need an entire page (e.g. the ternary or the null coalescing operators).

[lhsazevedo]

Thanks for bringing this up.

The main reason for skipping non-chunk entries was to avoid adding an sdesc to them, which would make them appear in the search results. The front-end does not properly handle non-chunk entries, and displaying them would lead to broken links. I realize I should have clarified this in the PR.

To properly integrate non-chunk entries into the search, we'd need to:

1. Include the container chunk ID in the index (unless parent_id is guaranteed to be a chunk).
2. Update the front-end to handle non-chunk links correctly, using the container chunk ID as the path and the entry ID as a hash property. For example, https://www.php.net/manual/en/language.operators.comparison.php#language.operators.comparison.ternary .
3. Determine the best value for the ldesc for non-chunk entries. Using the container chunk's sdesc might be preferable over the parent book/set sdesc.

Your question led me to some interesting discoveries:

1. There are 18,205 non-chunk entries currently missing sdesc, but 370 non-chunk entries actually have both sdesc and ldesc. These are <refentry> aliases (like DateTime::getTimezone, an alias for date_timezone_get()) and were removed from the index after the PR. I think they should be using the same chunk value as the entry they alias to, but it is hardcoded as false (0):
   

   phd/phpdotnet/phd/Index.php

   Lines 254 to 269 in ec49154

   	if ($array === true) {
   	foreach($lastChunk["sdesc"] as $sdesc) {
   	$this->commit[++$idx] = [
   	"docbook_id" => $lastChunkId,
   	"filename" => $lastChunk["filename"],
   	"parent_id" => $this->currentchunk,
   	"sdesc" => $sdesc,
   	"ldesc" => $lastChunk["ldesc"],
   	"element" => $lastChunk["element"],
   	"previous" => $lastChunk["previous"],
   	"next" => ($lastChunk["chunk"] ? "POST-REPLACEMENT" : ""),
   	"chunk" => 0,
   	];
   	$this->POST_REPLACEMENT_INDEXES[] = array("docbook_id" => $lastChunkId, "idx" => $idx);
   	}
   	}

   
   I'll create an issue so we can discuss how to handle this.

2. Another issue: while the entries' IDs are not unique, I've seen some search related code assuming they are, which definitely needs to be corrected. Will create an issue for this as well.

[lhsazevedo]

@kamil-tekiela would you mind adding the hacktoberfest-accepted label to this one?

[Girgias]

@kamil-tekiela would you mind adding the hacktoberfest-accepted label to this one?

Done