Add fast client-side search to Airflow documentation #59658

kaxil · 2025-12-20T00:38:12Z

Preview: https://airflow.staged.apache.org/docs/apache-airflow/3.1.5/

I have been frustrated by Sphinx search for a long-long time. So after adding dark-mode (and other related changes in https://github.com/apache/airflow-site), this was next in my list!

This PR/commit introduces a fast, fully client-side search experience for the Apache Airflow documentation, powered by Pagefind. The new search is keyboard-accessible (Cmd+K / Ctrl+K), works offline, and requires no external services.

Search indexes are generated automatically at documentation build time and loaded entirely in the browser, enabling sub-50 ms queries even on large docs.

~~I have kept the Sphinx search too as a backup and it will keep functioning.~~ Completely replaced Sphinx search now.

What’s included

New Sphinx extension: pagefind_search

Located in devel-common/src/sphinx_exts/pagefind_search/:

init.py: Extension setup with configuration values and event handlers
builder.py: Automatic index building with graceful fallback
static/css/pagefind.css: Search modal and button styling with dark mode support
static/js/search.js: Search functionality with keyboard shortcuts
templates/search-modal.html: Search modal HTML template

Features

Keyboard shortcut (Cmd+K/Ctrl+K) opens search modal
Arrow key navigation through results
Works offline (no external services)
Automatic indexing during documentation build
Dark mode support
Sub-50ms search performance
Configurable content indexing via conf.py

User Experience

Users can now:

Press Cmd+K from any documentation page to search
Navigate results with arrow keys, Enter to select, Esc to close
Search works immediately without network requests
Results show page title, breadcrumb, and excerpt

Configuration

Available in conf.py:

pagefind_enabled: Toggle search indexing
pagefind_verbose: Enable build logging
pagefind_root_selector: Define searchable content area
pagefind_exclude_selectors: Exclude navigation, headers, footers
pagefind_custom_records: Index non-HTML content (PDFs, etc.)

Ranking Optimization

I have also spent a lot of time tuning the search the below knows.. Now, the extension uses optimized ranking parameters in search.js which in my testing has produced better results:

termFrequency: 1.0 - Standard term occurrence weighting
termSaturation: 0.7 - Moderate saturation to prevent over-rewarding repetition
termSimilarity: 7.5 - Maximum boost for exact phrase matches and similar terms
pageLength: 0 - No penalty for longer pages (important for reference documentation)

closes apache/airflow-site#666

choo121600 · 2025-12-20T04:42:15Z

WoW, it looks really Cool😲

Dev-iL · 2025-12-20T05:18:20Z

Great work, Kaxil!

potiuk · 2025-12-20T07:11:17Z

NICE!

Dockerfile.ci

I have been frustrated by Sphinx search for a long-long time. So after adding dark-mode, this was next in my list! This PR/commit introduces a fast, fully client-side search experience for the Apache Airflow documentation, powered by [Pagefind](https://pagefind.app/). The new search is keyboard-accessible (Cmd+K / Ctrl+K), works offline, and requires no external services. Search indexes are generated automatically at documentation build time and loaded entirely in the browser, enabling sub-50 ms queries even on large docs. I have kept the Sphinx search too as a backup and it will keep functioning. ---- Add keyboard-accessible search (Cmd+K) to Apache Airflow documentation with automatic indexing and offline support. New Sphinx extension: `pagefind_search` Located in `devel-common/src/sphinx_exts/pagefind_search/`: - __init__.py: Extension setup with configuration values and event handlers - builder.py: Automatic index building with graceful fallback - static/css/pagefind.css: Search modal and button styling with dark mode support - static/js/search.js: Search functionality with keyboard shortcuts - templates/search-modal.html: Search modal HTML template - Keyboard shortcut (Cmd+K/Ctrl+K) opens search modal - Arrow key navigation through results - Works offline (no external services) - Automatic indexing during documentation build - Dark mode support - Sub-50ms search performance - Configurable content indexing via conf.py Users can now: - Press Cmd+K from any documentation page to search - Navigate results with arrow keys, Enter to select, Esc to close - Search works immediately without network requests - Results show page title, breadcrumb, and excerpt Available in conf.py: - pagefind_enabled: Toggle search indexing - pagefind_verbose: Enable build logging - pagefind_root_selector: Define searchable content area - pagefind_exclude_selectors: Exclude navigation, headers, footers - pagefind_custom_records: Index non-HTML content (PDFs, etc.)

Dockerfile.ci

kaxil · 2025-12-20T11:57:16Z

I completely replaced Sphinx search based on the mailing list feedback. This is how it would look now

I have been frustrated by Sphinx search for a long-long time. So after adding dark-mode, this was next in my list! This PR/commit introduces a fast, fully client-side search experience for the Apache Airflow documentation, powered by [Pagefind](https://pagefind.app/). The new search is keyboard-accessible (Cmd+K / Ctrl+K), works offline, and requires no external services. Search indexes are generated automatically at documentation build time and loaded entirely in the browser, enabling sub-50 ms queries even on large docs. I have kept the Sphinx search too as a backup and it will keep functioning. ---- Add keyboard-accessible search (Cmd+K) to Apache Airflow documentation with automatic indexing and offline support. New Sphinx extension: `pagefind_search` Located in `devel-common/src/sphinx_exts/pagefind_search/`: - __init__.py: Extension setup with configuration values and event handlers - builder.py: Automatic index building with graceful fallback - static/css/pagefind.css: Search modal and button styling with dark mode support - static/js/search.js: Search functionality with keyboard shortcuts - templates/search-modal.html: Search modal HTML template - Keyboard shortcut (Cmd+K/Ctrl+K) opens search modal - Arrow key navigation through results - Works offline (no external services) - Automatic indexing during documentation build - Dark mode support - Sub-50ms search performance - Configurable content indexing via conf.py Users can now: - Press Cmd+K from any documentation page to search - Navigate results with arrow keys, Enter to select, Esc to close - Search works immediately without network requests - Results show page title, breadcrumb, and excerpt Available in conf.py: - pagefind_enabled: Toggle search indexing - pagefind_verbose: Enable build logging - pagefind_root_selector: Define searchable content area - pagefind_exclude_selectors: Exclude navigation, headers, footers - pagefind_custom_records: Index non-HTML content (PDFs, etc.) (cherry picked from commit d0bd2df)

…che#59658) I have been frustrated by Sphinx search for a long-long time. So after adding dark-mode, this was next in my list! This PR/commit introduces a fast, fully client-side search experience for the Apache Airflow documentation, powered by [Pagefind](https://pagefind.app/). The new search is keyboard-accessible (Cmd+K / Ctrl+K), works offline, and requires no external services. Search indexes are generated automatically at documentation build time and loaded entirely in the browser, enabling sub-50 ms queries even on large docs. I have kept the Sphinx search too as a backup and it will keep functioning. ---- Add keyboard-accessible search (Cmd+K) to Apache Airflow documentation with automatic indexing and offline support. New Sphinx extension: `pagefind_search` Located in `devel-common/src/sphinx_exts/pagefind_search/`: - __init__.py: Extension setup with configuration values and event handlers - builder.py: Automatic index building with graceful fallback - static/css/pagefind.css: Search modal and button styling with dark mode support - static/js/search.js: Search functionality with keyboard shortcuts - templates/search-modal.html: Search modal HTML template - Keyboard shortcut (Cmd+K/Ctrl+K) opens search modal - Arrow key navigation through results - Works offline (no external services) - Automatic indexing during documentation build - Dark mode support - Sub-50ms search performance - Configurable content indexing via conf.py Users can now: - Press Cmd+K from any documentation page to search - Navigate results with arrow keys, Enter to select, Esc to close - Search works immediately without network requests - Results show page title, breadcrumb, and excerpt Available in conf.py: - pagefind_enabled: Toggle search indexing - pagefind_verbose: Enable build logging - pagefind_root_selector: Define searchable content area - pagefind_exclude_selectors: Exclude navigation, headers, footers - pagefind_custom_records: Index non-HTML content (PDFs, etc.) (cherry picked from commit d0bd2df) Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>

potiuk · 2025-12-20T14:53:40Z

Backport in #59669

Follow up after apache#59658 - there are few more places where canary builds used `pip check` and until the `pip` inconsistent behaviour is fixed (pypa/pip#13709) we should use `uv pip check` instead.

I have been frustrated by Sphinx search for a long-long time. So after adding dark-mode, this was next in my list! This PR/commit introduces a fast, fully client-side search experience for the Apache Airflow documentation, powered by [Pagefind](https://pagefind.app/). The new search is keyboard-accessible (Cmd+K / Ctrl+K), works offline, and requires no external services. Search indexes are generated automatically at documentation build time and loaded entirely in the browser, enabling sub-50 ms queries even on large docs. I have kept the Sphinx search too as a backup and it will keep functioning. ---- Add keyboard-accessible search (Cmd+K) to Apache Airflow documentation with automatic indexing and offline support. New Sphinx extension: `pagefind_search` Located in `devel-common/src/sphinx_exts/pagefind_search/`: - __init__.py: Extension setup with configuration values and event handlers - builder.py: Automatic index building with graceful fallback - static/css/pagefind.css: Search modal and button styling with dark mode support - static/js/search.js: Search functionality with keyboard shortcuts - templates/search-modal.html: Search modal HTML template - Keyboard shortcut (Cmd+K/Ctrl+K) opens search modal - Arrow key navigation through results - Works offline (no external services) - Automatic indexing during documentation build - Dark mode support - Sub-50ms search performance - Configurable content indexing via conf.py Users can now: - Press Cmd+K from any documentation page to search - Navigate results with arrow keys, Enter to select, Esc to close - Search works immediately without network requests - Results show page title, breadcrumb, and excerpt Available in conf.py: - pagefind_enabled: Toggle search indexing - pagefind_verbose: Enable build logging - pagefind_root_selector: Define searchable content area - pagefind_exclude_selectors: Exclude navigation, headers, footers - pagefind_custom_records: Index non-HTML content (PDFs, etc.) (cherry picked from commit d0bd2df)

Follow up after #59658 - there are few more places where canary builds used `pip check` and until the `pip` inconsistent behaviour is fixed (pypa/pip#13709) we should use `uv pip check` instead.

Follow up after apache#59658 - there are few more places where canary builds used `pip check` and until the `pip` inconsistent behaviour is fixed (pypa/pip#13709) we should use `uv pip check` instead. (cherry picked from commit 33e6d3e)

amoghrajesh · 2025-12-22T06:00:20Z

Looks really cool!

I have been frustrated by Sphinx search for a long-long time. So after adding dark-mode, this was next in my list! This PR/commit introduces a fast, fully client-side search experience for the Apache Airflow documentation, powered by [Pagefind](https://pagefind.app/). The new search is keyboard-accessible (Cmd+K / Ctrl+K), works offline, and requires no external services. Search indexes are generated automatically at documentation build time and loaded entirely in the browser, enabling sub-50 ms queries even on large docs. I have kept the Sphinx search too as a backup and it will keep functioning. ---- Add keyboard-accessible search (Cmd+K) to Apache Airflow documentation with automatic indexing and offline support. New Sphinx extension: `pagefind_search` Located in `devel-common/src/sphinx_exts/pagefind_search/`: - __init__.py: Extension setup with configuration values and event handlers - builder.py: Automatic index building with graceful fallback - static/css/pagefind.css: Search modal and button styling with dark mode support - static/js/search.js: Search functionality with keyboard shortcuts - templates/search-modal.html: Search modal HTML template - Keyboard shortcut (Cmd+K/Ctrl+K) opens search modal - Arrow key navigation through results - Works offline (no external services) - Automatic indexing during documentation build - Dark mode support - Sub-50ms search performance - Configurable content indexing via conf.py Users can now: - Press Cmd+K from any documentation page to search - Navigate results with arrow keys, Enter to select, Esc to close - Search works immediately without network requests - Results show page title, breadcrumb, and excerpt Available in conf.py: - pagefind_enabled: Toggle search indexing - pagefind_verbose: Enable build logging - pagefind_root_selector: Define searchable content area - pagefind_exclude_selectors: Exclude navigation, headers, footers - pagefind_custom_records: Index non-HTML content (PDFs, etc.) (cherry picked from commit d0bd2df)

Follow up after apache#59658 - there are few more places where canary builds used `pip check` and until the `pip` inconsistent behaviour is fixed (pypa/pip#13709) we should use `uv pip check` instead.

I have been frustrated by Sphinx search for a long-long time. So after adding dark-mode, this was next in my list! This PR/commit introduces a fast, fully client-side search experience for the Apache Airflow documentation, powered by [Pagefind](https://pagefind.app/). The new search is keyboard-accessible (Cmd+K / Ctrl+K), works offline, and requires no external services. Search indexes are generated automatically at documentation build time and loaded entirely in the browser, enabling sub-50 ms queries even on large docs. I have kept the Sphinx search too as a backup and it will keep functioning. ---- Add keyboard-accessible search (Cmd+K) to Apache Airflow documentation with automatic indexing and offline support. New Sphinx extension: `pagefind_search` Located in `devel-common/src/sphinx_exts/pagefind_search/`: - __init__.py: Extension setup with configuration values and event handlers - builder.py: Automatic index building with graceful fallback - static/css/pagefind.css: Search modal and button styling with dark mode support - static/js/search.js: Search functionality with keyboard shortcuts - templates/search-modal.html: Search modal HTML template - Keyboard shortcut (Cmd+K/Ctrl+K) opens search modal - Arrow key navigation through results - Works offline (no external services) - Automatic indexing during documentation build - Dark mode support - Sub-50ms search performance - Configurable content indexing via conf.py Users can now: - Press Cmd+K from any documentation page to search - Navigate results with arrow keys, Enter to select, Esc to close - Search works immediately without network requests - Results show page title, breadcrumb, and excerpt Available in conf.py: - pagefind_enabled: Toggle search indexing - pagefind_verbose: Enable build logging - pagefind_root_selector: Define searchable content area - pagefind_exclude_selectors: Exclude navigation, headers, footers - pagefind_custom_records: Index non-HTML content (PDFs, etc.) (cherry picked from commit d0bd2df)

Follow up after #59658 - there are few more places where canary builds used `pip check` and until the `pip` inconsistent behaviour is fixed (pypa/pip#13709) we should use `uv pip check` instead. (cherry picked from commit 33e6d3e)

I have been frustrated by Sphinx search for a long-long time. So after adding dark-mode, this was next in my list! This PR/commit introduces a fast, fully client-side search experience for the Apache Airflow documentation, powered by [Pagefind](https://pagefind.app/). The new search is keyboard-accessible (Cmd+K / Ctrl+K), works offline, and requires no external services. Search indexes are generated automatically at documentation build time and loaded entirely in the browser, enabling sub-50 ms queries even on large docs. I have kept the Sphinx search too as a backup and it will keep functioning. ---- Add keyboard-accessible search (Cmd+K) to Apache Airflow documentation with automatic indexing and offline support. New Sphinx extension: `pagefind_search` Located in `devel-common/src/sphinx_exts/pagefind_search/`: - __init__.py: Extension setup with configuration values and event handlers - builder.py: Automatic index building with graceful fallback - static/css/pagefind.css: Search modal and button styling with dark mode support - static/js/search.js: Search functionality with keyboard shortcuts - templates/search-modal.html: Search modal HTML template - Keyboard shortcut (Cmd+K/Ctrl+K) opens search modal - Arrow key navigation through results - Works offline (no external services) - Automatic indexing during documentation build - Dark mode support - Sub-50ms search performance - Configurable content indexing via conf.py Users can now: - Press Cmd+K from any documentation page to search - Navigate results with arrow keys, Enter to select, Esc to close - Search works immediately without network requests - Results show page title, breadcrumb, and excerpt Available in conf.py: - pagefind_enabled: Toggle search indexing - pagefind_verbose: Enable build logging - pagefind_root_selector: Define searchable content area - pagefind_exclude_selectors: Exclude navigation, headers, footers - pagefind_custom_records: Index non-HTML content (PDFs, etc.)

Follow up after apache#59658 - there are few more places where canary builds used `pip check` and until the `pip` inconsistent behaviour is fixed (pypa/pip#13709) we should use `uv pip check` instead.

boring-cyborg bot added the kind:documentation label Dec 20, 2025

kaxil mentioned this pull request Dec 20, 2025

Favour human-written documentation over machine-generated documentation apache/airflow-site#666

Closed

kaxil requested review from amoghrajesh, ashb, bugraoz93, gopidesupavan, jason810496, jscheffl and potiuk as code owners December 20, 2025 01:06

kaxil force-pushed the indexed-search branch from 14d4e2e to eac9892 Compare December 20, 2025 01:06

potiuk approved these changes Dec 20, 2025

View reviewed changes

potiuk reviewed Dec 20, 2025

View reviewed changes

Dockerfile.ci Show resolved Hide resolved

phanikumv approved these changes Dec 20, 2025

View reviewed changes

kaxil added 3 commits December 20, 2025 10:46

Make results more accurate

94716d5

fixup! Make results more accurate

609494f

kaxil force-pushed the indexed-search branch from eac9892 to 609494f Compare December 20, 2025 10:46

potiuk reviewed Dec 20, 2025

View reviewed changes

Dockerfile.ci Show resolved Hide resolved

kaxil added 2 commits December 20, 2025 12:05

Replace sphinx search completely

ca2b291

Add comment for uv pip check

eca9d4a

kaxil force-pushed the indexed-search branch from 7c15e3a to eca9d4a Compare December 20, 2025 12:05

kaxil added a commit to apache/airflow-site-archive that referenced this pull request Dec 20, 2025

Use client-side search from apache/airflow#59658

410f4d9

kaxil merged commit d0bd2df into apache:main Dec 20, 2025
126 checks passed

kaxil deleted the indexed-search branch December 20, 2025 13:01

kaxil added this to the Airflow 3.1.6 milestone Dec 20, 2025

kaxil added a commit to apache/airflow-site-archive that referenced this pull request Dec 20, 2025

[3.1.5] Use client-side search from apache/airflow#59658

50c3934

kaxil added a commit to apache/airflow-site-archive that referenced this pull request Dec 20, 2025

[stable] Use client-side search from apache/airflow#59658

617ae9e

potiuk mentioned this pull request Dec 20, 2025

Convert remaining pip check commands to uv pip check #59670

Merged

kaxil mentioned this pull request Dec 20, 2025

python: Write WHEEL tags in expanded form Pagefind/pagefind#989

Merged

ephraimbuddy added the type:doc-only Changelog: Doc Only label Jan 6, 2026

ephraimbuddy mentioned this pull request Jan 8, 2026

Status of testing of Apache Airflow 3.1.6rc1 and Task SDK 1.1.6rc1 #60262

Closed

61 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add fast client-side search to Airflow documentation #59658

Add fast client-side search to Airflow documentation #59658

Uh oh!

kaxil commented Dec 20, 2025 •

edited

Loading

Uh oh!

choo121600 commented Dec 20, 2025

Uh oh!

Dev-iL commented Dec 20, 2025

Uh oh!

potiuk commented Dec 20, 2025

Uh oh!

Uh oh!

Uh oh!

kaxil commented Dec 20, 2025

Uh oh!

Uh oh!

potiuk commented Dec 20, 2025

Uh oh!

amoghrajesh commented Dec 22, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Add fast client-side search to Airflow documentation #59658

Add fast client-side search to Airflow documentation #59658

Uh oh!

Conversation

kaxil commented Dec 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What’s included

Features

User Experience

Configuration

Ranking Optimization

Uh oh!

choo121600 commented Dec 20, 2025

Uh oh!

Dev-iL commented Dec 20, 2025

Uh oh!

potiuk commented Dec 20, 2025

Uh oh!

Uh oh!

Uh oh!

kaxil commented Dec 20, 2025

Uh oh!

Uh oh!

potiuk commented Dec 20, 2025

Uh oh!

amoghrajesh commented Dec 22, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

kaxil commented Dec 20, 2025 •

edited

Loading