Add linear hybrid search ranker

[willtai]

Description

This PR is a follow-up work from a previous PR to rework the Hybrid search ranking step. This PR allows users of the HybridRetriever and HybridCypherRetriever to specify ranker and alpha, which ranks the relevance of the retrieved results over the vector index and fulltext index search using the score:

final_score = alpha * (vector normalized score) + (1 - alpha) * (fulltext normalized score)

In practice, the vector index score is multiplied by alpha and fulltext index score is multiplied by 1 - alpha, then they are compared and ranked.

The default used is still naive which is the naive implementation of comparing normalized scores.

The pros and cons of using this ranker:

Pros

• Common scale when comparing scores. This reduces the risk of either vector or fulltext index score dominating the score due to its scale.
• Allow user to determine alpha.

Cons

• Assumption of linearity. This ranker assumes both index scores can be added linearly, when in reality the underlying score space may be non-linear.
• The dynamic normalization might introduce inconsistency in ranking across different queries.

Despite these tradeoffs, I think this is valuable for users as it is an upgrade over the naive ranker.

Type of Change

• New feature
• Bug fix
• Breaking change
• Documentation update
• Project configuration change

Complexity

Note

Please provide an estimated complexity of this PR of either Low, Medium or High

Complexity: Medium

How Has This Been Tested?

• Unit tests
• E2E tests
• Manual tests

Checklist

The following requirements should have been met (depending on the changes in the branch):

• Documentation has been updated
• Unit tests have been updated
• E2E tests have been updated
• Examples have been updated
• New files have copyright header
• CLA (https://neo4j.com/developer/cla/) has been signed
• CHANGELOG.md updated if appropriate

[willtai]

@CodiumAI-Agent /update_changelog

[QodoAI-Agent]

Changelog updates: 🔄

2025-02-24

Added

• Introduced a linear hybrid search ranker for HybridRetriever and HybridCypherRetriever, allowing customizable ranking with an alpha parameter.
• Added validation and error handling for invalid ranker types in hybrid search.
• Enhanced unit and E2E tests to cover the new linear ranker functionality.

to commit the new content to the CHANGELOG.md file, please type:
'/update_changelog --pr_update_changelog.push_changelog_changes=true'

[NathalieCharbel]

Thanks for having this feature implemented! Nicely done :D

[QodoAI-Agent]

Changelog updates: 🔄

2025-02-25

Added

• Introduced a linear hybrid search ranker for HybridRetriever and HybridCypherRetriever, allowing customizable ranking with an alpha parameter.

to commit the new content to the CHANGELOG.md file, please type:
'/update_changelog --pr_update_changelog.push_changelog_changes=true'

[stellasia]

Nice work, thanks for addressing this issue!

[QodoAI-Agent]

Changelog updates: 🔄

2025-02-26

Added

• Introduced a linear hybrid search ranker for HybridRetriever and HybridCypherRetriever, allowing customizable ranking with an alpha parameter.

to commit the new content to the CHANGELOG.md file, please type:
'/update_changelog --pr_update_changelog.push_changelog_changes=true'