#3220 fulltext index improvements

[robfrank]

No description provided.

[gemini-code-assist]

Summary of Changes

Hello @robfrank, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly upgrades ArcadeDB's full-text indexing capabilities, moving beyond basic text search to offer advanced features comparable to dedicated search engines. It introduces granular control over Lucene analyzers, supports complex Lucene query syntax, and enables multi-property indexing. Furthermore, it provides new SQL functions for flexible querying and exposes relevance scores directly within SQL results, empowering users to build more sophisticated search functionalities.

Highlights

• Configurable Full-Text Indexes: Introduced FullTextIndexMetadata and TypeFullTextIndexBuilder to allow detailed configuration of Lucene analyzers (default, index-specific, query-specific, per-field), leading wildcards, and default boolean operators via METADATA clause in CREATE INDEX statements.
• Multi-Property Full-Text Indexing: Enhanced LSMTreeFullTextIndex to support indexing multiple properties within a single full-text index, storing tokens with field prefixes for field-specific searches.
• Advanced Lucene Query Syntax: Implemented FullTextQueryExecutor to parse and execute Lucene query syntax (boolean operators, phrase queries, wildcards, field-specific queries) against the full-text index.
• New SQL Search Functions: Added SEARCH_INDEX(indexName, query) and SEARCH_FIELDS(fieldNames, query) SQL functions, enabling direct Lucene-style queries against full-text indexes and automatic index resolution for SEARCH_FIELDS.
• SQL Score Exposure: Integrated $score as a special projection variable in SQL queries, allowing users to retrieve and order results by relevance score from full-text searches.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[mergify]

🧪 CI Insights

Here's what we observed from your CI run for ac30be5.

🟢 All jobs passed!

But CI Insights is watching 👀

[gemini-code-assist]

Code Review

This pull request introduces significant enhancements to ArcadeDB's full-text indexing capabilities, including configurable analyzers, Lucene query syntax support, new search functions (SEARCH_INDEX, SEARCH_FIELDS), multi-property index support, and score exposure ($score). The changes are extensive and well-structured, following a detailed design and implementation plan. I've identified a critical issue in the boolean query logic within the new FullTextQueryExecutor, a high-severity issue related to how TypeIndex handles sharded full-text searches, and a couple of medium-severity maintainability improvements. Overall, this is a great addition, but the identified correctness issues should be addressed before merging.

[gemini-code-assist]

The logic for handling boolean queries with both MUST and SHOULD clauses appears to be incorrect. Currently, results from SHOULD clauses are added to scoreMap, and then results from MUST clauses are also added. This effectively performs a UNION between MUST and SHOULD results.

The correct behavior for a query like +term1 term2 (where term1 is MUST and term2 is SHOULD) is that documents must contain term1, and documents that also contain term2 get a higher score. The current implementation would return documents containing just term2, which is incorrect.

To fix this, if mustResults is not null (meaning there were MUST clauses), the scoreMap (containing SHOULD results) should be filtered to retain only the documents present in mustResults before merging the scores.

[gemini-code-assist]

The logic for get() in this part of the implementation plan appears to have a copy-paste error. Both the if and else branches for k.contains(":") execute the same code: rids = underlyingIndex.get(new String[] { k });. This seems incorrect for handling field-qualified searches versus unqualified searches.

The actual implementation in LSMTreeFullTextIndex.java uses a different, more correct logic by parsing terms into QueryTerm objects. To avoid confusion and ensure the plan accurately reflects the implementation, it would be beneficial to update this section of the plan.

[gemini-code-assist]

This withMetadata(JSONObject) method duplicates the logic from FullTextIndexMetadata.fromJSON(). To improve maintainability and avoid code duplication, this method could be simplified to directly call fromJSON() on the metadata object.

  public void withMetadata(final JSONObject json) {
    ((FullTextIndexMetadata) metadata).fromJSON(json);
  }

[codacy-production]

Coverage summary from Codacy

See diff coverage on Codacy

Coverage variation	Diff coverage
✅ +0.24%	✅ 79.57%

Coverage variation details

	Coverable lines	Covered lines	Coverage
Common ancestor commit (cc5dadd)	119127	64614	54.24%
Head commit (ac30be5)	119954 (+827)	65347 (+733)	54.48% (+0.24%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details

	Coverable lines	Covered lines	Diff coverage
Pull request (#3221)	1116	888	79.57%

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

See your quality gate settings    Change summary preferences

[robfrank]

Addressed the relevant review comments in commit 2227197:

1. Critical - Boolean MUST/SHOULD logic (FullTextQueryExecutor.java) ✅ Fixed

• Documents matching only SHOULD clauses are no longer returned when MUST clauses are present
• SHOULD clauses now correctly add bonus score to documents that already satisfy all MUST clauses
• Added mustWithShouldQuery test case to verify the fix

2. Medium - Code duplication (TypeFullTextIndexBuilder.java) ✅ Fixed

• Simplified withMetadata(JSONObject) to delegate to FullTextIndexMetadata.fromJSON()

Not addressed:

• High - TypeIndex sharding comment: This is a limitation in TypeIndex's bucket selection strategy for full-text queries, not a bug in this PR. The test comment documents this workaround. A proper fix would require changes to TypeIndex itself.
• Medium - Plan documentation: The plan file is historical documentation of the implementation approach, not production code.

[robfrank]

Response to Review Comments

All review comments have been addressed:

1. ✅ Critical - Boolean MUST/SHOULD logic (FullTextQueryExecutor.java:201)

Status: Fixed by @robfrank in commit 2227197

• Documents matching only SHOULD clauses are no longer returned when MUST clauses are present
• Added test case mustWithShouldQuery to verify the fix

2. ✅ High Priority - TypeIndex sharding (LSMTreeFullTextIndexTest.java:272)

Status: Fixed in latest commits

• Modified TypeIndex.getIndexesByKeys() to explicitly detect full-text indexes and bypass bucket selection
• Full-text searches now scan ALL buckets regardless of sharding strategy
• Removed workaround from multiPropertyIndexFieldSpecificSearch test - now uses TypeIndex directly
• Added new test fullTextIndexWithShardedType to verify correct behavior with sharded types
• All related tests pass (LSMTreeFullTextIndexTest, ACIDTransactionTest, LSMTreeIndexTest)

Technical details: The fix ensures that full-text queries containing search terms (e.g., "java", "title:java") are not subject to bucket selection based on hashing those strings. Instead, they always search all buckets to ensure complete results, which is critical for correctness on sharded types.

3. ✅ Medium Priority - Code duplication (TypeFullTextIndexBuilder.java:147)

Status: Fixed by @robfrank in commit 2227197

• Simplified withMetadata(JSONObject) to delegate to FullTextIndexMetadata.fromJSON()

4. ⏭️ Medium Priority - Plan documentation (docs/plans/2026-01-23-fulltext-index-implementation.md:704)

Status: Not addressed

• As noted by @robfrank, this is historical documentation of the implementation approach, not production code
• The actual implementation in LSMTreeFullTextIndex.java is correct
• Low priority since it doesn't affect functionality

---

All critical and high-priority issues have been resolved. The full-text index implementation now correctly handles sharded types and boolean query logic.

[robfrank]

Updated the previous comment - point #4 has now been addressed:

4. ✅ Medium Priority - Plan documentation (docs/plans/2026-01-23-fulltext-index-implementation.md:704)

Status: Fixed

• Removed copy-paste error in the plan document
• Added explanatory note that the actual implementation in FullTextQueryExecutor.java uses QueryTerm objects and constructs search keys appropriately (field + ":" + text for field-specific queries, text alone for unqualified queries)
• The simplified representation in the plan now references the actual implementation

All review comments have been addressed.