hybrid search with a Disjunction Max bm25

[Oaklight]

Hi, I'm trying to use hybrid search with a Disjunction Max bm25 query. Although the code works fine with either only bm25 or embeddings, the combined hybrid one does not work:

sql_query = """SELECT * FROM search_index.rank_hybrid(
            bm25_query => 'paradedb.disjunction_max(disjuncts => ARRAY[paradedb.parse(''answer:some-text''), paradedb.parse(''question:some-other-text'')])',
            similarity_query => '''[0.036399692, 0.055808958, ..., -0.057896737, -0.0030355677, 0.023476213]'' <=> embedding',
            bm25_weight => 0.9,
            similarity_weight => 0.1,
            bm25_limit_n => 10,
            similarity_limit_n => 10
        );"""

The error message reads:

Traceback (most recent call last):
  File "/projects/knowledge_base/search_data/search_hybrid.py", line 27, in <module>
    results = hybrid.search_hybrid(
              ^^^^^^^^^^^^^^^^^^^^^
  File "/projects/knowledge_base/core/hybrid.py", line 45, in search_hybrid
    cur.execute(sql_query)
psycopg2.errors.UndefinedFunction: function paradedb.minmax_bm25(jsonb, integer, integer) does not exist
LINE 20:                     FROM paradedb.minmax_bm25($1, NULL::int4...
                                  ^
HINT:  No function matches the given name and argument types. You might need to add explicit type casts.
QUERY:  
                WITH similarity AS (
                    SELECT
                        id as key_field,
                        CASE
                            WHEN (MAX('[0.036399692, 0.055808958, -0.03455947, -0.025994042, 0.01877212, -0.061296955, 0.029041132, -0.015619802, 0.024629243, 0.09458077, 0.006101706, 0.008676688, -0.018384498, -0.0470774, -0.047083315, -0.008698734, -0.008468424, -0.0009200693, -0.04134596, -0.0092298975, -0.02599103, 0.009642122, -0.005339215, 0.031209039, -0.003948931, -0.04629158, 0.012221733, 0.007058472, -0.019702172, -0.02662751, 0.012360383, 0.011164021, 0.015248464, -0.0072209775, 0.018585749, -0.010944223, 0.041158102, -0.024891185, 0.00050833065, 0.052900493, -0.0027208521, -0.048910707, -0.03633378, -0.024488114, -0.026672816, -0.006161673, 0.012983944, -0.021363186, 0.0020516564, 0.05076585, 0.058290347, 0.004753185, -0.053140085, -0.039731856, 0.0165522, 0.06761884, -0.0055074375, 0.008818247, 0.040325973, 0.048718978, 0.042723022, 0.004413631, -0.013172254, -0.0015622955, -0.05226788, 0.009603826, 0.004665336, -0.021174004, -0.029158412, -0.0217058

Any ideas why?

[rebasedming]

Hi! I just responded in Slack but I'll reply here as well.

I'm having trouble replicating. The following code works:

-- Create BM25 index over test table
CALL paradedb.create_bm25_test_table(
  schema_name => 'public',
  table_name => 'mock_items'
);

CALL paradedb.create_bm25(
    index_name => 'search_idx',
    schema_name => 'public',
    table_name => 'mock_items',
    key_field => 'id',
    text_fields => '{description: {tokenizer: {type: "en_stem"}}, category: {}}',
    numeric_fields => '{rating: {}}'
);

-- Create mock embeddings
ALTER TABLE mock_items ADD COLUMN embedding vector(3);
UPDATE mock_items m
SET embedding = ('[' ||
    ((m.id + 1) % 10 + 1)::integer || ',' ||
    ((m.id + 2) % 10 + 1)::integer || ',' ||
    ((m.id + 3) % 10 + 1)::integer || ']')::vector;

-- Hybrid search with disjunction max
SELECT * FROM search_idx.rank_hybrid(
    bm25_query => paradedb.disjunction_max(disjuncts => ARRAY[paradedb.parse('description:keyboard'), paradedb.parse('description:bluetooth')]),
    similarity_query => '''[1,2,3]'' <-> embedding',
    bm25_weight => 0.9,
    similarity_weight => 0.1
) LIMIT 5;

Could you check to see what's different/confirm this works for you?

[Oaklight]

when was 'paradedb.minmax_bm25' first introduced? I suspect the docker instance was initialized and persisted with an earlier version where the function was not supported.

Mine looks like this:

qa_database=> SELECT extversion FROM pg_extension WHERE extname = 'pg_search';
 extversion 
------------
 0.7.5
(1 row)

[philippemnoel]

The minmax_bm25 function was introduced a few months ago, before the release here, but it's possible the syntax changed. Is it working correctly with you on the latest version? (v0.8.2?`

[Oaklight]

After removing the persistent volume and recreating the database, it works as intended.

Strangely, my extension was 0.7.5, but I was using paradedb 0.8.2 before the volume recreation. Something went sideways.

As for the docker image upgrade, is ALTER EXTENSION pg_search UPDATE TO '0.8.2' required? I remember changing the tag from something to 0.8.2 a few days ago.