[META] Filtering/searching on tables woes

[chrisronline]

This is a meta ticket to describe the various issues around filtering/searching our common tables (visualizations, dashboards, saved objects, etc).

We need to define and implement a consistent search/filtering experience.

• Support wildcard in front of a search query Allow wildcard when searching saved objects #9767
• Default sort is whack Visualizations are always sorted alphabetically #14687
• Using quotes leads to sporadic results Filtering visualizations with double-quotes is intermittent #14688
• Saved Objects Search gone downhill in Kibana 6.x #15926

There are also a few tickets around using an analyzer on the title field for the saved objects.
#4563 #5734 #10424. We should probably have an analyzed field and a non analyzed field and be smart about choosing which to use when searching/filtering.

/cc @stacey-gammon

[stacey-gammon]

While not related to filtering or searching, another reason to have a non-analyzed title field is so we can implement paging with sorting.

[stacey-gammon]

Found another issue. Searching with a - doesn't return exact matches.

[chrisronline]

@stacey-gammon Reported here: #5734

[chrisronline]

I've been doing some investigation and more thinking on this and I think there is a line in the sand we can confidently draw.

For search queries that do not include spaces, using a non-analyzed field with an automatically appended * gives the best results.

Input (a total of 9 docs):

POST search/doc
{
  "text": "My Visualization"
}

POST search/doc
{
  "text": "My Dashboard"
}

POST search/doc
{
  "text": "My Saved Search"
}

POST search/doc
{
  "text": "My-Visualization"
}

POST search/doc
{
  "text": "My-Dashboard"
}

POST search/doc
{
  "text": "My-Saved-Search"
}

POST search/doc
{
  "text": "My_Visualization"
}

POST search/doc
{
  "text": "My_Dashboard"
}

POST search/doc
{
  "text": "My_Saved_Search"
}

Searches:

POST search/_search
{
  "query": {
    "query_string": {
      "query": "My*",
      "fields": ["text.keyword"]
    }
  }
}
-> 
"hits": {
    "total": 9,

POST search/_search
{
  "query": {
    "query_string": {
      "query": "My-*",
      "fields": ["text.keyword"]
    }
  }
}
->
"hits": {
    "total": 3,
    "hits": [
      {
        "_source": {
          "text": "My-Saved-Search"
        }
      },
      {
        "_source": {
          "text": "My-Visualization"
        }
      },
      {
        "_source": {
          "text": "My-Dashboard"
        }
      }
    ]

POST search/_search
{
  "query": {
    "query_string": {
      "query": "My_*",
      "fields": ["text.keyword"]
    }
  }
}
->
"hits": {
  "total": 3,
  "hits": [
    {
      "_source": {
        "text": "My_Dashboard"
      }
    },
    {
      "_source": {
        "text": "My_Saved_Search"
      }
    },
    {
      "_source": {
        "text": "My_Visualization"
      }
    }
  ]

However, introducing spaces causes issues with using the non analyzed field:

POST search/_search
{
  "query": {
    "query_string": {
      "query": "My S*",
      "fields": ["text.keyword"]
    }
  }
}
-> 
"hits": {
    "total": 0,

For spaces, we should use the analyzed field and let it the standard analyzer do it's work:

POST search/_search
{
  "query": {
    "query_string": {
      "query": "My S*",
      "fields": ["text"]
    }
  }
}
-> 
"hits": {
  "total": 6,
  "hits": [
    {
      "_source": {
        "text": "My-Saved-Search"
      }
    },
    {
      "_source": {
        "text": "My Saved Search"
      }
    },
    {
      "_source": {
        "text": "My Visualization"
      }
    },
    {
      "_source": {
        "text": "My-Visualization"
      }
    },
    {
      "_source": {
        "text": "My Dashboard"
      }
    },
    {
      "_source": {
        "text": "My-Dashboard"
      }
    }
  ]

It kinda works. But we probably only want the saved search results.

POST search/_search
{
  "query": {
    "query_string": {
      "query": "My S*",
      "analyze_wildcard": true, 
      "minimum_should_match": "2", 
      "fields": ["text"]
    }
  }
}
-> 
"hits": {
  "total": 2,
  "hits": [
    {
      "_source": {
        "text": "My-Saved-Search"
      }
    },
    {
      "_source": {
        "text": "My Saved Search"
      }
    }
  ]
}

That's better. But we also need to address #14687

POST search/doc
{
  "text": "Top Transactions For Me"
}

POST search/doc
{
  "text": "Top Transactions For You"
}

POST search/_search
{
  "query": {
    "query_string": {
      "query": "Top Transactions For Y*",
      "analyze_wildcard": true, 
      "minimum_should_match": "2", 
      "fields": ["text"]
    }
  }
}
->
"hits": {
  "total": 2,
  "hits": [
    {
      "_source": {
        "text": "Top Transactions For You"
      }
    },
    {
      "_source": {
        "text": "Top Transactions For Me"
      }
    }
  ]
}

We only want to show the first, so we need to ensure minimum_should_match matches our word count:

POST search/_search
{
  "query": {
    "query_string": {
      "query": "Top Transactions For Y*",
      "analyze_wildcard": true, 
      "minimum_should_match": "4", 
      "fields": ["text"]
    }
  }
}
->
"hits": {
  "total": 1,
  "hits": [
    {
      "_source": {
        "text": "Top Transactions For You"
      }
    }
  ]
}

Conclusion to follow...

[thomasneirynck]

this one's probably related too: #11403

[stratoula]

I am closing this as there is only one issue left and is already assigned in the correct team