[BUG] List of Available fields on Discover page is not correctly refreshed

[vzhgit]

Describe the bug

We are migrating to OpenSearch and found this strange behavior:
List of Available fields on Discover page is not correctly refreshed after pressing "Refresh" button or applying a filter if "Hide missing fields" is turned on.
But if refresh a browser's page (F5) the list will contain right list of fields.

To Reproduce
Steps to reproduce the behavior:

1. Go to Discover page with default search setting which returns some amount of documents.
2. Be sure that "Hide missing fields" option in "Filter by type" is turned on
3. Copy list of Available fields
4. Filter by _id field of any document (to get only one document in a search result)
5. Compare list of Available fields with list from step 2.
6. Now refresh browser's window
7. Compare list of Available fields with list from step 2.

Expected behavior
after applying a filter (Step 4) list of available fields list must contains fields only from documents from the search result (Step 6)

OpenSearch Version

  "version" : {
    "distribution" : "opensearch",
    "number" : "1.2.3",
    "build_type" : "tar",
    "build_hash" : "8a529d77c7432bc45b005ac1c4ba3b2741b57d4a",
    "build_date" : "2021-12-21T01:36:21.407473Z",
    "build_snapshot" : false,
    "lucene_version" : "8.10.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },

Dashboards Version
1.2.0

Plugins

$ bin/opensearch-dashboards-plugin list
alertingDashboards@1.2.0.0
anomalyDetectionDashboards@1.2.0.0
ganttChartDashboards@1.2.0.0
indexManagementDashboards@1.2.0.0
observabilityDashboards@1.2.0.0
queryWorkbenchDashboards@1.2.0.0
reportsDashboards@1.2.0.0
securityDashboards@1.2.0.0

Host/Environment (please complete the following information):

• OpenSearch server in k8s cluster. Host: Ubuntu 18.04.6 LTS
• Client's browser and version: Google Chrome 97.0.4692.99

[ashwin-pc]

Having taken a look at whats causing this, it seems almost intentional given how the code for this is written.

This is happening because we create a hash table of all the fields and how many times they have occurred in the search result. The count however is not reset when the search is updated. This causes the field present in the initial search to persist.

To me this seems like a useful feature if the user is filtering an index pattern and wants to continue exploring the existing fields. Changing the indexpattern, updating the search bar etc still trigger the available fields to still update.

We could make it more evident that the field is not present in the current set of filtered documents instead of hiding them all together. @ahopp @kgcreative What are your thoughts on this?

[kgcreative]

@ashwin-pc - Sounds like the bug is that the available fields are not updated when filters are added. I would expect fields to dynamically update when Hide missing fields is on. We can confirm this because a hard refresh does update the count.

[joshuarrrr]

@ashwin-pc @abbyhu2000 Do we have this same problem in VisBuilder? If not, and if we use the VisBuilder fields as a jumping-off point for de-angularizing discover, this may go away.

[ashwin-pc]

On VisBuilder the list of available fields is not dynamic. It does not represent all the available fields for the given search space, but rather all the available fields for the index in its mapping. A argument could be made whether we should hide missing fields there too, and I think we missed that during the initial implementation. Created an issue for that here #3074.

As for your comment about using VisBuilder as a jump off point, I think we could do that because solving this problem there correctly could be used to solve the problem for discover too since the Available fields section was built to mimic discovers