Index Patterns Refresh field list automatically update for new fields #913

teebu · 2021-11-05T17:05:43Z

Is your feature request related to a problem? Please describe.

I was using Dashboards, and was trying to figure out why new dynamic fields were not showing up in the selection.
It turns out this problem was solved in Kibana elastic/kibana#82223 a while ago, but I still had to manually hit the refresh fields in the Index Patterns in this version.

Describe the solution you'd like

Just like in Kibana, I believe you don't have to do anything, it just works. They solved it by removing field caching

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Other discussions on such topic:
https://discuss.elastic.co/t/refresh-field-list-through-api-call/198764
elastic/kibana#6498
elastic/kibana#82223

JacobBrandt · 2021-11-05T18:58:03Z

As another user pointed out after Kibana made this change...it broke some peoples use cases. In some instances people used this old behavior to purposely hide fields from their users. It's NOT a field security reason they do this either. Just want to make sure that this point is highlighted.

Removing field caching is a good feature but it should at the same time come with the ability to hide fields from users. Kibana never implemented a way to accommodate the use case that was possible with the old behavior.

This commits adds a projection click listener called onProjectionListener and available in the <Settings /> component with the following characteristics: - it will fire every time the user clicks on the projection area (everything within the axes) - the listener is called with an object that contains the nearest X and Ys values. These values are not screen coordinates but the real data domain values inverted computed inverting the x and y scales. - we prevent the onProjectionListener from firing an event if a onElementClick listener is available and it has fired a click event close opensearch-project#846

# [24.2.0](elastic/elastic-charts@v24.1.0...v24.2.0) (2020-11-25) ### Bug Fixes * near and far alignments for orthogonal rotations ([opensearch-project#911](elastic/elastic-charts#911)) ([6b162c6](elastic/elastic-charts@6b162c6)) ### Features * add projection click listener ([opensearch-project#913](elastic/elastic-charts#913)) ([19c14e4](elastic/elastic-charts@19c14e4)), closes [opensearch-project#846](elastic/elastic-charts#846)

florianakos · 2022-09-06T06:13:11Z

Any news on when this feature could land in OSD?

joshuarrrr · 2022-09-06T20:23:45Z

@ashwin-pc to research further

ashwin-pc · 2022-09-07T01:36:51Z

The main problem here is with Dashboards caching the index pattern field list. And to solve this multiple approaches have been proposed:

Refresh list API
Remove caching

Looking into this a lil more, this should be fairly simple to implement if we make caching optional. It should satisfy both requirements of maintaining the existing behavior while also giving users the flexibility to override this behavior and always update the field list.

This can also be a setting that one can set in the advanced setting's pane.

@ahopp do we need to prioritize this?

KannappanSomu · 2023-02-01T15:56:54Z

+1 ,

any news on this feature ?

Aigerim-ai · 2023-04-08T14:04:06Z

Hi, please assign this issue to me!

joshuarrrr · 2023-04-10T16:33:47Z

@Aigerim-ai Let's get at least one of your other open PRs merged first, before assigning. In the meantime, there are a number of other CCI PRs (https://github.com/opensearch-project/OpenSearch-Dashboards/pulls?q=is%3Aopen+is%3Apr+label%3ACCI) that you can help peer review.

djablonski-moia · 2023-09-19T12:12:34Z

Any progress here?

ashishvaishno · 2024-02-09T14:38:07Z

Are there any API's available to do the refresh programatically?
I tried to send a PUT request on the saved objects (_dashboards/api/saved_objects/index-pattern/index-pattern-id)
with the below parameters , but that doesnt work 👎
payload = {"attributes":{"title": "index-*", "refresh_fields": 'true'}}
headers = {"Content-Type": "application/json", "osd-xsrf": "true", "security_tenant": "global" }

yb-yu · 2024-03-12T02:31:53Z

How can I remove field cache manually by API?

cameronattard · 2024-03-12T02:42:29Z

Would be great for this to get some attention. It feels like a really bad user experience to have to manually refresh the cache very time a new field is added, particularly in use cases where fields are dynamic in nature like log analytics.

vshunkov · 2024-07-02T14:44:42Z

We are managing index static template deployment in automated manner, but still have to do manual operations. Looking forward for the proper implementation.

jphilaire · 2024-10-01T15:03:38Z

Are there any API's available to do the refresh programatically? I tried to send a PUT request on the saved objects (_dashboards/api/saved_objects/index-pattern/index-pattern-id) with the below parameters , but that doesnt work 👎 payload = {"attributes":{"title": "index-*", "refresh_fields": 'true'}} headers = {"Content-Type": "application/json", "osd-xsrf": "true", "security_tenant": "global" }

It could be nice to have an API like that. So people could automate or not the refresh of index patterns.
From my point of view, I would like to refresh all index patterns of each tenant one time a day.

yb-yu · 2024-10-02T15:28:23Z

I'm currently running a script to refresh the index pattern every 10 minutes.

This script mimics the same steps I'd take manually in the OpenSearch Dashboard to refresh the fields.

It fetches the current index pattern and retrieves all its fields using the GET index pattern API.

Instead of using Kibana API's refresh parameter, the OpenSearch Dashboard refresh process includes all existing fields from the index pattern, and the script follows that approach.

I’m not sure why it’s implemented this way, but I found that by performing the same steps, the index pattern gets refreshed...

Ideally, it would be great if it could be done with just one API call to refresh the fields, but for now, this script works as a workaround:

from urllib.parse import urljoin
import json

import boto3
import requests
from requests_aws4auth import AWS4Auth


DOMAIN = "https://your.opensearch.domain"
INDEX_PATTERN = "your-index-pattern-*"
TENANT = "your-tenant"
REGION = "us-east-1"

FIND_SAVED_OBJECTS_URI = urljoin(DOMAIN, "/_dashboards/api/saved_objects/_find")
INDEX_PATTERN_URI = urljoin(DOMAIN, "/_dashboards/api/saved_objects/index-pattern")
INDEX_PATTERN_FIELDS_URI = urljoin(DOMAIN, "/_dashboards/api/index_patterns/_fields_for_wildcard")

cred = boto3.Session().get_credentials()
auth = AWS4Auth(cred.access_key, cred.secret_key, REGION, "es", session_token=cred.token)
headers = {"Content-Type": "application/json", "osd-xsrf": "true", "securitytenant": TENANT}

# Step 1: Find the index pattern and get id & version
params = {"type": "index-pattern", "fields": "id", "search": INDEX_PATTERN}
index_pattern_info = requests.get(FIND_SAVED_OBJECTS_URI, auth=auth, headers=headers, params=params).json()

index_id = index_pattern_info["saved_objects"][0]["id"]
index_pattern_version = index_pattern_info["saved_objects"][0]["version"]

# Step 2: Get the index pattern fields
params = {"pattern": INDEX_PATTERN, "meta_fields": ["_source", "_id", "_type", "_index", "_score"]}
index_pattern_fields = requests.get(INDEX_PATTERN_FIELDS_URI, auth=auth, headers=headers, params=params).json()["fields"]

# Step 3: Update the index pattern
put_index_pattern_info = {
    "attributes": {
        "title": INDEX_PATTERN,
        "fields": json.dumps(index_pattern_fields),
        "version": index_pattern_version,
    }
}
put_index_pattern = requests.put(
    INDEX_PATTERN_URI + "/" + index_id, auth=auth, headers=headers, json=put_index_pattern_info
)

jphilaire · 2024-10-03T08:17:25Z

I'm currently running a script to refresh the index pattern every 10 minutes.

This script mimics the same steps I'd take manually in the OpenSearch Dashboard to refresh the fields.

It fetches the current index pattern and retrieves all its fields using the GET index pattern API.

Instead of using Kibana API's refresh parameter, the OpenSearch Dashboard refresh process includes all existing fields from the index pattern, and the script follows that approach.

I’m not sure why it’s implemented this way, but I found that by performing the same steps, the index pattern gets refreshed...

Ideally, it would be great if it could be done with just one API call to refresh the fields, but for now, this script works as a workaround:
from urllib.parse import urljoin
import json

import boto3
import requests
from requests_aws4auth import AWS4Auth
.
.
.

Well done ! It works fine !
Thank you @yb-yu for sharing !

niklasweimann · 2024-10-28T12:42:40Z

Any progress on this issue?

Its-Ankush · 2024-11-04T14:46:36Z

I'm currently running a script to refresh the index pattern every 10 minutes.

This script mimics the same steps I'd take manually in the OpenSearch Dashboard to refresh the fields.

It fetches the current index pattern and retrieves all its fields using the GET index pattern API.

Instead of using Kibana API's refresh parameter, the OpenSearch Dashboard refresh process includes all existing fields from the index pattern, and the script follows that approach.

I’m not sure why it’s implemented this way, but I found that by performing the same steps, the index pattern gets refreshed...

Ideally, it would be great if it could be done with just one API call to refresh the fields, but for now, this script works as a workaround:
from urllib.parse import urljoin
import json

import boto3
import requests
from requests_aws4auth import AWS4Auth
......

I believe it doesn't work now.

# Step 3: Update the index pattern
put_index_pattern_info = {
    "attributes": {
        "title": INDEX_PATTERN,
        "fields": json.dumps(index_pattern_fields),
        "version": index_pattern_version,
    }
}

I took a look at the original network request from chrome and it seems that it refreshing the index pattern needs a payload which looks like this -

{
    "attributes": {
        "title": "index_pattern_name",
        "timeFieldName": "date_field",
        "fields": "[{\"count\":0,\"name\":\"_id\",\"type\":\"string\",\"esTypes\":[\"_id\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":false}......]"
    },
    "version": "version_from_step_1"
}

Fields from Step 2 look like this

{
  'name': 'additional_field',
  'type': 'string',
  'esTypes': ['text'],
  'searchable': True,
  'aggregatable': False,
  'readFromDocValues': False
}

While it actually looks for

{
    "count": 0,
    "name": "additional_field",
    "type": "string",
    "esTypes": ["text"],
    "scripted": false,
    "searchable": true,
    "aggregatable": false,
    "readFromDocValues": false
}

Im getting blank screen when i click on my index pattern. Also this may not work for clusters with SAML/cognito enabled [I still need to check and confirm that]

@yb-yu is it working for you ?

Edit - this is what I see on Console

yb-yu · 2024-11-11T04:19:28Z

@Its-Ankush Hi,

The code is running smoothly on my end.

The actual response from step 2 looks like following in my case.

[
  {'name': '_id', 'type': 'string', 'esTypes': ['_id'], 'searchable': True, 'aggregatable': True, 'readFromDocValues': False}, 
  {'name': '_index', 'type': 'string', 'esTypes': ['_index'], 'searchable': True, 'aggregatable': True, 'readFromDocValues': False}, 
  {'name': '_score', 'type': 'number', 'searchable': False, 'aggregatable': False, 'readFromDocValues': False},
  ... # other custom fields
]

I have been using this code on aws managed opensearch since version 2.11 (now it's 2.15), and SAML is also enabled through the Microsoft Enterprise Application.

teebu added the enhancement New feature or request label Nov 5, 2021

mikespharss mentioned this issue Aug 1, 2022

Dashboards: Index Patterns field list requires manual refresh opensearch-project/OpenSearch#4046

Closed

joshuarrrr assigned ashwin-pc Sep 6, 2022

joshuarrrr added the needs research label Sep 6, 2022

ashwin-pc removed the needs research label Sep 7, 2022

ashwin-pc added the good first issue Good for newcomers label Mar 28, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Index Patterns Refresh field list automatically update for new fields #913

Index Patterns Refresh field list automatically update for new fields #913

teebu commented Nov 5, 2021

JacobBrandt commented Nov 5, 2021

florianakos commented Sep 6, 2022

joshuarrrr commented Sep 6, 2022

ashwin-pc commented Sep 7, 2022

KannappanSomu commented Feb 1, 2023

Aigerim-ai commented Apr 8, 2023

joshuarrrr commented Apr 10, 2023

djablonski-moia commented Sep 19, 2023

ashishvaishno commented Feb 9, 2024 •

edited

Loading

yb-yu commented Mar 12, 2024 •

edited

Loading

cameronattard commented Mar 12, 2024

vshunkov commented Jul 2, 2024

jphilaire commented Oct 1, 2024

yb-yu commented Oct 2, 2024 •

edited

Loading

jphilaire commented Oct 3, 2024 •

edited

Loading

niklasweimann commented Oct 28, 2024

Its-Ankush commented Nov 4, 2024 •

edited

Loading

yb-yu commented Nov 11, 2024

Index Patterns Refresh field list automatically update for new fields #913

Index Patterns Refresh field list automatically update for new fields #913

Comments

teebu commented Nov 5, 2021

JacobBrandt commented Nov 5, 2021

florianakos commented Sep 6, 2022

joshuarrrr commented Sep 6, 2022

ashwin-pc commented Sep 7, 2022

KannappanSomu commented Feb 1, 2023

Aigerim-ai commented Apr 8, 2023

joshuarrrr commented Apr 10, 2023

djablonski-moia commented Sep 19, 2023

ashishvaishno commented Feb 9, 2024 • edited Loading

yb-yu commented Mar 12, 2024 • edited Loading

cameronattard commented Mar 12, 2024

vshunkov commented Jul 2, 2024

jphilaire commented Oct 1, 2024

yb-yu commented Oct 2, 2024 • edited Loading

jphilaire commented Oct 3, 2024 • edited Loading

niklasweimann commented Oct 28, 2024

Its-Ankush commented Nov 4, 2024 • edited Loading

yb-yu commented Nov 11, 2024

ashishvaishno commented Feb 9, 2024 •

edited

Loading

yb-yu commented Mar 12, 2024 •

edited

Loading

yb-yu commented Oct 2, 2024 •

edited

Loading

jphilaire commented Oct 3, 2024 •

edited

Loading

Its-Ankush commented Nov 4, 2024 •

edited

Loading