[BUG] [Data Explorer] Modify columns on switch

[abbyhu2000]

Describe the bug
Advanced setting discover:modifyColumnsOnSwitch does not work for the new data explorer.

To Reproduce

Screen.Recording.2023-10-10.at.12.31.03.PM.mov

Expected behavior
When the setting discover:modifyColumnsOnSwitch is turned off, the columns from the previous index pattern should not be automatically cleared.

Screen.Recording.2023-10-10.at.12.33.58.PM.mov

[wbeckler]

What is the purpose of this switch?

[MadaniKK]

May I work on this issue? Ty.

[MadaniKK]

The workflow of how the fields are updated in the left side nav:

in the use effect, when a new indexPattern is selected, all the variables (fields, selected, unpopular, popular) used for rendering the elements in the left sidebar will get updated

OpenSearch-Dashboards/src/plugins/discover/public/application/components/sidebar/discover_sidebar.tsx

Lines 125 to 135 in 7def75c

	const {
	selected: selectedFields,
	popular: popularFields,
	unpopular: unpopularFields,
	} = useMemo(() => groupFields(fields, columns, popularLimit, fieldCounts, fieldFilterState), [
	fields,
	columns,
	popularLimit,
	fieldCounts,
	fieldFilterState,
	]);

we care most about the state of fields and columns(columns refers to the selected fields) because if MODIFY_COLUMN_ON_SWITCH is off, we want them to keep the fields & selected columns from the previously chosen data sample(indexPattern)

The state of fields gets modified here: function'getIndexPatternFieldList generates fields`

OpenSearch-Dashboards/src/plugins/discover/public/application/components/sidebar/discover_sidebar.tsx

Lines 104 to 107 in 7def75c

	useEffect(() => {
	const newFields = getIndexPatternFieldList(selectedIndexPattern, fieldCounts);
	setFields(newFields);
	}, [selectedIndexPattern, fieldCounts, hits, services]);

The state of columns gets modified here everytime the indexPattern is changed to a new one:

OpenSearch-Dashboards/src/plugins/discover/public/application/view_components/canvas/index.tsx

Lines 31 to 38 in 791210d

	const { columns } = useSelector((state) => state.discover);
	const filteredColumns = filterColumns(
	columns,
	indexPattern,
	uiSettings.get(DEFAULT_COLUMNS_SETTING)
	);
	const dispatch = useDispatch();
	const prevIndexPattern = useRef(indexPattern);

OpenSearch-Dashboards/src/plugins/discover/public/application/view_components/canvas/index.tsx

Lines 83 to 89 in 791210d

	useEffect(() => {
	if (indexPattern !== prevIndexPattern.current) {
	dispatch(setColumns({ columns: filteredColumns }));
	prevIndexPattern.current = indexPattern;
	}
	}, [dispatch, filteredColumns, indexPattern]);

Proposed solution

If we want to implement MODIFY_COLUMNS_ON_SWITCH, we just need to add in conditioning in the function'getIndexPatternFieldList generates fieldsand the functionfilterColumns` to keep previous column and fields

[ananzh]

What is the purpose of this switch?

One user story I could think about is that user has some similar index pattern, with most fields same but 1-2 key fields diff. When they nav from one index pattern to another one, they would like to keep all fields even though some of them will be grey. So they could know what index pattern they are using without distinguishing the index pattern name.

[ananzh]

@MadaniKK since we have go through the workflow today, could you help us add a clear summary on

• how index pattern is changed in Discover
• how does discover.state.columns update/refresh when index pattern is change

Thanks~ ❤️

[MadaniKK]

Updated the workflow

[ananzh]

closed it as it is resolved in #5508