Skip to content

Commit 93710dc

Browse files
abbyhu2000abbyhu2000
committed
Add readme for current plugin data persistence implementation
Added readme in root level doc folder Signed-off-by: abbyhu2000 <abigailhu2000@gmail.com>
1 parent 1cc8213 commit 93710dc

13 files changed

+197
-7
lines changed

‎docs/plugins/data_persistence.md

Lines changed: 197 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -18,17 +18,207 @@ There are two parts for data persistence:
1818
![img](./img/app_query.png)
1919

2020
2. app filters
21-
<img width="513" alt="Screen Shot 2022-11-15 at 1 19 14 AM" src="https://user-images.githubusercontent.com/43937633/201880353-df1bcfeb-9f77-4e1e-b689-e8894f9430e2.png">
21+
![img](./img/app_filter.png)
2222

23-
3. vis
24-
4. ui state
23+
3. vis & ui state
24+
![img](./img/visualization.png)
2525
2. global query state
2626
1. global state storage key: '_g'
2727
2. global query state is persistent across the entire application, values will persist when we refresh the page, or when we navigate across visualize, discover, timeline or dashboard page. For example, if we set time range to last 24 hours, and refresh intervals to every 30 min, the same time range and refresh intervals will be applied if we navigate to any of the other pages.
2828
3. params:
29-
1. filters
29+
1. global filters (Select `pin filter` to make the filters global)
30+
![img](./img/global_filter.png)
3031
2. refresh intervals
31-
<img width="465" alt="Screen Shot 2022-11-15 at 1 22 48 AM" src="https://user-images.githubusercontent.com/43937633/201881096-2f3422f5-264b-42ff-9af1-c6f7950720be.png">
32-
32+
![img](./img/refresh_interval.png)
3333
3. time range
34-
<img width="465" alt="Screen Shot 2022-11-15 at 1 22 13 AM" src="https://user-images.githubusercontent.com/43937633/201881002-033265f9-33fe-47f5-9329-b8d8c23e6428.png">
34+
![img](./img/time_range.png)
35+
36+
# URL breakdown & example
37+
38+
![img](./img/URL_example.png)
39+
# Global state persistence
40+
41+
1. In plugin.ts, during plugin setup, call `createOsdUrlTracker()`, listen to history changes and global state changes, then update the nav link URL. This also returns function such as `onMountApp()`, `onUnmountedApp()`
42+
```ts
43+
const {
44+
appMounted,
45+
appUnMounted,
46+
...
47+
} = createOsdUrlTracker({
48+
baseUrl: core.http.basePath.prepend('/app/visualize'),
49+
defaultSubUrl: '#/',
50+
storageKey: `lastUrl:${core.http.basePath.get()}:visualize`,
51+
navLinkUpdater$: this.appStateUpdater,
52+
stateParams: [
53+
{
54+
osdUrlKey: '_g',
55+
stateUpdate$: data.query.state$.pipe(
56+
filter(
57+
({ changes }) => !!(changes.globalFilters || changes.time || changes.refreshInterval)
58+
),
59+
map(({ state }) => ({
60+
...state,
61+
filters: state.filters?.filter(opensearchFilters.isFilterPinned),
62+
}))
63+
),
64+
},
65+
],
66+
....
67+
```
68+
69+
* when we enter the app and app is mounted, it initialize nav link by getting previously stored URL from storage instance: const storedUrl = storageInstance.getItem(storageKey). (Storage instance is a browser wide session storage instance.) Then it unsubscribes to global state$ and subscribes to URL$. The current app actively listens to history location changes. If there are changes, set the updated URL as the active URL
70+
71+
```ts
72+
function onMountApp() {
73+
unsubscribe();
74+
...
75+
// track current hash when within app
76+
unsubscribeURLHistory = historyInstance.listen((location) => {
77+
...
78+
setActiveUrl(location.hash.substr(1));
79+
}
80+
});
81+
}
82+
```
83+
84+
* when we are leaving the app and app is unmounted, unsubscribe URL$ and subscribe to global state$. If the global states are changed in another app, the global state listener will still get triggered in this app even though it is unmounted, it will set the updated URL in storage instance, so next time when we enter the app, it gets the URL from the storage instance thus the global state will persist.
85+
86+
```ts
87+
function onUnmountApp() {
88+
unsubscribe();
89+
// propagate state updates when in other apps
90+
unsubscribeGlobalState = stateParams.map(({ stateUpdate$, osdUrlKey }) =>
91+
stateUpdate$.subscribe((state) => {
92+
...
93+
const updatedUrl = setStateToOsdUrl( ... );
94+
...
95+
storageInstance.setItem(storageKey, activeUrl);
96+
})
97+
);
98+
}
99+
```
100+
101+
2. In app.tsx, call syncQueryStateWithUrl(query, osdUrlStateStorage) to sync '_g' portion of url with global state params
102+
* when we first enter the app, there is no initial state in the URL, then we initialize and put the _g key into url
103+
104+
```ts
105+
if (!initialStateFromUrl) {
106+
osdUrlStateStorage.set<QueryState>(GLOBAL_STATE_STORAGE_KEY, initialState, {
107+
replace: true,
108+
});
109+
}
110+
```
111+
112+
* when we enter the app, if there is some initial state in the URL(the previous saved URL in storageInstance), then we retrieve global state from '_g' URL
113+
114+
```ts
115+
// retrieve current state from `_g` url
116+
const initialStateFromUrl = osdUrlStateStorage.get<QueryState>(GLOBAL_STATE_STORAGE_KEY);
117+
// remember whether there was info in the URL
118+
const hasInheritedQueryFromUrl = Boolean(
119+
initialStateFromUrl && Object.keys(initialStateFromUrl).length
120+
);
121+
// prepare initial state, whatever was in URL takes precedences over current state in services
122+
const initialState: QueryState = {
123+
...defaultState,
124+
...initialStateFromUrl,
125+
};
126+
```
127+
128+
* if we make some changes to the global state: 1. stateUpdate$ get triggered for all other unmounted app(if we made the change in visualize plugin, then the stateUpdate$ will get triggered for dashboard, discover, timeline), then it will call setStateToOsdUrl() to set updatedURL in storageInstance so global state get updated for all unmounted app. 2. updateStorage() get triggered for currentApp to update current URL state storage, then global query state container will also be in sync with URL state storage
129+
130+
```ts
131+
const { start, stop: stopSyncingWithUrl } = syncState({
132+
stateStorage: osdUrlStateStorage,
133+
stateContainer: {
134+
...globalQueryStateContainer,
135+
set: (state) => {
136+
if (state) {
137+
// syncState utils requires to handle incoming "null" value
138+
globalQueryStateContainer.set(state);
139+
}
140+
},
141+
},
142+
storageKey: GLOBAL_STATE_STORAGE_KEY,
143+
});
144+
start();
145+
```
146+
147+
# App state persistence
148+
149+
1. we use `useVisualizeAppState()` hook to instantiate the visualize app state container, which is in sync with '_a' URL
150+
151+
```ts
152+
const { stateContainer, stopStateSync } = createVisualizeAppState({
153+
stateDefaults,
154+
osdUrlStateStorage: services.osdUrlStateStorage,
155+
byValue,
156+
});
157+
```
158+
2. when we first enter the app, there is no app state in the URL, so we set the default states into URL in `createDefaultVisualizeAppState()`: `osdUrlStateStorage.set(STATE_STORAGE_KEY, initialState, { replace: true });`
159+
160+
3. when we make changes to the app state, the `dirtyStateChange` event emitter will get triggered, then osd state container will call `updateStorage()` to update the URL state storage, then state container(appState) will also be in sync with URL state storage
161+
162+
```ts
163+
const onDirtyStateChange = ({ isDirty }: { isDirty: boolean }) => {
164+
if (!isDirty) {
165+
// it is important to update vis state with fresh data
166+
stateContainer.transitions.updateVisState(visStateToEditorState(instance, services).vis);
167+
}
168+
setHasUnappliedChanges(isDirty);
169+
};
170+
eventEmitter.on('dirtyStateChange', onDirtyStateChange);
171+
...
172+
const { start, stop: stopSyncingWithUrl } = syncState({
173+
stateStorage: osdUrlStateStorage,
174+
stateContainer: {
175+
...globalQueryStateContainer,
176+
set: (state) => {
177+
if (state) {
178+
globalQueryStateContainer.set(state);
179+
}
180+
},
181+
},
182+
storageKey: GLOBAL_STATE_STORAGE_KEY,
183+
});
184+
// start syncing the appState with the ('_a') url
185+
startStateSync();
186+
```
187+
188+
4. in `useEditorUpdates()`, we use the saved appState to load the visualize editor
189+
190+
# Refresh
191+
When we refresh the page, both app state and global state should persist:
192+
193+
1. `appMounted()` gets triggered for the current app, so current app subscribe to URL$
194+
2. `syncQueryStateWithUrl()` gets called within app.tsx for the current app, and we are getting the global states from URL '_g', and then `connectToQueryState()` gets called to sync global states and state container for the current app so the current app load the saved global states in top nav
195+
3. `stateUpdate$` will get triggered for every other unmounted app, so the global states are updated for their URL in storage instance as well by calling `setStateOsdUrl()`
196+
4. when we load the visualize editor, `createDefaultVisualizeAppState()` gets called, and it gets app state from URL '_a', and it updates appState based on URL
197+
5. in `useEditorUpdates()`, it uses the updated appState to load the visualization with previous saved states
198+
# Navigate to another app
199+
200+
When we navigate to another app from the current app, global state should persist:
201+
202+
1. `appUnmounted()` triggered for the current app, unsubscribe to `URLHistory$`, and subscribe to stateUpdate$
203+
2. `appMounted()` triggered for the app that we navigated to, so it unsubscribe its `stateUpdate$`, and subscribe to `URLHistory$`
204+
3. `syncQueryStateWithUrl` is triggered, it then gets the saved global state from the `osdurlstatestorage` and set the top nav global states by using `globalQueryStateContainer`
205+
206+
# Diagrams
207+
208+
1. When first navigate to the visualize app, initialize and sync state storage and state containers
209+
![img](./img/initialization.png)
210+
211+
2. When we navigate to another app, the browser wide storage instance stores the last active URL for each app and also updates the URL if there are any new global state values. This ensure global data persistence across different apps. For example, if we navigate from visualize app to discover app:
212+
![img](./img/navigate.png)
213+
214+
Here is an example of the storage instance:
215+
![img](./img/storage_instance.png)
216+
217+
3. When we made some changes to the global params, the global query state container will receive updates and push the updates to osd url state storage. Other unmounted app will update their last active URL in instance storage as well.
218+
![img](./img/global_persistence.png)
219+
220+
4. When we made some changes to the app params, the app state container will receive updates and also push the updates to osd url state storage.
221+
![img](./img/app_persistence.png)
222+
223+
5. When we refresh the page, we parse the information from the URL(_g for global state, _a for app state). We use the saved information to create new state containers and set up synchronization between state containers and state storage.
224+
![img](./img/refresh.png)

‎docs/plugins/img/URL_example.png

123 KB
Loading

‎docs/plugins/img/app_filter.png

23.2 KB
Loading

‎docs/plugins/img/app_persistence.png

251 KB
Loading

‎docs/plugins/img/global_filter.png

101 KB
Loading

‎docs/plugins/img/global_persistence.png

396 KB
Loading

‎docs/plugins/img/initialization.png

429 KB
Loading

‎docs/plugins/img/navigate.png

301 KB
Loading

‎docs/plugins/img/refresh.png

227 KB
Loading

‎docs/plugins/img/refresh_interval.png

24.2 KB
Loading

0 commit comments

Comments
 (0)