[Logs UI] Replace current timepicker with EUI SuperDatePicker

[katrin-freihofner]

Summary

This lays out the steps to replace the current point-in-time picker in the Logs UI with EUI's SuperDatePicker as discussed in #37217.

Task breakdown

• Record current performance (@afgomez)
• Refactor the log entry data fetching to hooks (@Zacqary)
• Refactor query bar state to hooks (@Zacqary)
• Refactor scroll position state to hooks (in expectation of the outer boundaries being added) (@Zacqary)
  • Investigate off-screen rendering vs static height calculation based on font dimensions and number of line breaks
  • use react-virtualized to render the rows
• Rewrite routes to HTTP+JSON (@afgomez) (Rewrite log stream routes to HTTP+JSON #51047)
  • logEntriesBetween not needed anymore
  • logEntriesAround remains (in expectation of the outer boundaries being added)
  • logSummary remains
  • highlights remains
  • logSummaryHighlights remains
  • logItem remains
• Replace the date picker with a date range picker.
  • use selected date range to make queries more efficient (instead of incremental "probing")
  • add "extend interval" components

Acceptance criteria

• The date picker...
  • is replaced with a date range picker (EUI's <SuperDatePicker>).
  • has the default value (now-1d, now).
  • allows for setting the auto-refresh interval.
• The log stream...
  • shows log lines that lie within the time frame selected in the date range picker.
  • aligns as follows when the time frame changes:
    • If the currently shown interval is within the new time frame, keep showing the same interval.
    • If the currently shown interval is outside of the new time frame, show the top of the new time frame.
  • continues to only load a subset of the log entries in the interval if they don't fit on the screen (on-demand loading and virtual scrolling).
  • contains indicators to extend the time frame, which...
    • are located above the first entry and below the last entry.
    • extend the time frame into their respective direction based on the current time frame size (e.g. if the selected timeframe is <= 1 day, extend by 1 hour. If it is <= 1 week extend by 1 day, etc).
      
  • contains loading indicators above the first and below the last entry to indicate that incremental loading is in progress.
    
  • contains a UI element to enable live streaming at the bottom when the end of the current time frame is now or in the future.
    
• The mini-map...
  • shows the summary of the time frame selected in the date range picker.
  • highlights the time interval currently visible in the log entry stream.
  • reacts to a click by jumping to that location in the log entry stream.
• The live streaming mode...
  • is triggered by the "stream live" button besides the date picker or at the bottom of the stream (when the end is now or in the future).
  • forces the stream to stay scrolled to the end.
  • sets the date picker to (now-1d, now) and auto-refresh to 5s.
  • is canceled when the user changes the start or end date in the date picker, extends the start of the time frame via the top "end of data" indicator or stops the auto-refresh.
  • sets the time frame end to the current timestamp when stopped.

[elasticmachine]

Pinging @elastic/logs-metrics-ui (Team:logs-metrics-ui)

[Zacqary]

Love these changes, it seems like it'll make for a much better UX while also reducing technical complexity of the rendering code. Few questions:

The live streaming mode...

• forces the stream to stay scrolled to the end.
  ...
• is canceled when the user changes the start or end date in the date picker or stops the auto-refresh.

Can the user still scroll up to cancel live streaming, or do they have to manually click the Stop button first before unlocking the scroll?

The log stream...

• aligns as follows when the time frame changes:
  • If the currently shown interval is within the new time frame, keep showing the same interval.
  • If the currently shown interval is outside of the new time frame, show the top of the new time frame.

Does this behavior also work for clicking and dragging the log minimap? It seems like it would make sense for the minimap-drag action to change the timeframe now rather than executing a jumpToTarget, but since it also has the feel of dragging a scrollbar I'm wondering if it should scroll the log stream proportionate to the drag distance?

[weltenwort]

Does this behavior also work for clicking and dragging the log minimap?

I suspect that trying to load the logs entries quickly enough to keep up with a large-scale drag operation on the minimap would be futile. If we limit the maximum drag distance to the boundaries of the minimap the first case should always apply, right?

[weltenwort]

Can the user still scroll up to cancel live streaming, or do they have to manually click the Stop button first before unlocking the scroll?

I could make an argument for either. @katrin-freihofner, any preference?

[katrin-freihofner]

Does this behavior also work for clicking and dragging the log minimap?
I'm not sure if we even need dragging. I'm fine without it.

Can the user still scroll up to cancel live streaming, or do they have to manually click the Stop button first before unlocking the scroll?
I don't think that we should stop the streaming. Our users should be able to scroll up while live streaming and scroll back to the bottom (where the new log lines appear in the meantime).

This is how I imagine it (it's 5pm in my example):

1. live streaming is enabled (this means we scroll to the very bottom, and we set the timestamp to (4pm - now)
2. user scrolls up until 4pm
3. user clicks Extend timeframe by 1 h
4. Scroll position stays. Minimap gets extended to 3pm - 5pm. Timepicker gets updated to 3pm - 5pm. User will be able to scroll further up. Live streaming ended.
   

I'm not sure what we do with the time that passes between all these steps. For example if a user streams for 1 hour, should we set the timeframe in step 5 to 3pm - 6pm? or 4pm - 6pm?

[weltenwort]

That sounds pretty intuitive to me, except for the 6pm at the end. Where does that come from?

[katrin-freihofner]

That sounds pretty intuitive to me, except for the 6pm at the end. Where does that come from?

If a user streams for one hour, it would be 6pm in the example above.

[weltenwort]

Ah, I see. We could either keep the end date as now or we could set it to the timestamp when the live streaming was stopped. The latter might help to make it more obvious that it's no longer continuously updating.

[katrin-freihofner]

The latter might help to make it more obvious that it's no longer continuously updating.

I agree! That sounds good.

[weltenwort]

👍 I've amended the description above

[Zacqary]

To summarize a discussion at EAH:

@afgomez and I are going to pair on the backend/frontend portions of this issue. We can implement this UI in an unoptimized way using the current GraphQL APIs, so that work can happen in parallel with building a more efficient backend.

Right now the backend fetches the 200 next or previous log entries whenever the user scrolls to "load new entries." This is inefficient because we don't have a start and end timerange to look for these 200 logs; we need to exponentially extend the timeframe until 200 of them actually get retrieved. This doesn't give Elasticsearch a lot of room to optimize our queries. Same with jumping to a new timestamp; we just pass a central timestamp, and the algorithm now has to use the same method to collect 200 surrounding log entries to display.

This UI will allow us to just pass a start and end timeframe to Elasticsearch, allowing query performance to improve dramatically.

When this issue is ready for implementation, we should probably convert it to a meta-issue and create frontend and backend sub-issues.