Enable live tailing #633

nojaf · 2025-05-08T15:15:34Z

What this PR does / why we need it:

I'm trying to add live tailing / streaming.

Which issue(s) this PR fixes:

Fixes #299

Special notes for your reviewer:

…frames

nojaf · 2025-05-12T09:17:44Z

Hello @idastambuk,

This is working for me locally. New logs are found via a polling mechanism. However, there is a difference in the UI: my tailed logs don’t display any colors or details, and I can't see any difference in the responses between a regular call and a WebSocket message.

To summarize, I would really appreciate a review from the team to help improve this.

Many thanks!

idastambuk

Hi @nojaf, here are some initial comments, will take a look at the rest and the data frame bug separately
Thanks for submitting this!

src/opensearchDatasource.ts

idastambuk · 2025-06-11T16:18:23Z

pkg/opensearch/opensearch.go

 // The QueryDataResponse contains a map of RefID to the response for each query, and each response
 // contains Frames ([]*Frame).
 func (ds *OpenSearchDatasource) QueryData(ctx context.Context, req *backend.QueryDataRequest) (*backend.QueryDataResponse, error) {
+	ds.logger.Info("QueryData called", "numQueries", len(req.Queries))


Could we do a review of all the lines where we log stuff, both FE and BE? They're good for debugging, but I think it's better to be conservative since they might be a burden on our cloud logs

I'm happy to remove or update the levels afterward. The main reason for adding them was to gauge how far along we are in the pipeline.

src/opensearchDatasource.ts

pkg/opensearch/resource_handlers.go

pkg/opensearch/lucene_handler.go

idastambuk · 2025-06-13T12:36:02Z

pkg/opensearch/stream_handler.go

+			queryExecutor := newQueryRequest(osClient, []backend.DataQuery{backendQuery}, req.PluginContext.DataSourceInstanceSettings)
+			queryDataResponse, err := queryExecutor.execute(ctx)
+
+			if err != nil {


Query response returned contains an .Error field instead of returning an error, so this error wouldn't get logged or passed down to the user

Would that be just passing things down as JSON?

json, err := respForRefId.MarshalJSON() if err != nil { o.logger.Error("RunStream: failed to marshal query response to JSON", "refId", refId, "error", err) return err } err = sender.SendJSON(json)

This looks good now. I think if we add the queryData() call, it will mostly take care of the error being added to the response already too

idastambuk · 2025-06-13T13:35:23Z

Hello @idastambuk,

This is working for me locally. New logs are found via a polling mechanism. However, there is a difference in the UI: my tailed logs don’t display any colors or details, and I can't see any difference in the responses between a regular call and a WebSocket message.

To summarize, I would really appreciate a review from the team to help improve this.

Many thanks!

I'm not sure how to fix this, I tried it in Loki and I see the same issue not being able to expand results:

src/opensearchDatasource.ts

nojaf · 2025-07-29T15:13:06Z

Hi @idastambuk, I think I made some progress.
I'm seeing more data now:

The UI is still not the same as with non-live queries, though 😔

idastambuk · 2025-08-29T13:08:07Z

pkg/opensearch/stream_handler.go

+
+			o.logger.Info("RunStream: Polling OpenSearch", "refId", refId, "from", backendQuery.TimeRange.From, "to", backendQuery.TimeRange.To, "duration", backendQuery.TimeRange.To.Sub(backendQuery.TimeRange.From))
+
+			osClient, err := client.NewClient(ctx, req.PluginContext.DataSourceInstanceSettings, o.httpClient, &backendQuery.TimeRange)


I think we we should be able to just run

queryDataResponse, err := o.QueryData(ctx, &backend.QueryDataRequest{ Queries: []backend.DataQuery{backendQuery}, PluginContext: req.PluginContext, Headers: req.Headers, })

instead of this and the following lines, since we have all the data to construct the data query request.

idastambuk · 2025-08-29T13:14:46Z

pkg/opensearch/stream_handler.go

+
+	lastTo := time.Now().Add(-1 * time.Second) // Start slightly in the past to catch any timing issues
+
+	for {


Could we extract OpenSearch polling logic (request and time range building & response processing) to a separate method on o *OpenSearchDatasource? It could return a channel, which we would just need to subscribe to in RunStream handler. It would probably be easier to add a test for opensearch streaming function it in that case then (which would be great to have!).

idastambuk · 2025-08-29T13:26:32Z

src/opensearchDatasource.ts

+    return `${query.datasource?.uid}/${hashArray.map((b) => b.toString(16).padStart(2, '0')).join('')}/${config.bootData.user.orgId}`;
+  }
+
+  runLiveQueryThroughBackend(request: DataQueryRequest<OpenSearchQuery>): Observable<DataQueryResponse> {


We could also probably extract this function to a separate file to keep the datasource class cleaner

idastambuk · 2025-08-29T13:49:43Z

Hello @idastambuk,

This is working for me locally. New logs are found via a polling mechanism. However, there is a difference in the UI: my tailed logs don’t display any colors or details, and I can't see any difference in the responses between a regular call and a WebSocket message.

To summarize, I would really appreciate a review from the team to help improve this.

Many thanks!

So I think this is just the feature and not a bug, it renders a basic table: LiveLogs.tsx#L146
and only when it stops streaming does it actually display the logs visualization with all the extra info: LogsContainer.tsx#L329 Not sure why it was implemented like that, I asked the team that did it if it could be improved

Edit: It's a pretty old component so it is possible that it was built before the Logs visualization got the improvements. However, we're wondering if there's really a use case to have the Logs visualization while streaming, maybe the simple table is enough data. However, if you have a strong use case to display all labels along with the message during streaming, please open a feature request in the grafana repo and the Logs squad will evaluate.

idastambuk · 2025-09-08T13:50:12Z

pkg/opensearch/stream_handler.go

+		}, fmt.Errorf("failed to get datasource info: %w", err)
+	}
+
+	o.logger.Info("SubscribeStream: path validated, authorizing stream", "path", req.Path)


We should add valiadation for OrgId (like here) too, to ensure that the requests are coming from the correct instance:

nojaf added 5 commits May 12, 2025 11:10

WIP, initial frontend changes

b58bfab

Add resolution for rxjs

0ecba3e

Initial StreamHandler skeleton

b87596a

Connecting live tail webSocket

4c301e3

Ensure all data is present in frame from websocket. Don't send empty …

a47271f

…frames

nojaf force-pushed the feat/live-tailing branch from 98468f9 to a47271f Compare May 12, 2025 09:13

nojaf marked this pull request as ready for review May 12, 2025 09:14

nojaf requested a review from a team as a code owner May 12, 2025 09:14

nojaf requested review from idastambuk and kevinwcyu and removed request for a team May 12, 2025 09:14

idastambuk added this to AWS Datasources May 13, 2025

github-project-automation bot moved this to Incoming in AWS Datasources May 13, 2025

idastambuk moved this from Incoming to Next in AWS Datasources May 13, 2025

idastambuk reviewed Jun 11, 2025

View reviewed changes

idastambuk moved this from Next to In Progress in AWS Datasources Jun 12, 2025

idastambuk self-assigned this Jun 12, 2025

idastambuk reviewed Jun 12, 2025

View reviewed changes

src/opensearchDatasource.ts Outdated Show resolved Hide resolved

pkg/opensearch/resource_handlers.go Outdated Show resolved Hide resolved

pkg/opensearch/lucene_handler.go Show resolved Hide resolved

idastambuk reviewed Jun 13, 2025

View reviewed changes

idastambuk moved this from In Progress to Waiting in AWS Datasources Jun 16, 2025

idastambuk reviewed Jun 20, 2025

View reviewed changes

src/opensearchDatasource.ts Outdated Show resolved Hide resolved

nojaf added 7 commits June 20, 2025 17:03

Apply code review feedback

9f65e59

Remove custom registration for live tailing, we don't need it.

a84ef8d

Revert lucene_handler

754af5e

Merge branch 'main' into feat/live-tailing

e55df52

Try and send error to the frontend

8f1a8b9

Merge branch 'main' into feat/live-tailing

edbfda3

Use getLiveStreamKey similar to Loki

699cec9

Try liveService.getStream

f22266e

idastambuk moved this from Waiting to In Progress in AWS Datasources Aug 4, 2025

idastambuk moved this from In Progress to Waiting in AWS Datasources Aug 20, 2025

idastambuk reviewed Aug 29, 2025

View reviewed changes

idastambuk reviewed Sep 8, 2025

View reviewed changes

Merge branch 'main' into feat/live-tailing

e951e90


		o.logger.Info("RunStream: Polling OpenSearch", "refId", refId, "from", backendQuery.TimeRange.From, "to", backendQuery.TimeRange.To, "duration", backendQuery.TimeRange.To.Sub(backendQuery.TimeRange.From))

		osClient, err := client.NewClient(ctx, req.PluginContext.DataSourceInstanceSettings, o.httpClient, &backendQuery.TimeRange)


		lastTo := time.Now().Add(-1 * time.Second) // Start slightly in the past to catch any timing issues

		for {

Enable live tailing #633

Are you sure you want to change the base?

Enable live tailing #633

Uh oh!

Conversation

nojaf commented May 8, 2025

Uh oh!

nojaf commented May 12, 2025

Uh oh!

idastambuk left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

idastambuk commented Jun 13, 2025

Uh oh!

Uh oh!

nojaf commented Jul 29, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

idastambuk commented Aug 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

idastambuk commented Aug 29, 2025 •

edited

Loading