[Error Handling] Backend - Event Log System

[xoscar]

NOTE: This ticket is partially blocked until the following tasks are completed:

• [Error Handling] Open API Specs #2246
• [Error Handling] Initial Event Type list #2247

This ticket is part of the epic #2040 to improve the UX/UI when running a test

Acceptance Criteria:

AC1
As a client triggering a new test run
I should be able to subscribe for live events from the server side using a WebSocket
And get live information about what is happening with the run

AC2
As a client that triggered a run
And I need to display the full list of events up until that point
I should be able to fetch the list from an endpoint to get the event history

AC3
As a developer working on adding event logs to the test run process
I should be able to use a wrapper subscription which will be listening to events happening at the test run level
And push those events to both the WebSocket and store them in the database

AC4
As a developer working on adding event logs to the test run process
I should be able to add extra information depending on the event (polling, data store connection)

AC5
As a developer working on adding event logs to the test run process
The event system should be extendable to add new event types depending on the requirements and updates

To find more information about the technical design for this task check out the following comment

[danielbdias]

Team, here are backend tasks that we can do to finish this task:

• Create EventEmitter facade that will persist events and send them to websockets #2261
• Implement Event database persistence migrations #2258
• Implement GetTestRunEvents endpoint logic #2260
• Differentiate between a trigger failed state vs a trace failed state in a test run #2285
• Implement Event publisher repository that will do the following operations:
  • Event Publisher/Emitter: Persist the event on the events table  #2262
  • Event Publisher/Emitter: Send the event JSON to subscribed clients #2263
• Add calls to the Event Publisher on:
  • Add calls to the EventEmitter on Trigger execution (CREATED_INFO, RESOLVE_ERROR, RESOLVE_SUCCESS, RESOLVE_START, EXECUTION_START, EXECUTION_SUCCESS events) #2264
  • Add calls to the EventEmitter on Trigger execution (HTTP_UNREACHABLE_HOST_ERROR and GRPC_UNREACHABLE_HOST_ERROR events) #2265
  • Add calls to the Event Publisher on Trigger execution (DOCKER_COMPOSE_HOST_MISMATCH_ERROR event) #2266
  • Add calls to the EventEmitter on Trace fetching (all events, except STOPPED_INFO) #2267
  • Add calls to the EventEmitter on Test spec execution (OUTPUT_GENERATION_WARNING event)  #2278
  • Add calls to the EventEmitter on Test spec execution (RESOLVE events)  #2286
  • Add calls to the EventEmitter on Test spec execution (TEST_SPECS_RUN events) #2268

[mathnogueira]

Build/re-use Websocket subscriber for Test events

I don't think this is the right approach TBH. One of the requirements for the epic is that Tracetest generates telemetry based on those events. This means that every time an event is created, it should have a span with that information too. The implication for that is that we get a new context.Context and have to use it in order to link the spans to their correct parents.

I don't think it's possible to have the subscriber manage those contexts in an easy way. I rather have a facade instead. This is how I imagine it (simplified version)

// facade.go
// ...
func (f facade) Emit(ctx context.Context, event Event) (context.Context, error) {
  span, ctx := f.tracer.Start(ctx, event.Name)
  // populate span with event data
  f.db.InsertEvent(ctx, event)
  f.subscriber.Publish(ctx, event)
}

[mathnogueira]

Should we add events to the otlp ingest endpoint? It's possible to get the testID and runID based on the traceID. @xoscar @jorgeepc @danielbdias @schoren

I can see why to add it, but I'm not sure if it adds any extra visibility when comparable to the polling events

[danielbdias]

At first glance, we could add it to have visibility in debugging, and to understand what happened during a test run.
We can do that on a v2, along with the Trace Stop event