Correlations

[jameskerr]

fixes #2443

This is exciting. We've heard from users who want to add custom correlation queries to brim. Before this PR, each correlation (zeek, suricata, md5) was implemented slightly differently and was hard-coded. This unifies the implementation and takes a huge step towards third-party correlations.

A significant change was extending the "Results" state slice. Now it can hold results for any number of queries, not just the main query. You provide an "id" whenever you set or select results.

I also created a simple api.query method on our BrimApi class. It cleans up after itself so that you don't have to think about it.

Finally, I've made a correlations/run() thunk that uses the above two features. Each correlation is given an id, the query is issued, and the results are saved to the Results state slice under that id.

Each correlation React component looks up the results by the relevant id.

The last piece of the puzzle is converting the brimcap-plugin to a new system. It currently relies on a thunk fetchCorrelation that needs to be deleted. We need to decide if we should issue a new query, or wait for the zeek query to find the conn record.

UPDATE:

1. All of the correlations now live in the plugins folder. 🎉
2. The brimcap plugin has been updated to not rely on the fetchCorrelation.

There is now a new api section called correlations.

It works like this:

// myplugin.ts

const md5Correlation: Correlation = {
  id: "zui-zeek/md5-correlation",
  when: hasMd5,
  query: (api) => {
    const md5 = api.current.value.get("md5").toJS()
    return zql`
        from ${api.current.poolName} 
        | md5==${md5} 
        | count() by md5 
        | sort -r 
        | head 5`
  },
}

api.correlations.add(md5Correlation)

Whenever the current value (selected) changes, we call dispatch(runCorrelations()). It will grab all the correlation objects off the api, loop through them, check if they should run with the when property, then run the query returned from the query property.

The results get saved to the results slice under the id correlation property.

Next Steps

To allow third-party javascript to add their own correlations, we only need to do two things:

1. Pick a "plugins folder" for the app to look for this code
2. Update the UI to loop through the configured correlation configurations and create the components automatically based off the configuration. We'll need to add more properties to the correlation configs like, title, caption, type, and so on. At first we'll just have two types: table and timeline. Each one will have its own configuration properties.

[mason-fish]

This is awesome, nice work!! 🔥

[mason-fish]

still want this?

[mason-fish]

🪵

[mason-fish]

So happy this is now handled by this PR :)

[mason-fish]

👏