Overhaul Reporting for Data Science

[Theodus]

I have made multiple attempts in the past to clean this up, which all failed to meaningfully improve the situation. The primary goal of this is to track information about queries as it becomes available in the code path, and moving the cumbersome backwards compatibility concerns somewhere else. This approach adds a custom layer to the tracing framework that collects events from special targets to eventually assemble the logs and Kafka messages for the data science team. It's a bit rough now. But I think it's finally a good starting point to facilitating the changing needs for standardization around gateway reporting.

This also adds subscription query Kafka messages.

[cmwhited]

Looks good. I like this layer approach. Super cool.

[cmwhited]

Just want to make sure, in the initial notion, this was an enum. I take it you decided against that?

[Theodus]

It wasn't since I didn't consider using an enum, but that's probably better. Fixed, updated Notion doc & reply below.

[cmwhited]

Can you send me the updated rust output from this .proto so I can include it in the graph-subscriptions-api? The fields look the same, but the order is different.

[Theodus]

#[allow(clippy::derive_partial_eq_without_eq)]
#[derive(Clone, PartialEq, ::prost::Message)]
pub struct GatewaySubscriptionQueryResult {
    /// Set to the value of the CF-Ray header, otherwise a generated UUID
    #[prost(string, tag = "1")]
    pub query_id: ::prost::alloc::string::String,
    #[prost(enumeration = "StatusCode", tag = "2")]
    pub status_code: i32,
    #[prost(string, tag = "3")]
    pub status_message: ::prost::alloc::string::String,
    #[prost(uint32, tag = "4")]
    pub response_time_ms: u32,
    /// `user` field from ticket payload, 0x-prefixed hex
    #[prost(string, tag = "5")]
    pub ticket_user: ::prost::alloc::string::String,
    /// `signer` field from ticket payload, 0x-prefixed hex
    #[prost(string, tag = "6")]
    pub ticket_signer: ::prost::alloc::string::String,
    /// `name` field from ticket payload
    #[prost(string, optional, tag = "7")]
    pub ticket_name: ::core::option::Option<::prost::alloc::string::String>,
    /// Subgraph Deployment ID, CIDv0 ("Qm" hash)
    #[prost(string, optional, tag = "8")]
    pub deployment: ::core::option::Option<::prost::alloc::string::String>,
    /// Chain name indexed by subgraph deployment
    #[prost(string, optional, tag = "9")]
    pub subgraph_chain: ::core::option::Option<::prost::alloc::string::String>,
    #[prost(uint32, optional, tag = "10")]
    pub query_count: ::core::option::Option<u32>,
    #[prost(float, optional, tag = "11")]
    pub query_budget: ::core::option::Option<f32>,
    #[prost(float, optional, tag = "12")]
    pub indexer_fees: ::core::option::Option<f32>,
}
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, PartialOrd, Ord, ::prost::Enumeration)]
#[repr(i32)]
pub enum StatusCode {
    Success = 0,
    InternalError = 1,
    UserError = 2,
    NotFound = 3,
}
impl StatusCode {
    /// String value of the enum field names used in the ProtoBuf definition.
    ///
    /// The values are not transformed in any way and thus are considered stable
    /// (if the ProtoBuf definition does not change) and safe for programmatic use.
    pub fn as_str_name(&self) -> &'static str {
        match self {
            StatusCode::Success => "SUCCESS",
            StatusCode::InternalError => "INTERNAL_ERROR",
            StatusCode::UserError => "USER_ERROR",
            StatusCode::NotFound => "NOT_FOUND",
        }
    }
    /// Creates an enum from field names used in the ProtoBuf definition.
    pub fn from_str_name(value: &str) -> ::core::option::Option<Self> {
        match value {
            "SUCCESS" => Some(Self::Success),
            "INTERNAL_ERROR" => Some(Self::InternalError),
            "USER_ERROR" => Some(Self::UserError),
            "NOT_FOUND" => Some(Self::NotFound),
            _ => None,
        }
    }
}