[FEATURE] Improve discoverability of integration data schema

[Swiddis]

Is your feature request related to a problem?

Integrations (particularly Flint ones) have a schema that's defined by an implicit create table query. The details of this schema are discoverable by viewing the source, but details aren't present or linked anywhere on the UI. There's the schema fields view (screenshot) that shows what the fields are after ingestion, but not what the expected input is. This makes it harder than necessary to get integrations set up, there's often confusion if a user tries to create an integration that fails due to an incompatible data format.

What solution would you like?

The bare minimum would be to add a detail saying what the supported data type is for each integration (parquet/json/csv), possibly linking to application docs. e.g. VPC is set up for parquet based on the documented Flow Logs format. This field could fit in the existing details table.

Implementation for this is still in the air -- adding it as a metadata field directly in the integration may have compatibility concerns if we later make it dynamic. Adding more links to the metadata would have maintenance overhead, we already have issues with just the one link.

What alternatives have you considered?

• Better than just the log encoding would be to link to the table creation somewhere so the user can view it directly, but setting up correct linking might be nontrivial.
• Better still would be to somehow render the fields in a pretty view directly in the UI like what we have for the schema fields view, but this would take some work.
• Balancing the two above, could we have a pop-up modal that displays the table creation code?

Do you have any additional context?

A related issue is that of adding dynamic substitution of the data type (e.g. you may be able to cleanly replace json with parquet with csv in some circumstances). This is out of scope for this particular issue.