[RFC]Getting Started Template catalog - schema suggestion

[YANG-DB]

Is your feature request related to a problem?

Today the Integrations catalog are becoming more popular due to its simple installation and opinionated schema governed by the Otel protocol.
Each integration has a TryMe button that has a small dataset which functions as a mock data-points for the dashboard to show some minimal visual appearance.

The need for a comprehensive documentation and instruction (including scripts or live docker image) of how to actually add the agent on top of the observed service, how to map the resulting logs into the Otel (simple schema) format and the actual setup of the agent that will be responsible of the data shipping is not covered and very much missing.

What solution would you like?
This is a proposal for the content of such getting-started template and format that will lead to a catalog of getting started component that are accompanied with the following :

• Documentation of the agent
• Instruction for setting up the agent
• Documentation service's log format
• Instruction for transforming the log's format into Otel designated format
• Documentation of the agent's shipping method and recommendations
• Instruction for setting up the agent's logs delivery

Each such component will have the following relationships:

• Associated Integrations
• Associated index name / index mapping (related to the signal type)
• Associated Docker-compose (live sample) that showcase all the instructions and can be run and ingest data using some build in mock that activate the monitored service

Do you have any additional context?

Getting Started Schema Specification

Getting-Started within the Integration Json Specification

• getting-started: An object containing the workflows for getting started.

Properties

getting-started

• Type: Object
• Description: Contains the workflows for getting started.

Metadata-Properties

• "ingestion": list of ingestion options for this asset (example fluent-bit)
• "structured": is this telemetry signal structured (bool)
• "technology": a list of related technologies associated with this asset
• "protocol": a list of protocols the signal comforms with (in case this is structured)
• "live-sample": has this getting-stared accompanied with a live (docker) sample

Properties

• workflows: An array of workflow objects.

workflows

• Type: Array
• Description: A list of workflow objects.
• Items: Objects representing individual workflows.

Workflow Object

• name: The name of the workflow.
  • Type: String
  • Example: "fluent-bit"
• description: A description of the workflow.
  • Type: String
  • Example: "This is a fluent-bit based getting started instructions tutorial"
• steps: An array of step objects.

steps

• Type: Array
• Description: A list of step objects within a workflow.
• Items: Objects representing individual steps.

Step Object

• name: The name of the step.
  • Type: String
  • Example: "Fluent-Bit Parser"
• type: The type of step.
  • Type: String
  • Example: "console-cmd"
• label: A label for the step.
  • Type: String
  • Example: "Log Parsing"
• info: An array of URLs providing additional information.
  • Type: Array
  • Items: String (URL)
  • Example: ["https://github.com/opensearch-project/opensearch-catalog/tree/main/integrations/observability/nginx/getting-started/fluent-bit/parsers.conf"]
• description: A description of the step.
  • Type: String
  • Example: "Setup Fluent-Bit parser config file parsing Nginx access log fields"
• content: The content of the step, such as commands or configurations.
  • Type: String
  • Example: "[PARSER]\n Name apache\n Format regex\n ..."

[ps48]

Can we add the field for getting_started in the existing integrations config type: https://github.com/opensearch-project/dashboards-observability/blob/b328720c11811d45da9d0594c5c28a18dd8fa2d0/server/adaptors/integrations/types.ts#L12-L29

Something like:

interface IntegrationConfig {
  name: string;
  version: string;
  displayName?: string;
  license: string;
  type: string;
  labels?: string[];
  author?: string;
  description?: string;
  sourceUrl?: string;
  workflows?: IntegrationWorkflow[];
  statics?: IntegrationStatics;
  components: IntegrationComponent[];
  assets: IntegrationAsset[];
  sampleData?: {
    path: string;
  };
gettingStarted?: GettingStartedType;
}

interface GettingStartedType {

// contains all the delta we add 
// workflow
// statics
// relatedAssets

}

Also Let's add type to the workflow steps; so content-type field can help UI component to tell what to render is it an image, video, code, etc.

[ps48]

@Swiddis Can you please enlighten us how validator behaves when an asset with gettingStarted field is added to older integrations code base?

[Swiddis]

It'd reject it, since additionalProperties: false is set

[TackAdam]

Need create schema section which describes the index template and name/alias to be generated as part of the preparation for ingestion.
The actual schema should come from the integration meta-deta component section. Which refers to the integration schema.