MINOR - Prepare 1.3 docs directories

openmetadata-docs/content/partials/v1.3/connectors/dashboard/configure-ingestion.md

@@ -0,0 +1,42 @@
+{% step srNumber=7 %}
+
+{% stepDescription title="7. Configure Metadata Ingestion" %}
+
+In this step we will configure the metadata ingestion pipeline,
+Please follow the instructions below
+
+{% /stepDescription %}
+
+{% stepVisualInfo %}
+
+{% image
+  src="/images/v1.2/connectors/configure-metadata-ingestion-dashboard.png"
+  alt="Configure Metadata Ingestion"
+  caption="Configure Metadata Ingestion Page" /%}
+
+{% /stepVisualInfo %}
+
+{% /step %}
+
+{% extraContent parentTagName="stepsContainer" %}
+
+#### Metadata Ingestion Options
+
+- **Name**: This field refers to the name of ingestion pipeline, you can customize the name or use the generated name.
+- **Dashboard Filter Pattern (Optional)**: Use it to control whether to include dashboard as part of metadata ingestion.
+  - **Include**: Explicitly include dashboards by adding a list of comma-separated regular expressions to the 'Include' field. OpenMetadata will include all dashboards with names matching one or more of the supplied regular expressions. All other dashboards will be excluded.
+  - **Exclude**: Explicitly exclude dashboards by adding a list of comma-separated regular expressions to the 'Exclude' field. OpenMetadata will exclude all dashboards with names matching one or more of the supplied regular expressions. All other dashboards will be included.
+- **Chart Pattern (Optional)**: Use it to control whether to include charts as part of metadata ingestion.
+  - **Include**: Explicitly include charts by adding a list of comma-separated regular expressions to the 'Include' field. OpenMetadata will include all charts with names matching one or more of the supplied regular expressions. All other charts will be excluded.
+  - **Exclude**: Explicitly exclude charts by adding a list of comma-separated regular expressions to the 'Exclude' field. OpenMetadata will exclude all charts with names matching one or more of the supplied regular expressions. All other charts will be included.
+- **Data Model Pattern (Optional)**: Use it to control whether to include data modes as part of metadata ingestion.
+  - **Include**: Explicitly include data models by adding a list of comma-separated regular expressions to the 'Include' field. OpenMetadata will include all data models with names matching one or more of the supplied regular expressions. All other data models will be excluded.
+  - **Exclude**: Explicitly exclude data models by adding a list of comma-separated regular expressions to the 'Exclude' field. OpenMetadata will exclude all data models with names matching one or more of the supplied regular expressions. All other data models will be included.
+- **Database Service Name (Optional)**: Enter the name of Database Service which is already ingested in OpenMetadata to create lineage between dashboards and database tables.
+- **Enable Debug Log (toggle)**: Set the 'Enable Debug Log' toggle to set the default log level to debug.
+- **Include Owners (toggle)**: Set the 'Include Owners' toggle to control whether to include owners to the ingested entity if the owner email matches with a user stored in the OM server as part of metadata ingestion. If the ingested entity already exists and has an owner, the owner will not be overwritten.
+- **Include Tags (toggle)**: Set the 'Include Tags' toggle to control whether to include tags in metadata ingestion.
+- **Include Data Models (toggle)**: Set the 'Include Data Models' toggle to control whether to include tags as part of metadata ingestion.
+- **Mark Deleted Dashboards (toggle)**: Set the 'Mark Deleted Dashboards' toggle to flag dashboards as soft-deleted if they are not present anymore in the source system.
+
+{% /extraContent %}

openmetadata-docs/content/partials/v1.3/connectors/database/advanced-configuration.md

@@ -0,0 +1,15 @@
+**Advanced Configuration**
+
+Database Services have an Advanced Configuration section, where you can pass extra arguments to the connector
+and, if needed, change the connection Scheme.
+
+This would only be required to handle advanced connectivity scenarios or customizations.
+
+- **Connection Options (Optional)**: Enter the details for any additional connection options that can be sent to Athena during the connection. These details must be added as Key-Value pairs.
+- **Connection Arguments (Optional)**: Enter the details for any additional connection arguments such as security or protocol configs that can be sent during the connection. These details must be added as Key-Value pairs.
+
+{% image
+  src="/images/v1.2/connectors/advanced-configuration.png"
+  alt="Advanced Configuration"
+  height="250px"
+  caption="Advanced Configuration" /%}

openmetadata-docs/content/partials/v1.3/connectors/database/configure-ingestion.md

@@ -0,0 +1,44 @@
+{% step srNumber=7 %}
+
+{% stepDescription title="7. Configure Metadata Ingestion" %}
+
+In this step we will configure the metadata ingestion pipeline,
+Please follow the instructions below
+
+{% /stepDescription %}
+
+{% stepVisualInfo %}
+
+{% image
+  src="/images/v1.2/connectors/configure-metadata-ingestion-database.png"
+  alt="Configure Metadata Ingestion"
+  caption="Configure Metadata Ingestion Page" /%}
+
+{% /stepVisualInfo %}
+
+{% /step %}
+
+{% extraContent parentTagName="stepsContainer" %}
+
+#### Metadata Ingestion Options
+
+- **Name**: This field refers to the name of ingestion pipeline, you can customize the name or use the generated name.
+- **Database Filter Pattern (Optional)**: Use to database filter patterns to control whether or not to include database as part of metadata ingestion.
+    - **Include**: Explicitly include databases by adding a list of comma-separated regular expressions to the Include field. OpenMetadata will include all databases with names matching one or more of the supplied regular expressions. All other databases will be excluded.
+    - **Exclude**: Explicitly exclude databases by adding a list of comma-separated regular expressions to the Exclude field. OpenMetadata will exclude all databases with names matching one or more of the supplied regular expressions. All other databases will be included.
+- **Schema Filter Pattern (Optional)**: Use to schema filter patterns to control whether to include schemas as part of metadata ingestion.
+    - **Include**: Explicitly include schemas by adding a list of comma-separated regular expressions to the Include field. OpenMetadata will include all schemas with names matching one or more of the supplied regular expressions. All other schemas will be excluded.
+    - **Exclude**: Explicitly exclude schemas by adding a list of comma-separated regular expressions to the Exclude field. OpenMetadata will exclude all schemas with names matching one or more of the supplied regular expressions. All other schemas will be included.
+- **Table Filter Pattern (Optional)**: Use to table filter patterns to control whether to include tables as part of metadata ingestion.
+    - **Include**: Explicitly include tables by adding a list of comma-separated regular expressions to the Include field. OpenMetadata will include all tables with names matching one or more of the supplied regular expressions. All other tables will be excluded.
+    - **Exclude**: Explicitly exclude tables by adding a list of comma-separated regular expressions to the Exclude field. OpenMetadata will exclude all tables with names matching one or more of the supplied regular expressions. All other tables will be included.
+- **Include views (toggle)**: Set the Include views toggle to control whether to include views as part of metadata ingestion.
+- **Include tags (toggle)**: Set the 'Include Tags' toggle to control whether to include tags as part of metadata ingestion.
+- **Enable Debug Log (toggle)**: Set the Enable Debug Log toggle to set the default log level to debug.
+
+- **Mark Deleted Tables (toggle)**: Set the Mark Deleted Tables toggle to flag tables as soft-deleted if they are not present anymore in the source system.
+- **Mark Deleted Tables from Filter Only (toggle)**: Set the Mark Deleted Tables from Filter Only toggle to flag tables as soft-deleted if they are not present anymore within the filtered schema or database only. This flag is useful when you have more than one ingestion pipelines. For example if you have a schema
+
+Note that the right-hand side panel in the OpenMetadata UI will also share useful documentation when configuring the ingestion.
+
+{% /extraContent %}

openmetadata-docs/content/partials/v1.3/connectors/database/related.md

@@ -0,0 +1,30 @@
+## Related
+
+{% tilesContainer %}
+
+{% tile
+  title="Usage Workflow"
+  description="Learn more about how to configure the Usage Workflow to ingest Query information from the UI."
+  link="/connectors/ingestion/workflows/usage" /%}
+
+{% tile
+  title="Lineage Workflow"
+  description="Learn more about how to configure the Lineage from the UI."
+  link="/connectors/ingestion/workflows/lineage" /%}
+
+{% tile
+  title="Profiler Workflow"
+  description="Learn more about how to configure the Data Profiler from the UI."
+  link="/connectors/ingestion/workflows/profiler" /%}
+
+{% tile
+  title="Data Quality Workflow"
+  description="Learn more about how to configure the Data Quality tests from the UI."
+  link="/connectors/ingestion/workflows/data-quality" /%}
+
+{% tile
+  title="dbt Integration"
+  description="Learn more about how to ingest dbt models' definitions and their lineage."
+  link="/connectors/ingestion/workflows/dbt" /%}
+
+{% /tilesContainer %}

openmetadata-docs/content/partials/v1.3/connectors/external-ingestion-deployment.md

@@ -0,0 +1,15 @@
+## How to Run the Connector Externally
+
+To run the Ingestion via the UI you'll need to use the OpenMetadata Ingestion Container, which comes shipped with
+custom Airflow plugins to handle the workflow deployment.
+
+If, instead, you want to manage your workflows externally on your preferred orchestrator, you can check
+the following docs to run the Ingestion Framework **anywhere**.
+
+{% tilesContainer %}
+  {% tile
+    title="External Schedulers"
+    description="Get more information about running the Ingestion Framework Externally"
+    link="/deployment/ingestion"
+  / %}
+{% /tilesContainer %}

openmetadata-docs/content/partials/v1.3/connectors/ingestion-modes-tiles.md

@@ -0,0 +1,26 @@
+## Ingestion Deployment
+
+To run the Ingestion via the UI you'll need to use the OpenMetadata Ingestion Container, which comes shipped with
+custom Airflow plugins to handle the workflow deployment. If you want to install it manually in an already existing
+Airflow host, you can follow [this](/deployment/ingestion/openmetadata) guide.
+
+If you don't want to use the OpenMetadata Ingestion container to configure the workflows via the UI, then you can check
+the following docs to run the Ingestion Framework in any orchestrator externally.
+
+{% tilesContainer %}
+{% tile
+    title="Run Connectors from the OpenMetadata UI"
+    description="Learn how to manage your deployment to run connectors from the UI"
+    link="/deployment/ingestion/openmetadata"
+  / %}
+{% tile
+    title="Run the Connector Externally"
+    description="Get the YAML to run the ingestion externally"
+    link=$yamlPath
+  / %}
+{% tile
+    title="External Schedulers"
+    description="Get more information about running the Ingestion Framework Externally"
+    link="/deployment/ingestion"
+  / %}
+{% /tilesContainer %}

openmetadata-docs/content/partials/v1.3/connectors/ingestion-schedule-and-deploy.md

@@ -0,0 +1,49 @@
+{% step srNumber=8 %}
+
+{% stepDescription title="8. Schedule the Ingestion and Deploy" %}
+
+Scheduling can be set up at an hourly, daily, weekly, or manual cadence. The
+timezone is in UTC. Select a Start Date to schedule for ingestion. It is
+optional to add an End Date.
+
+Review your configuration settings. If they match what you intended,
+click Deploy to create the service and schedule metadata ingestion.
+
+If something doesn't look right, click the Back button to return to the
+appropriate step and change the settings as needed.
+
+After configuring the workflow, you can click on Deploy to create the
+pipeline.
+
+{% /stepDescription %}
+
+{% stepVisualInfo %}
+
+{% image
+src="/images/v1.2/connectors/schedule.png"
+alt="Schedule the Workflow"
+caption="Schedule the Ingestion Pipeline and Deploy" /%}
+
+{% /stepVisualInfo %}
+
+{% /step %}
+
+{% step srNumber=9 %}
+
+{% stepDescription title="9. View the Ingestion Pipeline" %}
+
+Once the workflow has been successfully deployed, you can view the
+Ingestion Pipeline running from the Service Page.
+
+{% /stepDescription %}
+
+{% stepVisualInfo %}
+
+{% image
+src="/images/v1.2/connectors/view-ingestion-pipeline.png"
+alt="View Ingestion Pipeline"
+caption="View the Ingestion Pipeline from the Service Page" /%}
+
+{% /stepVisualInfo %}
+
+{% /step %}

openmetadata-docs/content/partials/v1.3/connectors/messaging/configure-ingestion.md

@@ -0,0 +1,33 @@
+{% step srNumber=7 %}
+
+{% stepDescription title="7. Configure Metadata Ingestion" %}
+
+In this step we will configure the metadata ingestion pipeline,
+Please follow the instructions below
+
+{% /stepDescription %}
+
+{% stepVisualInfo %}
+
+{% image
+src="/images/v1.2/connectors/configure-metadata-ingestion-messaging.png"
+alt="Configure Metadata Ingestion"
+caption="Configure Metadata Ingestion Page" /%}
+
+{% /stepVisualInfo %}
+
+{% /step %}
+
+{% extraContent parentTagName="stepsContainer" %}
+
+#### Metadata Ingestion Options
+
+- **Name**: This field refers to the name of ingestion pipeline, you can customize the name or use the generated name.
+- **Topic Filter Pattern (Optional)**: Use it to control whether to include topics as part of metadata ingestion.
+  - **Include**: Explicitly include topics by adding a list of comma-separated regular expressions to the 'Include' field. OpenMetadata will include all topics with names matching one or more of the supplied regular expressions. All other topics will be excluded.
+  - **Exclude**: Explicitly exclude topics by adding a list of comma-separated regular expressions to the 'Exclude' field. OpenMetadata will exclude all topics with names matching one or more of the supplied regular expressions. All other topics will be included.
+- **Ingest Sample Data (toggle)**: Set the 'Ingest Sample Data' toggle to ingest sample data from the topics.
+- **Enable Debug Log (toggle)**: Set the 'Enable Debug Log' toggle to set the default log level to debug.
+- **Mark Deleted Topics (toggle):** Set the 'Mark Deleted Topics' toggle to flag topics as soft-deleted if they are not present anymore in the source system.
+
+{% /extraContent %}

openmetadata-docs/content/partials/v1.3/connectors/metadata-ingestion-ui.md

@@ -0,0 +1,112 @@
+{% stepsContainer %}
+
+{% step srNumber=1 %}
+
+{% stepDescription title="1. Visit the Services Page" %}
+
+The first step is to ingest the metadata from your sources. To do that, you first need to create a Service connection first. 
+
+This Service will be the bridge between OpenMetadata and your source system.
+
+Once a Service is created, it can be used to configure your ingestion workflows.
+
+{% /stepDescription %}
+
+{% stepVisualInfo %}
+
+{% image
+  src="/images/v1.2/connectors/visit-services-page.png"
+  alt="Visit Services Page"
+  caption="Select your Service Type and Add a New Service" /%}
+
+{% /stepVisualInfo %}
+
+{% /step %}
+
+{% step srNumber=2 %}
+
+{% stepDescription title="2. Create a New Service" %}
+
+Click on _Add New Service_ to start the Service creation.
+
+{% /stepDescription %}
+
+{% stepVisualInfo %}
+
+{% image
+  src="/images/v1.2/connectors/create-new-service.png"
+  alt="Create a new Service"
+  caption="Add a new Service from the Services page" /%}
+
+{% /stepVisualInfo %}
+
+{% /step %}
+
+{% step srNumber=3 %}
+
+{% stepDescription title="3. Select the Service Type" %}
+
+Select {% $connector %} as the Service type and click _Next_.
+
+{% /stepDescription %}
+
+{% stepVisualInfo %}
+
+{% image
+  src=$selectServicePath
+  alt="Select Service"
+  caption="Select your Service from the list" /%}
+
+{% /stepVisualInfo %}
+
+{% /step %}
+
+{% step srNumber=4 %}
+
+{% stepDescription title="4. Name and Describe your Service" %}
+
+Provide a name and description for your Service.
+
+#### Service Name
+
+OpenMetadata uniquely identifies Services by their **Service Name**. Provide
+a name that distinguishes your deployment from other Services, including
+the other {% $connector %} Services that you might be ingesting metadata
+from.
+
+Note that when the name is set, it cannot be changed.
+
+{% /stepDescription %}
+
+{% stepVisualInfo %}
+
+{% image
+  src=$addNewServicePath
+  alt="Add New Service"
+  caption="Provide a Name and description for your Service" /%}
+
+{% /stepVisualInfo %}
+
+{% /step %}
+
+{% step srNumber=5 %}
+
+{% stepDescription title="5. Configure the Service Connection" %}
+
+In this step, we will configure the connection settings required for {% $connector %}.
+
+Please follow the instructions below to properly configure the Service to read from your sources. You will also find 
+helper documentation on the right-hand side panel in the UI.
+
+{% /stepDescription %}
+
+{% stepVisualInfo %}
+
+{% image
+  src=$serviceConnectionPath
+  alt="Configure Service connection"
+  caption="Configure the Service connection by filling the form" /%}
+
+{% /stepVisualInfo %}
+
+{% /step %}