Skip to content

Commit

Permalink
MINOR - Run ingestion externally docs (open-metadata#14751)
Browse files Browse the repository at this point in the history
* add docs

* MINOR - Run Ingestion Externally docs
  • Loading branch information
pmbrull authored and Abhishek332 committed Jan 25, 2024
1 parent 72205ff commit 4536c44
Show file tree
Hide file tree
Showing 4 changed files with 75 additions and 0 deletions.
38 changes: 38 additions & 0 deletions openmetadata-docs/content/v1.2.x/deployment/ingestion/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,41 @@ workflowConfig:
If you need to get the YAML shape of any connector, you can pick it up from its doc [page](/connectors).
Additionally, if you want to see your runs logged in the `Ingestions` tab of the connectors page as you would
when running the connectors natively with OpenMetadata, you can add the following configuration on your YAMLs:

```yaml
source:
type: mysql
serviceName: mysql
[...]
workflowConfig:
openMetadataServerConfig:
hostPort: 'http://localhost:8585/api'
authProvider: openmetadata
securityConfig:
jwtToken: ...
ingestionPipelineFQN: <serviceName>.<pipelineName> # E.g., mysql.marketing_metadata`
```

Adding the `ingestionPipelineFQN` - the Ingestion Pipeline Fully Qualified Name - will tell the Ingestion Framework
to log the executions and update the ingestion status, which will appear on the UI. Note that the action buttons
will be disabled, since OpenMetadata won't be able to interact with external systems.

### 3. (Optional) Disable the Pipeline Service Client

If you want to run your workflows **ONLY externally** without relying on OpenMetadata for any workflow management
or scheduling, you can update the following server configuration:

```yaml
pipelineServiceClientConfiguration:
enabled: ${PIPELINE_SERVICE_CLIENT_ENABLED:-true}
```
by setting `enabled: false` or setting the `PIPELINE_SERVICE_CLIENT_ENABLED=false` as an environment variable.

This will stop certain APIs and monitors related to the Pipeline Service Client (e.g., Airflow) from being operative.

### Examples

{% note %}
Expand Down Expand Up @@ -159,3 +194,6 @@ don't hesitate to reach to us in [Slack](https://slack.open-metadata.org/) or di
Run the ingestion process externally from GitHub Actions
{% /inlineCallout %}
{% /inlineCalloutContainer %}


###
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ slug: /connectors/database/mssql
| Query Usage | {% icon iconName="check" /%} |
| Data Profiler | {% icon iconName="check" /%} |
| Data Quality | {% icon iconName="check" /%} |
| Stored Procedures | {% icon iconName="check" /%} |
| Owners | {% icon iconName="cross" /%} |
| Tags | {% icon iconName="cross" /%} |
| DBT | {% icon iconName="check" /%} |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ slug: /connectors/database/mssql/yaml
| Metadata | {% icon iconName="check" /%} |
| Query Usage | {% icon iconName="check" /%} |
| Data Profiler | {% icon iconName="check" /%} |
| Stored Procedures | {% icon iconName="check" /%} |
| Data Quality | {% icon iconName="check" /%} |
| Owners | {% icon iconName="cross" /%} |
| Tags | {% icon iconName="cross" /%} |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,41 @@ workflowConfig:
If you need to get the YAML shape of any connector, you can pick it up from its doc [page](/connectors).
Additionally, if you want to see your runs logged in the `Ingestions` tab of the connectors page as you would
when running the connectors natively with OpenMetadata, you can add the following configuration on your YAMLs:

```yaml
source:
type: mysql
serviceName: mysql
[...]
workflowConfig:
openMetadataServerConfig:
hostPort: 'http://localhost:8585/api'
authProvider: openmetadata
securityConfig:
jwtToken: ...
ingestionPipelineFQN: <serviceName>.<pipelineName> # E.g., mysql.marketing_metadata`
```

Adding the `ingestionPipelineFQN` - the Ingestion Pipeline Fully Qualified Name - will tell the Ingestion Framework
to log the executions and update the ingestion status, which will appear on the UI. Note that the action buttons
will be disabled, since OpenMetadata won't be able to interact with external systems.

### 3. (Optional) Disable the Pipeline Service Client

If you want to run your workflows **ONLY externally** without relying on OpenMetadata for any workflow management
or scheduling, you can update the following server configuration:

```yaml
pipelineServiceClientConfiguration:
enabled: ${PIPELINE_SERVICE_CLIENT_ENABLED:-true}
```
by setting `enabled: false` or setting the `PIPELINE_SERVICE_CLIENT_ENABLED=false` as an environment variable.

This will stop certain APIs and monitors related to the Pipeline Service Client (e.g., Airflow) from being operative.

### Examples

{% note %}
Expand Down

0 comments on commit 4536c44

Please sign in to comment.