Skip to content
This repository has been archived by the owner on Sep 4, 2024. It is now read-only.

Add compression option to HttpTransport #318

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions docs/client/java/partials/java_transport.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ Allows sending events to HTTP endpoint, using [ApacheHTTPClient](https://hc.apac
- `type` - string specifying the "api_key" or the fully qualified class name of your TokenProvider. Required if `auth` is provided.
- `apiKey` - string setting the Authentication HTTP header as the Bearer. Required if `type` is `api_key`.
- `headers` - dictionary specifying HTTP request headers. Optional.
- `compression` - string, name of algorithm used by HTTP client to compress request body. Optional, default value `null`, allowed values: `gzip`. Added in v1.13.0.

#### Behavior

Expand Down Expand Up @@ -63,6 +64,7 @@ transport:
api_key: f38d2189-c603-4b46-bdea-e573a3b5a7d5
headers:
X-Some-Extra-Header: abc
compression: gzip
```

</TabItem>
Expand Down Expand Up @@ -96,6 +98,7 @@ spark.openlineage.transport.timeoutInMillis=5000
spark.openlineage.transport.auth.type=api_key
spark.openlineage.transport.auth.apiKey=f38d2189-c603-4b46-bdea-e573a3b5a7d5
spark.openlineage.transport.headers.X-Some-Extra-Header=abc
spark.openlineage.transport.compression=gzip
```

<details><summary>URL parsing within Spark integration</summary>
Expand Down Expand Up @@ -143,6 +146,7 @@ openlineage.transport.timeoutInMillis=5000
openlineage.transport.auth.type=api_key
openlineage.transport.auth.apiKey=f38d2189-c603-4b46-bdea-e573a3b5a7d5
openlineage.transport.headers.X-Some-Extra-Header=abc
openlineage.transport.compression=gzip
```

</TabItem>
Expand Down Expand Up @@ -214,6 +218,7 @@ httpConfig.setUrlParams(queryParams);
httpConfig.setAuth(apiKeyTokenProvider);
httpConfig.setTimeoutInMillis(headers);
httpConfig.setHeaders(5000);
httpConfig.setCompression(HttpConfig.Compression.GZIP);

OpenLineageClient client = OpenLineageClient.builder()
.transport(
Expand Down
5 changes: 4 additions & 1 deletion docs/client/python.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,7 @@ Allows sending events to HTTP endpoint, using [requests](https://requests.readth
- `auth` - dictionary specifying authentication options. Optional, by default no authorization is used. If set, requires the `type` property.
- `type` - string specifying the "api_key" or the fully qualified class name of your TokenProvider. Required if `auth` is provided.
- `apiKey` - string setting the Authentication HTTP header as the Bearer. Required if `type` is `api_key`.
- `compression` - string, name of algorithm used by HTTP client to compress request body. Optional, default value `null`, allowed values: `gzip`. Added in v1.13.0.

#### Behavior

Expand All @@ -106,21 +107,23 @@ transport:
auth:
type: api_key
apiKey: f048521b-dfe8-47cd-9c65-0cb07d57591e
compression: gzip
```

</TabItem>
<TabItem value="python" label="Python Code">

```python
from openlineage.client import OpenLineageClient
from openlineage.client.transport.http import ApiKeyTokenProvider, HttpConfig, HttpTransport
from openlineage.client.transport.http import ApiKeyTokenProvider, HttpConfig, HttpCompression, HttpTransport

http_config = HttpConfig(
url="https://backend:5000",
endpoint="api/v1/lineage",
timeout=5,
verify=False,
auth=ApiKeyTokenProvider({"apiKey": "f048521b-dfe8-47cd-9c65-0cb07d57591e"}),
compression=HttpCompression.GZIP,
)

client = OpenLineageClient(transport=HttpTransport(http_config))
Expand Down
Loading