Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Destination S3 Glue: struct fix #24675

Merged
merged 6 commits into from
Mar 29, 2023
Merged

Destination S3 Glue: struct fix #24675

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
50aa100
destination-s3-glue: Fix struct and array schema generation
blarghmatey Mar 10, 2023
b9220ac
rollback change
marcosmarxm Mar 29, 2023
8c17a82
bump connector version
marcosmarxm Mar 29, 2023
22a1f1b
Merge branch 'master' into marcos/s3-glue-struct-fix
marcosmarxm Mar 29, 2023
9e3aa72
return version to 0.1.3
marcosmarxm Mar 29, 2023
1404435
auto-bump connector version
octavia-squidington-iii Mar 29, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/destination-s3-glue/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,5 +14,5 @@ ENV APPLICATION destination-s3-glue

COPY --from=build /airbyte /airbyte

LABEL io.airbyte.version=0.1.3
LABEL io.airbyte.version=0.1.4
LABEL io.airbyte.name=airbyte/destination-s3-glue
9 changes: 6 additions & 3 deletions ...ion-s3-glue/src/main/java/io/airbyte/integrations/destination/s3_glue/GlueOperations.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -129,28 +129,31 @@ private String transformSchemaRecursive(JsonNode jsonNode) {
if (jsonNode.has("airbyte_type") && jsonNode.get("airbyte_type").asText().equals("integer")) {
yield "int";
}
yield "float";
yield "decimal"; // Default to use decimal as it is a more precise type and allows for large values
}
case "boolean" -> "boolean";
case "integer" -> "int";
case "array" -> {
String arrayType = "array<";
Set<String> itemTypes = filterTypes(jsonNode.get("items").get("type"));
Set<String> itemTypes;
if (jsonNode.has("items")) {
itemTypes = filterTypes(jsonNode.get("items").get("type"));
if (itemTypes.size() > 1) {
// TODO(itaseski) use union instead of array when having multiple types (rare occurrence)?
arrayType += "string>";
} else {
String subtype = transformSchemaRecursive(jsonNode.get("items"));
arrayType += (subtype + ">");
}
} else arrayType += "string>";
yield arrayType;
}
case "object" -> {
if (jsonNode.has("properties")) {
String objectType = "struct<";
Map<String, JsonNode> properties = objectMapper.convertValue(jsonNode.get("properties"), new TypeReference<>() {});
String columnTypes = properties.entrySet().stream()
.map(p -> p.getKey() + " : " + transformSchemaRecursive(p.getValue()))
.map(p -> p.getKey() + ":" + transformSchemaRecursive(p.getValue()))
.collect(Collectors.joining(","));
objectType += (columnTypes + ">");
yield objectType;
Expand Down
2 changes: 1 addition & 1 deletion ...glue/src/main/java/io/airbyte/integrations/destination/s3_glue/S3GlueConsumerFactory.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@ private static Function<ConfiguredAirbyteStream, S3GlueWriteConfig> toWriteConfi
final DestinationSyncMode syncMode = stream.getDestinationSyncMode();
final JsonNode jsonSchema = abStream.getJsonSchema();
((ObjectNode) jsonSchema.get("properties")).putPOJO(JavaBaseConstants.COLUMN_NAME_AB_ID, Map.of("type", "string"));
((ObjectNode) jsonSchema.get("properties")).putPOJO(JavaBaseConstants.COLUMN_NAME_EMITTED_AT, Map.of("type", "integer"));
((ObjectNode) jsonSchema.get("properties")).putPOJO(JavaBaseConstants.COLUMN_NAME_EMITTED_AT, Map.of("type", "number"));
final String location = "s3://" + s3Config.getBucketName() + "/" +
fullOutputPath.substring(0, fullOutputPath.lastIndexOf("/") + 1);
final S3GlueWriteConfig writeConfig =
Expand Down
1 change: 1 addition & 0 deletions docs/integrations/destinations/s3-glue.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -245,6 +245,7 @@ Output files can be compressed. The default option is GZIP compression. If compr

| Version | Date | Pull Request | Subject |
|:--------|:-----------|:---------------------------------------------------------|:----------------------------------------------------------------------------------------|
| 0.1.4 | 2023-03-10 | [23950](https://github.com/airbytehq/airbyte/pull/23950) | Fix schema syntax error for struct fields and handle missing `items` in array fields |
| 0.1.3 | 2023-02-10 | [22822](https://github.com/airbytehq/airbyte/pull/22822) | Fix data type for _ab_emitted_at column in table definition |
| 0.1.2 | 2023-02-01 | [22220](https://github.com/airbytehq/airbyte/pull/22220) | Fix race condition in test, table metadata, add Airbyte sync fields to table definition |
| 0.1.1 | 2022-12-13 | [19907](https://github.com/airbytehq/airbyte/pull/19907) | Fix parsing empty object in schema |
Expand Down