Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow complex types to be null when transforming to payload format #1318

Merged
merged 1 commit into from
Jul 6, 2020
Merged

Allow complex types to be null when transforming to payload format #1318

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 7 additions & 6 deletions ...c/main/java/com/mozilla/telemetry/ingestion/core/transform/PubsubMessageToObjectNode.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -330,8 +330,12 @@ private void processField(String jsonFieldName, Field field, JsonNode value, Obj
ObjectNode additionalProperties) {
final String bqFieldName = field.getName();

// A record of key and value indicates we need to transformForBqSchema a map to an array.
if (isMapType(field)) {
// null is valid for any type except an element of a list
if (value.isNull()) {
updateParent(parent, jsonFieldName, bqFieldName, value);

// A record of key and value indicates we need to transformForBqSchema a map to an array.
} else if (isMapType(field)) {
expandMapType(jsonFieldName, (ObjectNode) value, field, parent, additionalProperties);
Copy link
Contributor Author

@relud relud Jul 6, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the exception in #1317 is thrown by this line trying to cast a literal null to ObjectNode here.

This PR fixes that by allowing null for any value passed to processField, which I believe is called on every type that is not an element of a list.

The resulting behavior omits the field, which BigQuery will interpret as an empty list if necessary.


// A record with a single "list" field and a list value should be expanded appropriately.
Expand Down Expand Up @@ -516,10 +520,7 @@ private void expandNestedListType(String jsonFieldName, ArrayNode value, Field f
* field should be put to {@code additional_properties}.
*/
private Optional<JsonNode> coerceToBqType(JsonNode o, Field field) {
if (o.isNull()) {
// null is valid for any type, just not as an element of a list
return Optional.of(o);
} else if (field.getMode() == Field.Mode.REPEATED) {
if (field.getMode() == Field.Mode.REPEATED) {
if (o.isArray()) {
// We have not yet observed a case where an array type contains values that cannot be
// coerced to appropriate values, but if it does this will throw NoSuchElementException
Expand Down
2 changes: 1 addition & 1 deletion ingestion-core/src/test/resources/testdata/payload-format-input.ndjson
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,6 @@
{"attributeMap":{"document_namespace":"live-sink","document_type":"test","document_version":"1"},"payload":{"test_tuple":[{"key":"value"},"x",6]}}
{"attributeMap":{"document_namespace":"live-sink","document_type":"test","document_version":"1"},"payload":{"test_nested_list":[["a","b","c"]],"test_nested_csv":[{"list":"a,b,c"}],"test_list":["a","b","c"]}}
{"attributeMap":{"document_namespace":"live-sink","document_type":"test","document_version":"1"},"payload":{"test_list":null,"test_nested_list":[null],"test_record":null}}
{"attributeMap":{"document_namespace":"live-sink","document_type":"test","document_version":"1"},"payload":{"test_record":{"key":null}}}
{"attributeMap":{"document_namespace":"live-sink","document_type":"test","document_version":"1"},"payload":{"test_record":{"key":null},"test_map":null}}
{"attributeMap":{"document_namespace":"live-sink","document_type":"test","document_version":"1"},"payload":{"test_int64 ":null,"test_int64":7,"test_int64_":"invalid","test_int64+":8}}
{"attributeMap":{"document_namespace":"live-sink","document_type":"test","document_version":"1"},"payload":{"test_map":{"key":null,"value":null,"other":"other"}}}