Kazaam was created with the goal of supporting easy and fast transformations of JSON data with Golang. This functionality provides us with an easy mechanism for taking intermediate JSON message representations and transforming them to formats required by arbitrary third-party APIs.
Inspired by Jolt, Kazaam supports JSON to JSON transformation via a transform "specification" also defined in JSON. A specification is comprised of one or more "operations". See Specification Support, below, for more details.
API Documentation is available at http://godoc.org/gopkg.in/qntfy/kazaam.v3.
Kazaam is primarily designed to be used as a library for transforming arbitrary JSON. It ships with eleven built-in transform types, and twenty-two built-in converter types, described below, which provide significant flexibility in reshaping JSON data.
Also included when you go get Kazaam, is a binary implementation, kazaam that can be used for
development and testing of new transform specifications.
Finally, Kazaam supports the implementation of custom transform and converter types. We encourage and appreciate pull requests for new transform types so that they can be incorporated into the Kazaam distribution, but understand sometimes time-constraints or licensing issues prevent this. See the API documentation for details on how to write and register custom transforms.
Due to performance considerations, Kazaam does not fully validate that input data is valid JSON. The
IsJson() function is provided for convenience if this functionality is needed, it may significantly slow
down use of Kazaam.
Transforms are the main mechanism in Kazaam for shaping json documents. Transforms, unlike converters work at the document level, whereas converters work at the value level. There are many built-in transforms for you to shape your document, but there is also a mechanism for developing your own custom transforms when the need arises.
Kazaam currently supports the following built-in transforms:
- shift
- steps
- concat
- coalesce
- extract
- timestamp
- uuid
- default
- pass
- delete
- merge
The shift transform is the current Kazaam workhorse used for remapping of fields. It supports a "require" field that when
set to true, will throw an error if any of the paths in the source JSON are not present.
The shift transform by default is destructive. For in-place operation, an optional "inplace"
field may be set.
The specification supports jsonpath-esque JSON accesses and sets as well as a custom JSON Path Parameters.
Concretely
{
"operation": "shift",
"spec": {
"object.id": "doc.uid",
"gid2": "doc.guid[1]",
"allGuids": "doc.guidObjects[*].id"
}
}executed on a JSON message with format
{
"doc": {
"uid": 12345,
"guid": ["guid0", "guid2", "guid4"],
"guidObjects": [{"id": "guid0"}, {"id": "guid2"}, {"id": "guid4"}]
},
"top-level-key": null
}would result in
{
"object": {
"id": 12345
},
"gid2": "guid2",
"allGuids": ["guid0", "guid2", "guid4"]
}The JSON Path implementation supports a few special cases:
- Array accesses: Retrieve
nth element from array - Array wildcarding: indexing an array with
[*]will return every matching element in an array - Top-level object capture: Mapping
$into a field will nest the entire original object under the requested key - Array append/prepend and set: Append and prepend an array with
[+]and[-]. Attempting to write an array element that does not exist results in null padding as needed to add that element at the specified index (useful with"inplace"). - JSON Path Parameters: Conditional Expressions and chained value conversions through Converter expressions
JSON Path Conditionals allow value skipping based on document existence or conditional expression evaluation and take on the following forms:
| Path Structure | Description |
|---|---|
| path.existing.value ? | Return the existing value or skip the value if it is not defined. (NOTE: skipping is allowed with conditionals even when the "require" option is used with Shift) |
| path.missing.value ? "default value" path.missing.value ? 42 |
Default values can be provided, and when a value is missing, the default value that was provided is returned instead. |
| path.existing.value ? ston("other.value") > 3 && another.value == "test" : | Existing value is skipped unless the Conditional Expression evaluates to "true". Note: the colon is required here or the expression itself will attempt to be treated as a default value. |
| path.value ? other.value == "something" : "default value" | If the path exists and the expression evaluates to true, the existing value is returned. If the path is missing, the default is provided. Otherwise, if the expression evaluates to "false" the default is returned. |
Notes:
-
Default values are simple JSON Values only (no composites). Strings must be quoted (and when embedded in JSON, the quote will need to be escaped.) Strings, Boolean, Nulls and Numbers are supported.
e.g.
"gid2 ? \"default value\"": "guid2",
Notes:
-
Function calls in Conditional Expressions call to named Converters and require 1 or 2 string parameters. The first parameter must be a JSON Path (without parameters) as a string to a value that will be converted and the optional second parameter must be a string. If provided the arguments will be provided to the Converter as a single string for it to parse.
e.g.
"gid2 ? substr(\"guid2\",\"2 3\") == \"id\":": "guid2",
JSON Path Converter Expressions allow for values to be altered by chaining the existing (or default value) through Converter functions. The value returned from the last Converter in the chain will become the returned value for the JSON Path query.
| Path Structure | Description |
|---|---|
| path.existing.value | converter1 arguments | converter2 arguments | Chained Converter syntax |
| path.value ? other.value == "something" : "default value" | converter1 | Can be combined with Conditional Expressions |
Notes:
- If
|characters are required as part of the value, they can be escaped with a\\character, and\\characters themselves can also be escaped.
Notes:
-
The whitespace between the converter name and arguments, as well as surrounding the argument is ignored. Although whitespace within the arguments are preserved, if the whitespace around the arguments is required, it must be escaped:
e.g.
"path.value | converter1 \ arguments\" would causeargumentsto be the arguments string.
Arguments are passed to the converter functions as a single string, and will require the converter function to parse out any meaningful parameters.
The steps transform performs a series of shift transforms with each step working on the ouptput from the last step. This transform is very similar to the shift transform, and takes the same optional parameters.
The following example produces the same results as the Shift transform example presented earlier. The only difference
is that the each of the steps are guaranteed to transform in the specified order.
{
"operation": "steps",
"spec": {
"steps": [
{
"object.id": "doc.uid"
},
{
"gid2": "doc.guid[1]"
},
{
"allGuids": "doc.guidObjects[*].id"
}
]
}
}The concat transform allows the combination of fields and literal strings into a single string value.
{
"operation": "concat",
"spec": {
"sources": [{
"value": "TEST"
}, {
"path": "a.timestamp"
}],
"targetPath": "a.timestamp",
"delim": ","
}
}executed on a JSON message with format
{
"a": {
"timestamp": 1481305274
}
}would result in
{
"a": {
"timestamp": "TEST,1481305274"
}
}Notes:
- sources: list of items to combine (in the order listed)
- literal values are specified via
value - field values are specified via
path(supports the same addressing asshift)
- literal values are specified via
- targetPath: where to place the resulting string
- if this an existing path, the result will replace current value.
- delim: Optional delimiter
The concat transform also supports a "require" field. When set to true,
Kazaam will throw an error if any of the paths in the source JSON are not
present.
A coalesce transform provides the ability to check multiple possible keys to find a desired value. The first matching key found of those provided is returned.
{
"operation": "coalesce",
"spec": {
"firstObjectId": ["doc.guidObjects[0].uid", "doc.guidObjects[0].id"]
}
}executed on a json message with format
{
"doc": {
"uid": 12345,
"guid": ["guid0", "guid2", "guid4"],
"guidObjects": [{"id": "guid0"}, {"id": "guid2"}, {"id": "guid4"}]
}
}would result in
{
"doc": {
"uid": 12345,
"guid": ["guid0", "guid2", "guid4"],
"guidObjects": [{"id": "guid0"}, {"id": "guid2"}, {"id": "guid4"}]
},
"firstObjectId": "guid0"
}Coalesce also supports an ignore array in the spec. If an otherwise matching key has a value in ignore, it is not considered a match.
This is useful e.g. for empty strings
{
"operation": "coalesce",
"spec": {
"ignore": [""],
"firstObjectId": ["doc.guidObjects[0].uid", "doc.guidObjects[0].id"]
}
}An extract transform provides the ability to select a sub-object and have kazaam return that sub-object as the top-level object. For example
{
"operation": "extract",
"spec": {
"path": "doc.guidObjects[0].path.to.subobject"
}
}executed on a json message with format
{
"doc": {
"uid": 12345,
"guid": [
"guid0",
"guid2",
"guid4"
],
"guidObjects": [
{
"path": {
"to": {
"subobject": {
"name": "the.subobject",
"field": "field.in.subobject"
}
}
}
},
{
"id": "guid2"
},
{
"id": "guid4"
}
]
}
}would result in
{
"name": "the.subobject",
"field": "field.in.subobject"
}A timestamp transform parses and formats time strings using the golang
syntax. Note: this operation is done in-place. If you want to preserve the
original string(s), pair the transform with shift. This transform also
supports the $now operator for inputFormat, which will set the current
timestamp at the specified path, formatted according to the outputFormat.
$unix is supported for both input and output formats as a Unix time, the
number of seconds elapsed since January 1, 1970 UTC as an integer string.
{
"operation": "timestamp",
"timestamp[0]": {
"inputFormat": "Mon Jan _2 15:04:05 -0700 2006",
"outputFormat": "2006-01-02T15:04:05-0700"
},
"nowTimestamp": {
"inputFormat": "$now",
"outputFormat": "2006-01-02T15:04:05-0700"
},
"epochTimestamp": {
"inputFormat": "2006-01-02T15:04:05-0700",
"outputFormat": "$unix"
}
}
executed on a json message with format
{
"timestamp": [
"Sat Jul 22 08:15:27 +0000 2017",
"Sun Jul 23 08:15:27 +0000 2017",
"Mon Jul 24 08:15:27 +0000 2017"
]
}would result in
{
"timestamp": [
"2017-07-22T08:15:27+0000",
"Sun Jul 23 08:15:27 +0000 2017",
"Mon Jul 24 08:15:27 +0000 2017"
],
"nowTimestamp": "2017-09-08T19:15:27+0000"
}A uuid transform generates a UUID based on the spec. Currently supports UUIDv3, UUIDv4, UUIDv5.
For version 4 is a very simple spec
{
"operation": "uuid",
"spec": {
"doc.uuid": {
"version": 4
}
}
}executed on a json message with format
{
"doc": {
"author_id": 11122112,
"document_id": 223323,
"meta": {
"id": 23
}
}
}would result in
{
"doc": {
"author_id": 11122112,
"document_id": 223323,
"meta": {
"id": 23
},
"uuid": "f03bacc1-f4e0-4371-a5c5-e8160d3d6c0c"
}
}For UUIDv3 & UUIDV5 are a bit more complex. These require a Name Space which is a valid UUID already, and a set of paths, which generate UUID's based on the value of that path. If that path doesn't exist in the incoming document, a default field will be used instead. Note both of these fields must be strings.
Additionally you can use the 4 predefined namespaces such as DNS, URL, OID, & X500 in the name space field otherwise pass your own UUID.
{
"operation":"uuid",
"spec":{
"doc.uuid":{
"version":5,
"namespace":"DNS",
"names":[
{"path":"doc.author_name", "default":"some string"},
{"path":"doc.type", "default":"another string"}
]
}
}
}executed on a json message with format
{
"doc": {
"author_name": "jason",
"type": "secret-document",
"document_id": 223323,
"meta": {
"id": 23
}
}
}would result in
{
"doc": {
"author_name": "jason",
"type": "secret-document",
"document_id": 223323,
"meta": {
"id": 23
},
"uuid": "f03bacc1-f4e0-4371-a7c5-e8160d3d6c0c"
}
}A default transform provides the ability to set a key's value explicitly. For example
{
"operation": "default",
"spec": {
"type": "message"
}
}would ensure that the output JSON message includes {"type": "message"}.
A delete transform provides the ability to delete keys in place.
{
"operation": "delete",
"spec": {
"paths": ["doc.uid", "doc.guidObjects[1]"]
}
}executed on a json message with format
{
"doc": {
"uid": 12345,
"guid": ["guid0", "guid2", "guid4"],
"guidObjects": [{"id": "guid0"}, {"id": "guid2"}, {"id": "guid4"}]
}
}would result in
{
"doc": {
"guid": ["guid0", "guid2", "guid4"],
"guidObjects": [{"id": "guid0"}, {"id": "guid4"}]
}
}A pass transform, as the name implies, passes the input data unchanged to the output. This is used internally when a null transform spec is specified, but may also be useful for testing.
A merge transform will take multiple arrays and join them in to an array of objects joining them by keys. The arrays should be equal length.
{
"operation": "merge",
"spec": {
"merge1": [
{
"name": "prop_1",
"array": "array_a"
},
{
"name": "prop_2",
"array": "array_b"
},
{
"name": "prop_3",
"array": "array_c"
}
]
}
}executed on a json message with format:
{
"array_a": [
"a_1",
"a_2",
"a_3"
],
"array_b": [
"b_1",
"b_2",
"b_3"
],
"array_c": [
"c_1",
"c_2",
"c_3"
]
}would result in:
{
"merge1": [
{
"prop_1": "a_1",
"prop_2": "b_1",
"prop_3": "c_1"
},
{
"prop_1": "a_2",
"prop_2": "b_2",
"prop_3": "c_2"
},
{
"prop_1": "a_3",
"prop_2": "b_3",
"prop_3": "c_3"
}
]
}Converters in Kazaam allow for value level transformations and work within and extend the current Transform capabilities.
Kazaam currently supports the following built-in Conveters:
| Converter Name | Description |
|---|---|
add <num> |
adds a number value to a number value |
ceil |
converts the number value to the least integer greater than or equal to the number value |
div <num> |
divides a number value by a number value |
floor |
converts the number value to the greatest integer less than or equal to the number value |
format <string> |
converts the value to a string via a fmt string |
lower |
converts the string value to lowercase characters |
mapped <string> |
maps the string value to another string value using predefined named maps |
mul <num> |
multiples a number value by a number value |
ntos |
converts the number value to a string value |
regex |
alters the string value with named regex replacements |
round |
converts a number value to the closet integer value |
ston |
converts a string value to a number value |
substr <num> [<num>] |
converts a string value to a substring value |
trim |
converts a string value by removing the leading and trailing whitespace characters |
upper |
converts a string value to uppercase characters |
len |
converts a string to an integer value equal to the length of the string, also returns the length of an array if the value is an array |
splitn <string> <num> |
splits a string by a delimiter string and returns the Nth token (1 based) |
eqs <any> |
returns true or false based on whether the value matches the parameter |
not |
returns true if value is false and false if the value is anything other than false |
split <delim> |
returns array of values split on delimiter |
join <delim> |
joins an array of strings by the delimiter |
float <precision> |
converts a number to a floating point number with the specified precision (rounded) |
The following examples will use the same input JSON value:
{
"tests": {
"test_int": 500,
"test_float": 500.01,
"test_float2": 500.0,
"test_number": "750",
"test_fraction": 0.5,
"test_trim": " blah ",
"test_money": "$6,000,000",
"test_chars": "abcdefghijklmnopqrstuvwxyz",
"test_mapped": "Texas",
"test_null": null,
"pi": 3.141592653,
"test_true": true,
"test_false": false,
"test_null": null,
"test_string": "The quick brown fox",
"test_naics_code": "531312",
"test_split":"a|b|c",
"test_join":["a","b","c"]
},
"test_bool": true
}Adds a number to a number value
| Argument | Description |
|---|---|
| Number | Number value to add |
example:
{
"operation": "shift",
"spec": {
"output": "tests.test_int | add 1"
}
}produces:
{
"output": 501
}Converts a number value to the least closest integer greater than or equal to the number value
| Argument | Description |
|---|
example:
{
"operation": "shift",
"spec": {
"output": "tests.test_float | ceil"
}
}produces:
{
"output": 501
}Divides a number value by another number value
| Argument | Description |
|---|---|
| Number | dividend in a division operation |
example:
{
"operation": "shift",
"spec": {
"output1": "tests.test_float | div 2",
"output2": "tests.test_int | div .5"
}
}produces:
{
"output1": 250,
"output2": 1000
}Converts a number value to the greatest integer value less than or equal to the number value
| Argument | Description |
|---|
example:
{
"operation": "shift",
"spec": {
"output": "tests.test_float | floor"
}
}produces:
{
"output": 500
}Formats a value into a new string value using a fmt string
| Argument | Description |
|---|---|
| string | fmt string, if whitespace shouldn't be trimmed, it should be escaped with \ |
example:
{
"operation": "shift",
"spec": {
"output1": "tests.pi | format %.4f",
"output2": "tests.test_float | format %.0f",
"output3": "tests.test_string | format %s jumps over the lazy dog",
"output4": "tests.test_true | format %t is the value"
}
}produces:
{
"output1": "3.1416",
"output2": "500",
"output3": "The quick brown fox jumps over the lazy dog",
"output4": "true is the value"
}Converts a string value to lowsercase
| Argument | Description |
|---|
example:
{
"operation": "shift",
"spec": {
"output": "tests.test_string | lower"
}
}produces:
{
"output": "the quick brown fox"
}Maps a string value to another string value using a named JSON map defined in $.converters.mapped
| Argument | Description |
|---|---|
| string | name of the map to use |
example:
{
"operation": "shift",
"converters": {
"mapped": {
"states": {
"Ohio": "OH",
"Texas": "TX"
}
}
},
"spec": {
"output": "tests.test_mapped | mapped states"
}
}produces:
{
"output": "TX"
}Multiplies a number value by another number value
| Argument | Description |
|---|---|
| Number | multiplier value of a multiplication operation |
example:
{
"operation": "shift",
"spec": {
"output1": "tests.test_int | mul 2",
"output2": "tests.test_int | mul .5"
}
}produces:
{
"output1": 1000,
"output2": 250
}Converts a number value to a string value
| Argument | Description |
|---|
example:
{
"operation": "shift",
"spec": {
"output": "tests.test_int | ntos"
}
}produces:
{
"output": "500"
}Use Regexp ReplaceAll to match and replace values defined in the $.converters.regex configuration object. You can
also pass an array of configuration objects and they will all be applied in order, stopping after the first match is matched and replaced.
| Argument | Description |
|---|---|
| string | name of predefined regex match and replace |
example:
{
"operation": "shift",
"converters": {
"regex": {
"remove_dollar_sign": {
"match": "\\$\\s*(.*)",
"replace": "$1"
},
"remove_comma": {
"match": ",",
"replace": ""
},
"convert_naics": [
{
"match": "^8111.*",
"replace": "automotive_services"
},
{
"match": "^4413.*",
"replace": "automotive_services"
},
{
"match": "^531.*",
"replace": "real_estate"
}
]
}
},
"spec": {
"output": "tests.test_money | regex remove_dollar_sign | regex remove_comma"
}
}produces:
{
"output": "6000000"
}Rounds a number value to the closest integer value
| Argument | Description |
|---|
example:
{
"operation": "shift",
"spec": {
"output1": "tests.test_float | round",
"output2": "tests.test_fraction | round"
}
}produces:
{
"output1": 500,
"output2": 1
}Converts a string value to a number value
| Argument | Description |
|---|
example:
{
"operation": "shift",
"spec": {
"output": "tests.test_number | ston"
}
}produces:
{
"output": 750
}Returns a substring of a string value
| Argument | Description |
|---|---|
| number | 0 based index where to start the substring |
| number | (optional) index of last character + 1 in the substring, if omitted uses the string's length |
example:
{
"operation": "shift",
"spec": {
"output1": "tests.test_chars | substr 3 6",
"output2": "tests.test_string | substr 10"
}
}produces:
{
"output1": "def",
"output2": "brown fox"
}Removes whitespace from the start and end of a string value
| Argument | Description |
|---|
example:
{
"operation": "shift",
"spec": {
"output": "tests.test_trim | trim"
}
}produces:
{
"output": "blah"
}Converts a string value to uppercase
| Argument | Description |
|---|
example:
{
"operation": "shift",
"spec": {
"output": "tests.test_string | upper"
}
}produces:
{
"output": "THE QUICK BROWN FOX"
}Returns the length of a string value
| Argument | Description |
|---|
example:
{
"operation": "shift",
"spec": {
"output": "tests.test_string | len"
}
}produces:
{
"output": 19
}Returns the Nth token of a string split by a delimiter string
| Argument | Description |
|---|---|
| string | delimiter string |
| number | one based position of token to return |
example:
{
"operation": "shift",
"spec": {
"output": "tests.test_string | splitn o 2"
}
}produces:
{
"output": "wn f"
}Returns true or false based on whether the value equals the parameter
| Argument | Description |
|---|---|
| any | value to compare |
example:
{
"operation": "shift",
"spec": {
"output": "tests.test_string | eqs \"The quick brown fox\""
}
}produces:
{
"output": true
}Negates a false value returning true and returns false for everything else
| Argument | Description |
|---|
example:
{
"operation": "shift",
"spec": {
"output1": "tests.test_true | not",
"output2": "tests.test_false | not"
}
}produces:
{
"output1": false,
"output2": true
}| Argument | Description |
|---|---|
| delim | string delimiter on which to split the string |
example:
{
"operation": "shift",
"spec": {
"output1": "tests.test_split | split \\|"
}
}produces:
{
"output1": ["a","b","c"]
}| Argument | Description |
|---|---|
| delim | string delimiter on which to join the array into a string |
example:
{
"operation": "shift",
"spec": {
"output1": "tests.test_join | join \\|"
}
}produces:
{
"output1": "a|b|c"
}| Argument | Description |
|---|---|
| precision | number of decimals (it will round) |
example:
{
"operation": "shift",
"spec": {
"output1": "tests.test_float | float 1"
}
}produces:
{
"output1": 500.0
}To start, go get the versioned repository:
go get gopkg.in/qntfy/kazaam.v3If you want to create an executable binary from this project, follow
these steps (you'll need go installed and $GOPATH set):
go get gopkg.in/qntfy/kazaam.v3
cd $GOPATH/src/gopkg.in/qntfy.kazaam.v3/kazaam
go installThis will create an executable in $GOPATH/bin like you
would expect from the normal go build behavior.
See godoc examples.