diff --git a/README.md b/README.md index e98fc05..7b1f25d 100644 --- a/README.md +++ b/README.md @@ -21,13 +21,13 @@ API Documentation is available at http://godoc.org/gopkg.in/qntfy/kazaam.v3. ## Features Kazaam is primarily designed to be used as a library for transforming arbitrary JSON. -It ships with six built-in transform types, described below, which provide significant flexibility -in reshaping JSON data. +It ships with eleven built-in transform types, and twenty-two built-in converter types, +described below, which provide significant flexibility in reshaping JSON data. Also included when you `go get` Kazaam, is a binary implementation, `kazaam` that can be used for development and testing of new transform specifications. -Finally, Kazaam supports the implementation of custom transform types. We encourage and appreciate +Finally, Kazaam supports the implementation of custom transform and converter types. We encourage and appreciate pull requests for new transform types so that they can be incorporated into the Kazaam distribution, but understand sometimes time-constraints or licensing issues prevent this. See the API documentation for details on how to write and register custom transforms. @@ -36,9 +36,16 @@ Due to performance considerations, Kazaam does not fully validate that input dat `IsJson()` function is provided for convenience if this functionality is needed, it may significantly slow down use of Kazaam. -## Specification Support -Kazaam currently supports the following transforms: +## Transform Specification Support + +Transforms are the main mechanism in Kazaam for shaping json documents. Transforms, unlike converters work +at the document level, whereas converters work at the value level. There are many built-in transforms for +you to shape your document, but there is also a mechanism for developing your own custom transforms when the +need arises. + +Kazaam currently supports the following built-in transforms: - shift +- steps - concat - coalesce - extract @@ -47,11 +54,19 @@ Kazaam currently supports the following transforms: - default - pass - delete +- merge ### Shift -The shift transform is the current Kazaam workhorse used for remapping of fields. -The specification supports jsonpath-esque JSON accesses and sets. Concretely -```javascript +The shift transform is the current Kazaam workhorse used for remapping of fields. It supports a `"require"` field that when +set to `true`, will throw an error if *any* of the paths in the source JSON are not present. + +The shift transform by default is destructive. For in-place operation, an optional `"inplace"` +field may be set. + +The specification supports jsonpath-esque JSON accesses and sets as well as a custom JSON Path Parameters. + +Concretely +```json { "operation": "shift", "spec": { @@ -63,7 +78,7 @@ The specification supports jsonpath-esque JSON accesses and sets. Concretely ``` executed on a JSON message with format -```javascript +```json { "doc": { "uid": 12345, @@ -75,7 +90,7 @@ executed on a JSON message with format ``` would result in -```javascript +```json { "object": { "id": 12345 @@ -84,23 +99,96 @@ would result in "allGuids": ["guid0", "guid2", "guid4"] } ``` +##### JSON Path Syntax -The jsonpath implementation supports a few special cases: +The JSON Path implementation supports a few special cases: - *Array accesses*: Retrieve `n`th element from array - *Array wildcarding*: indexing an array with `[*]` will return every matching element in an array - *Top-level object capture*: Mapping `$` into a field will nest the entire original object under the requested key - *Array append/prepend and set*: Append and prepend an array with `[+]` and `[-]`. Attempting to write an array element that does not exist results in null padding as needed to add that element at the specified index (useful with `"inplace"`). +- *JSON Path Parameters*: Conditional Expressions and chained value conversions through Converter expressions -The shift transform also supports a `"require"` field. When set to `true`, -Kazaam will throw an error if *any* of the paths in the source JSON are not -present. +##### JSON Path Parameters + +###### JSON Path Conditional Expressions + +JSON Path Conditionals allow value skipping based on document existence or conditional expression evaluation and +take on the following forms: + +| Path Structure | Description | +|:--------------------------- |:---------------| +| _path.existing.value_ **?** | Return the existing value or skip the value if it is not defined. (NOTE: skipping is allowed with conditionals even when the `"require"` option is used with Shift) | +| _path.missing.value_ **? "default value"**
_path.missing.value_ **? 42** | Default values can be provided, and when a value is missing, the default value that was provided is returned instead. | +| _path.existing.value_ **? ston("other.value") > 3 && another.value == "test" :** | Existing value is skipped unless the *Conditional Expression* evaluates to `"true"`. **Note**: the colon is required here or the expression itself will attempt to be treated as a default value. | +| _path.value_ **? other.value == "something" : "default value"** | If the path exists and the expression evaluates to true, the existing value is returned. If the path is missing, the default is provided. Otherwise, if the expression evaluates to `"false"` the default is returned. | + +**Notes**: +* Default values are simple JSON Values only (no composites). Strings must be quoted (and when embedded in JSON, the quote will need to be escaped.) Strings, Boolean, Nulls and Numbers are supported. + + e.g. + `"gid2 ? \"default value\"": "guid2",` + +**Notes**: +* Function calls in Conditional Expressions call to named Converters and require 1 or 2 string parameters. The first parameter must be a JSON Path (without parameters) as a string to +a value that will be converted and the optional second parameter must be a string. If provided the arguments will be provided to the Converter as a single string for it to parse. + + e.g. + `"gid2 ? substr(\"guid2\",\"2 3\") == \"id\":": "guid2",` + +###### JSON Path Converter Expressions + +JSON Path Converter Expressions allow for values to be altered by chaining the existing (or default value) through Converter functions. The value +returned from the last Converter in the chain will become the returned value for the JSON Path query. + +| Path Structure | Description | +|:---------------|:------------| +| _path.existing.value_ **| converter1 arguments | converter2 arguments**| Chained Converter syntax | +| _path.value_ **? other.value == "something" : "default value" | converter1** | Can be combined with Conditional Expressions | + +**Notes**: +* If **`|`** characters are required as part of the value, they can be escaped with a **`\\`** character, and **`\\`** characters themselves +can also be escaped. + +**Notes**: +* The whitespace between the converter name and arguments, as well as surrounding the argument is ignored. Although whitespace +within the arguments are preserved, if the whitespace around the arguments is required, it must be escaped: + + e.g. + `"path.value | converter1 \ arguments\ `" would cause **` arguments `** to be the arguments string. + +Arguments are passed to the converter functions as a single string, and will require the converter function to parse out any meaningful parameters. + + +### Steps +The steps transform performs a series of shift transforms with each step working on the ouptput from the last step. This +transform is very similar to the shift transform, and takes the same optional parameters. + +The following example produces the same results as the `Shift` transform example presented earlier. The only difference +is that the each of the steps are guaranteed to transform in the specified order. + +```json +{ + "operation": "steps", + "spec": { + "steps": [ + { + "object.id": "doc.uid" + }, + { + "gid2": "doc.guid[1]" + }, + { + "allGuids": "doc.guidObjects[*].id" + } + ] + } +} +``` -Finally, shift by default is destructive. For in-place operation, an optional `"inplace"` -field may be set. ### Concat The concat transform allows the combination of fields and literal strings into a single string value. -```javascript +```json { "operation": "concat", "spec": { @@ -116,7 +204,7 @@ The concat transform allows the combination of fields and literal strings into a ``` executed on a JSON message with format -```javascript +```json { "a": { "timestamp": 1481305274 @@ -125,7 +213,7 @@ executed on a JSON message with format ``` would result in -```javascript +```json { "a": { "timestamp": "TEST,1481305274" @@ -133,7 +221,7 @@ would result in } ``` -Notes: +**Notes**: - *sources*: list of items to combine (in the order listed) - literal values are specified via `value` - field values are specified via `path` (supports the same addressing as `shift`) @@ -147,7 +235,7 @@ present. ### Coalesce A coalesce transform provides the ability to check multiple possible keys to find a desired value. The first matching key found of those provided is returned. -```javascript +```json { "operation": "coalesce", "spec": { @@ -157,7 +245,7 @@ A coalesce transform provides the ability to check multiple possible keys to fin ``` executed on a json message with format -```javascript +```json { "doc": { "uid": 12345, @@ -168,7 +256,7 @@ executed on a json message with format ``` would result in -```javascript +```json { "doc": { "uid": 12345, @@ -181,7 +269,7 @@ would result in Coalesce also supports an `ignore` array in the spec. If an otherwise matching key has a value in `ignore`, it is not considered a match. This is useful e.g. for empty strings -```javascript +```json { "operation": "coalesce", "spec": { @@ -193,7 +281,7 @@ This is useful e.g. for empty strings ### Extract An `extract` transform provides the ability to select a sub-object and have kazaam return that sub-object as the top-level object. For example -```javascript +```json { "operation": "extract", "spec": { @@ -203,18 +291,39 @@ An `extract` transform provides the ability to select a sub-object and have kaza ``` executed on a json message with format -```javascript +```json { "doc": { "uid": 12345, - "guid": ["guid0", "guid2", "guid4"], - "guidObjects": [{"path": {"to": {"subobject": {"name": "the.subobject", "field", "field.in.subobject"}}}}, {"id": "guid2"}, {"id": "guid4"}] + "guid": [ + "guid0", + "guid2", + "guid4" + ], + "guidObjects": [ + { + "path": { + "to": { + "subobject": { + "name": "the.subobject", + "field": "field.in.subobject" + } + } + } + }, + { + "id": "guid2" + }, + { + "id": "guid4" + } + ] } } ``` would result in -```javascript +```json { "name": "the.subobject", "field": "field.in.subobject" @@ -229,7 +338,7 @@ supports the `$now` operator for `inputFormat`, which will set the current timestamp at the specified path, formatted according to the `outputFormat`. `$unix` is supported for both input and output formats as a Unix time, the number of seconds elapsed since January 1, 1970 UTC as an integer string. -```javascript +```json { "operation": "timestamp", "timestamp[0]": { @@ -249,7 +358,7 @@ number of seconds elapsed since January 1, 1970 UTC as an integer string. ``` executed on a json message with format -```javascript +```json { "timestamp": [ "Sat Jul 22 08:15:27 +0000 2017", @@ -260,13 +369,13 @@ executed on a json message with format ``` would result in -```javascript +```json { "timestamp": [ "2017-07-22T08:15:27+0000", "Sun Jul 23 08:15:27 +0000 2017", "Mon Jul 24 08:15:27 +0000 2017" - ] + ], "nowTimestamp": "2017-09-08T19:15:27+0000" } ``` @@ -276,19 +385,19 @@ A `uuid` transform generates a UUID based on the spec. Currently supports UUIDv3 For version 4 is a very simple spec -```javascript +```json { "operation": "uuid", "spec": { "doc.uuid": { - "version": 4, //required + "version": 4 } } } ``` executed on a json message with format -```javascript +```json { "doc": { "author_id": 11122112, @@ -301,14 +410,14 @@ executed on a json message with format ``` would result in -```javascript +```json { "doc": { "author_id": 11122112, "document_id": 223323, "meta": { "id": 23 - } + }, "uuid": "f03bacc1-f4e0-4371-a5c5-e8160d3d6c0c" } } @@ -317,7 +426,7 @@ would result in For UUIDv3 & UUIDV5 are a bit more complex. These require a Name Space which is a valid UUID already, and a set of paths, which generate UUID's based on the value of that path. If that path doesn't exist in the incoming document, a default field will be used instead. **Note** both of these fields must be strings. **Additionally** you can use the 4 predefined namespaces such as `DNS`, `URL`, `OID`, & `X500` in the name space field otherwise pass your own UUID. -```javascript +```json { "operation":"uuid", "spec":{ @@ -326,7 +435,7 @@ For UUIDv3 & UUIDV5 are a bit more complex. These require a Name Space which is "namespace":"DNS", "names":[ {"path":"doc.author_name", "default":"some string"}, - {"path":"doc.type", "default":"another string"}, + {"path":"doc.type", "default":"another string"} ] } } @@ -334,11 +443,11 @@ For UUIDv3 & UUIDV5 are a bit more complex. These require a Name Space which is ``` executed on a json message with format -```javascript +```json { "doc": { "author_name": "jason", - "type": "secret-document" + "type": "secret-document", "document_id": 223323, "meta": { "id": 23 @@ -348,7 +457,7 @@ executed on a json message with format ``` would result in -```javascript +```json { "doc": { "author_name": "jason", @@ -365,7 +474,7 @@ would result in ### Default A default transform provides the ability to set a key's value explicitly. For example -```javascript +```json { "operation": "default", "spec": { @@ -378,7 +487,7 @@ would ensure that the output JSON message includes `{"type": "message"}`. ### Delete A delete transform provides the ability to delete keys in place. -```javascript +```json { "operation": "delete", "spec": { @@ -388,7 +497,7 @@ A delete transform provides the ability to delete keys in place. ``` executed on a json message with format -```javascript +```json { "doc": { "uid": 12345, @@ -399,7 +508,7 @@ executed on a json message with format ``` would result in -```javascript +```json { "doc": { "guid": ["guid0", "guid2", "guid4"], @@ -413,6 +522,738 @@ would result in A pass transform, as the name implies, passes the input data unchanged to the output. This is used internally when a null transform spec is specified, but may also be useful for testing. +### Merge +A merge transform will take multiple arrays and join them in to an array of objects joining them by keys. The arrays should be equal length. + +```json +{ + "operation": "merge", + "spec": { + "merge1": [ + { + "name": "prop_1", + "array": "array_a" + }, + { + "name": "prop_2", + "array": "array_b" + }, + { + "name": "prop_3", + "array": "array_c" + } + ] + } +} +``` + +executed on a json message with format: +```json +{ + "array_a": [ + "a_1", + "a_2", + "a_3" + ], + "array_b": [ + "b_1", + "b_2", + "b_3" + ], + "array_c": [ + "c_1", + "c_2", + "c_3" + ] +} +``` + +would result in: +```json +{ + "merge1": [ + { + "prop_1": "a_1", + "prop_2": "b_1", + "prop_3": "c_1" + }, + { + "prop_1": "a_2", + "prop_2": "b_2", + "prop_3": "c_2" + }, + { + "prop_1": "a_3", + "prop_2": "b_3", + "prop_3": "c_3" + } + ] +} +``` + + + +## Converter Specification Support + +Converters in Kazaam allow for value level transformations and work within and extend the current Transform +capabilities. + +Kazaam currently supports the following built-in Conveters: + +| Converter Name | Description | +|---------------:|:------------| +`add ` | adds a number value to a number value +`ceil` | converts the number value to the least integer greater than or equal to the number value +`div ` | divides a number value by a number value +`floor` | converts the number value to the greatest integer less than or equal to the number value +`format ` | converts the value to a string via a **`fmt`** string +`lower` | converts the string value to lowercase characters +`mapped `| maps the string value to another string value using predefined named maps +`mul ` | multiples a number value by a number value +`ntos` | converts the number value to a string value +`regex` | alters the string value with named regex replacements +`round` | converts a number value to the closet integer value +`ston` | converts a string value to a number value +`substr []` | converts a string value to a substring value +`trim` | converts a string value by removing the leading and trailing whitespace characters +`upper` | converts a string value to uppercase characters +`len` | converts a string to an integer value equal to the length of the string, also returns the length of an array if the value is an array +`splitn ` | splits a string by a delimiter string and returns the Nth token (1 based) +`eqs ` | returns `true` or `false` based on whether the value matches the parameter +`not` | returns `true` if value is `false` and `false` if the value is anything other than `false` +`split ` | returns array of values split on delimiter +`join ` | joins an array of strings by the delimiter +`float ` | converts a number to a floating point number with the specified precision (rounded) + +### Converter Examples ### + +The following examples will use the same input JSON value: + +```json +{ + "tests": { + "test_int": 500, + "test_float": 500.01, + "test_float2": 500.0, + "test_number": "750", + "test_fraction": 0.5, + "test_trim": " blah ", + "test_money": "$6,000,000", + "test_chars": "abcdefghijklmnopqrstuvwxyz", + "test_mapped": "Texas", + "test_null": null, + "pi": 3.141592653, + "test_true": true, + "test_false": false, + "test_null": null, + "test_string": "The quick brown fox", + "test_naics_code": "531312", + "test_split":"a|b|c", + "test_join":["a","b","c"] + }, + "test_bool": true +} +``` + +#### Add #### + +Adds a number to a number value + +Argument | Description +---------|------------ +Number | Number value to add + +example: +```json +{ + "operation": "shift", + "spec": { + "output": "tests.test_int | add 1" + } +} +``` + +produces: +```json +{ + "output": 501 +} +``` + +#### Ceil #### + +Converts a number value to the least closest integer greater than or equal to the number value + +Argument | Description +---------|------------ + + +example: +```json +{ + "operation": "shift", + "spec": { + "output": "tests.test_float | ceil" + } +} +``` + +produces: +```json +{ + "output": 501 +} +``` + +#### Div #### + +Divides a number value by another number value + +Argument | Description +---------|------------ +Number | dividend in a division operation + +example: +```json +{ + "operation": "shift", + "spec": { + "output1": "tests.test_float | div 2", + "output2": "tests.test_int | div .5" + } +} +``` + +produces: +```json +{ + "output1": 250, + "output2": 1000 +} +``` + +#### Floor #### + +Converts a number value to the greatest integer value less than or equal to the number value + +Argument | Description +---------|------------ + +example: +```json +{ + "operation": "shift", + "spec": { + "output": "tests.test_float | floor" + } +} +``` + +produces: +```json +{ + "output": 500 +} +``` + +#### Format #### + +Formats a value into a new string value using a **`fmt`** string + +Argument | Description +---------|------------ +string | fmt string, if whitespace shouldn't be trimmed, it should be escaped with \\ + +example: +```json +{ + "operation": "shift", + "spec": { + "output1": "tests.pi | format %.4f", + "output2": "tests.test_float | format %.0f", + "output3": "tests.test_string | format %s jumps over the lazy dog", + "output4": "tests.test_true | format %t is the value" + } +} +``` + +produces: +```json +{ + "output1": "3.1416", + "output2": "500", + "output3": "The quick brown fox jumps over the lazy dog", + "output4": "true is the value" +} +``` + +#### Lower #### + +Converts a string value to lowsercase + +Argument | Description +---------|------------ + +example: +```json +{ + "operation": "shift", + "spec": { + "output": "tests.test_string | lower" + } +} +``` + +produces: +```json +{ + "output": "the quick brown fox" +} +``` + +#### Mapped #### + +Maps a string value to another string value using a named JSON map defined in `$.converters.mapped` + +Argument | Description +---------|------------ +string | name of the map to use + +example: +```json +{ + "operation": "shift", + "converters": { + "mapped": { + "states": { + "Ohio": "OH", + "Texas": "TX" + } + } + }, + "spec": { + "output": "tests.test_mapped | mapped states" + } +} +``` + +produces: +```json +{ + "output": "TX" +} +``` + +#### Mul #### + +Multiplies a number value by another number value + +Argument | Description +---------|------------ +Number | multiplier value of a multiplication operation + +example: +```json +{ + "operation": "shift", + "spec": { + "output1": "tests.test_int | mul 2", + "output2": "tests.test_int | mul .5" + } +} +``` + +produces: +```json +{ + "output1": 1000, + "output2": 250 +} +``` + +#### Ntos #### + +Converts a number value to a string value + +Argument | Description +---------|------------ + +example: +```json +{ + "operation": "shift", + "spec": { + "output": "tests.test_int | ntos" + } +} +``` + +produces: +```json +{ + "output": "500" +} +``` + +#### Regex #### + +Use Regexp ReplaceAll to match and replace values defined in the `$.converters.regex` configuration object. You can +also pass an array of configuration objects and they will all be applied in order, stopping after the first match is matched and replaced. + +Argument | Description +---------|------------ +string | name of predefined regex match and replace + +example: +```json +{ + "operation": "shift", + "converters": { + "regex": { + "remove_dollar_sign": { + "match": "\\$\\s*(.*)", + "replace": "$1" + }, + "remove_comma": { + "match": ",", + "replace": "" + }, + "convert_naics": [ + { + "match": "^8111.*", + "replace": "automotive_services" + }, + { + "match": "^4413.*", + "replace": "automotive_services" + }, + { + "match": "^531.*", + "replace": "real_estate" + } + ] + } + }, + "spec": { + "output": "tests.test_money | regex remove_dollar_sign | regex remove_comma" + } +} +``` + +produces: +```json +{ + "output": "6000000" +} +``` + +#### Round #### + +Rounds a number value to the closest integer value + +Argument | Description +---------|------------ + +example: +```json +{ + "operation": "shift", + "spec": { + "output1": "tests.test_float | round", + "output2": "tests.test_fraction | round" + } +} +``` + +produces: +```json +{ + "output1": 500, + "output2": 1 +} +``` + +#### Ston #### + +Converts a string value to a number value + +Argument | Description +---------|------------ + +example: +```json +{ + "operation": "shift", + "spec": { + "output": "tests.test_number | ston" + } +} +``` + +produces: +```json +{ + "output": 750 +} +``` + +#### Substr #### + +Returns a substring of a string value + +Argument | Description +---------|------------ +number | 0 based index where to start the substring +number | (optional) index of last character + 1 in the substring, if omitted uses the string's length + +example: +```json +{ + "operation": "shift", + "spec": { + "output1": "tests.test_chars | substr 3 6", + "output2": "tests.test_string | substr 10" + } +} +``` + +produces: +```json +{ + "output1": "def", + "output2": "brown fox" +} +``` + +#### Trim #### + +Removes whitespace from the start and end of a string value + +Argument | Description +---------|------------ + +example: +```json +{ + "operation": "shift", + "spec": { + "output": "tests.test_trim | trim" + } +} +``` + +produces: +```json +{ + "output": "blah" +} +``` + +#### Upper #### + +Converts a string value to uppercase + +Argument | Description +---------|------------ + +example: +```json +{ + "operation": "shift", + "spec": { + "output": "tests.test_string | upper" + } +} +``` + +produces: +```json +{ + "output": "THE QUICK BROWN FOX" +} +``` + +#### Len #### + +Returns the length of a string value + +Argument | Description +---------|------------ + +example: +```json +{ + "operation": "shift", + "spec": { + "output": "tests.test_string | len" + } +} +``` + +produces: +```json +{ + "output": 19 +} +``` + +#### Splitn #### + +Returns the Nth token of a string split by a delimiter string + +Argument | Description +---------|------------ +string | delimiter string +number | one based position of token to return + +example: +```json +{ + "operation": "shift", + "spec": { + "output": "tests.test_string | splitn o 2" + } +} +``` + +produces: +```json +{ + "output": "wn f" +} +``` + +#### Eqs #### + +Returns `true` or `false` based on whether the value equals the parameter + +Argument | Description +---------|------------ +any | value to compare + + +example: +```json +{ + "operation": "shift", + "spec": { + "output": "tests.test_string | eqs \"The quick brown fox\"" + } +} +``` + +produces: +```json +{ + "output": true +} +``` + +#### Not #### + +Negates a `false` value returning `true` and returns `false` for everything else + +Argument | Description +---------|------------ + + + +example: +```json +{ + "operation": "shift", + "spec": { + "output1": "tests.test_true | not", + "output2": "tests.test_false | not" + } +} +``` + +produces: +```json +{ + "output1": false, + "output2": true +} +``` + +#### Split #### + + + +Argument | Description +---------|------------ +delim | string delimiter on which to split the string + + +example: +```json +{ + "operation": "shift", + "spec": { + "output1": "tests.test_split | split \\|" + } +} +``` + +produces: +```json +{ + "output1": ["a","b","c"] +} +``` + +#### Join #### + + + +Argument | Description +---------|------------ +delim | string delimiter on which to join the array into a string + + +example: +```json +{ + "operation": "shift", + "spec": { + "output1": "tests.test_join | join \\|" + } +} +``` + +produces: +```json +{ + "output1": "a|b|c" +} +``` + +#### Float #### + +Argument | Description +---------|------------ +precision | number of decimals (it will round) + + +example: +```json +{ + "operation": "shift", + "spec": { + "output1": "tests.test_float | float 1" + } +} +``` + +produces: +```json +{ + "output1": 500.0 +} +``` + + ## Usage To start, go get the versioned repository: diff --git a/converter/add.go b/converter/add.go new file mode 100644 index 0000000..70b1c2d --- /dev/null +++ b/converter/add.go @@ -0,0 +1,67 @@ +package converter + +import ( + "errors" + "github.com/mbordner/kazaam/transform" + "go/constant" + "go/token" +) + +type Add struct { + ConverterBase +} + +func (c *Add) Convert(jsonData []byte, value []byte, args []byte) (newValue []byte, err error) { + + var jsonValue, argsValue *transform.JSONValue + jsonValue, err = c.NewJSONValue(value) + if err != nil { + return + } + + argsValue, err = c.NewJSONValue(args) + if err != nil { + return + } + + if jsonValue.IsNumber() == false || argsValue.IsString() == false { + err = errors.New("invalid value or arguments for add converter") + return + } + + numStrVal := argsValue.GetStringValue() + if numStrVal[0] == '.' { + numStrVal = "0" + numStrVal + } + + // convert the string to number + argsValue, err = c.NewJSONValue([]byte(numStrVal)) + if err != nil { + return + } + + if argsValue.IsNumber() == false { + err = errors.New("arguments should be a number for add converter") + return + } + + var left, right constant.Value + + if jsonValue.GetType() == transform.JSONInt { + left = constant.MakeInt64(jsonValue.GetIntValue()) + } else { + left = constant.MakeFloat64(jsonValue.GetFloatValue()) + } + + if argsValue.GetType() == transform.JSONInt { + right = constant.MakeInt64(argsValue.GetIntValue()) + } else { + right = constant.MakeFloat64(argsValue.GetFloatValue()) + } + + result := constant.BinaryOp(left, token.ADD, right) + + newValue = []byte(result.String()) + + return +} diff --git a/converter/add_test.go b/converter/add_test.go new file mode 100644 index 0000000..7b294d3 --- /dev/null +++ b/converter/add_test.go @@ -0,0 +1,37 @@ +package converter + +import ( + "github.com/mbordner/kazaam/registry" + "strconv" + "testing" +) + +func TestAdd_Convert(t *testing.T) { + + registry.RegisterConverter("add", &Add{}) + c := registry.GetConverter("add") + + table := []struct { + value string + arguments string + expected string + }{ + {`5`, `1`, `6`,}, + {`5`, `-1`, `4`,}, + } + + for _, test := range table { + v, e := c.Convert(getTestData(), []byte(test.value), []byte(strconv.Quote(test.arguments))) + + if e != nil { + t.Error("error running convert function") + } + + if string(v) != test.expected { + t.Error("unexpected result from convert") + t.Log("Expected: {}", test.expected) + t.Log("Actual: {}", string(v)) + } + } + +} diff --git a/converter/ceil.go b/converter/ceil.go new file mode 100644 index 0000000..c29e094 --- /dev/null +++ b/converter/ceil.go @@ -0,0 +1,39 @@ +package converter + +import ( + "errors" + "github.com/mbordner/kazaam/transform" + "go/constant" + "math" +) + +type Ceil struct { + ConverterBase +} + +func (c *Ceil) Convert(jsonData []byte, value []byte, args []byte) (newValue []byte, err error) { + + newValue = value + + var jsonValue *transform.JSONValue + jsonValue, err = c.NewJSONValue(value) + if err != nil { + return + } + if jsonValue.IsNumber() == false { + err = errors.New("invalid value for ceil converter") + return + } + + if jsonValue.GetType() == transform.JSONInt { + return + } + + val := jsonValue.GetFloatValue() + + val = math.Ceil(val) + + newValue = []byte(constant.MakeInt64(int64(val)).String()) + + return +} diff --git a/converter/ceil_test.go b/converter/ceil_test.go new file mode 100644 index 0000000..a5be964 --- /dev/null +++ b/converter/ceil_test.go @@ -0,0 +1,36 @@ +package converter + +import ( + "github.com/mbordner/kazaam/registry" + "strconv" + "testing" +) + +func TestCeil_Convert(t *testing.T) { + registry.RegisterConverter("ceil", &Ceil{}) + c := registry.GetConverter("ceil") + + table := []struct { + value string + arguments string + expected string + }{ + {`5.1`, ``, `6`,}, + {`5.6`, ``, `6`,}, + {`0.01`, ``, `1`}, + } + + for _, test := range table { + v, e := c.Convert(getTestData(), []byte(test.value), []byte(strconv.Quote(test.arguments))) + + if e != nil { + t.Error("error running convert function") + } + + if string(v) != test.expected { + t.Error("unexpected result from convert") + t.Log("Expected: {}", test.expected) + t.Log("Actual: {}", string(v)) + } + } +} diff --git a/converter/div.go b/converter/div.go new file mode 100644 index 0000000..5b6e847 --- /dev/null +++ b/converter/div.go @@ -0,0 +1,67 @@ +package converter + +import ( + "errors" + "github.com/mbordner/kazaam/transform" + "go/constant" + "go/token" +) + +type Div struct { + ConverterBase +} + +func (c *Div) Convert(jsonData []byte, value []byte, args []byte) (newValue []byte, err error) { + + var jsonValue, argsValue *transform.JSONValue + jsonValue, err = c.NewJSONValue(value) + if err != nil { + return + } + + argsValue, err = c.NewJSONValue(args) + if err != nil { + return + } + + if jsonValue.IsNumber() == false || argsValue.IsString() == false { + err = errors.New("invalid value or arguments for div converter") + return + } + + numStrVal := argsValue.GetStringValue() + if numStrVal[0] == '.' { + numStrVal = "0" + numStrVal + } + + // convert the string to number + argsValue, err = c.NewJSONValue([]byte(numStrVal)) + if err != nil { + return + } + + if argsValue.IsNumber() == false { + err = errors.New("arguments should be a number for div converter") + return + } + + var left, right constant.Value + + if jsonValue.GetType() == transform.JSONInt { + left = constant.MakeInt64(jsonValue.GetIntValue()) + } else { + left = constant.MakeFloat64(jsonValue.GetFloatValue()) + } + + if argsValue.GetType() == transform.JSONInt { + right = constant.MakeInt64(argsValue.GetIntValue()) + } else { + right = constant.MakeFloat64(argsValue.GetFloatValue()) + } + + result := constant.BinaryOp(left, token.QUO, right) + + newValue = []byte(result.String()) + + return +} diff --git a/converter/div_test.go b/converter/div_test.go new file mode 100644 index 0000000..1e8cb71 --- /dev/null +++ b/converter/div_test.go @@ -0,0 +1,36 @@ +package converter + +import ( + "github.com/mbordner/kazaam/registry" + "strconv" + "testing" +) + +func TestDiv_Convert(t *testing.T) { + registry.RegisterConverter("div", &Div{}) + c := registry.GetConverter("div") + + table := []struct { + value string + arguments string + expected string + }{ + {`10`, `2`, `5`,}, + {`5`, `2`, `2.5`,}, + {`5`, `.5`, `10`}, + } + + for _, test := range table { + v, e := c.Convert(getTestData(), []byte(test.value), []byte(strconv.Quote(test.arguments))) + + if e != nil { + t.Error("error running convert function") + } + + if string(v) != test.expected { + t.Error("unexpected result from convert") + t.Log("Expected: {}", test.expected) + t.Log("Actual: {}", string(v)) + } + } +} \ No newline at end of file diff --git a/converter/eqs.go b/converter/eqs.go new file mode 100644 index 0000000..c32112c --- /dev/null +++ b/converter/eqs.go @@ -0,0 +1,28 @@ +package converter + +import ( + "bytes" + "github.com/mbordner/kazaam/transform" +) + +type Eqs struct { + ConverterBase +} + +func (c *Eqs) Convert(jsonData []byte, value []byte, args []byte) (newValue []byte, err error) { + + var argsValue *transform.JSONValue + argsValue, err = c.NewJSONValue(args) + if err != nil { + return + } + + argsBytes := []byte(argsValue.GetStringValue()) + + if bytes.Equal(value, argsBytes) == true { + return []byte("true"), nil + } + + return []byte("false"), nil + +} diff --git a/converter/eqs_test.go b/converter/eqs_test.go new file mode 100644 index 0000000..414e8d9 --- /dev/null +++ b/converter/eqs_test.go @@ -0,0 +1,37 @@ +package converter + +import ( + "strconv" + "testing" + "github.com/mbordner/kazaam/registry" +) + +func TestEqs_Convert(t *testing.T) { + + registry.RegisterConverter("eqs", &Eqs{}) + c := registry.GetConverter("eqs") + + table := []struct { + value string + arguments string + expected string + }{ + {`"The quick brown fox jumps over the lazy dog"`, `"The quick brown fox jumps over the lazy dog"`, `true`,}, + {`42`,`42`,`true`,}, + } + + for _, test := range table { + v, e := c.Convert(getTestData(), []byte(test.value), []byte(strconv.Quote(test.arguments))) + + if e != nil { + t.Error("error running convert function") + } + + if string(v) != test.expected { + t.Error("unexpected result from convert") + t.Log("Expected: {}", test.expected) + t.Log("Actual: {}", string(v)) + } + } + +} \ No newline at end of file diff --git a/converter/float.go b/converter/float.go new file mode 100644 index 0000000..8a491f0 --- /dev/null +++ b/converter/float.go @@ -0,0 +1,61 @@ +package converter + +import ( + "errors" + "github.com/mbordner/kazaam/transform" + "strconv" +) + +type Float struct { + ConverterBase +} + +func (c *Float) Convert(jsonData []byte, value []byte, args []byte) (newValue []byte, err error) { + + var jsonValue,argsValue *transform.JSONValue + jsonValue, err = c.NewJSONValue(value) + if err != nil { + return + } + + argsValue, err = c.NewJSONValue(args) + if err != nil { + return + } + + + if jsonValue.IsNumber() == false || argsValue.IsString() == false { + err = errors.New("invalid value or arguments for float converter") + return + } + + numStrVal := argsValue.GetStringValue() + if numStrVal[0] == '.' { + numStrVal = "0" + numStrVal + } + + // convert the string to number + argsValue, err = c.NewJSONValue([]byte(numStrVal)) + if err != nil { + return + } + + if argsValue.IsNumber() == false { + err = errors.New("arguments should be a number for float converter") + return + } + + precision := argsValue.GetIntValue() + + var val float64 + + if jsonValue.GetType() == transform.JSONInt { + val = float64(jsonValue.GetIntValue()) + } else { + val = jsonValue.GetFloatValue() + } + + newValue = []byte(strconv.FormatFloat(val, 'f', int(precision), 64)) + + return +} diff --git a/converter/float_test.go b/converter/float_test.go new file mode 100644 index 0000000..41078c3 --- /dev/null +++ b/converter/float_test.go @@ -0,0 +1,40 @@ +package converter + +import ( + "github.com/mbordner/kazaam/registry" + "strconv" + "testing" +) + +func TestFloat_Convert(t *testing.T) { + + registry.RegisterConverter("float", &Float{}) + c := registry.GetConverter("float") + + table := []struct { + value string + arguments string + expected string + }{ + {`5`, `1`, `5.0`,}, + {`5.01`, `2`, `5.01`,}, + {`5.012`,`1`,`5.0`}, + {`7.77`,`1`,`7.8`}, + {`500.01`,`1`,`500.0`}, + } + + for _, test := range table { + v, e := c.Convert(getTestData(), []byte(test.value), []byte(strconv.Quote(test.arguments))) + + if e != nil { + t.Error("error running convert function") + } + + if string(v) != test.expected { + t.Error("unexpected result from convert") + t.Log("Expected: {}", test.expected) + t.Log("Actual: {}", string(v)) + } + } + +} diff --git a/converter/floor.go b/converter/floor.go new file mode 100644 index 0000000..c67807c --- /dev/null +++ b/converter/floor.go @@ -0,0 +1,39 @@ +package converter + +import ( + "errors" + "github.com/mbordner/kazaam/transform" + "go/constant" + "math" +) + +type Floor struct { + ConverterBase +} + +func (c *Floor) Convert(jsonData []byte, value []byte, args []byte) (newValue []byte, err error) { + + newValue = value + + var jsonValue *transform.JSONValue + jsonValue, err = c.NewJSONValue(value) + if err != nil { + return + } + if jsonValue.IsNumber() == false { + err = errors.New("invalid value for floor converter") + return + } + + if jsonValue.GetType() == transform.JSONInt { + return + } + + val := jsonValue.GetFloatValue() + + val = math.Floor(val) + + newValue = []byte(constant.MakeInt64(int64(val)).String()) + + return +} diff --git a/converter/floor_test.go b/converter/floor_test.go new file mode 100644 index 0000000..2ca475a --- /dev/null +++ b/converter/floor_test.go @@ -0,0 +1,36 @@ +package converter + +import ( + "github.com/mbordner/kazaam/registry" + "strconv" + "testing" +) + +func TestFloor_Convert(t *testing.T) { + registry.RegisterConverter("floor", &Floor{}) + c := registry.GetConverter("floor") + + table := []struct { + value string + arguments string + expected string + }{ + {`5.1`, ``, `5`,}, + {`5.6`, ``, `5`,}, + {`0.01`, ``, `0`}, + } + + for _, test := range table { + v, e := c.Convert(getTestData(), []byte(test.value), []byte(strconv.Quote(test.arguments))) + + if e != nil { + t.Error("error running convert function") + } + + if string(v) != test.expected { + t.Error("unexpected result from convert") + t.Log("Expected: {}", test.expected) + t.Log("Actual: {}", string(v)) + } + } +} \ No newline at end of file diff --git a/converter/format.go b/converter/format.go new file mode 100644 index 0000000..aac6d35 --- /dev/null +++ b/converter/format.go @@ -0,0 +1,36 @@ +package converter + +import ( + "errors" + "fmt" + "github.com/mbordner/kazaam/transform" + "strconv" +) + +type Format struct { + ConverterBase +} + +func (c *Format) Convert(jsonData []byte, value []byte, args []byte) (newValue []byte, err error) { + newValue = value + + var jsonValue, argsValue *transform.JSONValue + jsonValue, err = c.NewJSONValue(value) + if err != nil { + return + } + + argsValue, err = transform.NewJSONValue(args) + if err != nil { + return + } + + if argsValue.IsString() == false { + err = errors.New("invalid value or arguments for substr converter") + return + } + + newValue = []byte(strconv.Quote(fmt.Sprintf(argsValue.GetStringValue(), jsonValue.GetValue()))) + + return +} diff --git a/converter/format_test.go b/converter/format_test.go new file mode 100644 index 0000000..d7ee0b4 --- /dev/null +++ b/converter/format_test.go @@ -0,0 +1,39 @@ +package converter + +import ( + "github.com/mbordner/kazaam/registry" + "strconv" + "testing" +) + +func TestFormat_Convert(t *testing.T) { + registry.RegisterConverter("format", &Format{}) + c := registry.GetConverter("format") + + table := []struct { + value string + arguments string + expected string + }{ + {`5.1`, `%.0f`, `"5"`,}, + {`5.23546`, `%.2f`, `"5.24"`,}, + {`0.01`, `%.4f`, `"0.0100"`}, + {`true`, `%t`, `"true"`}, + {`"the something fox"`, `%s jumped over something.`, `"the something fox jumped over something."`}, + {`42`, `%d`,`"42"`}, + } + + for _, test := range table { + v, e := c.Convert(getTestData(), []byte(test.value), []byte(strconv.Quote(test.arguments))) + + if e != nil { + t.Error("error running convert function") + } + + if string(v) != test.expected { + t.Error("unexpected result from convert") + t.Log("Expected: {}", test.expected) + t.Log("Actual: {}", string(v)) + } + } +} diff --git a/converter/join.go b/converter/join.go new file mode 100644 index 0000000..3a99dd6 --- /dev/null +++ b/converter/join.go @@ -0,0 +1,58 @@ +package converter + +import ( + "encoding/json" + "errors" + "github.com/mbordner/kazaam/transform" + "regexp" + "strconv" + "strings" +) + +type Join struct { + ConverterBase +} + +// |substr start end , end is optional, and will be the last char sliced's index + 1, +// start is the start index and required +func (c *Join) Convert(jsonData []byte, value []byte, args []byte) (newValue []byte, err error) { + newValue = value + + var stringValues []string + + err = json.Unmarshal(value,&stringValues) + if err != nil { + return + } + + argsValue, err := transform.NewJSONValue(args) + if err != nil { + return + } + + if argsValue.IsString() == false { + err = errors.New("invalid value or arguments for join converter") + return + } + + var re *regexp.Regexp + re, err = regexp.Compile(`(?Us)^(?:\s*)(.+)(?:\s*)$`) + if err != nil { + return + } + + argsString := argsValue.GetStringValue() + + newValue = []byte("null") + + if matches := re.FindStringSubmatch(argsString); matches != nil { + + val := strings.Join(stringValues,matches[1]) + + newValue = []byte(strconv.Quote(val)) + + } + + + return +} diff --git a/converter/join_test.go b/converter/join_test.go new file mode 100644 index 0000000..02d59c5 --- /dev/null +++ b/converter/join_test.go @@ -0,0 +1,36 @@ +package converter + +import ( + "github.com/mbordner/kazaam/registry" + "strconv" + "testing" +) + +func TestJoin_Convert(t *testing.T) { + + registry.RegisterConverter("join", &Join{}) + c := registry.GetConverter("join") + + table := []struct { + value string + arguments string + expected string + }{ + {`["a","b","c"]`, `|`, `"a|b|c"`,}, + } + + for _, test := range table { + v, e := c.Convert(getTestData(), []byte(test.value), []byte(strconv.Quote(test.arguments))) + + if e != nil { + t.Error("error running convert function") + } + + if string(v) != test.expected { + t.Error("unexpected result from convert") + t.Log("Expected: {}", test.expected) + t.Log("Actual: {}", string(v)) + } + } + +} \ No newline at end of file diff --git a/converter/len.go b/converter/len.go new file mode 100644 index 0000000..92280f5 --- /dev/null +++ b/converter/len.go @@ -0,0 +1,41 @@ +package converter + +import ( + "encoding/json" + "fmt" + "github.com/mbordner/kazaam/transform" + "github.com/pkg/errors" +) + +type Len struct { + ConverterBase +} + +func (c *Len) Convert(jsonData []byte, value []byte, args []byte) (newValue []byte, err error) { + newValue = value + + var arrayTest []interface{} + err = json.Unmarshal(value, &arrayTest) + if err == nil { + + newValue = []byte(fmt.Sprintf("%d", len(arrayTest))) + + } else { + var jsonValue *transform.JSONValue + jsonValue, err = c.NewJSONValue(value) + if err != nil { + return + } + + if jsonValue.IsString() == false { + err = errors.New("value must be string for len converter") + return + } + + origValue := jsonValue.GetStringValue() + + newValue = []byte(fmt.Sprintf("%d", len(origValue))) + } + + return +} diff --git a/converter/len_test.go b/converter/len_test.go new file mode 100644 index 0000000..af7a35f --- /dev/null +++ b/converter/len_test.go @@ -0,0 +1,38 @@ +package converter + +import ( + "github.com/mbordner/kazaam/registry" + "strconv" + "testing" +) + +func TestLen_Convert(t *testing.T) { + + registry.RegisterConverter("len", &Len{}) + c := registry.GetConverter("len") + + table := []struct { + value string + arguments string + expected string + }{ + {`"The quick brown fox jumps over the lazy dog"`, ``, `43`,}, + {`"the lazy dog"`, ``, `12`,}, + {`["one","two"]`,``,`2`}, + } + + for _, test := range table { + v, e := c.Convert(getTestData(), []byte(test.value), []byte(strconv.Quote(test.arguments))) + + if e != nil { + t.Error("error running convert function") + } + + if string(v) != test.expected { + t.Error("unexpected result from convert") + t.Log("Expected: {}", test.expected) + t.Log("Actual: {}", string(v)) + } + } + +} diff --git a/converter/lower.go b/converter/lower.go new file mode 100644 index 0000000..a8cb06b --- /dev/null +++ b/converter/lower.go @@ -0,0 +1,33 @@ +package converter + +import ( + "errors" + "github.com/mbordner/kazaam/transform" + "strconv" + "strings" +) + +type Lower struct { + ConverterBase +} + +func (c *Lower) Convert(jsonData []byte, value []byte, args []byte) (newValue []byte, err error) { + newValue = value + + var jsonValue *transform.JSONValue + jsonValue, err = c.NewJSONValue(value) + if err != nil { + return + } + + if jsonValue.IsString() == false { + err = errors.New("value must be string for lower converter") + return + } + + origValue := jsonValue.GetStringValue() + + newValue = []byte(strconv.Quote(strings.ToLower(origValue))) + + return +} diff --git a/converter/lower_test.go b/converter/lower_test.go new file mode 100644 index 0000000..ed95b9e --- /dev/null +++ b/converter/lower_test.go @@ -0,0 +1,34 @@ +package converter + +import ( + "github.com/mbordner/kazaam/registry" + "strconv" + "testing" +) + +func TestLower_Convert(t *testing.T) { + registry.RegisterConverter("lower", &Lower{}) + c := registry.GetConverter("lower") + + table := []struct { + value string + arguments string + expected string + }{ + {`"THIS IS A TEST"`, ``, `"this is a test"`,}, + } + + for _, test := range table { + v, e := c.Convert(getTestData(), []byte(test.value), []byte(strconv.Quote(test.arguments))) + + if e != nil { + t.Error("error running convert function") + } + + if string(v) != test.expected { + t.Error("unexpected result from convert") + t.Log("Expected: {}", test.expected) + t.Log("Actual: {}", string(v)) + } + } +} diff --git a/converter/mapped.go b/converter/mapped.go new file mode 100644 index 0000000..3249fee --- /dev/null +++ b/converter/mapped.go @@ -0,0 +1,53 @@ +package converter + +import ( + "encoding/json" + "errors" + "github.com/mbordner/kazaam/transform" + "strconv" +) + +var mappedSpecs map[string]map[string]string + +type Mapped struct { + ConverterBase +} + +func (c *Mapped) Init(config []byte) (err error) { + if err := json.Unmarshal(config, &mappedSpecs); err != nil { + mappedSpecs = make(map[string]map[string]string) + } + return +} + +func (c *Mapped) Convert(jsonData []byte, value []byte, args []byte) (newValue []byte, err error) { + + newValue = value + + var jsonValue, argsValue *transform.JSONValue + jsonValue, err = c.NewJSONValue(value) + if err != nil { + return + } + + argsValue, err = c.NewJSONValue(args) + if err != nil { + return + } + + if jsonValue.IsString() == false || argsValue.IsString() == false { + err = errors.New("invalid value or arguments for mapped converter") + return + } + + mappedCollectionName := argsValue.GetStringValue() + valueToMap := jsonValue.GetStringValue() + + if group, ok := mappedSpecs[mappedCollectionName]; ok { + if newValueStr, ok := group[valueToMap]; ok { + newValue = []byte(strconv.Quote(newValueStr)) + } + } + + return +} diff --git a/converter/mapped_test.go b/converter/mapped_test.go new file mode 100644 index 0000000..535d40d --- /dev/null +++ b/converter/mapped_test.go @@ -0,0 +1,44 @@ +package converter + +import ( + "github.com/mbordner/kazaam/registry" + "strconv" + "testing" +) + +func TestMapped_Convert(t *testing.T) { + registry.RegisterConverter("mapped", &Mapped{}) + c := registry.GetConverter("mapped") + + c.Init([]byte(` +{ + "states": { + "Ohio": "OH", + "Texas": "TX" + } +} +`)) + + table := []struct { + value string + arguments string + expected string + }{ + {`"Ohio"`,`states`,`"OH"`}, + {`"Kentucky"`,`states`,`"Kentucky"`}, + } + + for _, test := range table { + v, e := c.Convert(getTestData(), []byte(test.value), []byte(strconv.Quote(test.arguments))) + + if e != nil { + t.Error("error running convert function") + } + + if string(v) != test.expected { + t.Error("unexpected result from convert") + t.Log("Expected: {}", test.expected) + t.Log("Actual: {}", string(v)) + } + } +} \ No newline at end of file diff --git a/converter/mul.go b/converter/mul.go new file mode 100644 index 0000000..d0efd24 --- /dev/null +++ b/converter/mul.go @@ -0,0 +1,67 @@ +package converter + +import ( + "errors" + "github.com/mbordner/kazaam/transform" + "go/constant" + "go/token" +) + +type Mul struct { + ConverterBase +} + +func (c *Mul) Convert(jsonData []byte, value []byte, args []byte) (newValue []byte, err error) { + + var jsonValue, argsValue *transform.JSONValue + jsonValue, err = c.NewJSONValue(value) + if err != nil { + return + } + + argsValue, err = c.NewJSONValue(args) + if err != nil { + return + } + + if jsonValue.IsNumber() == false || argsValue.IsString() == false { + err = errors.New("invalid value or arguments for mul converter") + return + } + + numStrVal := argsValue.GetStringValue() + if numStrVal[0] == '.' { + numStrVal = "0" + numStrVal + } + + // convert the string to number + argsValue, err = c.NewJSONValue([]byte(numStrVal)) + if err != nil { + return + } + + if argsValue.IsNumber() == false { + err = errors.New("arguments should be a number for mul converter") + return + } + + var left, right constant.Value + + if jsonValue.GetType() == transform.JSONInt { + left = constant.MakeInt64(jsonValue.GetIntValue()) + } else { + left = constant.MakeFloat64(jsonValue.GetFloatValue()) + } + + if argsValue.GetType() == transform.JSONInt { + right = constant.MakeInt64(argsValue.GetIntValue()) + } else { + right = constant.MakeFloat64(argsValue.GetFloatValue()) + } + + result := constant.BinaryOp(left, token.MUL, right) + + newValue = []byte(result.String()) + + return +} diff --git a/converter/mul_test.go b/converter/mul_test.go new file mode 100644 index 0000000..cfe07f6 --- /dev/null +++ b/converter/mul_test.go @@ -0,0 +1,37 @@ +package converter + +import ( + "github.com/mbordner/kazaam/registry" + "strconv" + "testing" +) + +func TestMul_Convert(t *testing.T) { + registry.RegisterConverter("mul", &Mul{}) + c := registry.GetConverter("mul") + + table := []struct { + value string + arguments string + expected string + }{ + {`5`, `1`, `5`,}, + {`5`, `2`, `10`,}, + {`5`, `2.5`, `12.5`}, + {`10`, `.5`, `5`}, + } + + for _, test := range table { + v, e := c.Convert(getTestData(), []byte(test.value), []byte(strconv.Quote(test.arguments))) + + if e != nil { + t.Error("error running convert function") + } + + if string(v) != test.expected { + t.Error("unexpected result from convert") + t.Log("Expected: {}", test.expected) + t.Log("Actual: {}", string(v)) + } + } +} diff --git a/converter/not.go b/converter/not.go new file mode 100644 index 0000000..37eeddb --- /dev/null +++ b/converter/not.go @@ -0,0 +1,26 @@ +package converter + +import ( + "fmt" + "github.com/mbordner/kazaam/transform" +) + +type Not struct { + ConverterBase +} + +func (c *Not) Convert(jsonData []byte, value []byte, args []byte) (newValue []byte, err error) { + + var v *transform.JSONValue + v, err = c.NewJSONValue(value) + if err != nil { + return + } + + if v.IsBool() { + return []byte(fmt.Sprintf("%t", !v.GetBoolValue())), nil + } + + return []byte("false"), nil + +} diff --git a/converter/not_test.go b/converter/not_test.go new file mode 100644 index 0000000..5730986 --- /dev/null +++ b/converter/not_test.go @@ -0,0 +1,39 @@ +package converter + +import ( + "strconv" + "testing" + "github.com/mbordner/kazaam/registry" +) + +func TestNot_Convert(t *testing.T) { + + registry.RegisterConverter("not", &Not{}) + c := registry.GetConverter("not") + + table := []struct { + value string + arguments string + expected string + }{ + {`true"`, ``, `false`,}, + {`false`,``,`true`,}, + {`42"`, ``, `false`,}, + {`"42"`, ``, `false`,}, + } + + for _, test := range table { + v, e := c.Convert(getTestData(), []byte(test.value), []byte(strconv.Quote(test.arguments))) + + if e != nil { + t.Error("error running convert function") + } + + if string(v) != test.expected { + t.Error("unexpected result from convert") + t.Log("Expected: {}", test.expected) + t.Log("Actual: {}", string(v)) + } + } + +} \ No newline at end of file diff --git a/converter/ntos.go b/converter/ntos.go new file mode 100644 index 0000000..635580f --- /dev/null +++ b/converter/ntos.go @@ -0,0 +1,30 @@ +package converter + +import ( + "errors" + "github.com/mbordner/kazaam/transform" + "strconv" +) + +type Ntos struct { + ConverterBase +} + +func (c *Ntos) Convert(jsonData []byte, value []byte, args []byte) (newValue []byte, err error) { + var jsonValue *transform.JSONValue + jsonValue, err = c.NewJSONValue(value) + if err != nil { + return + } + + if jsonValue.IsString() { + newValue = value + } else if jsonValue.IsNumber() { + num := jsonValue.GetNumber() + newValue = []byte(strconv.Quote(num.String())) + } else { + err = errors.New("unexpected type") + } + + return +} diff --git a/converter/ntos_test.go b/converter/ntos_test.go new file mode 100644 index 0000000..0ccfbd1 --- /dev/null +++ b/converter/ntos_test.go @@ -0,0 +1,38 @@ +package converter + +import ( + "github.com/mbordner/kazaam/registry" + "strconv" + "testing" +) + +func TestNtos_Convert(t *testing.T) { + + registry.RegisterConverter("ntos", &Ntos{}) + c := registry.GetConverter("ntos") + + table := []struct { + value string + arguments string + expected string + }{ + {`5`, ``, `"5"`,}, + {`5.01`, ``, `"5.01"`,}, + {`-5.01`, ``, `"-5.01"`,}, + } + + for _, test := range table { + v, e := c.Convert(getTestData(), []byte(test.value), []byte(strconv.Quote(test.arguments))) + + if e != nil { + t.Error("error running convert function") + } + + if string(v) != test.expected { + t.Error("unexpected result from convert") + t.Log("Expected: {}", test.expected) + t.Log("Actual: {}", string(v)) + } + } + +} \ No newline at end of file diff --git a/converter/regex.go b/converter/regex.go new file mode 100644 index 0000000..5b94266 --- /dev/null +++ b/converter/regex.go @@ -0,0 +1,95 @@ +package converter + +import ( + "encoding/json" + "errors" + "github.com/mbordner/kazaam/transform" + "regexp" + "strconv" +) + +type regexSpecStruct struct { + Match *string `json:"match"` + Replace *string `json:"replace"` +} + +type regexSpecsList []regexSpecStruct + +func (rsl *regexSpecsList) UnmarshalJSON(config []byte) (err error) { + if config[0] == '{' { + spec := regexSpecStruct{} + err = json.Unmarshal(config, &spec) + if err != nil { + return err + } + *rsl = append(*rsl, spec) + return nil + } + + var list []regexSpecStruct + json.Unmarshal(config, &list) + + *rsl = regexSpecsList(list) + + return +} + +type regexSpecs map[string]regexSpecsList + +var specs regexSpecs + +type Regex struct { + ConverterBase +} + +func (r *Regex) Init(config []byte) (err error) { + specs = make(regexSpecs) + err = json.Unmarshal(config, &specs) + return +} + +func (r *Regex) Convert(jsonData []byte, value []byte, args []byte) (newValue []byte, err error) { + + var jsonValue, argsValue *transform.JSONValue + jsonValue, err = transform.NewJSONValue(value) + if err != nil { + return + } + + argsValue, err = transform.NewJSONValue(args) + if err != nil { + return + } + + if jsonValue.IsString() == false || argsValue.IsString() == false { + err = errors.New("invalid value or arguments for regex converter") + return + } + + reName := argsValue.GetStringValue() + if specs, ok := specs[reName]; ok { + for _, spec := range specs { + var re *regexp.Regexp + re, err = regexp.Compile(*spec.Match) + if err != nil { + return + } + + src := jsonValue.GetStringValue() + + if re.Match([]byte(src)) { + newValue = re.ReplaceAll([]byte(src), []byte(*spec.Replace)) + + newValue = []byte(strconv.Quote(string(newValue))) + break + } else { + newValue = []byte(strconv.Quote(string(src))) + } + } + } else { + err = errors.New("regex not defined") + return + } + + return +} diff --git a/converter/regex_test.go b/converter/regex_test.go new file mode 100644 index 0000000..8f0a221 --- /dev/null +++ b/converter/regex_test.go @@ -0,0 +1,69 @@ +package converter + +import ( + "github.com/mbordner/kazaam/registry" + "strconv" + "testing" +) + +func TestRegex_Convert(t *testing.T) { + registry.RegisterConverter("regex", &Regex{}) + c := registry.GetConverter("regex") + + c.Init([]byte(` +{ + "remove_dollar_sign": { + "match": "\\$\\s*(.*)", + "replace": "$1" + }, + "remove_comma": { + "match": ",", + "replace": "" + }, + "convert_naics": [ + { + "match": "^8111.*", + "replace": "automotive_services" + }, + { + "match": "^4413.*", + "replace": "automotive_services" + }, + { + "match": "^531.*", + "replace": "real_estate" + }, + { + "match": "real_estate", + "replace": "did not stop when matched" + } + ] +} +`)) + + table := []struct { + value string + arguments string + expected string + }{ + {`"$5,000,000"`, `remove_dollar_sign`, `"5,000,000"`,}, + {`"5,000,000"`, `remove_comma`, `"5000000"`,}, + {`"500"`, `remove_comma`, `"500"`,}, + {`"531312"`, `convert_naics`, `"real_estate"`,}, + {`"999"`, `convert_naics`, `"999"`,}, + } + + for _, test := range table { + v, e := c.Convert(getTestData(), []byte(test.value), []byte(strconv.Quote(test.arguments))) + + if e != nil { + t.Error("error running convert function") + } + + if string(v) != test.expected { + t.Error("unexpected result from convert") + t.Log("Expected: {}", test.expected) + t.Log("Actual: {}", string(v)) + } + } +} diff --git a/converter/round.go b/converter/round.go new file mode 100644 index 0000000..1e98b99 --- /dev/null +++ b/converter/round.go @@ -0,0 +1,39 @@ +package converter + +import ( + "errors" + "github.com/mbordner/kazaam/transform" + "go/constant" + "math" +) + +type Round struct { + ConverterBase +} + +func (c *Round) Convert(jsonData []byte, value []byte, args []byte) (newValue []byte, err error) { + + newValue = value + + var jsonValue *transform.JSONValue + jsonValue, err = c.NewJSONValue(value) + if err != nil { + return + } + if jsonValue.IsNumber() == false { + err = errors.New("invalid value for round converter") + return + } + + if jsonValue.GetType() == transform.JSONInt { + return + } + + val := jsonValue.GetFloatValue() + + val = math.Round(val) + + newValue = []byte(constant.MakeInt64(int64(val)).String()) + + return +} diff --git a/converter/round_test.go b/converter/round_test.go new file mode 100644 index 0000000..d9af329 --- /dev/null +++ b/converter/round_test.go @@ -0,0 +1,36 @@ +package converter + +import ( + "github.com/mbordner/kazaam/registry" + "strconv" + "testing" +) + +func TestRound_Convert(t *testing.T) { + registry.RegisterConverter("round", &Round{}) + c := registry.GetConverter("round") + + table := []struct { + value string + arguments string + expected string + }{ + {`5.1`, ``, `5`,}, + {`5.6`, ``, `6`,}, + {`0.01`, ``, `0`}, + } + + for _, test := range table { + v, e := c.Convert(getTestData(), []byte(test.value), []byte(strconv.Quote(test.arguments))) + + if e != nil { + t.Error("error running convert function") + } + + if string(v) != test.expected { + t.Error("unexpected result from convert") + t.Log("Expected: {}", test.expected) + t.Log("Actual: {}", string(v)) + } + } +} \ No newline at end of file diff --git a/converter/split.go b/converter/split.go new file mode 100644 index 0000000..47b1375 --- /dev/null +++ b/converter/split.go @@ -0,0 +1,57 @@ +package converter + +import ( + "encoding/json" + "errors" + "github.com/mbordner/kazaam/transform" + "regexp" + "strings" +) + +type Split struct { + ConverterBase +} + +// |substr start end , end is optional, and will be the last char sliced's index + 1, +// start is the start index and required +func (c *Split) Convert(jsonData []byte, value []byte, args []byte) (newValue []byte, err error) { + newValue = value + + var jsonValue, argsValue *transform.JSONValue + jsonValue, err = c.NewJSONValue(value) + if err != nil { + return + } + + argsValue, err = transform.NewJSONValue(args) + if err != nil { + return + } + + if jsonValue.IsString() == false || argsValue.IsString() == false { + err = errors.New("invalid value or arguments for split converter") + return + } + + var re *regexp.Regexp + re, err = regexp.Compile(`(?Us)^(?:\s*)(.+)(?:\s*)$`) + if err != nil { + return + } + + argsString := argsValue.GetStringValue() + origValue := jsonValue.GetStringValue() + + newValue = []byte("null") + + if matches := re.FindStringSubmatch(argsString); matches != nil { + + vals := strings.Split(origValue,matches[1]) + + newValue, err = json.Marshal(vals) + + } + + + return +} diff --git a/converter/split_test.go b/converter/split_test.go new file mode 100644 index 0000000..8c6b2dc --- /dev/null +++ b/converter/split_test.go @@ -0,0 +1,36 @@ +package converter + +import ( + "github.com/mbordner/kazaam/registry" + "strconv" + "testing" +) + +func TestSplit_Convert(t *testing.T) { + + registry.RegisterConverter("split", &Split{}) + c := registry.GetConverter("split") + + table := []struct { + value string + arguments string + expected string + }{ + {`"a|b|c"`, `|`, `["a","b","c"]`,}, + } + + for _, test := range table { + v, e := c.Convert(getTestData(), []byte(test.value), []byte(strconv.Quote(test.arguments))) + + if e != nil { + t.Error("error running convert function") + } + + if string(v) != test.expected { + t.Error("unexpected result from convert") + t.Log("Expected: {}", test.expected) + t.Log("Actual: {}", string(v)) + } + } + +} diff --git a/converter/splitn.go b/converter/splitn.go new file mode 100644 index 0000000..9d0315e --- /dev/null +++ b/converter/splitn.go @@ -0,0 +1,65 @@ +package converter + +import ( + "errors" + "github.com/mbordner/kazaam/transform" + "regexp" + "strconv" + "strings" +) + +type Splitn struct { + ConverterBase +} + +// |substr start end , end is optional, and will be the last char sliced's index + 1, +// start is the start index and required +func (c *Splitn) Convert(jsonData []byte, value []byte, args []byte) (newValue []byte, err error) { + newValue = value + + var jsonValue, argsValue *transform.JSONValue + jsonValue, err = c.NewJSONValue(value) + if err != nil { + return + } + + argsValue, err = transform.NewJSONValue(args) + if err != nil { + return + } + + if jsonValue.IsString() == false || argsValue.IsString() == false { + err = errors.New("invalid value or arguments for splintn converter") + return + } + + var re *regexp.Regexp + re, err = regexp.Compile(`(?Us)^(?:\s*)(.+)(?:\s*)(\d+)*(?:\s*)$`) + if err != nil { + return + } + + argsString := argsValue.GetStringValue() + origValue := jsonValue.GetStringValue() + + var n int64 + + newValue = []byte("null") + + if matches := re.FindStringSubmatch(argsString); matches != nil { + n, err = strconv.ParseInt(matches[2], 10, 64) + if err != nil { + return + } + + vals := strings.SplitN(origValue,matches[1],int(n)+1) + + if len(vals) >= int(n) { + newValue = []byte(strconv.Quote(vals[int(n)-1])) + } + + } + + + return +} diff --git a/converter/splitn_test.go b/converter/splitn_test.go new file mode 100644 index 0000000..b4668e7 --- /dev/null +++ b/converter/splitn_test.go @@ -0,0 +1,38 @@ +package converter + +import ( + "github.com/mbordner/kazaam/registry" + "strconv" + "testing" +) + +func TestSplitn_Convert(t *testing.T) { + + registry.RegisterConverter("splitn", &Splitn{}) + c := registry.GetConverter("splitn") + + table := []struct { + value string + arguments string + expected string + }{ + {`"aazbbzcczdd"`, `z 4`, `"dd"`,}, + {`"abc|def|ghi|jkl|mno"`, `| 2`, `"def"`,}, + {"\"abc\\ndef\\nghi\\njkl\\nmno\"", "\n 5", `"mno"`,}, + } + + for _, test := range table { + v, e := c.Convert(getTestData(), []byte(test.value), []byte(strconv.Quote(test.arguments))) + + if e != nil { + t.Error("error running convert function") + } + + if string(v) != test.expected { + t.Error("unexpected result from convert") + t.Log("Expected: {}", test.expected) + t.Log("Actual: {}", string(v)) + } + } + +} diff --git a/converter/ston.go b/converter/ston.go new file mode 100644 index 0000000..1fa86f1 --- /dev/null +++ b/converter/ston.go @@ -0,0 +1,33 @@ +package converter + +import ( + "errors" + "github.com/mbordner/kazaam/transform" +) + +type Ston struct { + ConverterBase +} + +func (c *Ston) Convert(jsonData []byte, value []byte, args []byte) (newValue []byte, err error) { + var jsonValue *transform.JSONValue + jsonValue, err = c.NewJSONValue(value) + if err != nil { + return + } + + if jsonValue.IsNumber() { + newValue = value + } else if jsonValue.IsString() { + jsonValue, err = c.NewJSONValue([]byte(jsonValue.GetStringValue())) + if jsonValue.IsNumber() { + newValue = jsonValue.GetData() + } else { + err = errors.New("string doesn't parse to number") + } + } else { + err = errors.New("unexpected type") + } + + return +} diff --git a/converter/ston_test.go b/converter/ston_test.go new file mode 100644 index 0000000..e6823fd --- /dev/null +++ b/converter/ston_test.go @@ -0,0 +1,40 @@ +package converter + +import ( + "github.com/mbordner/kazaam/registry" + "strconv" + "testing" +) + +func TestSton_Convert(t *testing.T) { + registry.RegisterConverter("ston", &Ston{}) + c := registry.GetConverter("ston") + + table := []struct { + value string + arguments string + expected string + }{ + {`"5"`, ``, `5`,}, + {`"5.01"`, ``, `5.01`,}, + {`"-5.01"`, ``, `-5.01`,}, + {`"09"`, ``, `9`,}, + {`"000.001"`,``,`0.001`,}, + {`"-000.001"`,``,`-0.001`,}, + {`"-042"`,``,`-42`,}, + } + + for _, test := range table { + v, e := c.Convert(getTestData(), []byte(test.value), []byte(strconv.Quote(test.arguments))) + + if e != nil { + t.Error("error running convert function") + } + + if string(v) != test.expected { + t.Error("unexpected result from convert") + t.Log("Expected: {}", test.expected) + t.Log("Actual: {}", string(v)) + } + } +} diff --git a/converter/substr.go b/converter/substr.go new file mode 100644 index 0000000..a76581d --- /dev/null +++ b/converter/substr.go @@ -0,0 +1,63 @@ +package converter + +import ( + "errors" + "github.com/mbordner/kazaam/transform" + "regexp" + "strconv" +) + +type Substr struct { + ConverterBase +} + +// |substr start end , end is optional, and will be the last char sliced's index + 1, +// start is the start index and required +func (c *Substr) Convert(jsonData []byte, value []byte, args []byte) (newValue []byte, err error) { + newValue = value + + var jsonValue, argsValue *transform.JSONValue + jsonValue, err = c.NewJSONValue(value) + if err != nil { + return + } + + argsValue, err = transform.NewJSONValue(args) + if err != nil { + return + } + + if jsonValue.IsString() == false || argsValue.IsString() == false { + err = errors.New("invalid value or arguments for substr converter") + return + } + + var re *regexp.Regexp + re, err = regexp.Compile(`(?:\s*)(\d+)(?:\s*)(\d+)*(?:\s*)`) + if err != nil { + return + } + + argsString := argsValue.GetStringValue() + origValue := jsonValue.GetStringValue() + + var start, end int64 + end = int64(len(origValue)) + + if matches := re.FindStringSubmatch(argsString); matches != nil { + start, err = strconv.ParseInt(matches[1], 10, 64) + if err != nil { + return + } + if len(matches) > 2 { + end, err = strconv.ParseInt(matches[2], 10, 64) + if err != nil { + return + } + } + } + + newValue = []byte(strconv.Quote(origValue[start:end])) + + return +} diff --git a/converter/substr_test.go b/converter/substr_test.go new file mode 100644 index 0000000..dc143b0 --- /dev/null +++ b/converter/substr_test.go @@ -0,0 +1,34 @@ +package converter + +import ( + "github.com/mbordner/kazaam/registry" + "strconv" + "testing" +) + +func TestSubstr_Convert(t *testing.T) { + registry.RegisterConverter("substr", &Substr{}) + c := registry.GetConverter("substr") + + table := []struct { + value string + arguments string + expected string + }{ + {`"THIS IS A TEST"`, `1 4`, `"HIS"`,}, + } + + for _, test := range table { + v, e := c.Convert(getTestData(), []byte(test.value), []byte(strconv.Quote(test.arguments))) + + if e != nil { + t.Error("error running convert function") + } + + if string(v) != test.expected { + t.Error("unexpected result from convert") + t.Log("Expected: {}", test.expected) + t.Log("Actual: {}", string(v)) + } + } +} \ No newline at end of file diff --git a/converter/trim.go b/converter/trim.go new file mode 100644 index 0000000..4546f74 --- /dev/null +++ b/converter/trim.go @@ -0,0 +1,33 @@ +package converter + +import ( + "github.com/pkg/errors" + "github.com/mbordner/kazaam/transform" + "strconv" + "strings" +) + +type Trim struct { + ConverterBase +} + +func (c *Trim) Convert(jsonData []byte, value []byte, args []byte) (newValue []byte, err error) { + newValue = value + + var jsonValue *transform.JSONValue + jsonValue, err = c.NewJSONValue(value) + if err != nil { + return + } + + if jsonValue.IsString() == false { + err = errors.New("value must be string for trim converter") + return + } + + origValue := jsonValue.GetStringValue() + + newValue = []byte(strconv.Quote(strings.Trim(origValue, " \t"))) + + return +} diff --git a/converter/trim_test.go b/converter/trim_test.go new file mode 100644 index 0000000..a4b89f7 --- /dev/null +++ b/converter/trim_test.go @@ -0,0 +1,34 @@ +package converter + +import ( + "github.com/mbordner/kazaam/registry" + "strconv" + "testing" +) + +func TestTrim_Convert(t *testing.T) { + registry.RegisterConverter("trim", &Trim{}) + c := registry.GetConverter("trim") + + table := []struct { + value string + arguments string + expected string + }{ + {`" THIS IS A TEST "`, ``, `"THIS IS A TEST"`,}, + } + + for _, test := range table { + v, e := c.Convert(getTestData(), []byte(test.value), []byte(strconv.Quote(test.arguments))) + + if e != nil { + t.Error("error running convert function") + } + + if string(v) != test.expected { + t.Error("unexpected result from convert") + t.Log("Expected: {}", test.expected) + t.Log("Actual: {}", string(v)) + } + } +} \ No newline at end of file diff --git a/converter/upper.go b/converter/upper.go new file mode 100644 index 0000000..191165d --- /dev/null +++ b/converter/upper.go @@ -0,0 +1,33 @@ +package converter + +import ( + "github.com/pkg/errors" + "github.com/mbordner/kazaam/transform" + "strconv" + "strings" +) + +type Upper struct { + ConverterBase +} + +func (c *Upper) Convert(jsonData []byte, value []byte, args []byte) (newValue []byte, err error) { + newValue = value + + var jsonValue *transform.JSONValue + jsonValue, err = c.NewJSONValue(value) + if err != nil { + return + } + + if jsonValue.IsString() == false { + err = errors.New("value must be string for upper converter") + return + } + + origValue := jsonValue.GetStringValue() + + newValue = []byte(strconv.Quote(strings.ToUpper(origValue))) + + return +} diff --git a/converter/upper_test.go b/converter/upper_test.go new file mode 100644 index 0000000..075fbe6 --- /dev/null +++ b/converter/upper_test.go @@ -0,0 +1,34 @@ +package converter + +import ( + "github.com/mbordner/kazaam/registry" + "strconv" + "testing" +) + +func TestUpper_Convert(t *testing.T) { + registry.RegisterConverter("upper", &Upper{}) + c := registry.GetConverter("upper") + + table := []struct { + value string + arguments string + expected string + }{ + {`"this is a test"`, ``, `"THIS IS A TEST"`,}, + } + + for _, test := range table { + v, e := c.Convert(getTestData(), []byte(test.value), []byte(strconv.Quote(test.arguments))) + + if e != nil { + t.Error("error running convert function") + } + + if string(v) != test.expected { + t.Error("unexpected result from convert") + t.Log("Expected: {}", test.expected) + t.Log("Actual: {}", string(v)) + } + } +} \ No newline at end of file diff --git a/converter/util.go b/converter/util.go new file mode 100644 index 0000000..2761d52 --- /dev/null +++ b/converter/util.go @@ -0,0 +1,24 @@ +package converter + +import ( + "github.com/mbordner/kazaam/transform" +) + +type ConverterBase struct{} + +func (c *ConverterBase) Init(config []byte) (err error) { + return +} + +func (c *ConverterBase) Convert(jsonData []byte, value []byte, args []byte) (newValue []byte, err error) { + newValue = value + return +} + +func (c *ConverterBase) GetJsonPathValue(jsonData []byte, path string) (value *transform.JSONValue, err error) { + return transform.GetJsonPathValue(jsonData, path) +} + +func (c *ConverterBase) NewJSONValue(data []byte) (value *transform.JSONValue, err error) { + return transform.NewJSONValue(data) +} diff --git a/converter/util_test.go b/converter/util_test.go new file mode 100644 index 0000000..6ba5f38 --- /dev/null +++ b/converter/util_test.go @@ -0,0 +1,21 @@ +package converter + +func getTestData() []byte { + data := []byte(` +{ + "tests": { + "test_int": 500, + "test_float": 500.01, + "test_float2": 500.0, + "test_fraction": 0.5, + "test_trim": " blah ", + "test_money": "$6,000,000", + "test_chars": "abcdefghijklmnopqrstuvwxyz", + "test_mapped": "Texas", + "test_null": null + }, + "test_bool": true +} +`) + return data +} diff --git a/kazaam.go b/kazaam.go index 7861e69..341711e 100644 --- a/kazaam.go +++ b/kazaam.go @@ -8,8 +8,10 @@ import ( "fmt" "strings" + "github.com/mbordner/kazaam/converter" + "github.com/mbordner/kazaam/registry" + "github.com/mbordner/kazaam/transform" "github.com/qntfy/jsonparser" - "github.com/qntfy/kazaam/transform" ) // TransformFunc defines the contract that any Transform function implementation @@ -23,10 +25,12 @@ import ( // Transforms should strive to fail gracefully whenever possible. type TransformFunc func(spec *transform.Config, data []byte) ([]byte, error) -var validSpecTypes map[string]TransformFunc +var defaultSpecTypes map[string]TransformFunc +var defaultConverters map[string]registry.Converter func init() { - validSpecTypes = map[string]TransformFunc{ + + defaultSpecTypes = map[string]TransformFunc{ "pass": transform.Pass, "shift": transform.Shift, "extract": transform.Extract, @@ -36,7 +40,39 @@ func init() { "coalesce": transform.Coalesce, "timestamp": transform.Timestamp, "uuid": transform.UUID, + "steps": transform.Steps, + "merge": transform.Merge, + } + + defaultConverters = map[string]registry.Converter{ + "ston": &converter.Ston{}, + "ntos": &converter.Ntos{}, + "regex": &converter.Regex{}, + "mapped": &converter.Mapped{}, + "upper": &converter.Upper{}, + "lower": &converter.Lower{}, + "trim": &converter.Trim{}, + "substr": &converter.Substr{}, + "add": &converter.Add{}, + "mul": &converter.Mul{}, + "round": &converter.Round{}, + "ceil": &converter.Ceil{}, + "floor": &converter.Floor{}, + "format": &converter.Format{}, + "div": &converter.Div{}, + "len": &converter.Len{}, + "splitn": &converter.Splitn{}, + "eqs": &converter.Eqs{}, + "not": &converter.Not{}, + "split": &converter.Split{}, + "join": &converter.Join{}, + "float": &converter.Float{}, } + + for name, conv := range defaultConverters { + registry.RegisterConverter(name, conv) + } + } // Error provides an error message (ErrMsg) and integer code (ErrType) for @@ -80,7 +116,7 @@ type Config struct { func NewDefaultConfig() Config { // make a copy, otherwise if new transforms are registered, they'll affect the whole package specTypes := make(map[string]TransformFunc) - for k, v := range validSpecTypes { + for k, v := range defaultSpecTypes { specTypes[k] = v } return Config{transforms: specTypes} @@ -178,6 +214,16 @@ func (k *Kazaam) Transform(data []byte) ([]byte, error) { return d, err } +func initConverters(config *map[string]interface{}) { + for name, converterConfig := range *config { + c := registry.GetConverter(name) + if c != nil { + bytes, _ := json.Marshal(converterConfig) + c.Init(bytes) + } + } +} + // TransformInPlace takes the byte slice `data`, transforms it according // to the loaded spec, and modifies the byte slice in place. // @@ -194,6 +240,11 @@ func (k *Kazaam) TransformInPlace(data []byte) ([]byte, error) { var err error for _, specObj := range k.specJSON { + + if specObj.ConvertersConfig != nil { + initConverters(specObj.ConvertersConfig) + } + if specObj.Config != nil && specObj.Over != nil { var transformedDataList [][]byte _, err = jsonparser.ArrayEach(data, func(value []byte, dataType jsonparser.ValueType, offset int, err error) { @@ -233,6 +284,7 @@ func (k *Kazaam) TransformInPlace(data []byte) ([]byte, error) { return data, transformErrorType(err) } } + } return data, transformErrorType(err) } diff --git a/kazaam/main.go b/kazaam/main.go index 0eefd74..fdeca4f 100644 --- a/kazaam/main.go +++ b/kazaam/main.go @@ -10,7 +10,7 @@ import ( "log" "os" - "github.com/qntfy/kazaam" + "github.com/mbordner/kazaam" ) var ( diff --git a/kazaam_benchmarks_test.go b/kazaam_benchmarks_test.go index 0eb807c..c291251 100644 --- a/kazaam_benchmarks_test.go +++ b/kazaam_benchmarks_test.go @@ -3,7 +3,7 @@ package kazaam_test import ( "testing" - "github.com/qntfy/kazaam" + "github.com/mbordner/kazaam" ) const ( diff --git a/kazaam_int_test.go b/kazaam_int_test.go index ee18355..f756451 100644 --- a/kazaam_int_test.go +++ b/kazaam_int_test.go @@ -6,8 +6,8 @@ import ( "testing" "github.com/qntfy/jsonparser" - "github.com/qntfy/kazaam" - "github.com/qntfy/kazaam/transform" + "github.com/mbordner/kazaam" + "github.com/mbordner/kazaam/transform" ) const testJSONInput = `{"rating":{"example":{"value":3},"primary":{"value":3}}}` diff --git a/kazaam_test.go b/kazaam_test.go index 261e636..df8f21d 100644 --- a/kazaam_test.go +++ b/kazaam_test.go @@ -5,7 +5,7 @@ import ( "testing" "github.com/qntfy/jsonparser" - "github.com/qntfy/kazaam/transform" + "github.com/mbordner/kazaam/transform" ) func TestDefaultKazaamGetUnknownTransform(t *testing.T) { @@ -38,7 +38,7 @@ func TestReregisterKazaamTransform(t *testing.T) { } func TestDefaultTransformsSetCardinarily(t *testing.T) { - if len(validSpecTypes) != 9 { + if len(defaultSpecTypes) != 11 { t.Error("Unexpected number of default transforms. Missing tests?") } } diff --git a/registry/registry.go b/registry/registry.go new file mode 100644 index 0000000..c607b9f --- /dev/null +++ b/registry/registry.go @@ -0,0 +1,31 @@ +package registry + +import "errors" + +type Converter interface { + Init(config []byte) (error) + Convert(jsonData []byte, value []byte, args []byte) (newValue []byte, err error) +} + +var converters map[string]Converter + +func init() { + converters = make(map[string]Converter) +} + +func RegisterConverter(name string, conv Converter) error { + _, ok := converters[name] + if ok { + return errors.New("converter with that name already registered") + } + converters[name] = conv + return nil +} + +func GetConverter(name string) Converter { + conv, ok := converters[name] + if ok { + return conv + } + return nil +} diff --git a/spec.go b/spec.go index 85d35df..49c496c 100644 --- a/spec.go +++ b/spec.go @@ -3,7 +3,7 @@ package kazaam import ( "encoding/json" - "github.com/qntfy/kazaam/transform" + "github.com/mbordner/kazaam/transform" ) // Spec represents an individual spec element. It describes the name of the operation, @@ -11,8 +11,9 @@ import ( // describes the configuration of the transform. type spec struct { *transform.Config - Operation *string `json:"operation"` - Over *string `json:"over,omitempty"` + Operation *string `json:"operation"` + Over *string `json:"over,omitempty"` + ConvertersConfig *map[string]interface{} `json:"converters"` } type specInt spec diff --git a/transform/expr.go b/transform/expr.go new file mode 100644 index 0000000..e57f956 --- /dev/null +++ b/transform/expr.go @@ -0,0 +1,299 @@ +package transform + +import ( + "errors" + "github.com/mbordner/kazaam/registry" + "go/ast" + "go/constant" + "go/parser" + "go/token" + "strconv" +) + +// support for basic expression evaluation support. +// NOTE: expressions must evaluate to a bool value, i.e. true or false +// supports +// ! - unary not operator +// && - binary logical and +// || - binary logical or +// == - binary equality +// != - binary equality negated +// < - binary less than +// > - binary greater than +// <= - binary less than or equal to +// >= - binary greater than or equal to +// ( ) - parentheses grouping +// true - boolean true constant +// false - boolean false constant +// - number constants +// - string constants ( "" ) +// - converter function call, f( jsonPath, stringArgs ) where, f is a converter id, jsonPath is a json path, and args are arguments to the converter +// - a json path.. does not support path parameters, i.e. ? conditionals and converter extensions +type BasicExpr struct { + fullExpression string + data []byte + astExpr ast.Expr +} + +// json data for evaluating json paths, and the expression string to evaluate to a boolean true/false constant value +// err will be returned if the expression can't be parsed +func NewBasicExpr(data []byte, exprStr string) (expr *BasicExpr, err error) { + astExpr, err := parser.ParseExpr(exprStr) + if err != nil { + return + } + + expr = new(BasicExpr) + + expr.data = data + expr.fullExpression = exprStr + expr.astExpr = astExpr + + return +} + +// returns true|false, otherwise error if the expression can not be evaluated +func (expr *BasicExpr) Eval() (val bool, err error) { + var evaluation constant.Value + evaluation, err = expr.evalExpr(expr.astExpr) + if err != nil { + return + } + val = constant.BoolVal(evaluation) + return +} + +func (expr *BasicExpr) evalBinaryExpr(exp *ast.BinaryExpr) (val constant.Value, err error) { + var left, right constant.Value + + left, err = expr.evalExpr(exp.X) + if err != nil { + return + } + + right, err = expr.evalExpr(exp.Y) + if err != nil { + return + } + + switch exp.Op { + // logical operators + case token.LOR: + fallthrough + case token.LAND: + + l := left.Kind() + r := right.Kind() + + if l != constant.Bool || r != constant.Bool { + err = errors.New("logical operators require bool values") + return + } + + val = constant.BinaryOp(left, exp.Op, right) + return + + // comparison operators + case token.GTR: + fallthrough + case token.LSS: + fallthrough + case token.EQL: + fallthrough + case token.NEQ: + fallthrough + case token.GEQ: + fallthrough + case token.LEQ: + + l := left.Kind() + r := right.Kind() + + if l != r && !(expr.isNumKind(l) && expr.isNumKind(r)) { + err = errors.New("comparison operators require types to be the same") + return + } + + val = constant.MakeBool(constant.Compare(left, exp.Op, right)) + return + } + + err = errors.New("unsupported operator") + + return +} + +func (expr *BasicExpr) isNumKind(k constant.Kind) bool { + return k == constant.Int || k == constant.Float +} + +func (expr *BasicExpr) evalExpr(exp ast.Expr) (val constant.Value, err error) { + switch exp := exp.(type) { + case *ast.UnaryExpr: + if exp.Op == token.NOT { + val, err = expr.evalExpr(exp.X) + if err != nil { + return + } + val = constant.MakeBool(!(constant.BoolVal(val))) + return + } + case *ast.ParenExpr: + val, err = expr.evalExpr(exp.X) + return + case *ast.BinaryExpr: + val, err = expr.evalBinaryExpr(exp) + return + case *ast.Ident: + if exp.Name == "true" || exp.Name == "false" { + val = constant.MakeBool(exp.Name == "true") + return + } else if exp.Name == "null" || exp.Name == "nil" { + val = constant.MakeUnknown() + return + } else { + // assumed to be a json path variable -- without selector syntax, e.g top level prop + val, err = expr.evalJsonPath(exp.Name) + return + } + case *ast.SelectorExpr: + pos := exp.Pos() + end := exp.End() + path := expr.fullExpression[ pos-1 : end-1 ] + val, err = expr.evalJsonPath(path) + return + case *ast.BasicLit: + switch exp.Kind { + case token.STRING: + val = constant.MakeFromLiteral(exp.Value, exp.Kind, 0) + return + case token.INT: + val = constant.MakeFromLiteral(exp.Value, exp.Kind, 0) + return + case token.FLOAT: + val = constant.MakeFromLiteral(exp.Value, exp.Kind, 0) + return + } + case *ast.CallExpr: + val, err = expr.evalConverterFunc(exp.Fun.(*ast.Ident).Name, exp.Args) + return + } + + err = errors.New("unsupported expression syntax") + + return +} + +func (expr *BasicExpr) evalJsonPath(path string) (val constant.Value, err error) { + var jsonPathSimpleValue *JSONValue + + jsonPathSimpleValue, err = GetJsonPathValue(expr.data, path) + if err != nil { + return + } + + switch jsonPathSimpleValue.valueType { + case JSONNull: + val = constant.MakeUnknown() + case JSONString: + val = constant.MakeString(jsonPathSimpleValue.GetStringValue()) + case JSONInt: + val = constant.MakeInt64(jsonPathSimpleValue.GetIntValue()) + case JSONFloat: + val = constant.MakeFloat64(jsonPathSimpleValue.GetFloatValue()) + case JSONBool: + val = constant.MakeBool(jsonPathSimpleValue.GetBoolValue()) + default: + err = errors.New("unsupported type") + } + + return +} + +func (expr *BasicExpr) evalConverterFunc(name string, args []ast.Expr) (val constant.Value, err error) { + var jsonPath, converterArgs string + + if len(args) > 0 { + argValues := make([]constant.Value, 0, len(args)) + for _, a := range args { + v, e := expr.evalExpr(a) + if e != nil { + err = e + return + } + argValues = append(argValues, v) + } + if argValues[0].Kind() == constant.String { + jsonPath = constant.StringVal(argValues[0]) + } else { + err = errors.New("expected json path as string") + return + } + if len(args) > 1 { + if argValues[1].Kind() == constant.String { + converterArgs = constant.StringVal(argValues[1]) + } else { + err = errors.New("expected converter arguments as string") + return + } + } + } else { + err = errors.New("expected path string, and optional arguments as string") + return + } + + conv := registry.GetConverter(name) + if conv != nil { + + var jsonPathBytes, converterArgsBytes []byte + + // get json path (string)'s value as simple value + jsonPathValue, e := GetJsonPathValue(expr.data, jsonPath) + if e != nil { + err = e + return + } + + // get the bytes for this value + jsonPathBytes = jsonPathValue.GetData() + + if len(converterArgs) > 0 { + converterArgsBytes = []byte(strconv.Quote(converterArgs)) + } + + // run them through the converter, and get the new value's bytes + var newValue []byte + newValue, err = conv.Convert(expr.data, jsonPathBytes, converterArgsBytes) + if err != nil { + return + } + + // convert to simple value + var simpleValue *JSONValue + simpleValue, err = NewJSONValue(newValue) + if err != nil { + return + } + + switch simpleValue.valueType { + case JSONNull: + val = constant.MakeUnknown() + case JSONString: + val = constant.MakeString(simpleValue.GetStringValue()) + case JSONInt: + val = constant.MakeInt64(simpleValue.GetIntValue()) + case JSONFloat: + val = constant.MakeFloat64(simpleValue.GetFloatValue()) + case JSONBool: + val = constant.MakeBool(simpleValue.GetBoolValue()) + default: + err = errors.New("unsupported type") + } + + } else { + val = constant.MakeUnknown() + err = errors.New("missing converter") + } + + return +} diff --git a/transform/expr_test.go b/transform/expr_test.go new file mode 100644 index 0000000..672d85c --- /dev/null +++ b/transform/expr_test.go @@ -0,0 +1,204 @@ +package transform + +import ( + "github.com/mbordner/kazaam/registry" + "testing" +) + +type ExprConverterTest struct{} + +func (c *ExprConverterTest) Init(config []byte) (err error) { + return +} +func (c *ExprConverterTest) Convert(jsonData []byte, value []byte, args []byte) (newValue []byte, err error) { + v,e := NewJSONValue(args) + if e != nil { + err = e + } else { + v,e := NewJSONValue([]byte(v.GetStringValue())) + if e != nil { + err = e + } else { + if v.GetType() == JSONString { + newValue = args + } else { + newValue =[]byte(v.String()) + } + } + } + return +} + +func TestExpressions(t *testing.T) { + + registry.RegisterConverter("exprConvA", &ExprConverterTest{}) + registry.RegisterConverter("exprConvB", &ExprConverterTest{}) + + data := []byte(` +{ + "tests": { + "test_int": 500, + "test_float": 500.01, + "test_float2": 500.0, + "test_fraction": 0.5, + "test_trim": " blah ", + "test_money": "$6,000,000", + "test_chars": "abcdefghijklmnopqrstuvwxyz", + "test_mapped": "Texas", + "test_null": null + }, + "test_bool": true +} +`) + + table := []struct { + expr string + expected bool + expectedError bool + }{ + { + ` ( tests.test_int == 500 ) `, + true, + false, + }, + { + `tests.test_int > 400 `, + true, + false, + }, + { + `tests.test_int < 400`, + false, + false, + }, + { + `tests.test_int >= 500`, + true, + false, + }, + { + `tests.test_int <= 500`, + true, + false, + }, + { + `!(tests.test_int != 500)`, + true, + false, + }, + { + `tests.test_float == 500 || tests.test_int == 500`, + true, + false, + }, + { + `tests.test_float2 == tests.test_int`, + true, + false, + }, + { + `true && false`, + false, + false, + }, + { + `false || true`, + true, + false, + }, + { + `"string1" != "string2"`, + true, + false, + }, + { + `"string1" == "string2" || (exprConvA("tests.test_mapped","\"Texas\"") == "Texas") && true || false`, + true, + false, + }, + { + `1 && 1`, + false, // && and || needs boolean expressions + true, + }, + { + `"string1" == 1`, + false, + true, + }, + { + `1 ^ 1`, + false, + true, + }, + { + `50.0 == 50`, + true, + false, + }, + { + `exprConvA("tests.test_int","null") == null`, + true, + false, + }, + { + `tests.test_null == null && tests.test_null == nil`, + true, + false, + }, + { + `tests.test_chars == "abcdefghijklmnopqrstuvwxyz"`, + true, + false, + }, + { + `exprConvA("tests.test_int","true") == true`, + true, + false, + }, + { + `exprConvA("tests.test_int","5") == 5`, + true, + false, + }, + { + `exprConvA("tests.test_int","5.01") == 5.01`, + true, + false, + }, + { + `blah("tests.test_int","test") == true`, + false, + true, + }, + { + `exprConvA("tests.test_int",1) == 1`, // will fail because the arguments are not a string, and this is required + false, + true, + }, + { + `exprConvA(1,1) == 1`, // will fail because the path is not a string, and this is required + false, + true, + }, + { + `exprConvA() == 1`, // will fail because expects path string + false, + true, + }, + } + + for _, test := range table { + be, err := NewBasicExpr(data, test.expr) + if err != nil { + t.Error("error parsing expression: {}", test.expr) + } + + if evaluation, err := be.Eval(); err != nil || evaluation != test.expected { + if err != nil && test.expectedError == false { + t.Error("unexpected expression evaluation value") + } + } + } + +} diff --git a/transform/merge.go b/transform/merge.go new file mode 100644 index 0000000..aa46973 --- /dev/null +++ b/transform/merge.go @@ -0,0 +1,86 @@ +package transform + +import ( + "encoding/json" +) + +// Merge joins multiple array values into array of objects with property names containing their matching array. Arrays +// should be the same length. +func Merge(spec *Config, data []byte) ([]byte, error) { + var outData []byte + + if spec.InPlace { + outData = data + } else { + outData = []byte(`{}`) + } + + // iterate through the spec + for k, v := range *spec.Spec { + + // map[prop_name] = [ values... ] + arrayVals := make(map[string][]interface{}) + outVals := make([]map[string]interface{}, 0) + + mergeSpec, ok := v.([]interface{}) + if !ok { + return nil, SpecError("Invalid Spec for Merge") + } + + l := 0 + + for i, v := range mergeSpec { + arraySpec := v.(map[string]interface{}) + var name, array string + name, ok = arraySpec["name"].(string) + if !ok { + return nil, SpecError("Array spec missing name for Merge") + } + array, ok = arraySpec["array"].(string) + if !ok { + return nil, SpecError("Array spec missing array for Merge") + } + + var dataForV []byte + var err error + + dataForV, err = getJSONRaw(data, array, true) + if err != nil { + return nil, err + } + + var arrayValues []interface{} + err = json.Unmarshal(dataForV, &arrayValues) + + arrayVals[name] = arrayValues + if i == 0 { + l = len(arrayValues) + } else if l != len(arrayValues) { + return nil, SpecError("Arrays must be the same length for Merge") + } + } + + for i := 0; i < l; i++ { + m := make(map[string]interface{}) + for k, v := range arrayVals { + m[k] = v[0] + arrayVals[k] = v[1:] + } + outVals = append(outVals, m) + } + + dataForV, err := json.Marshal(outVals) + if err != nil { + return nil, err + } + + outData, err = setJSONRaw(outData, dataForV, k) + if err != nil { + return nil, err + } + + } + + return outData, nil + +} diff --git a/transform/merge_test.go b/transform/merge_test.go new file mode 100644 index 0000000..4b34caf --- /dev/null +++ b/transform/merge_test.go @@ -0,0 +1,37 @@ +package transform + +import "testing" + +func TestMerge(t *testing.T) { + spec := `{"merge1":[{"name":"prop_1","array":"array_a"},{"name":"prop_2","array":"array_b"},{"name":"prop_3","array":"array_c"}]}` + jsonIn := `{"array_a":["a_1","a_2","a_3"],"array_b":["b_1","b_2","b_3"],"array_c":["c_1","c_2","c_3"]}` + jsonOut := `{"merge1":[{"prop_1":"a_1","prop_2":"b_1","prop_3":"c_1"},{"prop_1":"a_2","prop_2":"b_2","prop_3":"c_2"},{"prop_1":"a_3","prop_2":"b_3","prop_3":"c_3"}]}` + + cfg := getConfig(spec, false) + kazaamOut, _ := getTransformTestWrapper(Merge, cfg, jsonIn) + areEqual, _ := checkJSONBytesEqual(kazaamOut, []byte(jsonOut)) + + if !areEqual { + t.Error("Transformed data does not match expectation.") + t.Log("Expected: ", jsonOut) + t.Log("Actual: ", kazaamOut) + t.FailNow() + } +} + +func TestMergeSingleArray(t *testing.T) { + spec := `{"merge1":[{"name":"prop_1","array":"array_a"}]}` + jsonIn := `{"array_a":["a_1","a_2","a_3"]}` + jsonOut := `{"merge1":[{"prop_1":"a_1"},{"prop_1":"a_2"},{"prop_1":"a_3"}]}` + + cfg := getConfig(spec, false) + kazaamOut, _ := getTransformTestWrapper(Merge, cfg, jsonIn) + areEqual, _ := checkJSONBytesEqual(kazaamOut, []byte(jsonOut)) + + if !areEqual { + t.Error("Transformed data does not match expectation.") + t.Log("Expected: ", jsonOut) + t.Log("Actual: ", kazaamOut) + t.FailNow() + } +} diff --git a/transform/shift.go b/transform/shift.go index c99f75d..c4f7c79 100644 --- a/transform/shift.go +++ b/transform/shift.go @@ -50,6 +50,9 @@ func Shift(spec *Config, data []byte) ([]byte, error) { } else { dataForV, err = getJSONRaw(data, v, spec.Require) if err != nil { + if _, ok := err.(CPathSkipError); ok { // was a conditional path, + continue + } return nil, err } } diff --git a/transform/shift_test.go b/transform/shift_test.go index 4f34223..1904e1e 100644 --- a/transform/shift_test.go +++ b/transform/shift_test.go @@ -234,3 +234,19 @@ func TestShiftWithEndArrayAccess(t *testing.T) { t.FailNow() } } + +func TestSkipConditionalPath(t *testing.T) { + jsonOut := `{"Rating":3,"example":{"old":{"value":3}}}` + spec := `{"Rating": "rating.primary.value","example.old": "rating.example","conditional.skip":"path.not.found?"}` + + cfg := getConfig(spec, false) + kazaamOut, _ := getTransformTestWrapper(Shift, cfg, testJSONInput) + areEqual, _ := checkJSONBytesEqual(kazaamOut, []byte(jsonOut)) + + if !areEqual { + t.Error("Transformed data does not match expectation.") + t.Log("Expected: ", jsonOut) + t.Log("Actual: ", kazaamOut) + t.FailNow() + } +} diff --git a/transform/steps.go b/transform/steps.go new file mode 100644 index 0000000..0cad779 --- /dev/null +++ b/transform/steps.go @@ -0,0 +1,90 @@ +package transform + +import ( + "fmt" +) + +// Shift moves values from one provided json path to another in raw []byte. +func Steps(spec *Config, data []byte) ([]byte, error) { + var outData []byte + if spec.InPlace { + outData = data + } else { + outData = []byte(`{}`) + } + + if steps, ok := (*spec.Spec)["steps"]; ok { + + for _, s := range steps.([]interface{}) { + stepSpec := s.(map[string]interface{}) + + for k, v := range stepSpec { + array := true + var keyList []string + + // check if `v` is a string or list and build a list of keys to evaluate + switch v.(type) { + case string: + keyList = append(keyList, v.(string)) + array = false + case []interface{}: + for _, vItem := range v.([]interface{}) { + vItemStr, found := vItem.(string) + if !found { + return nil, ParseError(fmt.Sprintf("Warn: Unable to coerce element to json string: %v", vItem)) + } + keyList = append(keyList, vItemStr) + } + default: + return nil, ParseError(fmt.Sprintf("Warn: Unknown type in message for key: %s", k)) + } + + // iterate over keys to evaluate + // Note: this could be sped up significantly (especially for large shift transforms) + // by using `jsonparser.EachKey()` to iterate through data once and pick up all the + // needed underlying data. It would be a non-trivial update since you'd have to make + // recursive calls and keep track of all the key paths at each level. + // Currently we iterate at worst once per key in spec, with a better design it would be once + // per spec. + for _, v := range keyList { + var dataForV []byte + var err error + + // grab the data + if v == "$" { + dataForV = data + } else { + dataForV, err = getJSONRaw(data, v, spec.Require) + if err != nil { + if _, ok := err.(CPathSkipError); ok { // was a conditional path, + continue + } + return nil, err + } + } + + // if array flag set, encapsulate data + if array { + // bookend() is destructive to underlying slice, need to copy. + // extra capacity saves an allocation and copy during bookend. + tmp := make([]byte, len(dataForV), len(dataForV)+2) + copy(tmp, dataForV) + dataForV = bookend(tmp, '[', ']') + } + // Note: following pattern from current Shift() - if multiple elements are included in an array, + // they will each successively overwrite each other and only the last element will be included + // in the transformed data. + outData, err = setJSONRaw(outData, dataForV, k) + if err != nil { + return nil, err + } + } + } + + data = outData + } + + } + + return outData, nil +} diff --git a/transform/steps_test.go b/transform/steps_test.go new file mode 100644 index 0000000..26d6805 --- /dev/null +++ b/transform/steps_test.go @@ -0,0 +1,19 @@ +package transform + +import "testing" + +func TestSteps(t *testing.T) { + jsonOut := `{"example":{"old":{"value":3}},"Rating":3}` + spec := `{"steps":[{"example.old": "rating.example"},{"Rating": "example.old.value"}]}` + + cfg := getConfig(spec, false) + kazaamOut, _ := getTransformTestWrapper(Steps, cfg, `{"rating":{"example":{"value":3},"primary":{"value":3}}}`) + areEqual, _ := checkJSONBytesEqual(kazaamOut, []byte(jsonOut)) + + if !areEqual { + t.Error("Transformed data does not match expectation.") + t.Log("Expected: ", jsonOut) + t.Log("Actual: ", string(kazaamOut)) + t.FailNow() + } +} \ No newline at end of file diff --git a/transform/util.go b/transform/util.go index bbd9c2e..dd0d2f1 100644 --- a/transform/util.go +++ b/transform/util.go @@ -3,7 +3,10 @@ package transform import ( "bytes" + "encoding/json" + "errors" "fmt" + "github.com/mbordner/kazaam/registry" "regexp" "strconv" "strings" @@ -11,6 +14,9 @@ import ( "github.com/qntfy/jsonparser" ) +// go test "github.com/qntfy/kazaam/transform" -coverprofile cover.out +// go tool cover -html=cover.out -o cover.html + // ParseError should be thrown when there is an issue with parsing any of the specification or data type ParseError string @@ -18,6 +24,13 @@ func (p ParseError) Error() string { return string(p) } +// CPathNoDefaultError thrown when a path is missing, and marked conditional, and didn't have a default set +type CPathSkipError string + +func (c CPathSkipError) Error() string { + return string(c) +} + // RequireError should be thrown if a required key is missing in the data type RequireError string @@ -32,6 +45,200 @@ func (s SpecError) Error() string { return string(s) } +const ( + JSONNull = iota + JSONString + JSONInt + JSONFloat + JSONBool +) + +var ( + ConditionalPathSkip = CPathSkipError("Conditional Path missing and without a default value") + NonExistentPath = RequireError("Path does not exist") + jsonPathRe = regexp.MustCompile(`([^\[\]]+)\[(.*?)\]`) + leadingZeroRe = regexp.MustCompile(`^(-)*(0+)([\.1-9])`) + // matches converter groups separated by pipe delimiter characters (|), ignoring escaped delimiters + // (odd number of \ chars proceeding the delimiter) + // json path, e.g. path.path?conditional default|converter1 args|converter2 args + // allows also || to be ignored incase we need to support OR operators + // phoneNumbers[?(@.type =||= "iPhone")].number ? blah.blah == " blah:bla \" h " : "b\"l\tah" |convert||\|\\\|er1 "args args" |converter2 \ args\|\\\| args |converter3 2 2 + pathConverterSplitRe = regexp.MustCompile(`(?:\|)?(?:[^\||\\]*(?:(?:\|\|)|(?:\\(?:\\\\)*.?))*)*[^\|]`) // with forward scanning: (?<=(?|<|>=|<=)(?:\s*)(\w*|".*:.*")(?:\s*):(?:\s*)(.*?)(?:\s*)$`) // blah.blah == " blah:bla \" h " : "blah" + conditionMatchRe = regexp.MustCompile(`(?:\s*)(.*?)(?:\s*):(?:\s*)(\w*|".*")(?:\s*)$`) // blah.blah == " blah:bla \" h " : "b\"lah" + + // match the slashes, and the next character, to split and unescape characters + unescapeTokensRe = regexp.MustCompile(`(?:[^\\]+)|\\(.)`) + + // used to parse out converter name and arguments + converterParsingRe = regexp.MustCompile(`(?:^\|\s*)(\w+)(?:\s*)((?:.*?)(?:\\\s?)*)(?:\s*)$`) // | blah \ blah blah blah \ +) + +type JSONValue struct { + valueType int + value interface{} + num json.Number + data []byte + floatStrPrecision int +} + +func (v *JSONValue) GetType() int { + return v.valueType +} + +func (v *JSONValue) GetValue() interface{} { + return v.value +} + +func (v *JSONValue) GetStringValue() string { + return v.value.(string) +} + +func (v *JSONValue) GetQuotedStringValue() string { + return strconv.Quote(v.GetStringValue()) +} + +func (v *JSONValue) GetIntValue() int64 { + return v.value.(int64) +} + +func (v *JSONValue) GetFloatValue() float64 { + return v.value.(float64) +} + +func (v *JSONValue) GetBoolValue() bool { + return v.value.(bool) +} + +func (v *JSONValue) GetNumber() json.Number { + return v.num +} + +func (v *JSONValue) IsNumber() bool { + return v.valueType == JSONInt || v.valueType == JSONFloat +} + +func (v *JSONValue) IsBool() bool { + return v.valueType == JSONBool +} + +func (v *JSONValue) IsString() bool { + return v.valueType == JSONString +} + +func (v *JSONValue) IsNull() bool { + return v.valueType == JSONNull +} + +func (v *JSONValue) GetData() []byte { + return v.data +} + +func (v *JSONValue) SetFloatStringPrecision(p int) { + v.floatStrPrecision = p +} + +func (v *JSONValue) getFloatStringFormat() string { + if v.floatStrPrecision < 0 { + return "%f" + } + return fmt.Sprintf("%%.%df", v.floatStrPrecision) +} + +func (v *JSONValue) String() string { + switch v.valueType { + default: + fallthrough + case JSONNull: + return "null" + case JSONString: + return strconv.Quote(v.GetStringValue()) + case JSONBool: + return fmt.Sprintf("%t", v.GetBoolValue()) + case JSONInt: + return fmt.Sprintf("%d", v.GetIntValue()) + case JSONFloat: + return fmt.Sprintf(v.getFloatStringFormat(), v.GetFloatValue()) + } +} + +// returns a Value (json value type) from the json data byte array, and string path +func GetJsonPathValue(jsonData []byte, path string) (value *JSONValue, err error) { + var data []byte + + data, err = GetJSONRaw(jsonData, path, true) + if err != nil { + return + } + + value, err = NewJSONValue(data) + if err != nil { + return + } + + return +} + +// returns a Value (json value type) from the byte array data for a value +func NewJSONValue(data []byte) (value *JSONValue, err error) { + + // remove leading zeros that are invalid in jaon if it's a number value + tmp := string(data) + if m := leadingZeroRe.FindStringSubmatch(tmp); m != nil { + tmp = tmp[len(m[1])+len(m[2]):] + if rune(m[3][0]) == '.' { + m[2] = "0" + } else { + m[2] = "" + } + tmp = m[1] + m[2] + tmp + data = []byte(tmp) + } + + value = new(JSONValue) + value.data = data + value.floatStrPrecision = -1 + + reader := bytes.NewReader(data) + decoder := json.NewDecoder(reader) + decoder.UseNumber() + + if err = decoder.Decode(&value.value); err != nil { + return + } + + switch value.value.(type) { + case string: + value.valueType = JSONString + case bool: + value.valueType = JSONBool + case nil: + value.valueType = JSONNull + case json.Number: + value.num = value.value.(json.Number) + if strings.Contains(value.num.String(), ".") { + value.valueType = JSONFloat + value.value, err = value.num.Float64() + if err != nil { + return + } + } else { + value.valueType = JSONInt + value.value, err = value.num.Int64() + if err != nil { + return + } + } + } + + return +} + // Config contains the options that dictate the behavior of a transform. The internal // `spec` object can be an arbitrary json configuration for the transform. type Config struct { @@ -40,13 +247,230 @@ type Config struct { InPlace bool `json:"inplace,omitempty"` } -var ( - NonExistentPath = RequireError("Path does not exist") - jsonPathRe = regexp.MustCompile("([^\\[\\]]+)\\[(.*?)\\]") -) +// all it does is remove \ characters for now. +func unescapeString(s string) string { + if matches := unescapeTokensRe.FindAllStringSubmatch(s, -1); matches != nil { + tokens := make([]string, len(matches)) + for i, match := range matches { + if match[0][0] == '\\' { + tokens[i] = match[1] + } else { + tokens[i] = match[0] + } + } + return strings.Join(tokens, "") + } + return s +} + +type JSONPathConverter struct { + converter string + name string + arguments string +} + +func NewJSONPathConverter(converter string) *JSONPathConverter { // ||stoi \ no \| + params := new(JSONPathConverter) + params.converter = converter + if matches := converterParsingRe.FindStringSubmatch(converter); matches != nil { + if len(matches) > 1 { + params.name = matches[1] + if len(matches) > 2 { + params.arguments = strconv.Quote(unescapeString(matches[2])) // store as a quoted string, so that it will be unpacked as a string value in Convert + } + } + } + return params +} + +func (converter *JSONPathConverter) isValid() bool { + return len(converter.name) > 0 +} +func (converter *JSONPathConverter) getName() string { + return converter.name +} +func (converter *JSONPathConverter) getArguments() string { + return converter.arguments +} + +/* + [jsonPath]?[conditional]|[converter] + e.g. root.node1[1].property ? root.node0.prop1 == "true" | regex remove_commas + + the path will be broken into 3 components: + 1. the json parser path to the node value + 2. the conditional component + 3. and a series of converter expressions to pipe the value through. + + The conditional component has 4 forms: + + root.prop1 ? + // if the value exists, return it, otherwise skip it, regardless on + // whether paths are required. + root.prop1 ? defaultVal + // if value exists, use that, otherwise return the default value. this is JSON syntax + // so strings, will require double quotes. + root.prop1 ? root.node1.prop2 == true : + // if value exists, and the expression is true, return the existing value + // if the value exists, and the expression is false, skip the existing value + // note that the : is required here to end the expression. + root.prop1 ? root.node1.prop2 == true : defaultValue + // if the value exists, and the expression is true, return the existing value + // if the value exists and the expression is false, + // return the default value (JSON syntax) + + Conditional Expressions support () , <, >, >=, <=, ==, !=, !, true, false, && and || + Conditional expressions must evaluate to a boolean true or false value. + Identifiers in the expression are assumed to be JSON paths within the document, and will evaluate + to their current value. Only non composite JSON values are supported: boolean, number, + string (i.e. not arrays or objects). + + Function calls of the form ( json.path, "converter arguments") are also supported, e.g. + root.prop1 ? ston(root.node1.prop1) == 3 && regex(root.node2.prop2,"remove_commas) == "1000" : + + The converters component will define a series of value conversions +*/ +type JSONPathParameters struct { + data []byte + originalPath string // original path string + jsonPath string // parsed out and trimmed json path + + condition *BasicExpr + conditional bool + defaultValue []byte + conditionParseError bool + + converters []*JSONPathConverter +} + +func NewJSONPathParameters(data []byte, path string) *JSONPathParameters { + jsonPathParams := new(JSONPathParameters) + + jsonPathParams.data = data + jsonPathParams.originalPath = path + + // this is basically parsing out the converter tokens, but the first token will be the + // path along with any conditional/default value markup + tokens := pathConverterSplitRe.FindAllString(path, -1) + + // check if the first token contains a conditional declaration, path.path?conditional default + // when ? is after the path, it means it's conditional on the path existing, and will be skipped + // otherwise + if matches := conditionalMatchRe.FindStringSubmatch(tokens[0]); matches != nil { + jsonPathParams.conditional = true + + jsonPathParams.jsonPath = matches[1] + + if len(matches[2]) > 0 { + if conditionMatchRe.MatchString(matches[2]) { // looking for condition : default value + + if matches := conditionMatchRe.FindStringSubmatch(matches[2]); matches != nil { + + jsonPathParams.defaultValue = []byte(unescapeString(matches[2])) + + expr, e := NewBasicExpr(data, matches[1]) + if e == nil { + jsonPathParams.condition = expr + } else { + jsonPathParams.conditionParseError = true + } + } + + } else { // or just default value + jsonPathParams.defaultValue = []byte(unescapeString(matches[2])) + } + } + } else { + jsonPathParams.jsonPath = strings.Trim(tokens[0], " \t") + } + + jsonPathParams.converters = make([]*JSONPathConverter, 0, len(tokens)-1) + + // parse out the converter + if len(tokens) > 1 { + for _, token := range tokens[1:] { + converter := NewJSONPathConverter(token) + if converter.isValid() { + jsonPathParams.converters = append(jsonPathParams.converters, converter) + } + } + } + + return jsonPathParams +} + +func (params *JSONPathParameters) getJsonPath() string { + return params.jsonPath +} + +// means that ? existed in the json path +func (params *JSONPathParameters) isConditional() bool { + return params.conditional +} + +// means that there was an expression in the conditional component of the path, e.g. path ? : +func (params *JSONPathParameters) hasConditionExpression() bool { + return params.condition != nil || params.conditionParseError +} + +func (params *JSONPathParameters) evalConditionExpression() (results bool, err error) { + if params.conditionParseError == false { + return params.condition.Eval() + } + err = errors.New("condition parse error") + return +} + +// has a value after the ? in the conditional component, or a value after the : if there is a conditional expression +func (params *JSONPathParameters) hasDefaultValue() bool { + if params.defaultValue != nil && len(params.defaultValue) > 0 { + _, err := NewJSONValue([]byte(params.defaultValue)) + if err == nil { + return true + } + } + return false +} + +func (params *JSONPathParameters) getDefaultValue() ([]byte, error) { + if params.hasDefaultValue() { + return params.defaultValue, nil + } + return nil, errors.New("invalid or non existent default value") +} + +// converters will modify the value, and can be chained +func (params *JSONPathParameters) hasConverters() bool { + return len(params.converters) > 0 +} + +func (params *JSONPathParameters) convert(value []byte) (newValue []byte, err error) { + newValue = value + for _, c := range params.converters { + converter := registry.GetConverter(c.getName()) + var args []byte + if len(c.getArguments()) > 0 { + args = []byte(c.getArguments()) + } + newValue, err = converter.Convert(params.data, newValue, args) + if err != nil { + return + } + } + return +} + +func GetJSONRaw(data []byte, path string, pathRequired bool) ([]byte, error) { + return getJSONRaw(data, path, pathRequired) +} // Given a json byte slice `data` and a kazaam `path` string, return the object at the path in data if it exists. func getJSONRaw(data []byte, path string, pathRequired bool) ([]byte, error) { + jsonPathParams := NewJSONPathParameters(data, path) + return getProcessedJSONRaw(data, jsonPathParams.getJsonPath(), pathRequired, jsonPathParams) +} + +func getProcessedJSONRaw(data []byte, path string, pathRequired bool, params *JSONPathParameters) ([]byte, error) { objectKeys := strings.Split(path, ".") numOfInserts := 0 for element, k := range objectKeys { @@ -82,7 +506,7 @@ func getJSONRaw(data []byte, path string, pathRequired bool) ([]byte, error) { // GetJSONRaw() the rest of path for each element in results if newPath != "" { for i, value := range results { - intermediate, err := getJSONRaw(value, newPath, pathRequired) + intermediate, err := getProcessedJSONRaw(value, newPath, pathRequired, params) if err == jsonparser.KeyPathNotFoundError { if pathRequired { return nil, NonExistentPath @@ -127,15 +551,49 @@ func getJSONRaw(data []byte, path string, pathRequired bool) ([]byte, error) { result = []byte("null") } if err == jsonparser.KeyPathNotFoundError { - if pathRequired { + if params.isConditional() { + if params.hasDefaultValue() { + result, _ = params.getDefaultValue() + } else { + return nil, ConditionalPathSkip + } + } else if pathRequired { return nil, NonExistentPath } + } else if params.isConditional() && params.hasConditionExpression() { + // path exists, but there is a conditional expression + evaluation, err := params.evalConditionExpression() + if err == nil { + + if !evaluation { + if params.hasDefaultValue() { + result, _ = params.getDefaultValue() + } else { + return nil, ConditionalPathSkip + } + } + + } else if params.hasDefaultValue() { + result, _ = params.getDefaultValue() + } else { + return nil, ConditionalPathSkip // because there was an error, it should be evaluated as false + } + } else if err != nil { return nil, err } + + if params.hasConverters() { + result, err = params.convert(result) + } + return result, nil } +func SetJSONRaw(data, out []byte, path string) ([]byte, error) { + return setJSONRaw(data, out, path) +} + // setJSONRaw sets the value at a key and handles array indexing func setJSONRaw(data, out []byte, path string) ([]byte, error) { var err error @@ -208,6 +666,10 @@ func setJSONRaw(data, out []byte, path string) ([]byte, error) { return data, nil } +func DelJSONRaw(data []byte, path string, pathRequired bool) ([]byte, error) { + return delJSONRaw(data, path, pathRequired) +} + // delJSONRaw deletes the value at a path and handles array indexing func delJSONRaw(data []byte, path string, pathRequired bool) ([]byte, error) { var err error diff --git a/transform/util_test.go b/transform/util_test.go index 5ddebc7..3b07490 100644 --- a/transform/util_test.go +++ b/transform/util_test.go @@ -1,8 +1,11 @@ package transform import ( + "bytes" "encoding/json" + "github.com/mbordner/kazaam/registry" "reflect" + "strconv" "testing" ) @@ -164,3 +167,293 @@ func TestGetJSONRawBadIndex(t *testing.T) { t.FailNow() } } + +func TestGetJsonPathValue(t *testing.T) { + data := []byte(`{"data":{"subData":[{"key": "value"}, {"key": "value"}]}}`) + jv, err := GetJsonPathValue(data, `data.subData[0].key`) + + if err != nil { + t.Error("failed to get JSONValue") + t.FailNow() + } else { + if jv.IsString() == false { + t.Error("unexpected json value") + } + } +} + +func TestJSONValueParsing(t *testing.T) { + + table := []struct { + data string + dataType int + }{ + {`null`, JSONNull}, + {`"this is a string"`, JSONString}, + {`42`, JSONInt}, + {`3.14`, JSONFloat}, + {`true`, JSONBool}, + {`false`, JSONBool}, + } + + pi := `3.141592653` + jv, e := NewJSONValue([]byte(pi)) + if e != nil { + t.Error("failed parsing [{}]", pi) + t.FailNow() + } + + jv.SetFloatStringPrecision(4) + if jv.String() != "3.1416" { + t.Error("float print precision issue with rounding") + } + + for _, test := range table { + jv, e := NewJSONValue([]byte(test.data)) + if e != nil { + t.Error("failed parsing [{}]", test.data) + t.FailNow() + } else { + if jv.GetType() != test.dataType { + t.Error("json value data type mismatch") + t.Log("Exepcted: {}", test.dataType) + t.Log("Actual: {}", jv.GetType()) + } + + jv.SetFloatStringPrecision(2) + + valBytes, _ := json.Marshal(jv.GetValue()) + if bytes.Compare(valBytes, jv.GetData()) != 0 { + t.Error("original bytes data not matching json marshal") + } + + if string(valBytes) != test.data { + t.Error("GetValue didnt return the original value") + } + + if test.data != jv.String() { + t.Error("expected String() call to produce original string") + t.Log("Expected: {}", test.data) + t.Log("Actual: {}", jv.String()) + } + + switch jv.GetType() { + case JSONNull: + if !jv.IsNull() { + t.Error("null test function failed") + } + case JSONString: + if !jv.IsString() { + t.Error("string test function failed") + } + if jv.GetQuotedStringValue() != test.data { + t.Error("not returning expected quoted string value") + t.Log("Expected: {}", test.data) + t.Log("Actual: {}", jv.GetQuotedStringValue()) + } + tmp, _ := strconv.Unquote(test.data) + if jv.GetStringValue() != tmp { + t.Error("not returning expected string value") + t.Log("Expected: {}", test.data) + t.Log("Actual: {}", jv.GetStringValue()) + } + case JSONInt: + if !jv.IsNumber() { + t.Error("number test function failed for int") + } + tmp, _ := strconv.ParseInt(test.data, 10, 64) + if jv.GetIntValue() != tmp { + t.Error("not returning expected int value") + } + if jv.GetNumber().String() != jv.String() { + t.Error("expected number string to be same as jv string") + } + case JSONFloat: + if !jv.IsNumber() { + t.Error("number test function failed for int") + } + tmp, _ := strconv.ParseFloat(test.data, 64) + if jv.GetFloatValue() != tmp { + t.Error("not returning expected float value") + } + case JSONBool: + if !jv.IsBool() { + t.Error("bool test function failed") + } + if jv.GetBoolValue() && test.data == "false" || !jv.GetBoolValue() && test.data == "true" { + t.Error("failed getting bool value") + } + } + + } + } + +} + +func TestUnescapeString(t *testing.T) { + + s := `\ blah blah\ \n\t\\ \` + + if unescapeString(s) != ` blah blah nt\ ` { + t.Error("unexpected behavior from unescapeString") + } +} + +type ConverterTest struct{} + +func (c *ConverterTest) Init(config []byte) (err error) { + return +} +func (c *ConverterTest) Convert(jsonData []byte, value []byte, args []byte) (newValue []byte, err error) { + newValue = args + return +} + +func TestJsonPathParameters(t *testing.T) { + + registry.RegisterConverter("convA", &ConverterTest{}) + registry.RegisterConverter("convB", &ConverterTest{}) + + data := []byte(` +{ + "tests": { + "test_int": 500, + "test_float": 500.01, + "test_fraction": 0.5, + "test_trim": " blah ", + "test_money": "$6,000,000", + "test_chars": "abcdefghijklmnopqrstuvwxyz", + "test_mapped": "Texas", + "test_array": [ "one", "two" ], + "test_array2": [ { "one": 1, "two": 2 }, { "one": 1, "two": 2 } ] + }, + "test_bool": true +} +`) + + table := []struct { + path string + expectSkip bool + expected interface{} + }{ + { + `tests.test_array[1]?`, + false, + "two", + }, + { + `tests.test_array[2]?`, + true, + nil, + }, + { + `tests.test_array[2]?"three"`, + true, + "three", + }, + { + `tests.test_array2[1].one?`, + false, + 1, + }, + { + `tests.test_array2[3].one?4`, + true, + 4, + }, + { + `path.not.found?`, + true, + nil, + }, + { + `path.not.found? 1 `, + false, + 1, + }, + { + `path.not.found? -2.2`, + false, + -2.2, + }, + { + `path.not.found? "blah"`, + false, + "blah", + }, + { + `path.not.found? true`, + false, + true, + }, + { + `tests.test_float ? tests.test_int == 500 && convA("tests.test_trim","blah") == "bleh" : `, + true, + nil, + }, + { + `tests.test_float ? tests.test_int == 500 && convA("path.not.found","blah") == "blah" : "expression error, so returns default value, even though exists" `, + false, + "expression error, so returns default value, even though exists", + }, + { + `path.not.found ? invalid_expr( can't even parse : "default value because expression had syntax errors and is treated as false evaluation" `, + false, + "default value because expression had syntax errors and is treated as false evaluation", + }, + { + `tests.test_float ? invalid_expr( : `, + true, + nil, + }, + { + `path.not.found ? invalid_expr( but forgot colon so it's treated like default value, and skipped cause it's invalid json' `, + true, + nil, + }, + { + `tests.test_float ? (tests.test_int == 500 && convA("tests.test_trim","blah") == "bleh") && true : "default value" `, + false, + "default value", + }, + { + `tests.test_money ? (tests.test_int == 500 && test_bool ) : "$7,000,000" | convA test1 | convB test2`, + false, + "test2", + }, + { // white space is ignored around the arguments, unless escaped with a slash.. NOTE in json, this would require extra \ for escaping + `tests.test_money ? (tests.test_int == 500 && test_bool ) && (convA("tests.test_trim","blah") == "blah") : "$7,000,000" | convA test1 | convB \ test2 \ `, + false, + " test2 ", + }, + } + + for _, test := range table { + + val, err := GetJSONRaw(data, test.path, true) + + if err != nil { + if _, ok := err.(CPathSkipError); ok { // was a conditional path, and no default + if test.expectSkip == false { + t.Error("unexpected conditional skip error") + } + + if string(err.Error()) != "Conditional Path missing and without a default value" { + t.Error("unexpected cpath error message") + } + + } else { + t.Error("unexpected error parsing json path [{}]", test.path) + } + } else { + expextedBytes, _ := json.Marshal(test.expected) + + if bytes.Compare(val, expextedBytes) != 0 { + t.Error("value {} doesn't match expected {}", string(val), test.expected) + } + + } + + } + +}