|
@@ -1,506 +0,0 @@
|
|
|
-# Functions
|
|
|
-
|
|
|
-eKuiper has many built-in functions for performing calculations on data.
|
|
|
-
|
|
|
-## Aggregate Functions
|
|
|
-Aggregate functions perform a calculation on a set of values and return a single value. Aggregate functions can be used as expressions only in the following:
|
|
|
-* The select list of a SELECT statement (either a sub-query or an outer query).
|
|
|
-* A HAVING clause.
|
|
|
-
|
|
|
-| Function | Example | Description |
|
|
|
-|-----------------|----------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
|
|
-| avg | avg(col1) | The average of the values in a group. The null values will be ignored. |
|
|
|
-| count | count(*) | The number of items in a group. The null values will be ignored. |
|
|
|
-| max | max(col1) | The maximum value in a group. The null values will be ignored. |
|
|
|
-| min | min(col1) | The minimum value in a group. The null values will be ignored. |
|
|
|
-| sum | sum(col1) | The sum of all the values in a group. The null values will be ignored. |
|
|
|
-| collect | collect(*), collect(col1) | Returns an array with all column or the whole record (when the parameter is *) values from the group. |
|
|
|
-| deduplicate | deduplicate(col, false) | Returns the deduplicate results in the group, usually a window. The first argument is the column as the key to deduplicate; the second argument is whether to return all items or just the latest item which is not duplicate. If the latest item is a duplicate, the sink will receive an empty map. Set the sink property [omitIfEmpty](../guide/sinks/overview.md#common-properties) to the sink to not triggering the action. |
|
|
|
-| stddev | stddev(col) | Returns the population standard deviation of expression in the group, usually a window. The argument is the column as the key to stddev. |
|
|
|
-| stddevs | stddevs(col) | Returns the sample standard deviation of expression in the group, usually a window. The argument is the column as the key to stddevs. |
|
|
|
-| var | var(col) | Returns the population variance (square of the population standard deviation) of expression in the group, usually a window. The argument is the column as the key to var. |
|
|
|
-| vars | vars(col) | Returns the sample variance (square of the sample standard deviation) of expression in the group, usually a window. The argument is the column as the key to vars. |
|
|
|
-| percentile | percentile(col, percentile) | Returns the percentile value based on a continuous distribution of expression in the group, usually a window. The first argument is the column as the key to percentile. The second argument is the percentile of the value that you want to find. The percentile must be a constant between 0.0 and 1.0. |
|
|
|
-| percentile_disc | percentile_disc(col, percentile) | Returns the percentile value based on a discrete distribution of expression in the group, usually a window. The first argument is the column as the key to percentile_disc. The second argument is the percentile of the value that you want to find. The percentile must be a constant between 0.0 and 1.0. |
|
|
|
-
|
|
|
-### Collect() Examples
|
|
|
-
|
|
|
-- Get an array of column `a` of the current window. Assume the column `a` is of int type, the result will be like: `[{"r1":[32, 45]}]`
|
|
|
- ```sql
|
|
|
- SELECT collect(a) as r1 FROM test GROUP BY TumblingWindow(ss, 10)
|
|
|
- ```
|
|
|
-- Get the whole array of the current window. The result will be like: `[{"r1":{"a":32, "b":"hello"}, {"a":45, "b":"world"}}]`
|
|
|
- ```sql
|
|
|
- SELECT collect(*) as r1 FROM test GROUP BY TumblingWindow(ss, 10)
|
|
|
- ```
|
|
|
-
|
|
|
-- Get the second element's column 'a' value within the current window. The result will be like: `[{"r1":32}]`
|
|
|
- ```sql
|
|
|
- SELECT collect(*)[1]->a as r1 FROM test GROUP BY TumblingWindow(ss, 10)
|
|
|
- ```
|
|
|
-
|
|
|
-### Deduplicate() Examples
|
|
|
-
|
|
|
- - Get the whole array of the current window which is deduplicated by column `a`. The result will be like: `[{"r1":{"a":32, "b":"hello"}, {"a":45, "b":"world"}}]`
|
|
|
- ```sql
|
|
|
- SELECT deduplicate(a, true) as r1 FROM test GROUP BY TumblingWindow(ss, 10)
|
|
|
- ```
|
|
|
- - Get the column `a` value which is not duplicate during the last hour. The result will be like: `[{"r1":32}]`, `[{"r1":45}]` and `[{}]` if a duplicate value arrives. Use the omitIfEmpty sink property to filter out those empty results.
|
|
|
- ```sql
|
|
|
- SELECT deduplicate(a, false)->a as r1 FROM demo GROUP BY SlidingWindow(hh, 1)
|
|
|
- ```
|
|
|
-
|
|
|
-
|
|
|
-## Mathematical Functions
|
|
|
-| Function | Example | Description |
|
|
|
-|----------|--------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
|
|
-| abs | abs(col1) | The absolute value of a value |
|
|
|
-| acos | acos(col1) | The inverse cosine of a number in radians |
|
|
|
-| asin | asin(col1) | The inverse sine of a number in radians |
|
|
|
-| atan | atan(col1) | The inverse tangent of a number in radians |
|
|
|
-| atan2 | atan2(col1, col2) | The angle, in radians, between the positive x-axis and the (x, y) point defined in the two arguments |
|
|
|
-| bitand | bitand(col1, col2) | Performs a bitwise AND on the bit representations of the two Int(-converted) arguments |
|
|
|
-| bitor | bitor(col1, col2) | Performs a bitwise OR of the bit representations of the two arguments |
|
|
|
-| bitxor | bitxor(col1, col2) | Performs a bitwise XOR on the bit representations of the two Int(-converted) arguments |
|
|
|
-| bitnot | bitnot(col1) | Performs a bitwise NOT on the bit representations of the Int(-converted) argument |
|
|
|
-| ceil | ceil(col1) | Round a value up to the nearest BIGINT value. |
|
|
|
-| cos | cos(col1) | Returns the cosine of a number in radians. |
|
|
|
-| cosh | cosh(col1) | Returns the hyperbolic cosine of a number in radians. |
|
|
|
-| exp | exp(col1) | Returns e raised to the Decimal argument. |
|
|
|
-| ln | ln(col1) | Returns the natural logarithm of the argument. |
|
|
|
-| log | log(col1) | Returns the base 10 logarithm of the argument. |
|
|
|
-| mod | mod(col1, col2) | Returns the remainder of the division of the first argument by the second argument. |
|
|
|
-| power | power(x, y) | Pow returns x**y, the base-x exponential of y. |
|
|
|
-| rand | rand() | Returns a pseudorandom, uniformly distributed double between 0.0 and 1.0. |
|
|
|
-| round | round(col1) | Round a value to the nearest BIGINT value. |
|
|
|
-| sign | sign(col1) | Returns the sign of the given number. When the sign of the argument is positive, 1 is returned. When the sign of the argument is negative, -1 is returned. If the argument is 0, 0 is returned. |
|
|
|
-| sin | sin(col1) | Returns the sine of a number in radians. |
|
|
|
-| sinh | sinh(col1) | Returns the hyperbolic sine of a number in radians. |
|
|
|
-| sqrt | sqrt(col1) | Returns the square root of a number. |
|
|
|
-| tan | tan(col1) | Returns the tangent of a number in radians. |
|
|
|
-| tanh | tanh(col1) | Returns the hyperbolic tangent of a number in radians. |
|
|
|
-
|
|
|
-## String Functions
|
|
|
-
|
|
|
-| Function | Example | Description |
|
|
|
-|----------------|----------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
|
|
-| concat | concat(col1...) | Concatenates arrays or strings. This function accepts any number of arguments and returns a String or an Array |
|
|
|
-| endswith | endswith(col1, col2) | Returns a Boolean indicating whether the first String argument ends with the second String argument. |
|
|
|
-| format_time | format_time(col1, format) | Format a datetime to string. The 'col1' will be [casted to datetime type](#cast-to-datetime) if it is bigint, float or string type before formatting. Please check [format patterns](#formattime-patterns) for how to compose the format. |
|
|
|
-| indexof | indexof(col1, col2) | Returns the first index (0-based) of the second argument as a substring in the first argument. |
|
|
|
-| length | length(col1) | Returns the number of characters in the provided string. |
|
|
|
-| lower | lower(col1) | Returns the lowercase version of the given String. |
|
|
|
-| lpad | lpad(col1, 2) | Returns the String argument, padded on the left side with the number of spaces specified by the second argument. |
|
|
|
-| ltrim | ltrim(col1) | Removes all leading whitespace (tabs and spaces) from the provided String. |
|
|
|
-| numbytes | numbytes(col1) | Returns the number of bytes in the UTF-8 encoding of the provided string. |
|
|
|
-| regexp_matches | regexp_matches(col1, regex) | Returns true if the string (first argument) contains a match for the regular expression. |
|
|
|
-| regexp_replace | regexp_replace(col1, regex, str) | Replaces all occurrences of the second argument (regular expression) in the first argument with the third argument. |
|
|
|
-| regexp_substr | regexp_substr(col1, regex) | Finds the first match of the 2nd parameter (regex) in the first parameter. |
|
|
|
-| rpad | rpad(col1, 2) | Returns the String argument, padded on the right side with the number of spaces specified by the second argument. |
|
|
|
-| rtrim | rtrim(col1) | Removes all trailing whitespace (tabs and spaces) from the provided String. |
|
|
|
-| substring | substring(col1, start, end) | returns the substring of the provided String from the provided Int index (0-based, inclusive) to the end of the String. |
|
|
|
-| startswith | startswith(col1, str) | Returns Boolean, whether the first string argument starts with the second string argument. |
|
|
|
-| split_value | split_value(col1, str_splitter, index) | Split the value of the 1st parameter with the 2nd parameter, and return the value of split array that indexed with the 3rd parameter.<br />`split_value("/test/device001/message","/",0) AS a`, the returned value of function is empty; <br />`split_value("/test/device001/message","/",3) AS a`, the returned value of function is `message`; |
|
|
|
-| trim | trim(col1) | Removes all leading and trailing whitespace (tabs and spaces) from the provided String. |
|
|
|
-| upper | upper(col1) | Returns the uppercase version of the given String. |
|
|
|
-
|
|
|
-### Format_time patterns
|
|
|
-
|
|
|
-A pattern is used to create a format string. Patterns are based on a simple sequence of letters and symbols which is common in many languages like Java etc. The supported symbols in Kuiper are
|
|
|
-
|
|
|
-| Symbol | Meaning | Example |
|
|
|
-|--------|---------------------------|---------------------------------------|
|
|
|
-| G | era | G(AD) |
|
|
|
-| Y | year | YYYY(2004), YY(04) |
|
|
|
-| M | month | M(1), MM(01), MMM(Jan), MMMM(January) |
|
|
|
-| d | day of month | d(2), dd(02) |
|
|
|
-| E | day of week | EEE(Mon), EEEE(Monday) |
|
|
|
-| H | hour in 24 hours format | HH(15) |
|
|
|
-| h | hour in 12 hours format | h(2), hh(03) |
|
|
|
-| a | AM or PM | a(PM) |
|
|
|
-| m | minute | m(4), mm(04) |
|
|
|
-| s | second | s(5), ss(05) |
|
|
|
-| S | fraction of second | S(.0), SS(.00), SSS(.000) |
|
|
|
-| z | time zone name | z(MST) |
|
|
|
-| Z | 4 digits time zone offset | Z(-0700) |
|
|
|
-| X | time zone offset | X(-07), XX(-0700), XXX(-07:00) |
|
|
|
-
|
|
|
-Examples:
|
|
|
-
|
|
|
-- YYYY-MM-dd T HH:mm:ss -> 2006-01-02 T 15:04:05
|
|
|
-- YYYY/MM/dd HH:mm:ssSSS XXX -> 2006/01/02 15:04:05.000 -07:00
|
|
|
-
|
|
|
-## Conversion Functions
|
|
|
-
|
|
|
-| Function | Example | Description |
|
|
|
-|------------------|----------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
|
|
-| cast | cast(col, "bigint") | Converts a value from one data type to another. The supported types includes: bigint, float, string, boolean, bytea and datetime. |
|
|
|
-| chr | chr(col1) | Returns the ASCII character that corresponds to the given Int argument |
|
|
|
-| encode | encode(col1, "base64") | Use the encode function to encode the payload, which potentially might be non-JSON data, into its string representation based on the encoding scheme. Currently, only "base64" encoding type is supported. |
|
|
|
-| decode | decode(col1, "base64") | Decode the input string with specified decoding method. Currently, only "base64" encoding type is supported. |
|
|
|
-| trunc | trunc(dec, int) | Truncates the first argument to the number of Decimal places specified by the second argument. If the second argument is less than zero, it is set to zero. If the second argument is greater than 34, it is set to 34. Trailing zeroes are stripped from the result. |
|
|
|
-
|
|
|
-### Cast to datetime
|
|
|
-
|
|
|
-When casting to datetime type, the supported column type and casting rule are:
|
|
|
-
|
|
|
-1. If column is datetime type, just return the value.
|
|
|
-2. If column is bigint or float type, the number will be treated as the milliseconds elapsed since January 1, 1970 00:00:00 UTC and converted.
|
|
|
-3. If column is string, it will be parsed to datetime with the default format: `"2006-01-02T15:04:05.000Z07:00"`.
|
|
|
-4. Other types are not supported.
|
|
|
-
|
|
|
-## Hashing Functions
|
|
|
-| Function | Example | Description |
|
|
|
-|----------|--------------|------------------------------|
|
|
|
-| md5 | md5(col1) | Hashed value of the argument |
|
|
|
-| sha1 | sha1(col1) | Hashed value of the argument |
|
|
|
-| sha256 | sha256(col1) | Hashed value of the argument |
|
|
|
-| sha384 | sha384(col1) | Hashed value of the argument |
|
|
|
-| sha512 | sha512(col1) | Hashed value of the argument |
|
|
|
-
|
|
|
-## JSON Functions
|
|
|
-| Function | Example | Description |
|
|
|
-|-----------------------|---------------------------------------|------------------------------------------------------------------------------------------------------------------------------|
|
|
|
-| to_json | to_json(col1) | Converts a value to a string containing the JSON representation of the value. If the input is NULL, the result is also NULL. |
|
|
|
-| parse_json | parse_json(col1) | Converts a JSON string to a value. If the input is NULL, the result is also NULL. |
|
|
|
-| json_path_exists | json_path_exists(col1, "$.name") | Checks whether JSON path returns any item for the specified JSON value. Return bool value. |
|
|
|
-| json_path_query | json_path_query(col1, "$.name") | Gets all items returned by JSON path for the specified JSON value. |
|
|
|
-| json_path_query_first | json_path_query_first(col1, "$.name") | Gets the first item returned by JSON path for the specified JSON value. |
|
|
|
-
|
|
|
-**Please refer to [json path functions](./json_expr.md#json-path-functions) for how to compose a json path.**
|
|
|
-
|
|
|
-## Compression/Decompression Functions
|
|
|
-
|
|
|
-| Function | Example | Description |
|
|
|
-|------------|---------------------------|-----------------------------------------------------------------------|
|
|
|
-| compress | compress(input, "zlib") | Compress the input string or binary value with a compression method |
|
|
|
-| decompress | decompress(input, "zlib") | Decompress the input string or binary value with a compression method |
|
|
|
-
|
|
|
-Currently, 'zlib', 'gzip', 'flate' and 'zstd' method are supported.
|
|
|
-
|
|
|
-## Array Functions
|
|
|
-
|
|
|
-| Function | Example | Description |
|
|
|
-|---------------------|-----------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------|
|
|
|
-| array_position | array_position(array, value) | Return a 0-based index of the first occurrence of val if it is found within array. If val does not exist within array, it returns -1 |
|
|
|
-| element_at | element_at(array, index) | Returns element of array at index val. If val < 0, this function accesses elements from the last to the first |
|
|
|
-| array_contains | array_contains(array, value) | Returns true if array contains the element |
|
|
|
-| array_create | array_create(value1, ......) | Construct an array from literals |
|
|
|
-| array_remove | array_remove(array, value) | Returns the array with all occurrences of value removed |
|
|
|
-| array_last_position | array_last_position(array, val) | Return a 0-based index of the last occurrence of val if it is found within array. If val does not exist within array, it returns -1 |
|
|
|
-| array_contain_any | array_contain_any(array1, array2) | Returns true if array1 and array2 have any elements in common |
|
|
|
-| array_intersect | array_intersect(array1, array2) | Returns an intersection of the two arrays, with all duplicates removed |
|
|
|
-| array_union | array_union(array1, array2) | Returns a union of the two arrays, with all duplicates removed |
|
|
|
-| array_max | array_max(array) | Returns an element which is greater than or equal to all other elements of the array. If an element of the array is null, it returns null |
|
|
|
-| array_min | array_min(array) | Returns an element which is less than or equal to all other elements of the array. If an element of the array is null, it returns null |
|
|
|
-| array_except | array_except(array1, array2) | Returns an array of elements in array1 but not in array2, without duplicates |
|
|
|
-| repeat | repeat(val, count) | Constructs an array of val repeated count times |
|
|
|
-| sequence | sequence(start, stop[, step]) | Constructs an array from start to stop with each value increasing or decreasing by step. If step is not provided, it defaults to 1 if start is less than stop, or -1 if start is greater than stop. Notice that step is not allowed to be 0 |
|
|
|
-
|
|
|
-
|
|
|
-## Object Functions
|
|
|
-
|
|
|
-| Function | Example | Description |
|
|
|
-|------------------|----------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
|
|
-| keys | keys(map[string]interface{}) | Return an array containing the keys of the map |
|
|
|
-| values | values(map[string]interface{}) | Return an array containing the values of the map |
|
|
|
-| object | object(arr1, arr2) | Construct an object from an array of keys and an array of values. keys must be an array of strings. values must be an arbitrary array of the same length as keys. |
|
|
|
-| zip | zip([arr1, arr2], ......) | Construct an object from an array of entries. Each entry must itself be an array of size 2: the first element is the key (and must be a string), and the second element is the value. |
|
|
|
-| items | items(map[string]interface{}) | Return an array containing the entries of obj. Each entry is a 2-element array; the first is the key, the second is the value. |
|
|
|
-| object_construct | object_construct(key1, col, ...) | Return a struct type object/map constructed by the arguments. The arguments are series of key value pairs, thus the arguments count must be an odd number. The key must a string and the value can be of any type. If the value is null, the key/value pair will not present in the final object. |
|
|
|
-
|
|
|
-## Analytic Functions
|
|
|
-
|
|
|
-Analytic functions always use state to do analytic jobs. In streaming processing, analytic functions are evaluated first so that they are not affected by predicates in WHERE clause.
|
|
|
-
|
|
|
-Analytic function call format is, over is optional
|
|
|
-
|
|
|
-```
|
|
|
-AnalyticFuncName(<arguments>...) OVER ([PARTITION BY <partition key>] [WHEN <Expression>])
|
|
|
-```
|
|
|
-
|
|
|
-Analytic function computations are performed over all the input events of the current query input, optionally you can limit analytic function to only consider events that match the partition_by_clause.
|
|
|
-
|
|
|
-The syntax is like:
|
|
|
-
|
|
|
-```
|
|
|
-AnalyticFuncName(<arguments>...) OVER ([PARTITION BY <partition key>])
|
|
|
-```
|
|
|
-
|
|
|
-The analysis function can use the WHEN clause to determine whether the current event is a valid event based on whether the condition is met.
|
|
|
-When it is a valid event, calculate the result and update the state according to the analysis function semantics. When it is an invalid event, ignore the event value and reuse the saved state value.
|
|
|
-
|
|
|
-```
|
|
|
-AnalyticFuncName(<arguments>...) OVER ([WHEN <Expression>])
|
|
|
-```
|
|
|
-
|
|
|
-
|
|
|
-
|
|
|
-| Function | Example | Description |
|
|
|
-|-------------|--------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
|
|
-| lag | lag(expr, [offset], [default value]) | Return the former result of expression at offset, if not found, return the default value specified , if default value not set, return nil. if offset and default value not specified, offset is 1 and default value is nil |
|
|
|
-| latest | latest(expr, [default value]) | Return the lastest non null value of the expression. If not found, return the default value specified , if default value not set, return nil. |
|
|
|
-| changed_col | changed_col(true, col) | Return the column value if it has changed from the last execution. |
|
|
|
-| had_changed | had_changed(true, expr1, expr2, ...) | Return if any of the columns had changed since the last run. The expression could be * to easily detect the change status of all columns. |
|
|
|
-
|
|
|
-Example function call to get the previous temperature value:
|
|
|
-
|
|
|
-```text
|
|
|
-lag(temperature)
|
|
|
-```
|
|
|
-
|
|
|
-Example function call to get the previous temperature value with the same device id:
|
|
|
-
|
|
|
-```text
|
|
|
-lag(temperature) OVER (PARTITION BY deviceId)
|
|
|
-```
|
|
|
-
|
|
|
-Example function call to calculate duration of events: ts is timestamp, and statusCode1 and statusCode2 are device status in the same event
|
|
|
-
|
|
|
-```text
|
|
|
-select lag(Status) as Status, ts - lag(ts, 1, ts) OVER (WHEN had_changed(true, statusCode)) as duration from demo
|
|
|
-```
|
|
|
-
|
|
|
-## Other Functions
|
|
|
-| Function | Example | Description |
|
|
|
-|-----------------|--------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
|
|
-| isNull | isNull(col1) | Returns true if the argument is the Null value. |
|
|
|
-| coalesce | coalesce(expr1, expr2, ...) | Return the first non null value. If all expr are null,return nil. |
|
|
|
-| cardinality | cardinality(col1) | The number of members in the group. The null value is 0. |
|
|
|
-| newuuid | newuuid() | Returns a random 16-byte UUID. |
|
|
|
-| tstamp | tstamp() | Returns the current timestamp in milliseconds from 00:00:00 Coordinated Universal Time (UTC), Thursday, 1 January 1970 |
|
|
|
-| rule_id | rule_id() | Returns the ID of the currently matched rule |
|
|
|
-| mqtt | mqtt(topic) | Returns the MQTT meta-data of specified key. The current supported keys<br />- topic: return the topic of message. If there are multiple stream source, then specify the source name in parameter. Such as `mqtt(src1.topic)`<br />- messageid: return the message id of message. If there are multiple stream source, then specify the source name in parameter. Such as `mqtt(src2.messageid)` |
|
|
|
-| meta | meta(topic) | Returns the meta-data of specified key. The key could be:<br/> - a standalone key if there is only one source in the from clause, such as `meta(device)`<br />- A qualified key to specify the stream, such as `meta(src1.device)` <br />- A key with arrow for multi level meta data, such as `meta(src1.reading->device->name)` This assumes reading is a map structure meta data. |
|
|
|
-| window_start | window_start() | Return the window start timestamp in int64 format. If there is no time window, it returns 0. The window time is aligned with the timestamp notion of the rule. If the rule is using processing time, then the window start timestamp is the processing timestamp. If the rule is using event time, then the window start timestamp is the event timestamp. |
|
|
|
-| window_end | window_end() | Return the window end timestamp in int64 format. If there is no time window, it returns 0. The window time is aligned with the timestamp notion of the rule. If the rule is using processing time, then the window start timestamp is the processing timestamp. If the rule is using event time, then the window start timestamp is the event timestamp. |
|
|
|
-| get_keyed_state | get_keyed_state(expr1, expr2, expr3) | Return the keyed value in database. First parameter is the key, second is data format of the value, support bigint, float, string, boolean and datetime. third is the default value if key not exist. Default database is sqlite, users can change the database by this [configuration](../configuration/global_configurations.md#external-state). |
|
|
|
-| delay | delay(delayTime, returnVal) | Delay the execution of the rule for a specified time and then return the returnVal. |
|
|
|
-
|
|
|
-## Multiple Column Functions
|
|
|
-
|
|
|
-A multiple column function is a function that returns multiple columns. Contrast to normal scalar function, which returns a single column of a single row.
|
|
|
-
|
|
|
-Multiple column function can only be used in the `SELECT` clause of a query.
|
|
|
-
|
|
|
-| Function | Example | Description |
|
|
|
-|--------------|----------------------------------------------|-------------------------------------------------------------------------------------------------------------|
|
|
|
-| changed_cols | changed_cols(prefix, ignoreNull, colA, colB) | Return the changed columns whose name are prefixed. Check [changed_cols](#changedcols-function) for detail. |
|
|
|
-
|
|
|
-## Multiple Row Functions
|
|
|
-
|
|
|
-A multiple row function is a function that returns multiple rows.
|
|
|
-
|
|
|
-Multiple row function can only be used in the `SELECT` clause of a query and only allowed 1 multiple rows function in the clause for now.
|
|
|
-
|
|
|
-| Function | Example | Description |
|
|
|
-|----------|---------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
|
|
-| unnest | unnest(array) | The argument column must be an array. This function will expand the array into multiple rows as returned result. If the item in the array is map[string]interface object, then it will be built as columns in the result rows. |
|
|
|
-
|
|
|
-### Examples
|
|
|
-
|
|
|
-Create a stream demo and have below inputs
|
|
|
-
|
|
|
-```json lines
|
|
|
-{"a": [1,2], "b": 3}
|
|
|
-```
|
|
|
-
|
|
|
-Rule to get the unnest values:
|
|
|
-
|
|
|
-```text
|
|
|
-SQL: SELECT unnest(a) FROM demo
|
|
|
-___________________________________________________
|
|
|
-{"unnest":1}
|
|
|
-{"unnest":2}
|
|
|
-```
|
|
|
-
|
|
|
-Rule to get the unnest values with other columns:
|
|
|
-
|
|
|
-```text
|
|
|
-SQL: SELECT unnest(a), b FROM demo
|
|
|
-___________________________________________________
|
|
|
-{"unnest":1, "b":3}
|
|
|
-{"unnest":2, "b":3}
|
|
|
-```
|
|
|
-
|
|
|
-Create a stream demo and have below inputs
|
|
|
-
|
|
|
-```json lines
|
|
|
-{"x": [{"a": 1,"b": 2}, {"a": 3,"b": 4}], "c": 5}
|
|
|
-```
|
|
|
-
|
|
|
-Rule to get the unnest values with other columns:
|
|
|
-
|
|
|
-```text
|
|
|
-SQL: SELECT unnest(x), c FROM demo
|
|
|
-___________________________________________________
|
|
|
-{"a":1, "b":2, "c": 5}
|
|
|
-{"a":3, "b":4, "c": 5}
|
|
|
-```
|
|
|
-
|
|
|
-## Functions to detect changes
|
|
|
-
|
|
|
-### Changed_col function
|
|
|
-
|
|
|
-This function is a normal scalar function, so it can be used in any clause including SELECT and WHERE.
|
|
|
-
|
|
|
-**Syntax**
|
|
|
-
|
|
|
-```CHANGED_COL(<ignoreNull>, <expr>)```
|
|
|
-
|
|
|
-**Arguments**
|
|
|
-
|
|
|
-**ignoreNull**: whether to ignore null values when comparing for changes. If true, the null value won’t emit a change.
|
|
|
-
|
|
|
-**expr**: An expression to be selected and monitored for the changed status.
|
|
|
-
|
|
|
-**Returns**
|
|
|
-
|
|
|
-Return the changed value or nil with column name changed_col by default like any other functions. Use `as alias` to rename the column.
|
|
|
-
|
|
|
-### Changed_cols function
|
|
|
-
|
|
|
-This function returns multiple columns, so it is only allowed in the SELECT clause.
|
|
|
-
|
|
|
-**Syntax**
|
|
|
-
|
|
|
-```CHANGED_COLS (<prefix>, <ignoreNull>, <expr> [,...,<exprN>])```
|
|
|
-
|
|
|
-**Arguments**
|
|
|
-
|
|
|
-**prefix**: The prefix of the selected column name. By default, the selected name will be the same as select the expr directly. For example, `CHANGED_COLS("", true, col1)` will return `col1` as the name. If setting a prefix, the return name will have that prefix. For example, `CHANGED_COLS("changed_", true, col1)` will return `changed_col1` as the name.
|
|
|
-
|
|
|
-**ignoreNull**: whether to ignore null values when detecting changes. If true, the null value won’t trigger a change.
|
|
|
-
|
|
|
-**expr**: An expression to be selected and monitored for the changed status. Allow any expression that can be used in select clause. The expression can be a `*` which will return multiple columns by one expression.
|
|
|
-
|
|
|
-**Returns**
|
|
|
-
|
|
|
-Return all changed values compared to the previous sink result. So if using in a scalar rule, it will compare to the previous value emit. If using in a window, it will compare to the previous window result.
|
|
|
-
|
|
|
-In the first run, all expressions will be returned because there is no previous result.
|
|
|
-
|
|
|
-In the consequent runs, if nothing changed, it can emit nothing. And if the sink has the default omitEmpty, the sink will not be triggerred.
|
|
|
-
|
|
|
-**Notice**
|
|
|
-
|
|
|
-The multiple column outputs can only be used in the select clause. Even the selected result cannot be access in WHERE or other place. If filter based on the value is needed, use CHANGED_COL or set the result of multiple column outputs as the prior rule in a rule chain.
|
|
|
-
|
|
|
-For multiple column outputs, the alias can only be set generally with the prefix. To set alias for each column separately, try to call the changed function for each column respectively and use as to set alias.
|
|
|
-
|
|
|
-### Had_changed function
|
|
|
-
|
|
|
-This function is a scalar function with one or more arguments.
|
|
|
-
|
|
|
-```HAD_CHANGED (<ignoreNull>, <expr> [,...,<exprN>])```
|
|
|
-
|
|
|
-**Arguments**
|
|
|
-
|
|
|
-**ignoreNull**: whether to ignore null values when detecting changes. If true, the null value won’t trigger a change.
|
|
|
-
|
|
|
-**expr**: An expression to be monitored for the changed status. Allow any expression that can be used in select clause. The expression can be a `*` to detect changes of all columns easily.
|
|
|
-
|
|
|
-**Returns**
|
|
|
-
|
|
|
-Return a bool value to indicate the changed status if any of the arguments had changed since the last run. The multiple arguments' version is a handy way to check HAD_CHANGED(expr1) OR HAD_CHANGED(expr2) ... OR HAD_CHANGED(exprN). To detect other relationship, just use separate HAD_CHANGED functions. For example, to check if all expressions are changed HAD_CHANGED(expr1) AND HAD_CHANGED(expr2) ... AND HAD_CHANGED(exprN).
|
|
|
-
|
|
|
-### Examples
|
|
|
-
|
|
|
-Create a stream demo and have below inputs
|
|
|
-
|
|
|
-```json lines
|
|
|
-{"ts":1, "temperature":23, "humidity":88}
|
|
|
-{"ts":2, "temperature":23, "humidity":88}
|
|
|
-{"ts":3, "temperature":23, "humidity":88}
|
|
|
-{"ts":4, "temperature":25, "humidity":88}
|
|
|
-{"ts":5, "temperature":25, "humidity":90}
|
|
|
-{"ts":6, "temperature":25, "humidity":91}
|
|
|
-{"ts":7, "temperature":25, "humidity":91}
|
|
|
-{"ts":8, "temperature":25, "humidity":91}
|
|
|
-```
|
|
|
-
|
|
|
-Rule to get the changed temperature values:
|
|
|
-
|
|
|
-```text
|
|
|
-SQL: SELECT CHANGED_COLS("", true, temperature) FROM demo
|
|
|
-___________________________________________________
|
|
|
-{"temperature":23}
|
|
|
-{"temperature":25}
|
|
|
-```
|
|
|
-
|
|
|
-Rule to get the changed temperature and humidity values, and rename the changed value in a unified prefix:
|
|
|
-
|
|
|
-```text
|
|
|
-SQL: SELECT CHANGED_COLS("c_", true, temperature, humidity) FROM demo
|
|
|
-_________________________________________________________
|
|
|
-{"c_ts":1, "c_temperature":23, "c_humidity":88}
|
|
|
-{"c_ts":2}
|
|
|
-{"c_ts":3}
|
|
|
-{"c_ts":4, "c_temperature":25}
|
|
|
-{"c_ts":5, "c_humidity":90}
|
|
|
-{"c_ts":6, "c_humidity":91}
|
|
|
-{"c_ts":7}
|
|
|
-{"c_ts":8}
|
|
|
-```
|
|
|
-
|
|
|
-Rule to get the changed values of all columns and do not ignore null:
|
|
|
-
|
|
|
-```text
|
|
|
-SQL: SELECT CHANGED_COLS("c_", false, *) FROM demo
|
|
|
-_________________________________________________________
|
|
|
-{"c_temperature":23,"c_humidity":88}
|
|
|
-{"c_temperature":25}
|
|
|
-{"c_humidity":90}
|
|
|
-{"c_humidity":91}
|
|
|
-```
|
|
|
-
|
|
|
-Rule to get the average value change in a window:
|
|
|
-
|
|
|
-```text
|
|
|
-SQL: SELECT CHANGED_COLS("t", true, avg(temperature)) FROM demo GROUP BY CountWindow(2)
|
|
|
-_________________________________________________________________
|
|
|
-{"tavg":23}
|
|
|
-{"tavg":24}
|
|
|
-{"tavg":25}
|
|
|
-```
|
|
|
-
|
|
|
-Rule to get the events when temperature or humidity changed:
|
|
|
-
|
|
|
-```text
|
|
|
-SQL: SELECT ts, temperature, humidity FROM demo
|
|
|
-WHERE HAD_CHANGED(true, temperature, humidity) = true
|
|
|
-_________________________________________________________
|
|
|
-{"ts":1,temperature":23,"humidity":88}
|
|
|
-{"ts":4,temperature":25,"humidity":88}
|
|
|
-{"ts":5,temperature":25,"humidity":90}
|
|
|
-{"ts":6,temperature":25,"humidity":91}
|
|
|
-```
|
|
|
-
|
|
|
-Rule to get the events when temperature has changed but humidity has NOT changed:
|
|
|
-
|
|
|
-```text
|
|
|
-SQL: SELECT ts, temperature, humidity FROM demo
|
|
|
-WHERE HAD_CHANGED(true, temperature) = true AND HAD_CHANGED(true, humidity) = false
|
|
|
-_________________________________________________________
|
|
|
-{"ts":4,temperature":25,"humidity":88}
|
|
|
-```
|
|
|
-
|
|
|
-Rule to get the changed temperature and humidity value with customized names:
|
|
|
-
|
|
|
-```text
|
|
|
-SQL: SELECT CHANGED_COL(true, temperature) AS myTemp, CHANGED_COL(true, humidity) AS myHum FROM demo
|
|
|
-_________________________________________________________
|
|
|
-{"myTemp":23,"myHum":88}
|
|
|
-{"myTemp":25}
|
|
|
-{"myHum":90}
|
|
|
-{"myHum":91}
|
|
|
-```
|
|
|
-
|
|
|
-Rule to get the changed values when the temperature had changed to value bigger than 24:
|
|
|
-
|
|
|
-```text
|
|
|
-SQL: SELECT ts, temperature, humidity FROM demo
|
|
|
-WHERE CHANGED_COL(true, temperature) > 24
|
|
|
-_________________________________________________________
|
|
|
-{"ts":4,temperature":25,"humidity":88}
|
|
|
-```
|