Преглед изворни кода

doc(source, sink): add rule pipeline doc

1. Memory source/sink conf file and doc
2. Rule pipeline doc

Signed-off-by: Jiyong Huang <huangjy@emqx.io>
Jiyong Huang пре 3 година
родитељ
комит
68310c4103

+ 58 - 0
docs/en_US/rules/rule_pipeline.md

@@ -0,0 +1,58 @@
+# Rule Pipeline
+
+We can form rule pipelines by importing results of prior rule into the following rule. This is possible by employing intermediate storage or MQ such as mqtt broker. By using the pair of [memory source](./sources/memory.md) and [sink](./sinks/memory.md), we can create rule pipelines without external dependencies.
+
+## Usage
+
+Rule pipeline will be implicit. Each rule can use an memory sink / source. This means that each step will be created separately using existing api (example below).
+
+```shell
+#1 Create the source stream
+{"sql" : "create stream demo () WITH (DATASOURCE=\"demo\", FORMAT=\"JSON\")"}
+
+#2 Create rule and sink to memory
+{
+  "id": "rule1",
+  "sql": "SELECT * FROM demo WHERE isNull(temperature)=false",
+  "actions": [{
+    "log": {
+    },
+    "memory": {
+      "topic": "home/ch1/sensor1"
+    }
+  }]
+}
+
+#3 Create a stream from the memory topic
+{"sql" : "create stream sensor1 () WITH (DATASOURCE=\"home/+/sensor1\", FORMAT=\"JSON\", TYPE=\"memory\")"}
+
+#4 Create another rules to consume from the memory topic
+{
+  "id": "rule2-1",
+  "sql": "SELECT avg(temperature) FROM sensor1 GROUP BY CountWindow(10)",
+  "actions": [{
+    "log": {
+    },
+    "memory": {
+      "topic": "analytic/sensors"
+    }
+  }]
+}
+
+{
+  "id": "rule2-2",
+  "sql": "SELECT temperature + 273.15 as k FROM sensor1",
+  "actions": [{
+    "log": {
+    }
+  }]
+}
+
+```
+
+By using the memory topic as the bridge, we now form a rule pipeline:
+`rule1->{rule2-1, rule2-2}`. The pipeline can be multiple to multiple and very flexible. 
+
+Notice that, the memory sink can be used together with other sinks to create multiple rule actions for a rule. And the memory source topic can use wildcard to subscirbe to a filtered topic list.
+
+     

+ 17 - 0
docs/en_US/rules/sinks/memory.md

@@ -0,0 +1,17 @@
+# Memory action
+
+The action is used to flush the result into an in-memory topic so that it can be consumed by the [memory source](../sources/memory.md). The topic is like pubsub topic such as mqtt, so that there could be multiple memory sinks which publish to the same topic and multiple memory sources which subscribe to the same topic. The typical usage for memory action is to form [rule pipelines](../rule_pipeline.md).
+
+| Property name      | Optional | Description                                                  |
+| ------------------ | -------- | ------------------------------------------------------------ |
+| topic              | false    | The in-memory topic, such as `analysis/result`                    |
+
+Below is a sample memory action configuration:
+
+```json
+{
+  "memory": {
+    "topic": "devices/result"
+  }
+}
+```

+ 24 - 0
docs/en_US/rules/sources/memory.md

@@ -0,0 +1,24 @@
+# Memory Source
+
+Memory source is provided to consume events produced by the [memory sink](../sinks/memory.md) through topics. The topic is like pubsub topic such as mqtt, so that there could be multiple memory sinks which publish to the same topic and multiple memory sources which subscribe to the same topic. The typical usage for memory action is to form [rule pipelines](../rule_pipeline.md).
+
+There is no configuration properties. The topic is specified by the stream data source property like below examples:
+
+```text
+CREATE TABLE table1 (
+    name STRING,
+    size BIGINT,
+    id BIGINT
+) WITH (DATASOURCE="devices/result", FORMAT="json", TYPE="memory");
+```
+
+## Topic Wildcard
+
+Similar to mqtt topic, memory source also supports topic wildcards. Currently, there are two wildcards supported.
+
+**+** : Single level wildcard replaces one topic level. 
+**#**: Multi level wildcard covers multiple topic levels, and it can only be used at the end.
+
+Examples:
+1. `home/device1/+/sensor1`
+2. `home/device1/#`

+ 57 - 0
docs/zh_CN/rules/rule_pipeline.md

@@ -0,0 +1,57 @@
+# 规则管道
+
+我们可以通过将先前规则的结果导入后续规则来形成规则管道。 这可以通过使用中间存储或 MQ(例如 mqtt 消息服务器)来实现。 通过同时使用 [内存源](./sources/memory.md) 和 [目标](./sinks/memory.md),我们可以创建没有外部依赖的规则管道。
+
+## 使用
+
+规则管道将是隐式的。 每个规则都可以使用一个内存目标/源。 这意味着每个步骤将使用现有的 api 单独创建(示例如下所示)。
+
+```shell
+#1 创建源流
+{"sql" : "create stream demo () WITH (DATASOURCE=\"demo\", FORMAT=\"JSON\")"}
+
+#2 创建规则和内存目标
+{
+  "id": "rule1",
+  "sql": "SELECT * FROM demo WHERE isNull(temperature)=false",
+  "actions": [{
+    "log": {
+    },
+    "memory": {
+      "topic": "home/ch1/sensor1"
+    }
+  }]
+}
+
+#3 从内存主题创建一个流
+{"sql" : "create stream sensor1 () WITH (DATASOURCE=\"home/+/sensor1\", FORMAT=\"JSON\", TYPE=\"memory\")"}
+
+#4 从内存主题创建另一个要使用的规则
+{
+  "id": "rule2-1",
+  "sql": "SELECT avg(temperature) FROM sensor1 GROUP BY CountWindow(10)",
+  "actions": [{
+    "log": {
+    },
+    "memory": {
+      "topic": "analytic/sensors"
+    }
+  }]
+}
+
+{
+  "id": "rule2-2",
+  "sql": "SELECT temperature + 273.15 as k FROM sensor1",
+  "actions": [{
+    "log": {
+    }
+  }]
+}
+
+```
+
+通过使用内存主题作为桥梁,我们现在创建一个规则管道:`rule1->{rule2-1, rule2-2}`。 管道可以是多对多的,而且非常灵活。 
+
+请注意,内存目标可以与其他目标一起使用,为一个规则创建多个规则动作。 并且内存源主题可以使用通配符订阅过滤后的主题列表。
+
+​     

+ 17 - 0
docs/zh_CN/rules/sinks/memory.md

@@ -0,0 +1,17 @@
+# 内存动作
+
+该动作用于将结果刷新到内存中的主题中,以便 [内存源](../sources/memory.md) 可以使用它。 该主题类似于 pubsub 主题,例如 mqtt,因此可能有多个内存目标发布到同一主题,也可能有多个内存源订阅同一主题。 内存动作的典型用途是形成[规则管道](../rule_pipeline.md)。
+
+| 属性名称 | 是否可选 | 描述                                 |
+| -------- | -------- | ------------------------------------ |
+| topic    | 否       | 内存中的主题,例如 `analysis/result` |
+
+下面是一个内存动作配置示例:
+
+```json
+{
+  "memory": {
+    "topic": "devices/result"
+  }
+}
+```

+ 24 - 0
docs/zh_CN/rules/sources/memory.md

@@ -0,0 +1,24 @@
+# 内存源
+
+内存源通过主题消费由 [内存目标](../sinks/memory.md) 生成的事件。该主题类似于 pubsub 主题,例如 mqtt,因此可能有多个内存目标发布到同一主题,也可能有多个内存源订阅同一主题。 内存动作的典型用途是形成[规则管道](../rule_pipeline.md)。
+
+主题没有配置属性,由流数据源属性指定,如以下示例所示:
+
+```text
+CREATE TABLE table1 (
+    name STRING,
+    size BIGINT,
+    id BIGINT
+) WITH (DATASOURCE="devices/result", FORMAT="json", TYPE="memory");
+```
+
+## 主题通配符
+
+内存源也支持主题通配符,与 mqtt 主题类似。 目前,支持两种通配符。
+
+**+** : 单级通配符替换一个主题等级。
+**#**: 多级通配符涵盖多个主题级别,只能在结尾使用。
+
+示例:
+1. `home/device1/+/sensor1`
+2. `home/device1/#`

+ 36 - 0
etc/sinks/memory.json

@@ -0,0 +1,36 @@
+{
+  "about": {
+    "trial": false,
+    "author": {
+      "name": "EMQ",
+      "email": "contact@emqx.io",
+      "company": "EMQ Technologies Co., Ltd",
+      "website": "https://www.emqx.io"
+    },
+    "helpUrl": {
+      "en_US": "https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/rules/sinks/memory.md",
+      "zh_CN": "https://github.com/lf-edge/ekuiper/blob/master/docs/zh_CN/rules/sinks/memory.md"
+    },
+    "description": {
+      "en_US": "The action is used to flush the result into an in-memory topic so that it can be consumed by the memory source.",
+      "zh_CN": "该操作用于将输出消息发送到内存主题中,以便内存源可以消费。"
+    }
+  },
+  "properties": [
+    {
+      "name": "topic",
+      "optional": false,
+      "control": "text",
+      "default": "",
+      "type": "string",
+      "hint": {
+        "en_US": "The topic, such as analysis/result",
+        "zh_CN": "主题,例如 analysis/result"
+      },
+      "label": {
+        "en_US": "Topic",
+        "zh_CN": "主题"
+      }
+    }
+  ]
+}

+ 22 - 0
etc/sources/memory.json

@@ -0,0 +1,22 @@
+{
+  "about": {
+    "trial": true,
+    "author": {
+      "name": "EMQ",
+      "email": "contact@emqx.io",
+      "company": "EMQ Technologies Co., Ltd",
+      "website": "https://www.emqx.io"
+    },
+    "helpUrl": {
+      "en_US": "https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/plugins/sources/memory.md",
+      "zh_CN": "https://github.com/lf-edge/ekuiper/blob/master/docs/zh_CN/plugins/sources/memory.md"
+    },
+    "description": {
+      "en_US": "The source subscribes to topics to consume events from memory sink to form a rule pipeline.",
+      "zh_CN": "内存源可通过订阅主题来消费内存sink里的消息,从而实现规则流水线。"
+    }
+  },
+  "libs": [],
+  "properties": {
+  }
+}

+ 0 - 0
etc/sources/memory.yaml