Browse Source

add docs for edgex source and test script

RockyJin 5 years atrás
parent
commit
d0fe630ab1
3 changed files with 142 additions and 2 deletions
  1. 3 1
      docs/en_US/rules/overview.md
  2. 112 0
      docs/en_US/rules/sources/edgex.md
  3. 27 1
      fvt_scripts/README.md

+ 3 - 1
docs/en_US/rules/overview.md

@@ -39,7 +39,9 @@ The identification of the rule. The rule name cannot be duplicated in the same K
 
 The sql query to run for the rule. 
 
-- Kuiper provides embeded support MQTT source, see  [MQTT source stream](sources/mqtt.md) for more detailed info.
+- Kuiper provides embeded following 2 sources,
+  - MQTT source, see  [MQTT source stream](sources/mqtt.md) for more detailed info.
+  - EdgeX source by default is shipped in [docker images](https://hub.docker.com/r/emqx/kuiper), but NOT included in single download binary files, you use ``make pkg_with_edgex`` command to build a binary package that supports EdgeX source. Please see [EdgeX source stream](sources/edgex.md) for more detailed info.
 - See [SQL](../sqls/overview.md) for more info of Kuiper SQL.
 - Sources can be customized, see [extension](../extension/overview.md) for more detailed info.
 

+ 112 - 0
docs/en_US/rules/sources/edgex.md

@@ -0,0 +1,112 @@
+# EdgeX Source
+
+Kuiper provides built-in support for EdgeX source stream, which can subscribe the message from [EdgeX message bus](https://github.com/edgexfoundry/go-mod-messaging) and feed into the Kuiper streaming process pipeline.  
+
+## Stream definition for EdgeX
+
+EdgeX already defines data types in [value descriptors](https://github.com/edgexfoundry/go-mod-core-contracts), so it's recommeded to use schema-less stream definition in EdgeX source as in below.
+
+```shell
+# cd $kuiper_base
+# bin/cli CREATE STREAM demo'() with(format="json", datasource="demo" type="edgex")'
+```
+
+EdgeX source will try to get the data type of a field, 
+
+- convert to releated data type if field of a type can be found in value descriptors service;
+- or keep original value if  field of a type can not be found in value descriptors service;
+- or if failed to conver the value, then the value will be **dropped**, and a warning message print in the log;
+
+The types defined in EdgeX value descriptors will be converted into related [data types](../../sqls/streams.md) that supported in Kuiper.
+
+### Boolean
+
+If  ``Type`` value of ``ValueDescriptor`` is ``B``, ``Bool`` or ``Boolean``, then Kuiper tries to convert to ``boolean`` type. Following values will be converted into ``true``.
+
+- "1", "t", "T", "true", "TRUE", "True" 
+
+Following will be converted into ``false``.
+
+- "0", "f", "F", "false", "FALSE", "False"
+
+### Bigint
+
+If  ``Type`` value of ``ValueDescriptor`` is ``I``, ``INT``,  ``INT8`` , ``INT16``, ``INT32``,  ``INT64``,``UINT`` , ``UINT8`` , ``UINT16`` ,  ``UINT32`` , ``UINT64`` then Kuiper tries to convert to ``Bigint`` type. 
+
+### Float
+
+If  ``Type`` value of ``ValueDescriptor`` is ``F``, ``FLOAT``,  ``FLOAT16`` , ``FLOAT32``, ``FLOAT64``then Kuiper tries to convert to ``Float`` type. 
+
+### String
+
+If  ``Type`` value of ``ValueDescriptor`` is ``S``, ``String``, then Kuiper tries to convert to ``String`` type. 
+
+### Struct
+
+If  ``Type`` value of ``ValueDescriptor`` is ``J``, ``Json``, then Kuiper tries to convert to ``Struct`` type. 
+
+# Global configurations
+
+The configuration file of EdgeX source is at ``$kuiper/etc/sources/edgex.yaml``. Below is the file format.
+
+```yaml
+#Global Edgex configurations
+default:
+  protocol: tcp
+  server: localhost
+  port: 5570
+  topic: events
+  serviceServer: http://localhost:10080
+#  optional:
+#    ClientId: client1
+#    Username: user1
+#    Password: password
+```
+
+
+
+Use can specify the global EdgeX settings here. The configuration items specified in ``default`` section will be taken as default settings for all EdgeX source. 
+
+## protocol
+
+The protocol connect to EdgeX message bus, default value is ``tcp``.
+
+## server
+
+The server address of  EdgeX message bus, default value is ``localhost``.
+
+## port
+
+The port of EdgeX message bus, default value is ``5570``.
+
+## topic
+
+The topic name of EdgeX message bus,  default value is ``events``.
+
+## serviceServer
+
+The base service address for getting value descriptors, the value of ``serviceServer`` will be concatenated to ``/api/v1/valuedescriptor`` to get all of value descriptors of EdgeX server.
+
+## Override the default settings
+
+```yaml
+#Override the global configurations
+demo_conf: #Conf_key
+  protocol: tcp
+  server: 10.211.55.6
+  port: 5570
+  topic: events
+```
+
+If you have a specific connection that need to overwrite the default settings, you can create a customized section. In the previous sample, we create a specific setting named with ``demo``.  Then you can specify the configuration with option ``CONF_KEY`` when creating the stream definition (see [stream specs](../../sqls/streams.md) for more info).
+
+**Sample**
+
+```
+demo (
+		...
+	) WITH (DATASOURCE="device1", FORMAT="JSON", type="edgex", CONF_KEY="demo");
+```
+
+The configuration keys used for these specific settings are the same as in ``default`` settings, any values specified in specific settings will overwrite the values in ``default`` section.
+

+ 27 - 1
fvt_scripts/README.md

@@ -133,4 +133,30 @@ For most of scripts, you can just start JMeter by default way, such as ``bin/jme
 
   - Another JMeter mock-up user subscribes MQTT result topic, and assert the order for device_id field is descending, and temperature is ascending.
 
-  
+- [EdgeX source with condition](select_edgex_condition_rule.jmx)
+
+  The test script is used for testing [Kuiper EdgeX source](../docs/en_US/rules/sources/edgex.md). To run the script, 
+
+  - A mockup EdgeX value descriptor service should be compiled and run before test.
+
+    ```shell
+    # go build -o fvt_scripts/edgex/valuedesc/vdmocker fvt_scripts/edgex/valuedesc/vd_server.go
+    
+    # fvt_scripts/edgex/valuedesc/vdmocker > vdmocker.out 2>&1 &
+    ```
+
+  - An EdgeX message bus publish tool should be compiled and run during running test.
+
+    ```shell
+    # go build -o fvt_scripts/edgex/pub fvt_scripts/edgex/pub.go
+    ```
+
+  - Run the JMeter with following command, and specify the ``fvt`` property in the JMeter command line, the ``fvt`` is where you develop Kuiper, script will search ``fvt_scripts/edgex/pub`` from the location.
+
+    ```shell
+    bin/jmeter.sh -Dfvt="/Users/rockyjin/Downloads/workspace/edge/src/kuiper"
+    ```
+
+  - The processing SQL is ``SELECT * FROM demo WHERE temperature > 30``, so all of the data that with temperature less than 30 will be fitered. 
+  - Another JMeter mock-up user subscribes MQTT result topic, and assert message number and contents.
+