Merge pull request #161 from emqx/update_docker

Update docker

deploy/chart/kuiper/README.md

@@ -67,15 +67,7 @@ Kuiper can be deployed at k3s/k8s cluster through Helm chart. Below takes k3s as
   | `nodeSelector`                 | Node selector                                        | {}                       |
   | `tolerations`                  | Tolerations                                          | []                       |
   | `affinity`                     | Affinity                                             | {}                       |
-  | `mqtt.servers`                 | MQTT broker address                                  | `[tcp://127.0.0.1:1883]` |
-  | `mqtt.qos`                     | QoS of message subscription                          | 1                        |
-  | `mqtt.sharedSubscription`      | Use shared subscription or not                       | true                     |
-  | `mqtt.username`                | MQTT connection user name                            |                          |
-  | `mqtt.password`                | MQTT connection password                             |                          |
-  | `mqtt.certificationSecretName` | Secret resource name created for certification file. |                          |
-  | `mqtt.privateKeySecretName`    | Secret resource name created fro private key file    |                          |
-  | `mqtt.certificationPath`       | Certification path for MQTT connection               |                          |
-  | `mqtt.privateKeyPath`          | Private key path for MQTT connection                 |                          |
+  | `kuiperConfig`                 | Configuration file in the Kuiper `etc` directory     |                          |
 
 ## Deploy Kuiper through Helm
 
@@ -169,10 +161,25 @@ Kuiper can be deployed at k3s/k8s cluster through Helm chart. Below takes k3s as
 
 + Open and edit `values.yaml` file
 
-  + Set `mqtt.certificationSecretName` certification Secret resource: `mqtt.certificationSecretName: client-cert`
-  + Set `mqtt.privateKeySecretName` private key Secret resource:`mqtt.privateKeySecretName: client-key`
-  + Set certification file path: `mqtt.certificationPath: /var/kuiper/certificate.pem`
-  + Set private key file path: `mqtt.privateKeyPath: /var/kuiper/private.pem.key`
+  ```shell
+  $ vim value.yaml
+  kuiperConfig:
+  ...
+    "mqtt_source.yaml":
+      #Global MQTT configurations
+      default:
+        qos: 1
+        sharedSubscription: true
+        servers: [tcp://127.0.0.1:1883]
+        concurrency: 1
+        #username: user1
+        #password: password
+        certificationSecretName: client-cert  # Set certification Secret resource name
+        certificationPath: /var/kuiper/certificate.pem # Set certification file path
+        privateKeySecretName: client-key  # Set private key Secret resource name
+        privateKeyPath: /var/kuiper/xyz-private.pem.key # Set private key file path
+  ...
+  ```
 
 + Deploy Kuiper through Helm 
 

deploy/chart/kuiper/README_zh.md

@@ -67,15 +67,7 @@ Kuiper 可以通过 Helm chart 部署在 k3s / k8s 集群上。下面以 k3s 为
   | `nodeSelector`                 | 节点选择                            | {}                       |
   | `tolerations`                  | 污点容忍                            | []                       |
   | `affinity`                     | 节点亲和性                          | {}                       |
-  | `mqtt.servers`                 | mqtt服务器的代理地址                | `[tcp://127.0.0.1:1883]` |
-  | `mqtt.qos`                     | 消息转发的服务质量                  | 1                        |
-  | `mqtt.sharedSubscription`      | 是否使用共享订阅                    | true                     |
-  | `mqtt.username`                | 连接用户名                          |                          |
-  | `mqtt.password`                | 连接密码                            |                          |
-  | `mqtt.certificationSecretName` | 通过证书文件创建的 Secre 资源的名字 |                          |
-  | `mqtt.privateKeySecretName`    | 通过私钥文件创建的 Secre 资源的名字 |                          |
-  | `mqtt.certificationPath`       | 证书路径。必须是绝对路径。          |                          |
-  | `mqtt.privateKeyPath`          | 私钥路径。必须绝对路径。            |                          |
+  | `kuiperConfig`                 | Kuiper `etc` 目录下的配置文件           |                        |
 
 ## 通过 Helm 部署 Kuiper
 
@@ -169,10 +161,25 @@ Kuiper 可以通过 Helm chart 部署在 k3s / k8s 集群上。下面以 k3s 为
 
 + 编辑 `values.yaml` 文件
 
-  + 设置 `mqtt.certificationSecretName` 为证书文件 Secret 资源： `mqtt.certificationSecretName: client-cert`
-  + 设置 `mqtt.privateKeySecretName` 为私钥文件 Secret 资源：`mqtt.privateKeySecretName: client-key`
-  + 设置证书文件部署路径：`mqtt.certificationPath: /var/kuiper/certificate.pem`
-  + 设置私钥文件部署路径：`mqtt.privateKeyPath: /var/kuiper/private.pem.key`
+  ```shell
+  $ vim value.yaml
+  kuiperConfig:
+  ...
+    "mqtt_source.yaml":
+      #Global MQTT configurations
+      default:
+        qos: 1
+        sharedSubscription: true
+        servers: [tcp://127.0.0.1:1883]
+        concurrency: 1
+        #username: user1
+        #password: password
+        certificationSecretName: client-cert  # 设置证书文件 Secret resource name
+        certificationPath: /var/kuiper/certificate.pem # 设置证书文件部署路径
+        privateKeySecretName: client-key  # 设置私钥文件的 Secret resource name
+        privateKeyPath: /var/kuiper/xyz-private.pem.key # 设置私钥文件部署路径
+  ...
+  ```
 
 + 使用 Helm 部署 Kuiper
 

deploy/chart/kuiper/templates/StatefulSet.yaml

@@ -49,21 +49,33 @@ spec:
           claimName: {{ tpl . $ }}
         {{- end }}
       {{- end }}
-      - name: mqtt
+      - name: kuiper-config
         configMap:
           name: {{ include "kuiper.fullname" . }}
           items:
-          - key: mqtt.yaml
-            path: mqtt.yaml
-      {{- if .Values.mqtt.certificationSecretName }}
+          - key: mqtt_source.yaml
+            path: mqtt_source.yaml
+          - key: kuiper.yaml
+            path: kuiper.yaml
+          - key: client.yaml
+            path: client.yaml
+          - key: edgex.yaml
+            path: edgex.yaml
+          - key: random.yaml
+            path: random.yaml
+          - key: zmq.yaml
+            path: zmq.yaml
+      {{ $certificationSecretName := index .Values "kuiperConfig" "mqtt_source.yaml" "default" "certificationSecretName" }}
+      {{- if $certificationSecretName }}
       - name: kuiper-certification
         secret:
-          secretName: {{ .Values.mqtt.certificationSecretName }}
+          secretName: {{ $certificationSecretName }}
       {{- end }}
-      {{- if .Values.mqtt.privateKeySecretName }}
+      {{ $privateKeySecretName := index .Values "kuiperConfig" "mqtt_source.yaml" "default" "privateKeySecretName" }}
+      {{- if $privateKeySecretName }}
       - name: kuiper-private-key
         secret:
-          secretName: {{ .Values.mqtt.privateKeySecretName }}
+          secretName: {{ $privateKeySecretName }}
       {{- end }}
       containers:
         - name: kuiper
@@ -72,17 +84,34 @@ spec:
           volumeMounts:
           - name: kuiper-data
             mountPath: "/kuiper/data"
-          - name: mqtt
-            mountPath: "/kuiper/etc/sources/mqtt.yaml"
-            subPath: "mqtt.yaml"
-          {{ if .Values.mqtt.certificationSecretName  }}
+          - name: kuiper-config
+            mountPath: "/kuiper/etc/mqtt_source.yaml"
+            subPath: "mqtt_source.yaml"
+          - name: kuiper-config
+            mountPath: "/kuiper/etc/kuiper.yaml"
+            subPath: "kuiper.yaml"
+          - name: kuiper-config
+            mountPath: "/kuiper/etc/client.yaml"
+            subPath: "client.yaml"
+          - name: kuiper-config
+            mountPath: "/kuiper/etc/sources/edgex.yaml"
+            subPath: "edgex.yaml"
+          - name: kuiper-config
+            mountPath: "/kuiper/etc/sources/random.yaml"
+            subPath: "random.yaml"
+          - name: kuiper-config
+            mountPath: "/kuiper/etc/sources/zmq.yaml"
+            subPath: "zmq.yaml"
+          {{ $certificationSecretName := index .Values "kuiperConfig" "mqtt_source.yaml" "default" "certificationSecretName" }}
+          {{- if $certificationSecretName }}
           - name: kuiper-certification
-            mountPath: {{ .Values.mqtt.certificationPath | default "/var/kuiper/certificate.pem" }}
+            mountPath: {{ index .Values "kuiperConfig" "mqtt_source.yaml" "default" "certificationPath" | default "/var/kuiper/certificate.pem" }}
             readOnly: true
           {{ end }}
-          {{ if .Values.mqtt.privateKeySecretName  }}
+          {{ $privateKeySecretName := index .Values "kuiperConfig" "mqtt_source.yaml" "default" "privateKeySecretName" }}
+          {{- if $privateKeySecretName }}
           - name: kuiper-private-key
-            mountPath: {{ .Values.mqtt.privateKeyPath | default "/var/kuiper/private.pem.key" }}
+            mountPath: {{ index .Values "kuiperConfig" "mqtt_source.yaml" "default" "privateKeyPath" | default "/var/kuiper/private.pem.key" }}
             readOnly: true
           {{ end }}
           resources:

deploy/chart/kuiper/templates/configmap.yaml

@@ -6,34 +6,21 @@ metadata:
   labels:
     {{- include "kuiper.labels" . | nindent 4 }}
 data:
-    "random.yaml": |
-      default:
-        interval: 1000
-        pattern:
-          count: 50
-      ext:
-        interval: 300
-        seed: 1
-        pattern:
-          count: 50
-    "zmq.yaml": |
-      #Global Zmq configurations
-      default:
-        server: tcp://127.0.0.1:5563
-    "mqtt.yaml": |
-      #Global MQTT configurations
-      default:
-        {{- toYaml .Values.mqtt | nindent 8 }}
-      #Override the global configurations
-      demo_conf: #Conf_key
-        qos: 0
-        servers: [tcp://10.211.55.6:1883, tcp://127.0.0.1]
-    "client.yaml": |
-      basic:
-        host: 127.0.0.1
-        port: 20498
-    "kuiper.yaml": |
-      basic:
-        # true|false, with debug level, it prints more debug info
-        debug: false
-        port: 20498
+  "client.yaml": |
+    {{ $client := index .Values "kuiperConfig" "client.yaml" }}
+    {{- toYaml $client | nindent 6 }}
+  "kuiper.yaml": |
+    {{ $kuiper := index .Values "kuiperConfig" "kuiper.yaml" }}
+    {{- toYaml $kuiper | nindent 6 }}
+  "mqtt_source.yaml": |
+    {{ $mqtt := index .Values "kuiperConfig" "mqtt_source.yaml" }}
+    {{- toYaml $mqtt | nindent 6 }}
+  "edgex.yaml": |
+    {{ $edgex := index .Values "kuiperConfig" "sources/edgex.yaml" }}
+    {{- toYaml $edgex | nindent 6 }}
+  "random.yaml": |
+    {{ $random := index .Values "kuiperConfig" "sources/random.yaml" }}
+    {{- toYaml $random | nindent 6 }}
+  "zmq.yaml": |
+    {{ $zmq := index .Values "kuiperConfig" "sources/zmq.yaml" }}
+    {{- toYaml $zmq | nindent 6 }}

deploy/chart/kuiper/values.yaml

@@ -38,14 +38,65 @@ tolerations: []
 
 affinity: {}
 
-mqtt:
-  #Global MQTT configurations
-  qos: 1
-  sharedSubscription: true
-  servers: [tcp://127.0.0.1:1883]
-  #username: user1
-  #password: password
-  certificationSecretName: client-cert
-  privateKeySecretName: client-key
-  #certificationPath: /var/kuiper/certificate.pem
-  #privateKeyPath: /var/kuiper/private.pem.key
+kuiperConfig:
+  "client.yaml":
+    basic:
+      host: 127.0.0.1
+      port: 20498
+  "kuiper.yaml":
+    basic:
+      # true|false, with debug level, it prints more debug info
+      debug: false
+      port: 20498
+      restPort: 9081
+      prometheus: false
+      prometheusPort: 20499
+  "mqtt_source.yaml":
+    #Global MQTT configurations
+    default:
+      qos: 1
+      sharedSubscription: true
+      servers: [tcp://127.0.0.1:1883]
+      concurrency: 1
+      #username: user1
+      #password: password
+      #certificationSecretName: Secret resource name created for certification file
+      #certificationPath: /var/kuiper/xyz-certificate.pem
+      #privateKeySecretName: Secret resource name created for certification file
+      #privateKeyPath: /var/kuiper/xyz-private.pem.key
+    #Override the global configurations
+    demo_conf: #Conf_key
+      qos: 0
+      servers: [tcp://10.211.55.6:1883, tcp://127.0.0.1]
+  "sources/edgex.yaml":
+    #Global Edgex configurations
+    default:
+      protocol: tcp
+      server: localhost
+      port: 5570
+      topic: events
+      serviceServer: http://localhost:10080
+    #  optional:
+    #    ClientId: client1
+    #    Username: user1
+    #    Password: password
+    #Override the global configurations
+    application_conf: #Conf_key
+      protocol: tcp
+      server: localhost
+      port: 5571
+      topic: application
+  "sources/random.yaml":
+    default:
+      interval: 1000
+      pattern:
+        count: 50
+    ext:
+      interval: 300
+      seed: 1
+      pattern:
+        count: 50
+  "sources/zmq.yaml":
+    #Global Zmq configurations
+    default:
+      server: tcp://127.0.0.1:5563

deploy/docker/Dockerfile

@@ -11,7 +11,7 @@ FROM alpine:3.10
 COPY --from=builder /go/kuiper/_build/kuiper-* /kuiper/
 COPY ./deploy/docker/docker-entrypoint.sh /usr/bin/docker-entrypoint.sh
 
-RUN apk add sed
+RUN apk add sed libzmq
 
 WORKDIR /kuiper
 

deploy/docker/README.md

@@ -144,19 +144,32 @@ docker run -d --name kuiper -e MQTT_BROKER_ADDRESS=$MQTT_BROKER_ADDRESS emqx/kui
 
 ### Configuration
 
-Use the environment variable to configure `etc/sources/mqtt.yaml`  on the Kuiper container.
-
-| Options                    | Default            | Mapped                    |
-| ---------------------------| ------------------ | ------------------------- |
-| MQTT_BROKER_ADDRESS         | tcp://127.0.0.1:1883 | default.servers |
-| MQTT_BROKER_SHARED_SUBSCRIPTION | true   | default.sharedSubscription |
-| MQTT_BROKER_QOS | 1                 | default.qos    |
-| MQTT_BROKER_USERNAME |   | default.username |
-| MQTT_BROKER_PASSWORD |                | default.password |
-| MQTT_BROKER_CER_PATH |                | default.certificationPath |
-| MQTT_BROKER_KEY_PATH |     | default.privateKeyPath |
-
-If you want to configure more options, you can mount the configuration file into Kuiper container.
+Use the environment variable to configure `etc/mqtt_sources.yaml`  on the Kuiper container.
+
+| Options                         | Default               | Mapped                      |
+| ------------------------------- | --------------------- | --------------------------- |
+| MQTT_BROKER_ADDRESS             | tcp://127.0.0.1:1883  | default.servers             |
+| MQTT_BROKER_SHARED_SUBSCRIPTION | true                  | default.sharedSubscription  |
+| MQTT_BROKER_QOS                 | 1                     | default.qos                 |
+| MQTT_BROKER_USERNAME            |                       | default.username            |
+| MQTT_BROKER_PASSWORD            |                       | default.password            |
+| MQTT_BROKER_CER_PATH            |                       | default.certificationPath   |
+| MQTT_BROKER_KEY_PATH            |                       | default.privateKeyPath      |
+
+Use the environment variable to configure `etc/sources/edgex.yaml`  on the Kuiper container.
+
+| Options                    | Default                  | Mapped                    |
+| ---------------------------| -------------------------| ------------------------- |
+| EDGEX_PROTOCOL             | tcp                      | default.protocol          |
+| EDGEX_SERVER               | localhost                | default.server            |
+| EDGEX_PORT                 | 5570                     | default.port              |
+| EDGEX_TOPIC                | events                   | default.topic             |
+| EDGEX_SERVICE_SERVER       | http://localhost:10080   | default.serviceServer     |
+
+If you want to configure more options, you can mount the configuration file into Kuiper container, like this:
+```
+$ docker run --name kuiper -v /path/to/mqtt_sources.yaml:/kuiper/etc/mqtt_sources.yaml -v /path/to/edgex.yaml:/kuiper/etc/sources/edgex.yaml emqx/kuiper:$tag
+```
 
 # More
 

deploy/docker/docker-entrypoint.sh

@@ -12,43 +12,75 @@ CONFIG="$KUIPER_HOME/etc/mqtt_source.yaml"
 if [ ! -z "$MQTT_BROKER_ADDRESS" ]; then
     sed -i '/default:/ ,/servers/{/servers/d}' $CONFIG
     sed -i "/default:/a\  servers: [$MQTT_BROKER_ADDRESS]" $CONFIG
-    echo "default.servers = $MQTT_BROKER_ADDRESS"
+    echo "mqtt.default.servers = $MQTT_BROKER_ADDRESS"
 fi
 
 if [ ! -z "$MQTT_BROKER_SHARED_SUBSCRIPTION" ]; then
     sed -i '/default:/ ,/sharedSubscription/{/sharedSubscription/d}' $CONFIG
     sed -i "/default:/a\  sharedSubscription: $MQTT_BROKER_SHARED_SUBSCRIPTION" $CONFIG
-    echo "default.sharedSubscription = $MQTT_BROKER_SHARED_SUBSCRIPTION"
+    echo "mqtt.default.sharedSubscription = $MQTT_BROKER_SHARED_SUBSCRIPTION"
 fi
 
 if [ ! -z "$MQTT_BROKER_QOS" ]; then
     sed -i '/default:/ ,/qos/{/qos/d}' $CONFIG
     sed -i "/default:/a\  qos: $MQTT_BROKER_QOS" $CONFIG
-    echo "default.qos = $MQTT_BROKER_QOS"
+    echo "mqtt.default.qos = $MQTT_BROKER_QOS"
 fi
 
 if [ ! -z "$MQTT_BROKER_USERNAME" ]; then
     sed -i '/default:/ ,/username/{/username/d}' $CONFIG
     sed -i "/default:/a\  username: $MQTT_BROKER_USERNAME" $CONFIG
-    echo "default.username = $MQTT_BROKER_USERNAME"
+    echo "mqtt.default.username = $MQTT_BROKER_USERNAME"
 fi
 
 if [ ! -z "$MQTT_BROKER_PASSWORD" ]; then
     sed -i '/default:/ ,/password/{/password/d}' $CONFIG
     sed -i "/default:/a\  password: $MQTT_BROKER_PASSWORD" $CONFIG
-    echo "default.password = $MQTT_BROKER_PASSWORD"
+    echo "mqtt.default.password = $MQTT_BROKER_PASSWORD"
 fi
 
 if [ ! -z "$MQTT_BROKER_CER_PATH" ]; then
     sed -i '/default:/ ,/certificationPath/{/certificationPath/d}' $CONFIG
     sed -i "/default:/a\  certificationPath: $MQTT_BROKER_CER_PATH" $CONFIG
-    echo "default.certificationPath = $MQTT_BROKER_CER_PATH"
+    echo "mqtt.default.certificationPath = $MQTT_BROKER_CER_PATH"
 fi
 
 if [ ! -z "$MQTT_BROKER_KEY_PATH" ]; then
     sed -i '/default:/ ,/privateKeyPath/{/privateKeyPath/d}' $CONFIG
     sed -i "/default:/a\  privateKeyPath: $MQTT_BROKER_KEY_PATH" $CONFIG
-    echo "default.privateKeyPath = $MQTT_BROKER_KEY_PATH"
+    echo "mqtt.default.privateKeyPath = $MQTT_BROKER_KEY_PATH"
+fi
+
+EDGEX_CONFIG="$KUIPER_HOME/etc/sources/edgex.yaml"
+
+if [ ! -z "$EDGEX_PROTOCOL" ]; then
+    sed -i '/default:/ ,/protocol/{/protocol/d}' $EDGEX_CONFIG
+    sed -i "/default:/a\  protocol: $EDGEX_PROTOCOL" $EDGEX_CONFIG
+    echo "edgex.default.protocol = $EDGEX_PROTOCOL"
+fi
+
+if [ ! -z "$EDGEX_SERVER" ]; then
+    sed -i '/default:/ ,/server/{/server/d}' $EDGEX_CONFIG
+    sed -i "/default:/a\  server: $EDGEX_SERVER" $EDGEX_CONFIG
+    echo "edgex.default.server = $EDGEX_SERVER"
+fi
+
+if [ ! -z "$EDGEX_PORT" ]; then
+    sed -i '/default:/ ,/port/{/port/d}' $EDGEX_CONFIG
+    sed -i "/default:/a\  port: $EDGEX_PORT" $EDGEX_CONFIG
+    echo "edgex.default.port = $EDGEX_PORT"
+fi
+
+if [ ! -z "$EDGEX_TOPIC" ]; then
+    sed -i '/default:/ ,/topic/{/topic/d}' $EDGEX_CONFIG
+    sed -i "/default:/a\  topic: $EDGEX_TOPIC" $EDGEX_CONFIG
+    echo "edgex.default.topic = $EDGEX_TOPIC"
+fi
+
+if [ ! -z "$EDGEX_SERVICE_SERVER" ]; then
+    sed -i '/default:/ ,/serviceServer/{/serviceServer/d}' $EDGEX_CONFIG
+    sed -i "/default:/a\  serviceServer: $EDGEX_SERVICE_SERVER" $EDGEX_CONFIG
+    echo "edgex.default.serviceServer = $EDGEX_SERVICE_SERVER"
 fi
 
 exec "$@"