Use of Kuiper management console

Overview

From Kuiper version 0.9.1, whenever a new version of Kuiper is released, the corresponding version of the management console will be released. This article uses a practical example to illustrate how to use the management console to operate and manage Kuiper nodes. In the article, the data will be subscribed from the MQTT server, and be sent to the specified file after processing through the rules written by Kuiper. The demonstration is as follows:

Create a Kuiper node through the management console
Create a stream to subscribe to the data in the MQTT server. This example demonstrates subscribing to the MQTT server. The relevant information is shown below.
- Address: tcp://broker.emqx.io:1883，
- Topic: devices/device_001/messages，
- Data: {"temperature": 40, "humidity" : 20}
Create a rule to calculate the subscribed data and write the data to the sink "This example demonstrates writing the subscribed message to a file".
Kuiper currently supports multiple sources and sinks. Users only need to install the corresponding plugins to achieve the corresponding function. "The source of this example is MQTT source, which has built-in support without the need of installation; the sink is a file, which does not have built-in support and needs to be installed separately."

Architecture design

UI end: a visual interface, easy for users to operate
Kuiper-manager: Management console, which essentially is a reverse HTTP proxy service, providing the services of user management, permission verification. It can be deployed in the cloud or at the edge
Kuiper instance: managed Kuiper node instance, Kuiper-manager can manage multiple Kuiper nodes at the same time

Install the management console

Install Kuiper

Pull Kuiper's Docker image from Docker Image Library. Since it is required to install the plugin in this article, you must use the kuiper:0.9.1-slim image (kuiper:0.9.1-alpine image is relatively small and easy to install, but due to the lack of some necessary library files, the plug-in cannot run normally. The kuiper:0.9.1 image is the development version, which is suitable for use in the development phase).

  docker pull emqx/kuiper:0.9.1-slim

Run the Kuiper container (for convenience, we will use the public MQTT server provided by EMQ, and the address can be set by the -e option when running the container). If you want to access the Kuiper instance through the host, you can expose port 9081 by adding the -p 9081:9081 parameter when starting the container.

  # docker run -d --name kuiper -e MQTT_SOURCE__DEFAULT__SERVERS=[tcp://broker.emqx.io:1883] emqx/kuiper:0.9.1-slim

When the container is running, the MQTT server address can be set through the -e option, and the data is written to the MQTT source configuration file, which can be viewed by the following command:

  # docker exec -it kuiper sh
  # cat etc/mqtt_source.yaml

Some output of this file is shown below, and the value of servers is set to tcp://broker.emqx.io:1883.

  default:
    concurrency: 1
    qos: 1
    servers:
    - tcp://broker.emqx.io:1883
    sharedSubscription: true
  ....

Install management console

Pull the Docker image of kuiper-manager from Docker Image Library, and 0.9.1-ief is a dedicated image for Huawei IEF users, This example uses the 0.9.1 image.

  docker pull emqx/kuiper-manager:0.9.1

Run the Kuiper-manager container and expose port 9082.

  docker run --name kuiperManager -d -p 9082:9082 emqx/kuiper-manager:0.9.1

Getting started

Login to kuiper-manager

You need to provide the address, user name, and password of kuiper-manager when logging in, which is shown below:

Address: http://$yourhost:9082
User name: admin
Password: public

Create a Kuiper node

When creating a kuiper node, you need to fill in the "node type", "node name" and "endpoint URL".

Node Type: Select Direct Connect Node (Huawei IEF Node is dedicated to Huawei users).
Node name: self-made, this example uses example.
Endpoint URL: http://$IP:9081, the IP acquisition command is as follows:

  docker inspect kuiper |  grep IPAddress

The example of creating a Kuiper node is shown in the figure below. If the port is exposed to the host, then the 9081 port address on the host can also be used directly.

Install the plugin

In our scenario, the target plugin named file will be used. Select "Plugins" > "Install plugin", and the following dialog box will pop up: Select the target plugin named file in the drop-down list to download and install it, and the plugin will write data to the file specified by the user. As shown in the figure below, after the reader selects the plug-in with the corresponding name, the input box of "File" will automatically be filled with the corresponding plug-in download address. After clicking the "Submit" button, Kuiper will automatically download the corresponding plugin from the relevant address on https://www.emqx.io/downloads and install it into the system automatically.

**Note: After the plug-in is installed and used through the rules, the plug-in has been loaded into the memory. Due to the limitations of the Golang language, when the plug-in is deleted, it cannot be uninstalled in practice. Therefore, if you want to reinstall the plug-in, you must restart Kuiper before it can take effect. Currently only the installation in the Docker environment of debian is supported, and other environments are not supported temporarily. **

Create a stream

Create a stream named demoStream, as shown below:

Used to subscribe to MQTT server messages with the address tcp://broker.emqx.io:1883
The message topic is devices/device_001/messages
The stream structure definition contains the following two fields.
- temperature: bigint
- humidity: bigint

Users can also remove "whether it is a stream with structure" to define a schemaless data source.

"Stream Type" can be left unselected. If not selected, it is "mqtt" by default, or you can directly select "mqtt" as shown in the figure below
"Configuration Group", similar to "Stream Type". If not selected, "default" is used by default
"Stream format", similar to "Stream Type". If not selected, the default "json" is used

As shown above, the "default" configuration group is used. Users can also write their own configuration according to their needs. The specific operation is to click Source Configuration on the page of creating a steam, and a dialog box will pop up as shown in the figure below. If the user enters the value of any configuration item here, the corresponding default configuration will be overwritten; if no value is entered, the default configuration value will be used.

Create a rule

As shown in the figure below, a rule named demoRule is created to filter out the data with temperature> 30 in the data. The SQL editor can give hints during the user's SQL writing process to facilitate the user to complete the SQL writing.

Click the "Add" button and a dialog box will pop up as shown below. The file path of /kuiper/demoFile where the input result is stored is input . More information about the file sink can be found in Help File. The target file is in the Beta state and cannot be used as an actual production environment.

After the rule is created, if everything goes well, the rule is running.

View execution results

Enter the Kuiper container to create a file:

# docker exec -it kuiper sh
# touch demoFile
# tail -f demoFile

Use the MQTT client tool mosquitto_pub to send the sensor data to the topic devices/device_001/messages of the MQTT server tcp://broker.emqx.io:1883, and the command is as follows. If everything goes well, the file named demoFile will receive data: {"temperature": 40, "humidity": 20}.

# mosquitto_pub -h broker.emqx.io -m '{"temperature": 40, "humidity" : 20}' -t devices/device_001/messages

View more information of the rules

As shown in the figure below, there are three buttons in the options. Readers can click to try it out.

Rule running status
Restart rules
Delete rules

Extended reading

How to display custom plugins in the installation list of the management console: Kuiper provides a plugin extension mechanism, and users can implement custom plugins based on the extended interface. On the management console, users can install plugins directly through the interface. If readers have customized plugins and want to show them in the installation list of the management console, this article can give readers some reference.
If the readers want to develop their own plug-in, they can refer to Plugin Development Tutorial for more information.
EMQ edge-stack project: This project allows users to install and test EMQ edge series products more easily, realize industrial data analysis, edge data aggregation, and Kuiper-based edge data analysis and other one-stop edge solutions.

overview.md 9.5 KB تاريخچه خام