|
@@ -36,7 +36,7 @@ All currently supported formats, their supported codec methods and modes are sho
|
|
|
|
|
|
When using `custom` format or `protobuf` format, the user can customize the codec and schema in the form of a go language plugin. Among them, `protobuf` only supports custom codecs, and the schema needs to be defined by `*.proto` file. The steps for customizing the format are as follows:
|
|
When using `custom` format or `protobuf` format, the user can customize the codec and schema in the form of a go language plugin. Among them, `protobuf` only supports custom codecs, and the schema needs to be defined by `*.proto` file. The steps for customizing the format are as follows:
|
|
|
|
|
|
-1. implement codec-related interfaces. The Encode function encodes the incoming data (currently always `map[string]interface{}`) into a byte array. The Decode function, on the other hand, decodes the byte array into `map[string]interface{}`. The decode function is called in source, while the encode function will be called in sink.
|
|
|
|
|
|
+1. Implement codec-related interfaces. The Encode function encodes the incoming data (currently always `map[string]interface{}`) into a byte array. The Decode function, on the other hand, decodes the byte array into `map[string]interface{}`. The decode function is called in source, while the encode function will be called in sink.
|
|
```go
|
|
```go
|
|
// Converter converts bytes & map or []map according to the schema
|
|
// Converter converts bytes & map or []map according to the schema
|
|
type Converter interface {
|
|
type Converter interface {
|
|
@@ -50,7 +50,7 @@ When using `custom` format or `protobuf` format, the user can customize the code
|
|
GetSchemaJson() string
|
|
GetSchemaJson() string
|
|
}
|
|
}
|
|
```
|
|
```
|
|
-3. Compile as a plugin so file. Usually, format extensions do not need to depend on the main eKuiper project. Due to the limitations of the Go language plugin system, the compilation of the plugin still needs to be done in the same compilation environment as the main eKuiper application, including the same operations, Go language version, etc. If you need to deploy to the official docker, you can use the corresponding docker image for compilation.
|
|
|
|
|
|
+3. Compile as a plugin so file. Usually, format extensions do not need to depend on the main eKuiper project. Due to the limitations of the Go language plugin system, the compilation of the plugin still needs to be done in the same compilation environment as the main eKuiper application, including the same operations, Go language version, etc. If you need to [deploy to the official docker](#build-format-plugin-with-docker), you can use the corresponding docker image for compilation.
|
|
```shell
|
|
```shell
|
|
go build -trimpath --buildmode=plugin -o data/test/myFormat.so internal/converter/custom/test/*.go
|
|
go build -trimpath --buildmode=plugin -o data/test/myFormat.so internal/converter/custom/test/*.go
|
|
```
|
|
```
|
|
@@ -69,6 +69,26 @@ When using `custom` format or `protobuf` format, the user can customize the code
|
|
|
|
|
|
The complete custom format can be found in [myFormat.go](https://github.com/lf-edge/ekuiper/blob/master/internal/converter/custom/test/myformat.go). This file defines a simple custom format where the codec actually only calls JSON for serialization. It returns a data structure that can be used to infer the data structure of the eKuiper source.
|
|
The complete custom format can be found in [myFormat.go](https://github.com/lf-edge/ekuiper/blob/master/internal/converter/custom/test/myformat.go). This file defines a simple custom format where the codec actually only calls JSON for serialization. It returns a data structure that can be used to infer the data structure of the eKuiper source.
|
|
|
|
|
|
|
|
+#### Build Format Plugin with Docker
|
|
|
|
+
|
|
|
|
+Due to go plugin limitations, it is better to build the format plugin in the same environment as the eKuiper build environment. The official eKuiper docker image and binaries are built in two os: debian and alpine. Except the default docker image like `1.8.0` and `1.8.0-apline`, other images and binaries are using debian.
|
|
|
|
+
|
|
|
|
+To build for debian environment, please use the corresponding dev image to build. For example, to build format plugin for 1.8.0, use `1.8.0-dev` image.
|
|
|
|
+
|
|
|
|
+To build for alpine environment, we can use the golang alpine image as the base environment. The steps are as below:
|
|
|
|
+
|
|
|
|
+1. In your plugin project, create a Makefile and make sure the plugin can be built by `make` command. Check the [sample project](https://github.com/lf-edge/ekuiper/tree/master/internal/converter/custom/test) for reference.
|
|
|
|
+2. Check the golang version of your eKuiper. Check the `GO_VERSION` arg in the [docker file](https://github.com/lf-edge/ekuiper/blob/master/deploy/docker/Dockerfile) of the corresponding eKuiper version. For example, if the version is `1.18.5`, use `golang:1.18.5-alpine` docker image for build.
|
|
|
|
+3. Switch to your project location then start the golang docker container with your project, install dependencies then execute `make`, make sure build is successful.
|
|
|
|
+ ```shell
|
|
|
|
+ cd ${yourProjectLoc}
|
|
|
|
+ docker run --rm -it -v "$PWD":/usr/src/myapp -w /usr/src/myapp golang:1.18.5-alpine sh
|
|
|
|
+ ### inside docker container
|
|
|
|
+ /usr/src/myapp # apk add gcc make libc-dev
|
|
|
|
+ /usr/src/myapp # make
|
|
|
|
+ ```
|
|
|
|
+4. You should find the built *.so file (test.so in this example) for you plugin in your project. Use that to register the format plugin.
|
|
|
|
+
|
|
### Static Protobuf
|
|
### Static Protobuf
|
|
|
|
|
|
When using the Protobuf format, we support both dynamic and static parsing. With dynamic parsing, the user only needs to
|
|
When using the Protobuf format, we support both dynamic and static parsing. With dynamic parsing, the user only needs to
|