ソースを参照

doc: add more elaboration for custom static format extension

Signed-off-by: Jiyong Huang <huangjy@emqx.io>
Jiyong Huang 2 年 前
コミット
04e524c39c

+ 22 - 2
docs/en_US/guide/serialization/serialization.md

@@ -36,7 +36,7 @@ All currently supported formats, their supported codec methods and modes are sho
 
 
 When using `custom` format or `protobuf` format, the user can customize the codec and schema in the form of a go language plugin. Among them, `protobuf` only supports custom codecs, and the schema needs to be defined by `*.proto` file. The steps for customizing the format are as follows:
 When using `custom` format or `protobuf` format, the user can customize the codec and schema in the form of a go language plugin. Among them, `protobuf` only supports custom codecs, and the schema needs to be defined by `*.proto` file. The steps for customizing the format are as follows:
 
 
-1. implement codec-related interfaces. The Encode function encodes the incoming data (currently always `map[string]interface{}`) into a byte array. The Decode function, on the other hand, decodes the byte array into `map[string]interface{}`. The decode function is called in source, while the encode function will be called in sink.
+1. Implement codec-related interfaces. The Encode function encodes the incoming data (currently always `map[string]interface{}`) into a byte array. The Decode function, on the other hand, decodes the byte array into `map[string]interface{}`. The decode function is called in source, while the encode function will be called in sink.
     ```go
     ```go
     // Converter converts bytes & map or []map according to the schema
     // Converter converts bytes & map or []map according to the schema
     type Converter interface {
     type Converter interface {
@@ -50,7 +50,7 @@ When using `custom` format or `protobuf` format, the user can customize the code
 	    GetSchemaJson() string
 	    GetSchemaJson() string
     }
     }
     ```
     ```
-3. Compile as a plugin so file. Usually, format extensions do not need to depend on the main eKuiper project. Due to the limitations of the Go language plugin system, the compilation of the plugin still needs to be done in the same compilation environment as the main eKuiper application, including the same operations, Go language version, etc. If you need to deploy to the official docker, you can use the corresponding docker image for compilation.
+3. Compile as a plugin so file. Usually, format extensions do not need to depend on the main eKuiper project. Due to the limitations of the Go language plugin system, the compilation of the plugin still needs to be done in the same compilation environment as the main eKuiper application, including the same operations, Go language version, etc. If you need to [deploy to the official docker](#build-format-plugin-with-docker), you can use the corresponding docker image for compilation.
     ```shell
     ```shell
     go build -trimpath --buildmode=plugin -o data/test/myFormat.so internal/converter/custom/test/*.go
     go build -trimpath --buildmode=plugin -o data/test/myFormat.so internal/converter/custom/test/*.go
     ```
     ```
@@ -69,6 +69,26 @@ When using `custom` format or `protobuf` format, the user can customize the code
 
 
 The complete custom format can be found in [myFormat.go](https://github.com/lf-edge/ekuiper/blob/master/internal/converter/custom/test/myformat.go). This file defines a simple custom format where the codec actually only calls JSON for serialization. It returns a data structure that can be used to infer the data structure of the eKuiper source.
 The complete custom format can be found in [myFormat.go](https://github.com/lf-edge/ekuiper/blob/master/internal/converter/custom/test/myformat.go). This file defines a simple custom format where the codec actually only calls JSON for serialization. It returns a data structure that can be used to infer the data structure of the eKuiper source.
 
 
+#### Build Format Plugin with Docker
+
+Due to go plugin limitations, it is better to build the format plugin in the same environment as the eKuiper build environment. The official eKuiper docker image and binaries are built in two os: debian and alpine. Except the default docker image like `1.8.0` and `1.8.0-apline`, other images and binaries are using debian.
+
+To build for debian environment, please use the corresponding dev image to build. For example, to build format plugin for 1.8.0, use `1.8.0-dev` image.
+
+To build for alpine environment, we can use the golang alpine image as the base environment. The steps are as below:
+
+1. In your plugin project, create a Makefile and make sure the plugin can be built by `make` command. Check the [sample project](https://github.com/lf-edge/ekuiper/tree/master/internal/converter/custom/test) for reference.
+2. Check the golang version of your eKuiper. Check the `GO_VERSION` arg in the [docker file](https://github.com/lf-edge/ekuiper/blob/master/deploy/docker/Dockerfile) of the corresponding eKuiper version. For example, if the version is `1.18.5`, use `golang:1.18.5-alpine` docker image for build.
+3. Switch to your project location then start the golang docker container with your project, install dependencies then execute `make`, make sure build is successful.
+   ```shell
+   cd ${yourProjectLoc}
+   docker run --rm -it -v "$PWD":/usr/src/myapp -w /usr/src/myapp golang:1.18.5-alpine sh
+   ### inside docker container
+   /usr/src/myapp # apk add gcc make libc-dev
+   /usr/src/myapp # make
+   ```
+4. You should find the built *.so file (test.so in this example) for you plugin in your project. Use that to register the format plugin.
+
 ### Static Protobuf
 ### Static Protobuf
 
 
 When using the Protobuf format, we support both dynamic and static parsing. With dynamic parsing, the user only needs to
 When using the Protobuf format, we support both dynamic and static parsing. With dynamic parsing, the user only needs to

+ 21 - 1
docs/zh_CN/guide/serialization/serialization.md

@@ -50,7 +50,7 @@ eKuiper 计算过程中使用的是基于 Map 的数据结构,因此 source/si
 	    GetSchemaJson() string
 	    GetSchemaJson() string
     }
     }
     ```
     ```
-3. 编译为插件 so 文件。通常格式的扩展无需依赖 eKuiper 的主项目。由于 Go 语言插件系统的限制,插件的编译仍然需要在与 eKuiper 主程序相同的编译环境中进行,包括操作相同,Go 语言版本等。若需要部署到官方 docker 中,则可使用对应的 docker 镜像进行编译。
+3. 编译为插件 so 文件。通常格式的扩展无需依赖 eKuiper 的主项目。由于 Go 语言插件系统的限制,插件的编译仍然需要在与 eKuiper 主程序相同的编译环境中进行,包括操作相同,Go 语言版本等。若需要部署到官方 docker 中,则可使用对应的 docker 镜像进行编译。详细步骤,请参考[格式插件编译](#使用-docker-编译格式插件).
     ```shell
     ```shell
     go build -trimpath --buildmode=plugin -o data/test/myFormat.so internal/converter/custom/test/*.go
     go build -trimpath --buildmode=plugin -o data/test/myFormat.so internal/converter/custom/test/*.go
     ```
     ```
@@ -69,6 +69,26 @@ eKuiper 计算过程中使用的是基于 Map 的数据结构,因此 source/si
 
 
 完整的自定义格式可参考 [myFormat.go](https://github.com/lf-edge/ekuiper/blob/master/internal/converter/custom/test/myformat.go)。该文件定义了一个简单的自定义格式,编解码实际上仅调用 JSON 进行序列化。它返回了一个数据结构,可用于 eKuiper source 的数据结构推断。
 完整的自定义格式可参考 [myFormat.go](https://github.com/lf-edge/ekuiper/blob/master/internal/converter/custom/test/myformat.go)。该文件定义了一个简单的自定义格式,编解码实际上仅调用 JSON 进行序列化。它返回了一个数据结构,可用于 eKuiper source 的数据结构推断。
 
 
+#### 使用 Docker 编译格式插件
+
+由于go插件的限制,最好在与 eKuiper 构建环境相同的环境中构建格式插件。官方的 eKuiper docker 镜像和二进制文件都是在两个操作系统中构建的:debian 和 alpine。除了默认的 docker镜像如 `1.8.0` 和 `1.8.0-apline` 之外,其他镜像和二进制文件都使用 debian 系统构建。
+
+要在 debian 环境下构建,请使用相应的 dev 镜像来构建。例如,要构建 1.8.0 的格式插件,请使用`1.8.0-dev` 镜像。
+
+要在 alpine 环境下构建,我们可以使用 golang alpine 镜像作为基础环境。具体步骤如下:
+
+1. 在你的插件项目中,创建一个 Makefile,并确保该插件能被 `make` 命令构建。请查看[示例项目](https://github.com/lf-edge/ekuiper/tree/master/internal/converter/custom/test)以获得参考。
+2. 检查你的 eKuiper 的版本所使用的 golang 版本。检查相应 eKuiper 版本的[ docker 文件](https://github.com/lf-edge/ekuiper/blob/master/deploy/docker/Dockerfile)中的 `GO_VERSION` 参数。如果版本是 `1.18.5`,则使用 `golang:1.18.5-alpine` docker 镜像进行构建。
+3. 切换到你的项目位置,然后在你的项目位置启动 golang docker 容器,安装依赖项,然后执行`make`,确保构建成功。
+   ```shell
+   cd ${yourProjectLoc}
+   docker run --rm -it -v "$PWD":/usr/src/myapp -w /usr/src/myapp golang:1.18.5-alpine sh
+   ### inside docker container
+   /usr/src/myapp # apk add gcc make libc-dev
+   /usr/src/myapp # make
+   ```
+4. 你应该在你的项目中找到为你的插件建立的 *.so 文件(在这个例子中是 test.so)。用它来注册格式插件。
+
 ### 静态 Protobuf
 ### 静态 Protobuf
 
 
 使用 Protobuf 格式时,我们支持动态解析和静态解析两种方式。使用动态解析时,用户仅需要在注册模式时指定 proto 文件。在解析性能要求更高的条件下,用户可采用静态解析的方式。静态解析需要开发解析插件,其步骤如下:
 使用 Protobuf 格式时,我们支持动态解析和静态解析两种方式。使用动态解析时,用户仅需要在注册模式时指定 proto 文件。在解析性能要求更高的条件下,用户可采用静态解析的方式。静态解析需要开发解析插件,其步骤如下:

+ 5 - 0
internal/converter/custom/test/Makefile

@@ -0,0 +1,5 @@
+.PHONY: build
+build:
+	@echo "Start build"
+	CGO_ENABLED=1 go build -trimpath --buildmode=plugin
+	@echo "Build successfully"

+ 5 - 0
internal/converter/custom/test/go.mod

@@ -0,0 +1,5 @@
+module test
+
+go 1.18
+
+require github.com/mitchellh/mapstructure v1.5.0

+ 2 - 0
internal/converter/custom/test/go.sum

@@ -0,0 +1,2 @@
+github.com/mitchellh/mapstructure v1.5.0 h1:jeMsZIYE/09sWLaz43PL7Gy6RuMjD2eJVyuac5Z2hdY=
+github.com/mitchellh/mapstructure v1.5.0/go.mod h1:bFUtVrKA4DC2yAKiSyO/QUcy7e+RRV2QTWOzhPopBRo=

+ 5 - 0
internal/converter/protobuf/test/Makefile

@@ -0,0 +1,5 @@
+.PHONY: build
+build:
+	@echo "Start build"
+	CGO_ENABLED=1 go build -trimpath --buildmode=plugin
+	@echo "Build successfully"

+ 8 - 0
internal/converter/protobuf/test/go.mod

@@ -0,0 +1,8 @@
+module helloworld
+
+go 1.18
+
+require (
+	github.com/golang/protobuf v1.5.2
+	google.golang.org/protobuf v1.28.0
+)

+ 11 - 0
internal/converter/protobuf/test/go.sum

@@ -0,0 +1,11 @@
+github.com/golang/protobuf v1.5.0/go.mod h1:FsONVRAS9T7sI+LIUmWTfcYkHO4aIWwzhcaSAoJOfIk=
+github.com/golang/protobuf v1.5.2 h1:ROPKBNFfQgOUMifHyP+KYbvpjbdoFNs+aK7DXlji0Tw=
+github.com/golang/protobuf v1.5.2/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY=
+github.com/google/go-cmp v0.5.5 h1:Khx7svrCpmxxtHBq5j2mp/xVjsi8hQMfNLvJFAlrGgU=
+github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
+golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543 h1:E7g+9GITq07hpfrRu66IVDexMakfv52eLZ2CXBWiKr4=
+golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
+google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw=
+google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc=
+google.golang.org/protobuf v1.28.0 h1:w43yiav+6bVFTBQFZX0r7ipe9JQ1QsbMgHwbBziscLw=
+google.golang.org/protobuf v1.28.0/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I=