golang 故障模拟工具failpoint的使用

测试是功能上线之前的重要环节。

测试过程中，要尽量覆盖各种场景。故障情况或异常情况下的场景测试，也是必不可少的。

如何模拟故障呢？

在FreeBSD 中, failpoints经常用来模拟故障。

在golang，也有failpoint的实现，就是目前pingcap 公司的开源项目failpoint.

下面介绍golang中的failpoint的实现。

1.安装failpoint工具

cd $GOPATH/src

mkdir -p github.com/pingcap

cd github.com/pingcap

git clone https://github.com/pingcap/failpoint.git

cd failpoint

make
GO111MODULE=on CGO_ENABLED=0 GO111MODULE=on go build  -ldflags '-X "github.com/pingcap/failpoint/failpoint-ctl/version.releaseVersion=12f4ac2-dev" -X "github.com/pingcap/failpoint/failpoint-ctl/version.buildTS=2019-11-15 09:41:49" -X "github.com/pingcap/failpoint/failpoint-ctl/version.gitHash=12f4ac2fd11dfc6b2f7018b00bb90f61a5b6b692" -X "github.com/pingcap/failpoint/failpoint-ctl/version.gitBranch=master" -X "github.com/pingcap/failpoint/failpoint-ctl/version.goVersion=go version go1.13 darwin/amd64"' -o bin/failpoint-ctl failpoint-ctl/main.go
failpoint-ctl build successfully :-) !

注意：仔细确认这些步骤。如果搞错，会出现无法转换代码的情况。

编译后，生成可执行文件failpoint-ctl：

ll bin
total 6840
-rwxr-xr-x  1 lanyang  staff   3.3M 11 15 17:41 failpoint-ctl

2. 测试代码

代码结构如下：

cd $GOPATH/src

mkdir fp_example

cd fp_example

# tree ./
./
├── fruit
│   └── banana.go
├── main.go
└── util
    ├── apple.go
    └── orange.go

其中，main.go文件内容：

package main

import (
        "fmt"
        "fp_example/hot_fruit"
        "fp_example/fruit"
)

func main() {

        hot_fruit.Apple()
        hot_fruit.Orange()
        fruit.Banana()

        fmt.Println("main end")
}

fruit/banana.go：

package fruit

import (
        "fmt"
        "github.com/pingcap/failpoint"
)

func Banana() {
        fmt.Println("banana....")

        failpoint.Inject("bananaPanic", func() {
                panic("banana failpoint triggerd")
        })

}

注入一个panic。

hot_fruit/apple.go：

package hot_fruit

import (
        "fmt"
        "github.com/pingcap/failpoint"
)

func Apple() {
        fmt.Println("apple....")

        failpoint.Inject("applePanic", func() {
                panic("apple failpoint triggerd")
        })

}

再次注入一个panic。

hot_fruit/orange.go：

package hot_fruit

import (
        "fmt"
        "github.com/pingcap/failpoint"

)
func Orange() {
        fmt.Println("orange....")

        failpoint.Inject("orangePanic", func() {
                panic("orange failpoint triggerd")
        })

}

再次注入一个panic。

3. 使用failpoint转换代码

$GOPATH/src/github.com/pingcap/failpoint/bin/failpoint-ctl enable

代码转换之后，看下代码文件结构：

 tree ./
./
├── fruit
│   ├── banana.go
│   ├── banana.go__failpoint_stash__
│   └── binding__failpoint_binding__.go
├── hot_fruit
│   ├── apple.go
│   ├── apple.go__failpoint_stash__
│   ├── binding__failpoint_binding__.go
│   ├── orange.go
│   └── orange.go__failpoint_stash__
├── main.go

2 directories, 10 files

可以看到多了几个文件。

如果执行disable，这些文件又会消失。

$GOPATH/src/github.com/pingcap/failpoint/bin/failpoint-ctl disable

4.编译和执行

使用装换之后的代码，进行编译：

go build -o simple main.go

生成可执行文件simple.

正常执行:

./simple
apple....
orange....
banana....
main end

触发bananaPanic故障执行：

GO_FAILPOINTS="fp_example/fruit/bananaPanic=return(true)" ./simple
apple....
orange....
banana....
panic: banana failpoint triggerd

goroutine 1 [running]:
fp_example/fruit.Banana()
	/Users/lanyang/workspace/go_projects/src/fp_example/fruit/banana.go:12 +0x10e
main.main()
	/Users/lanyang/workspace/go_projects/src/fp_example/main.go:13 +0x2c

其中，fp_example/fruit 是package名称, bananaPanic是注入的故障名称。

触发applePanic故障执行：

GO_FAILPOINTS="fp_example/hot_fruit/applePanic=return(true)" ./simple

apple....
panic: apple failpoint triggerd

goroutine 1 [running]:
fp_example/hot_fruit.Apple()
	/Users/lanyang/workspace/go_projects/src/fp_example/hot_fruit/apple.go:12 +0x10e
main.main()
	/Users/lanyang/workspace/go_projects/src/fp_example/main.go:11 +0x22

触发orangePanic故障执行：

GO_FAILPOINTS="fp_example/hot_fruit/orangePanic=return(true)" ./simple
apple....
orange....
panic: orange failpoint triggerd

goroutine 1 [running]:
fp_example/hot_fruit.Orange()
	/Users/lanyang/workspace/go_projects/src/fp_example/hot_fruit/orange.go:12 +0x10e
main.main()
	/Users/lanyang/workspace/go_projects/src/fp_example/main.go:12 +0x27

5.参考

pingcap/failpoint

Golang Failpoint 的设计与实现