iOS自动化探索（四）自动化测试框架pytest

自动化测试框架 - pytest

pytest是Python最流行的单元测试框架之一, 帮助更便捷的编写测试脚本，并支持多种功能复杂的测试场景，能用来做app测试也能用作函数测试

官方文档: https://docs.pytest.org/en/latest/

pytest具有以下优点:

允许使用assert进行断言
自动识别测试脚本、类、函数
可用于管理小型或者参数类型的测试数据或资源
兼容unittest和nose测试框架
支持Python2.7/Python3.4+
丰富的插件支持，超过315个插件支持

pytest安装

pip install -U pytest

如果提示下面的错误，说明是pip的版本太老了，要更新下：

  Could not find a version that satisfies the requirement pytest (from versions: )

No matching distribution found for pytest

更新方式:

easy_install --upgrade pip

官方示例

准备一个test_sample.py, 内容如下：

def inc(x):

    return x + 

def test_answer():

    assert inc() ==

在文件所在目录执行:

pytest

这里我们做下说明：

pytest脚本都以test_xxx.py为文件名；

inc方法是我们定义的一个自增函数，该函数将传递进来的参数加1后返回；

test_answer是我们编写的一个测试函数，其中我们使用基本的断言语句assert来对结果进行验证，测试函数以test_xxx作为命名

执行结果如下：

============================================================ test session starts ============================================================

platform darwin -- Python 2.7., pytest-4.1., py-1.7., pluggy-0.8.

rootdir: /Users/jackey/Documents/iOS/code/iOS-Auto/Agent_Test, inifile:

collected  item                                                                                                                            

test_sample.py F                                                                                                                      [%]

================================================================= FAILURES ==================================================================

________________________________________________________________ test_answer ________________________________________________________________

    def test_answer():

>       assert inc() ==

E       assert  ==

E        +  where  = inc()

test_sample.py:: AssertionError

=========================================================  failed in 0.05 seconds ==========================================================

(wda_python) bash-3.2$

当执行到assert inc(3) == 5时，报错

执行pytest会在当前目录和子目录中寻找test_xx.py的测试文件，并进入到测试文件中寻找test_xx开头的测试函数开始执行

执行pytest -q test_xxx.py是执行执行的脚本

在看一个例子,测试指定错误： (Assert that a certain exception is raised)

import pytest

def f():

    raise SystemExit()

def test_mytest():

    with pytest.raises(SystemExit):

        f()

执行指令：

pytest -q test_sysexit.py

输出:

(wda_python) bash-3.2$ pytest -q test_sysexit.py

.                                                                                                                                      [%]

 passed in 0.04 seconds

(wda_python) bash-3.2$

如果要开发多个测试方法，可以把方法写进一个class中

class TestClass(object):

    def test_one(self):

        x = 'this'

        assert 'h' in x

    def test_two(self):

        x = 'hello'

        assert hasattr(x, 'check')

pytest能够自动识别类中的测试方法，也不用我们去创建子类或者实实例, 运行结果如下：

(wda_python) bash-3.2$ pytest -q test_sample.py

.F                                                                                                                                     [%]

================================================================== FAILURES ==================================================================

_____________________________________________________________ TestClass.test_two _____________________________________________________________

self = <test_sample.TestClass object at 0x102e151d0>

    def test_two(self):

        x = 'hello'

>       assert hasattr(x, 'check')

E       AssertionError: assert False

E        +  where False = hasattr('hello', 'check')

test_sample.py:: AssertionError

 failed,  passed in 0.08 seconds

(wda_python) bash-3.2$

除了直接在脚本路径执行pytest外，还可以用以下方式

python -m pytest xxx.py

出现第一个（或第N个）错误时停止

pytest -x            # stop after first failure

pytest --maxfail=    # stop after two failures

运行执行测试脚本

pytest test_mod.py

运行指定目录下的所有脚本

pytest testing/

运行包含指定关键字的测试方法, 可以是文件名、类名、测试函数名

pytest -k "MyClass and not method"

执行node id运行测试脚本，每一个被收集的测试方法都会分配一个指定的id，我们可以用一下方式运行执行的测试方法:

# To run a specific test within a module

pytest test_mod.py::test_func     

# To run a test within a class

pytest test_mod.py::TestClass::test_method

日志打印的不同方式

pytest --showlocals # show local variables in tracebacks

pytest -l           # show local variables (shortcut)

pytest --tb=auto    # (default) 'long' tracebacks for the first and last

                     # entry, but 'short' style for the other entries

pytest --tb=long    # exhaustive, informative traceback formatting

pytest --tb=short   # shorter traceback format

pytest --tb=line    # only one line per failure

pytest --tb=native  # Python standard library formatting

pytest --tb=no      # no traceback at all

测试报告

pytest默认是完整的测试报告，我们可以加上-r标签显示简短测试报告，可再搭配一下参数

Here is the full list of available characters that can be used:

f - failed

E - error

s - skipped

x - xfailed

X - xpassed

p - passed

P - passed with output

a - all except pP

可以多个参数一起使用

Debug模式

pytest --pdb

示例:

(wda_python) bash-3.2$ pytest --pdb

========================================================== test session starts ===========================================================

platform darwin -- Python 2.7., pytest-4.1., py-1.7., pluggy-0.8.

rootdir: /Users/jackey/Documents/iOS/code/iOS-Auto/Agent_Test, inifile:

collected  items                                                                                                                        

test_sample.py .F

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> traceback >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

self = <test_sample.TestClass object at 0x10e928610>

    def test_two(self):

        x = 'hello'

>       assert hasattr(x, 'check')

E       AssertionError: assert False

E        +  where False = hasattr('hello', 'check')

test_sample.py:: AssertionError

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

> /Users/jackey/Documents/iOS/code/iOS-Auto/Agent_Test/test_sample.py()test_two()

-> assert hasattr(x, 'check')

(Pdb) print x

hello

(Pdb) print hasattr(x,'check')

False

(Pdb)

还可以指定第几次失败开始进入debug:

pytest -x --pdb   # drop to PDB on first failure, then end test session

pytest --pdb --maxfail=  # drop to PDB for first three failures

任何失败的异常信息都会存储在sys.last_value，sys.last_type 以及 sys_last_traceback

在debug中可以通过以下方式获取最后报错的内容

(Pdb) import sys

(Pdb) sys.last_traceback.tb_lineno

(Pdb) sys.last_value

AssertionError(u"assert False\n +  where False = hasattr('hello', 'check')",)

(Pdb)

在执行一开始就进入到debug模式

pytest --trace

输入next执行下一步, exit退出

脚本中设置断点

import pdb

pdb.set_trace()

例如：

import pdb

class TestClass(object):

    def test_one(self):

        x = 'this'

        pdb.set_trace()

        assert 'h' in x

    def test_two(self):

        x = 'hello'

        assert hasattr(x, 'check')

获取执行最慢的n个测试步骤

pytest --durations=

======================================================= slowest  test durations ========================================================

(0.00 durations hidden.  Use -vv to show these durations.)

但如果所有脚本的运行时间都小于0.01s, 就不显示了，除非带上-vv参数

pytest --durations= -vv

输出结果:

======================================================= slowest  test durations ========================================================

.00s call     test_sample.py::TestClass::test_two

.00s setup    test_sysexit.py::test_mytest

.00s setup    test_sample.py::TestClass::test_two

.00s setup    test_sample.py::TestClass::test_one

.00s teardown test_sample.py::TestClass::test_two

.00s teardown test_sample.py::TestClass::test_one

.00s call     test_sysexit.py::test_mytest

.00s teardown test_sysexit.py::test_mytest

.00s call     test_sample.py::TestClass::test_one

===================================================  failed,  passed in 0.06 seconds ===================================================

(wda_python) bash-3.2$

将日志保存到指定文件

pytest --resultlog=path

Disabling plugins

To disable loading specific plugins at invocation time, use the -p option together with the prefix no:.

Example: to disable loading the plugin doctest, which is responsible for executing doctest tests from text files, invoke pytest like this:

pytest -p no:doctest

我们也可以在pytestdemo脚本中去启动pytest:

import pytest

pytest.main()

执行python pytestdemo.py就可以执行pytest

main()不会抛出SystemExit的异常, 但会返回exitcode, 一共有6种exitcode

Exit code :    All tests were collected and passed successfully

Exit code :    Tests were collected and run but some of the tests failed

Exit code :    Test execution was interrupted by the user

Exit code :    Internal error happened while executing tests

Exit code :    pytest command line usage error

Exit code :    No tests were collected

我们试着加上打印

import pytest

print pytest.main()

输出：

(wda_python) bash-3.2$ python pytestDemo.py

========================================================== test session starts ===========================================================

platform darwin -- Python 2.7., pytest-4.1., py-1.7., pluggy-0.8.

rootdir: /Users/jackey/Documents/iOS/code/iOS-Auto/Agent_Test, inifile:

collected  items                                                                                                                        

test_sample.py .F                                                                                                                  [ %]

test_sysexit.py .                                                                                                                  [%]

================================================================ FAILURES ================================================================

___________________________________________________________ TestClass.test_two ___________________________________________________________

self = <test_sample.TestClass object at 0x1038ba650>

    def test_two(self):

        x = 'hello'

>       assert hasattr(x, 'check')

E       AssertionError: assert False

E        +  where False = hasattr('hello', 'check')

test_sample.py:: AssertionError

===================================================  failed,  passed in 0.05 seconds ===================================================

(wda_python) bash-3.2$

我们还可以在main中传递参数:

pytest.main(['-q','test_sample.py'])

给pytest.main添加plugin, 如下示例在执行的开头和结尾，添加打印信息

import pytest

class MyPlugin(object):
    def pytest_sessionfinish(self):
        print '*** Test run reporting finishing'

    def pytest_sessionstart(self):
        print '*** Test run report beginning'

pytest.main(['-q','test_sample.py'], plugins=[MyPlugin()])

输出：

(wda_python) bash-3.2$ python pytestDemo.py
*** Test run report beginning
.F [100%]*** Test run reporting finishing

================================================================ FAILURES ================================================================
___________________________________________________________ TestClass.test_two ___________________________________________________________

self = <test_sample.TestClass object at 0x1090843d0>

def test_two(self):
x = 'hello'
> assert hasattr(x, 'check')
E AssertionError: assert False
E + where False = hasattr('hello', 'check')

test_sample.py:11: AssertionError
1 failed, 1 passed in 0.05 seconds