当断言失败时继续在 Python 的单元测试中

Continuing in Python's unittest when an assertion fails

提问by Bruce Christensen

EDIT: switched to a better example, and clarified why this is a real problem.

编辑：切换到一个更好的例子，并澄清为什么这是一个真正的问题。

I'd like to write unit tests in Python that continue executing when an assertion fails, so that I can see multiple failures in a single test. For example:

我想用 Python 编写单元测试，当断言失败时继续执行，以便我可以在单个测试中看到多个失败。例如：

class Car(object):
  def __init__(self, make, model):
    self.make = make
    self.model = make  # Copy and paste error: should be model.
    self.has_seats = True
    self.wheel_count = 3  # Typo: should be 4.

class CarTest(unittest.TestCase):
  def test_init(self):
    make = "Ford"
    model = "Model T"
    car = Car(make=make, model=model)
    self.assertEqual(car.make, make)
    self.assertEqual(car.model, model)  # Failure!
    self.assertTrue(car.has_seats)
    self.assertEqual(car.wheel_count, 4)  # Failure!

Here, the purpose of the test is to ensure that Car's __init__sets its fields correctly. I could break it up into four methods (and that's often a great idea), but in this case I think it's more readable to keep it as a single method that tests a single concept ("the object is initialized correctly").

在这里，测试的目的是确保 Car's__init__正确设置其字段。我可以将它分解为四个方法（这通常是一个好主意），但在这种情况下，我认为将其作为测试单个概念的单个方法（“对象已正确初始化”）更具可读性。

If we assume that it's best here to not break up the method, then I have a new problem: I can't see all of the errors at once. When I fix the modelerror and re-run the test, then the wheel_counterror appears. It would save me time to see both errors when I first run the test.

如果我们假设这里最好不要中断方法，那么我有一个新问题：我无法一次看到所有错误。当我修复model错误并重新运行测试时，就会wheel_count出现错误。当我第一次运行测试时，它会节省我看到这两个错误的时间。

For comparison, Google's C++ unit testing framework distinguishes betweenbetween non-fatal EXPECT_*assertions and fatal ASSERT_*assertions:

为了比较，Google 的 C++ 单元测试框架区分了非致命EXPECT_*断言和致命ASSERT_*断言：

The assertions come in pairs that test the same thing but have different effects on the current function. ASSERT_* versions generate fatal failures when they fail, and abort the current function. EXPECT_* versions generate nonfatal failures, which don't abort the current function. Usually EXPECT_* are preferred, as they allow more than one failures to be reported in a test. However, you should use ASSERT_* if it doesn't make sense to continue when the assertion in question fails.

断言成对出现，测试相同的事物但对当前函数有不同的影响。ASSERT_* 版本在失败时会产生致命的失败，并中止当前函数。EXPECT_* 版本生成非致命故障，不会中止当前函数。通常首选 EXPECT_*，因为它们允许在测试中报告多个失败。但是，如果在相关断言失败时继续操作没有意义，您应该使用 ASSERT_*。

Is there a way to get EXPECT_*-like behavior in Python's unittest? If not in unittest, then is there another Python unit test framework that does support this behavior?

有没有办法EXPECT_*在 Python 中获得类似行为unittest？如果不在 中unittest，那么是否有另一个支持这种行为的 Python 单元测试框架？

---

---

Incidentally, I was curious about how many real-life tests might benefit from non-fatal assertions, so I looked at some code examples(edited 2014-08-19 to use searchcode instead of Google Code Search, RIP). Out of 10 randomly selected results from the first page, all contained tests that made multiple independent assertions in the same test method. All would benefit from non-fatal assertions.

顺便说一句，我很好奇有多少现实生活中的测试可以从非致命断言中受益，所以我查看了一些代码示例（编辑 2014-08-19 以使用搜索代码而不是谷歌代码搜索，RIP）。在第一页随机选择的 10 个结果中，所有结果都包含在同一测试方法中进行多个独立断言的测试。所有人都将从非致命断言中受益。

采纳答案by dietbuddha

What you'll probably want to do is derive unittest.TestCasesince that's the class that throws when an assertion fails. You will have to re-architect your TestCaseto not throw (maybe keep a list of failures instead). Re-architecting stuff can cause other issues that you would have to resolve. For example you may end up needing to derive TestSuiteto make changes in support of the changes made to your TestCase.

您可能想要做的是派生，unittest.TestCase因为这是断言失败时抛出的类。你将不得不重新构建你TestCase的不抛出（也许保留一个失败列表）。重新架构的东西可能会导致您必须解决的其他问题。例如，您可能最终需要派生TestSuite来进行更改以支持对TestCase.

回答by Steven

It is considered an anti-pattern to have multiple asserts in a single unit test. A single unit test is expected to test only one thing. Perhaps you are testing too much. Consider splitting this test up into multiple tests. This way you can name each test properly.

在单个单元测试中具有多个断言被认为是一种反模式。单个单元测试预计只测试一件事。也许你测试的太多了。考虑将此测试拆分为多个测试。这样您就可以正确命名每个测试。

Sometimes however, it is okay to check multiple things at the same time. For instance when you are asserting properties of the same object. In that case you are in fact asserting whether that object is correct. A way to do this is to write a custom helper method that knows how to assert on that object. You can write that method in such a way that it shows all failing properties or for instance shows the complete state of the expected object and the complete state of the actual object when an assert fails.

但是，有时可以同时检查多项内容。例如，当您断言同一对象的属性时。在这种情况下，您实际上是在断言该对象是否正确。一种方法是编写一个自定义帮助器方法，该方法知道如何对该对象进行断言。您可以编写该方法以显示所有失败的属性，或者例如在断言失败时显示预期对象的完整状态和实际对象的完整状态。

回答by Johnsyweb

I don't think there is a way to do this with PyUnit and wouldn't want to see PyUnit extended in this way.

我认为 PyUnit 没有办法做到这一点，也不希望看到 PyUnit 以这种方式扩展。

I prefer to stick to one assertion per test function (or more specifically asserting one concept per test) and would rewrite test_addition()as four separate test functions. This would give more useful information on failure, viz:

我更喜欢坚持每个测试函数一个断言（或更具体地说，每个测试断言一个概念），并将重写test_addition()为四个独立的测试函数。这将提供有关失败的更多有用信息，即：

.FF.
======================================================================
FAIL: test_addition_with_two_negatives (__main__.MathTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_addition.py", line 10, in test_addition_with_two_negatives
    self.assertEqual(-1 + (-1), -1)
AssertionError: -2 != -1

======================================================================
FAIL: test_addition_with_two_positives (__main__.MathTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_addition.py", line 6, in test_addition_with_two_positives
    self.assertEqual(1 + 1, 3)  # Failure!
AssertionError: 2 != 3

----------------------------------------------------------------------
Ran 4 tests in 0.000s

FAILED (failures=2)

If you decide that this approach isn't for you, you may find this answerhelpful.

如果您认为这种方法不适合您，您可能会发现此答案很有帮助。

Update

更新

It looks like you are testing two concepts with your updated question and I would split these into two unit tests. The first being that the parameters are being stored on the creation of a new object. This would have two assertions, one for makeand one for model. If the first fails, the that clearly needs to be fixed, whether the second passes or fails is irrelevant at this juncture.

看起来您正在使用更新的问题测试两个概念，我会将它们分成两个单元测试。首先是在创建新对象时存储参数。这将有两个断言，一个 formake和一个 for model。如果第一个失败，那显然需要修复，第二个通过还是失败在这个时刻无关紧要。

The second concept is more questionable... You're testing whether some default values are initialised. Why? It would be more useful to test these values at the point that they are actually used (and if they are not used, then why are they there?).

第二个概念更值得怀疑......您正在测试是否初始化了一些默认值。为什么？在实际使用时测试这些值会更有用（如果不使用它们，那么它们为什么会在那里？）。

Both of these tests fail, and both should. When I am unit-testing, I am far more interested in failure than I am in success as that is where I need to concentrate.

这两个测试都失败了，而且都应该失败。当我进行单元测试时，我对失败比对成功更感兴趣，因为那是我需要集中精力的地方。

FF
======================================================================
FAIL: test_creation_defaults (__main__.CarTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_car.py", line 25, in test_creation_defaults
    self.assertEqual(self.car.wheel_count, 4)  # Failure!
AssertionError: 3 != 4

======================================================================
FAIL: test_creation_parameters (__main__.CarTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_car.py", line 20, in test_creation_parameters
    self.assertEqual(self.car.model, self.model)  # Failure!
AssertionError: 'Ford' != 'Model T'

----------------------------------------------------------------------
Ran 2 tests in 0.000s

FAILED (failures=2)

回答by Lennart Regebro

Do each assert in a separate method.

在单独的方法中执行每个断言。

class MathTest(unittest.TestCase):
  def test_addition1(self):
    self.assertEqual(1 + 0, 1)

  def test_addition2(self):
    self.assertEqual(1 + 1, 3)

  def test_addition3(self):
    self.assertEqual(1 + (-1), 0)

  def test_addition4(self):
    self.assertEqaul(-1 + (-1), -1)

回答by Anthony Batchelor

Another way to have non-fatal assertions is to capture the assertion exception and store the exceptions in a list. Then assert that that list is empty as part of the tearDown.

获得非致命断言的另一种方法是捕获断言异常并将异常存储在列表中。然后断言该列表是空的，作为拆卸的一部分。

import unittest

class Car(object):
  def __init__(self, make, model):
    self.make = make
    self.model = make  # Copy and paste error: should be model.
    self.has_seats = True
    self.wheel_count = 3  # Typo: should be 4.

class CarTest(unittest.TestCase):
  def setUp(self):
    self.verificationErrors = []

  def tearDown(self):
    self.assertEqual([], self.verificationErrors)

  def test_init(self):
    make = "Ford"
    model = "Model T"
    car = Car(make=make, model=model)
    try: self.assertEqual(car.make, make)
    except AssertionError, e: self.verificationErrors.append(str(e))
    try: self.assertEqual(car.model, model)  # Failure!
    except AssertionError, e: self.verificationErrors.append(str(e))
    try: self.assertTrue(car.has_seats)
    except AssertionError, e: self.verificationErrors.append(str(e))
    try: self.assertEqual(car.wheel_count, 4)  # Failure!
    except AssertionError, e: self.verificationErrors.append(str(e))

if __name__ == "__main__":
    unittest.main()

回答by hwiechers

One option is assert on all the values at once as a tuple.

一种选择是将所有值作为元组一次断言。

For example:

例如：

class CarTest(unittest.TestCase):
  def test_init(self):
    make = "Ford"
    model = "Model T"
    car = Car(make=make, model=model)
    self.assertEqual(
            (car.make, car.model, car.has_seats, car.wheel_count),
            (make, model, True, 4))

The output from this tests would be:

此测试的输出将是：

======================================================================
FAIL: test_init (test.CarTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\temp\py_mult_assert\test.py", line 17, in test_init
    (make, model, True, 4))
AssertionError: Tuples differ: ('Ford', 'Ford', True, 3) != ('Ford', 'Model T', True, 4)

First differing element 1:
Ford
Model T

- ('Ford', 'Ford', True, 3)
?           ^ -          ^

+ ('Ford', 'Model T', True, 4)
?           ^  ++++         ^

This shows that both the model and the wheel count are incorrect.

这表明模型和轮数都不正确。

回答by Zoro_77

I liked the approach by @Anthony-Batchelor, to capture the AssertionError exception. But a slight variation to this approach using decorators and also a way to report the tests cases with pass/fail.

我喜欢@Anthony-Batchelor 的方法来捕获 AssertionError 异常。但是这种使用装饰器的方法略有不同，也是一种报告测试用例通过/失败的方法。

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import unittest

class UTReporter(object):
    '''
    The UT Report class keeps track of tests cases
    that have been executed.
    '''
    def __init__(self):
        self.testcases = []
        print "init called"

    def add_testcase(self, testcase):
        self.testcases.append(testcase)

    def display_report(self):
        for tc in self.testcases:
            msg = "=============================" + "\n" + \
                "Name: " + tc['name'] + "\n" + \
                "Description: " + str(tc['description']) + "\n" + \
                "Status: " + tc['status'] + "\n"
            print msg

reporter = UTReporter()

def assert_capture(*args, **kwargs):
    '''
    The Decorator defines the override behavior.
    unit test functions decorated with this decorator, will ignore
    the Unittest AssertionError. Instead they will log the test case
    to the UTReporter.
    '''
    def assert_decorator(func):
        def inner(*args, **kwargs):
            tc = {}
            tc['name'] = func.__name__
            tc['description'] = func.__doc__
            try:
                func(*args, **kwargs)
                tc['status'] = 'pass'
            except AssertionError:
                tc['status'] = 'fail'
            reporter.add_testcase(tc)
        return inner
    return assert_decorator



class DecorateUt(unittest.TestCase):

    @assert_capture()
    def test_basic(self):
        x = 5
        self.assertEqual(x, 4)

    @assert_capture()
    def test_basic_2(self):
        x = 4
        self.assertEqual(x, 4)

def main():
    #unittest.main()
    suite = unittest.TestLoader().loadTestsFromTestCase(DecorateUt)
    unittest.TextTestRunner(verbosity=2).run(suite)

    reporter.display_report()


if __name__ == '__main__':
    main()

Output from console:

控制台输出：

(awsenv)$ ./decorators.py 
init called
test_basic (__main__.DecorateUt) ... ok
test_basic_2 (__main__.DecorateUt) ... ok

----------------------------------------------------------------------
Ran 2 tests in 0.000s

OK
=============================
Name: test_basic
Description: None
Status: fail

=============================
Name: test_basic_2
Description: None
Status: pass

回答by Ken

expect is very useful in gtest. This is python way in gist, and code:

expect 在 gtest 中非常有用。这是gist 中的python方式，代码：

import sys
import unittest


class TestCase(unittest.TestCase):
    def run(self, result=None):
        if result is None:
            self.result = self.defaultTestResult()
        else:
            self.result = result

        return unittest.TestCase.run(self, result)

    def expect(self, val, msg=None):
        '''
        Like TestCase.assert_, but doesn't halt the test.
        '''
        try:
            self.assert_(val, msg)
        except:
            self.result.addFailure(self, sys.exc_info())

    def expectEqual(self, first, second, msg=None):
        try:
            self.failUnlessEqual(first, second, msg)
        except:
            self.result.addFailure(self, sys.exc_info())

    expect_equal = expectEqual

    assert_equal = unittest.TestCase.assertEqual
    assert_raises = unittest.TestCase.assertRaises


test_main = unittest.main

回答by skia.heliou

There is a soft assertion package in PyPI called softestthat will handle your requirements. It works by collecting the failures, combining exception and stack trace data, and reporting it all as part of the usual unittestoutput.

PyPI 中有一个软断言包softest可以处理您的需求。它的工作原理是收集故障、组合异常和堆栈跟踪数据，并将其作为通常unittest输出的一部分进行报告。

For instance, this code:

例如，这段代码：

import softest

class ExampleTest(softest.TestCase):
    def test_example(self):
        # be sure to pass the assert method object, not a call to it
        self.soft_assert(self.assertEqual, 'Worf', 'wharf', 'Klingon is not ship receptacle')
        # self.soft_assert(self.assertEqual('Worf', 'wharf', 'Klingon is not ship receptacle')) # will not work as desired
        self.soft_assert(self.assertTrue, True)
        self.soft_assert(self.assertTrue, False)

        self.assert_all()

if __name__ == '__main__':
    softest.main()

...produces this console output:

...产生这个控制台输出：

======================================================================
FAIL: "test_example" (ExampleTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\...\softest_test.py", line 14, in test_example
    self.assert_all()
  File "C:\...\softest\case.py", line 138, in assert_all
    self.fail(''.join(failure_output))
AssertionError: ++++ soft assert failure details follow below ++++

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
The following 2 failures were found in "test_example" (ExampleTest):
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Failure 1 ("test_example" method)
+--------------------------------------------------------------------+
Traceback (most recent call last):
  File "C:\...\softest_test.py", line 10, in test_example
    self.soft_assert(self.assertEqual, 'Worf', 'wharf', 'Klingon is not ship receptacle')
  File "C:\...\softest\case.py", line 84, in soft_assert
    assert_method(*arguments, **keywords)
  File "C:\...\Python\Python36-32\lib\unittest\case.py", line 829, in assertEqual
    assertion_func(first, second, msg=msg)
  File "C:\...\Python\Python36-32\lib\unittest\case.py", line 1203, in assertMultiLineEqual
    self.fail(self._formatMessage(msg, standardMsg))
  File "C:\...\Python\Python36-32\lib\unittest\case.py", line 670, in fail
    raise self.failureException(msg)
AssertionError: 'Worf' != 'wharf'
- Worf
+ wharf
 : Klingon is not ship receptacle

+--------------------------------------------------------------------+
Failure 2 ("test_example" method)
+--------------------------------------------------------------------+
Traceback (most recent call last):
  File "C:\...\softest_test.py", line 12, in test_example
    self.soft_assert(self.assertTrue, False)
  File "C:\...\softest\case.py", line 84, in soft_assert
    assert_method(*arguments, **keywords)
  File "C:\...\Python\Python36-32\lib\unittest\case.py", line 682, in assertTrue
    raise self.failureException(msg)
AssertionError: False is not true


----------------------------------------------------------------------
Ran 1 test in 0.000s

FAILED (failures=1)

NOTE: I created and maintain softest.

注意：我创建并维护了softest.

回答by user

I had a problem with the answer from @Anthony Batchelorbecause it would have forced me to use try...catchinside my unit tests. Instead, I encapsulated the try...catchlogic in an override of the TestCase.assertEqualmethod. Here is the code:

我对@Anthony Batchelor的回答有疑问，因为它会迫使我try...catch在单元测试中使用。相反，我将try...catch逻辑封装在TestCase.assertEqual方法的覆盖中。这是代码：

import unittest
import traceback

class AssertionErrorData(object):

    def __init__(self, stacktrace, message):
        super(AssertionErrorData, self).__init__()
        self.stacktrace = stacktrace
        self.message = message

class MultipleAssertionFailures(unittest.TestCase):

    def __init__(self, *args, **kwargs):
        self.verificationErrors = []
        super(MultipleAssertionFailures, self).__init__( *args, **kwargs )

    def tearDown(self):
        super(MultipleAssertionFailures, self).tearDown()

        if self.verificationErrors:
            index = 0
            errors = []

            for error in self.verificationErrors:
                index += 1
                errors.append( "%s\nAssertionError %s: %s" % ( 
                        error.stacktrace, index, error.message ) )

            self.fail( '\n\n' + "\n".join( errors ) )
            self.verificationErrors.clear()

    def assertEqual(self, goal, results, msg=None):

        try:
            super( MultipleAssertionFailures, self ).assertEqual( goal, results, msg )

        except unittest.TestCase.failureException as error:
            goodtraces = self._goodStackTraces()
            self.verificationErrors.append( 
                    AssertionErrorData( "\n".join( goodtraces[:-2] ), error ) )

    def _goodStackTraces(self):
        """
            Get only the relevant part of stacktrace.
        """
        stop = False
        found = False
        goodtraces = []

        # stacktrace = traceback.format_exc()
        # stacktrace = traceback.format_stack()
        stacktrace = traceback.extract_stack()

        # https://stackoverflow.com/questions/54499367/how-to-correctly-override-testcase
        for stack in stacktrace:
            filename = stack.filename

            if found and not stop and \
                    not filename.find( 'lib' ) < filename.find( 'unittest' ):
                stop = True

            if not found and filename.find( 'lib' ) < filename.find( 'unittest' ):
                found = True

            if stop and found:
                stackline = '  File "%s", line %s, in %s\n    %s' % ( 
                        stack.filename, stack.lineno, stack.name, stack.line )
                goodtraces.append( stackline )

        return goodtraces

# class DummyTestCase(unittest.TestCase):
class DummyTestCase(MultipleAssertionFailures):

    def setUp(self):
        self.maxDiff = None
        super(DummyTestCase, self).setUp()

    def tearDown(self):
        super(DummyTestCase, self).tearDown()

    def test_function_name(self):
        self.assertEqual( "var", "bar" )
        self.assertEqual( "1937", "511" )

if __name__ == '__main__':
    unittest.main()

Result output:

结果输出：

F
======================================================================
FAIL: test_function_name (__main__.DummyTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\User\Downloads\test.py", line 77, in tearDown
    super(DummyTestCase, self).tearDown()
  File "D:\User\Downloads\test.py", line 29, in tearDown
    self.fail( '\n\n' + "\n\n".join( errors ) )
AssertionError: 

  File "D:\User\Downloads\test.py", line 80, in test_function_name
    self.assertEqual( "var", "bar" )
AssertionError 1: 'var' != 'bar'
- var
? ^
+ bar
? ^
 : 

  File "D:\User\Downloads\test.py", line 81, in test_function_name
    self.assertEqual( "1937", "511" )
AssertionError 2: '1937' != '511'
- 1937
+ 511
 : 

More alternative solutions for the correct stacktrace capture could be posted on How to correctly override TestCase.assertEqual(), producing the right stacktrace?

更多关于正确堆栈跟踪捕获的替代解决方案可以发布在如何正确覆盖 TestCase.assertEqual()，生成正确的堆栈跟踪？