Java 在 JUnit 中将单元测试标记为预期失败

Mark unit test as an expected failure in JUnit

提问by Kaleb Pederson

How can I mark a test as an expected failure in JUnit 4?

如何在 JUnit 4 中将测试标记为预期失败？

In this case I want to continue to run this test until something is patched upstream. Ignoring the test goes a little too far, as then I might forget about it. I may be able to add an @expectedannotation and catch the exception thrown by assertThat, but that also seems to lie about the expected behavior.

在这种情况下，我想继续运行此测试，直到上游修补某些内容为止。忽略测试有点过头了，因为那样我可能会忘记它。我也许可以添加一个@expected注释并捕获由 抛出的异常assertThat，但这似乎也与预期的行为有关。

Here's what my current test looks like:

这是我当前的测试：

@Test
public void unmarshalledDocumentHasExpectedValue() 
{
    doc = unmarshaller.unmarshal(getResourceAsStream("mydoc.xml"));
    final ST title = doc.getTitle();
    assertThat(doc.getTitle().toStringContent(), equalTo("Expected"));
}

That assert should succeed, but because of an upstream bug it doesn't. Yet, that test is correct; it should succeed. Virtually all the alternatives that I've found are misleading. Right now I think @Ignore("This test should pass once fixed upstream")is my best bet, but I still have to remember to come back to it. I'd prefer that the test run.

该断言应该成功，但由于上游错误，它没有成功。然而，那个测试是正确的。它应该会成功。我发现的几乎所有替代方案都具有误导性。现在我认为这@Ignore("This test should pass once fixed upstream")是我最好的选择，但我仍然要记得回来。我更喜欢测试运行。

In Python I can use the expectedFailuredecorator:

在 Python 中，我可以使用expectedFailure装饰器：

class ExpectedFailureTestCase(unittest.TestCase):
    @unittest.expectedFailure
    def test_fail(self):
        self.assertEqual(1, 0, "broken")

With Qt's QTestLibin C++, you can use QEXPECT_FAIL:

使用C++ 中的Qt 的QTestLib，您可以使用QEXPECT_FAIL：

QEXPECT_FAIL("", "Will be fixed next version", Continue);
QCOMPARE(i, 42);

In both cases above, the unit test runs which is what I'm hoping to have happen. Am I missing something in JUnit?

在上述两种情况下，单元测试都会运行，这正是我希望发生的。我在 JUnit 中遗漏了什么吗？

采纳答案by Matthew Farwell

I'm assuming here that you want the test to pass if your assert fails, but if the assert succeeds, then the test should pass as well.

我在这里假设您希望在断言失败时测试通过，但如果断言成功，则测试也应该通过。

The easiest way to do this is to use a TestRule. TestRule gives the opportunity to execute code before and after a test method is run. Here is an example:

最简单的方法是使用TestRule。TestRule 提供了在运行测试方法之前和之后执行代码的机会。下面是一个例子：

public class ExpectedFailureTest {
    public class ExpectedFailure implements TestRule {
        public Statement apply(Statement base, Description description) {
            return statement(base, description);
        }

        private Statement statement(final Statement base, final Description description) {
            return new Statement() {
                @Override
                public void evaluate() throws Throwable {
                    try {
                        base.evaluate();
                    } catch (Throwable e) {
                        if (description.getAnnotation(Deprecated.class) != null) {
                            // you can do whatever you like here.
                            System.err.println("test failed, but that's ok:");
                        } else {
                            throw e;
                        }
                    }
                }
            };
        }
    }

    @Rule public ExpectedFailure expectedFailure = new ExpectedFailure();

    // actually fails, but we catch the exception and make the test pass.
    @Deprecated
    @Test public void testExpectedFailure() {
        Object o = null;
        o.equals("foo");
    }

    // fails
    @Test public void testExpectedFailure2() {
        Object o = null;
        o.equals("foo");
    }
}

First, note that the first method is marked as @Deprecated. I'm using this as a marker for the method for which I want to ignore any assertion failures. You can do whatever you like to identify the methods, this is just an example.

首先，请注意第一种方法被标记为@Deprecated。我使用它作为我想忽略任何断言失败的方法的标记。你可以做任何你喜欢识别的方法，这只是一个例子。

Next, in the ExpectedFailure#apply(), when I do the base.evaluate(), I'm catching any Throwable (which includes AssertionError) and if the method is marked with the annotation @Deprecated, I ignore the error. You can perform whatever logic you like to decide whether you should ignore the error or not, based on version number, some text, etc. You can also pass a dynamically determined flag into ExpectedFailure to allow it to fail for certain version numbers:

接下来，在 中ExpectedFailure#apply()，当我执行 base.evaluate() 时，我会捕获任何 Throwable（包括 AssertionError），如果该方法标有注解 @Deprecated，我将忽略该错误。您可以根据版本号、某些文本等执行任何您喜欢的逻辑来决定是否应该忽略错误。您还可以将动态确定的标志传递给 ExpectedFailure 以允许它在某些版本号上失败：

public void unmarshalledDocumentHasExpectedValue() {
    doc = unmarshaller.unmarshal(getResourceAsStream("mydoc.xml"));

    expectedFailure.setExpectedFailure(doc.getVersionNumber() < 3000);

    final ST title = doc.getTitle();
    assertThat(doc.getTitle().toStringContent(), equalTo("Expected"));
}

For further examples, see ExternalResource, and ExpectedException

有关更多示例，请参阅ExternalResource和ExpectedException

Ignoring an expected failure test rather than passing it

忽略预期的失败测试而不是通过它

If you want to mark you tests as Ignored rather than Success, it becomes a bit more complex, because tests are ignored before they are executed, so you have to retrospectively mark a test as ignored, which would involve constructing your own Runner. To give you a start, see my answer to How to define JUnit method rule in a suite?. Or ask another question.

如果你想将你的测试标记为忽略而不是成功，它会变得有点复杂，因为测试在执行之前被忽略，所以你必须回顾性地将测试标记为忽略，这将涉及构建你自己的 Runner。首先，请参阅我对如何在套件中定义 JUnit 方法规则的回答？. 或者再问一个问题。

回答by Brad Mace

I'm not quite getting the specifics of your scenario, but here's how I generally test for expected failure:

我不太了解您的场景的细节，但这是我通常测试预期失败的方法：

The slick new way:

光滑的新方法：

@Test(expected=NullPointerException.class)
public void expectedFailure() {
    Object o = null;
    o.toString();
}

for older versions of JUnit:

对于旧版本的 JUnit：

public void testExpectedFailure() {
    try {
        Object o = null;
        o.toString();
        fail("shouldn't get here");
    }
    catch (NullPointerException e) {
        // expected
    }
}

If you have a bunch of things that you want to ensure throw an exception, you may also want to use this second technique inside a loop rather than creating a separate test method for each case. If you were just to loop through a bunch of cases in a single method using expected, the first one to throw an exception would end the test, and the subsequent cases wouldn't get checked.

如果您有一堆要确保抛出异常的事情，您可能还想在循环中使用第二种技术，而不是为每种情况创建单独的测试方法。如果您只是在一个方法中使用 循环遍历一堆案例expected，第一个抛出异常的案例将结束测试，并且不会检查后续案例。

回答by Aaron McIver

One option is mark the test as @Ignoreand put text in there that is a bug perhaps and awaiting a fix. That way it won't run. It will then become skipped. You could also make use of the extensionsto suit your need in a potentially different way.

一种选择是将测试标记为@Ignore并在其中放置可能是错误并等待修复的文本。这样就跑不起来了。然后它会被跳过。您还可以使用这些扩展以一种可能不同的方式满足您的需求。

回答by topchef

Use mocked upstream class if possible. Stub it with correct result. Optionally, replace mock with real object after bug is fixed.

如果可能，请使用模拟的上游类。用正确的结果存根。或者，在修复错误后用真实对象替换模拟。

回答by pholser

What about explicitly expecting an AssertionError?

明确期望 AssertionError 怎么样？

@Test(expected = AssertionError.class)
public void unmarshalledDocumentHasExpectedValue() {
    // ...
}

If you're reasonably confident that only the JUnit machinery within the test would raise AssertionError, this seems as self-documenting as anything.

如果您有理由相信只有测试中的 JUnit 机器会引发 AssertionError，那么这似乎是自记录的。

You'd still run the risk of forgetting about such a test. I wouldn't let such tests into version control for long, if ever.

你仍然会冒着忘记这样一个测试的风险。如果有的话，我不会让这样的测试长期进入版本控制。

回答by Stefan Haberl

I've taken Matthew's answer a step further and actually implemented an @Optionalannotation you could use instead of the @Deprecatedmarker annotation he mentions in his answer. Although simple, I'll share the code with you, maybe it's of help for someone:

我已经采取了更进一步的 Matthew's answer 并实际实现了一个@Optional注释，您可以使用它来代替@Deprecated他在答案中提到的标记注释。虽然简单，我还是把代码分享给大家，也许对某人有帮助：

@Target(ElementType.METHOD)
@Retention(RetentionPolicy.RUNTIME)
@Documented
public @interface Optional {

  /**
   * Specify a Throwable, to cause a test method to succeed even if an exception
   * of the specified class is thrown by the method.
   */
  Class<? extends Throwable>[] exception();
}

With a simple alteration of Matt's ExpectedFailureclass:

对马特的ExpectedFailure课程做一个简单的改动：

public class ExpectedFailure implements TestRule {

  @Override
  public Statement apply(final Statement base, final Description description) {
    return statement(base, description);
  }

  private Statement statement(final Statement base, final Description description) {
    return new Statement() {

      @Override
      public void evaluate() throws Throwable {
        try {
          base.evaluate();
        } catch (Throwable e) {
          // check for certain exception types
          Optional annon = description.getAnnotation(Optional.class);
          if (annon != null && ArrayUtils.contains(annon.exception(), e.getClass())) {
            // ok
          } else {
            throw e;
          }
        }
      }
    };
  }
}

You can now annotate your test method with @Optionaland it will not fail, even if the given type of exception is raised (provide one or more types you would like the test method to pass):

您现在可以使用注释测试方法@Optional并且它不会失败，即使引发了给定类型的异常（提供您希望测试方法通过的一种或多种类型）：

public class ExpectedFailureTest {

  @Rule public ExpectedFailure expectedFailure = new ExpectedFailure();

  // actually fails, but we catch the exception and make the test pass.
  @Optional(exception = NullPointerException.class)
  @Test public void testExpectedFailure() {
      Object o = null;
      o.equals("foo");
  }

}

[UPDATE]

[更新]

You could also rewrite your tests using JUnit's org.junit.Assumeinstead of the tradtional org.junit.Assert, if you want your tests to pass even if the assumption does not hold.

如果您希望您的测试通过，即使假设不成立，您也可以使用 JUnitorg.junit.Assume而不是 tradtional重写您的测试org.junit.Assert。

From Assume's JavaDoc:

来自Assume的 JavaDoc：

A set of methods useful for stating assumptions about the conditions in which a test is meaningful.A failed assumption does not mean the code is broken, but that the test provides no useful information. The default JUnit runner treats tests with failing assumptions as ignored.

一组方法，用于陈述有关测试有意义的条件的假设。失败的假设并不意味着代码被破坏，而是测试没有提供有用的信息。默认的 JUnit 运行器将假设失败的测试视为忽略。

Assumeis available since JUnit 4.4

Assume自 JUnit 4.4 起可用