You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Test with comment 'This test fails because ... java.lang.ArithmeticException' does not fail #1272
UTBot could find situations which leads to arithmetic overflow and generate tests for such cases adding comment that such test would fail, but actually such test pass.
To Reproduce
Steps to reproduce the behavior:
Open utbot project in IntelliJ Idea
In UTBot settings set Overflow detection: to Treat overflows as errors
Set Test generated method to 100 % Symbolic
Generate tests for org.utbot.examples.math.OverflowExamples#intOverflow method
Open the generated test
Expected behavior
Either UTBot generates test which causes ArithmeticException during execution
Or it doesn't write that test should fail
Actual behavior
Test which are supposed to generate errors are generated, but during execution they pass.
Sample of such test:
@Test@DisplayName("intOverflow: x * x * x > 0 -> ThrowArithmeticException")
publicvoidtestIntOverflow_ThrowArithmeticException() {
OverflowExamplesoverflowExamples = newOverflowExamples();
/* This test fails because method [org.utbot.examples.math.OverflowExamples.intOverflow] produces [java.lang.ArithmeticException: int * overflow] */overflowExamples.intOverflow(-11086, 0);
}
Visual proofs (screenshots, logs, images)
Additional context
So this case is more about how to properly present results of UTBot finding. Currently there are at least two problems:
incorrect comment that test fails
as test pass it would be difficult to find such test in case of bulk test generation, so overflow problem could be left unnoticed.
Probably we should add fail() to such test, so it would actually fail and draw user attention.
Activity
denis-fokin commentedon Dec 21, 2022
Not enough resources, move to the next release
New treating of artificial errors #1272 (#1573)
New treating of artificial errors #1272 (#1573)
New treating of artificial errors #1272 (#1573)