1berry/qtbase
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Tweak a selftest blacklist to exercise more of the blacklisting code

Browse Source 
In the process, corrected an inaccurate XFAIL message (an XPASS is
normally an error, unless blacklisting ignores it so turns it into a
BPASS). Added the missing .tap file to its expected output.
Documented the similarity to the silent/ selftest.

Task-number: QTPM-1385
Change-Id: Id74a1353d54af2f3bfe2c764e33c1f051958ab21
Reviewed-by: Jędrzej Nowacki <jedrzej.nowacki@qt.io>
This commit is contained in:
Edward Welbourne 2018-10-11 15:11:38 +02:00
parent f00b72f133
commit 33d7f76f0e
8 changed files with 63 additions and 11 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

14
tests/auto/testlib/selftests/blacklisted/BLACKLIST
View File

@ -1,12 +1,20 @@
[pass]
obscure # no such platform; is ignored
*
[pass]
!*
[skip]
*
[fail]
*
[xpass]
*
[xfail]
*
[xpass]
*
[messages]
*

5
tests/auto/testlib/selftests/blacklisted/tst_blacklisted.cpp
View File

@ -45,7 +45,8 @@ private slots:
void messages();
};
// All the tests below have been blacklisted in blacklisted/BLACKLIST
// All the tests below except pass() have been blacklisted in blacklisted/BLACKLIST
// Contrast with ../silent/, for the same tests without blacklisting but with -silent
void tst_Blacklisted::pass()
{
@ -64,7 +65,7 @@ void tst_Blacklisted::fail()
void tst_Blacklisted::xfail()
{
QEXPECT_FAIL("", "This test should XFAIL then BFAIL", Abort);
QEXPECT_FAIL("", "This test should XFAIL then BPASS", Abort);
QVERIFY(false);
}

2
tests/auto/testlib/selftests/expected_blacklisted.lightxml
View File

@ -25,7 +25,7 @@
</TestFunction>
<TestFunction name="xfail">
<Incident type="xfail" file="qtbase/tests/auto/testlib/selftests/blacklisted/tst_blacklisted.cpp" line="0">
<Description><![CDATA[This test should XFAIL then BFAIL]]></Description>
<Description><![CDATA[This test should XFAIL then BPASS]]></Description>
</Incident>
<Incident type="bpass" file="" line="0" />
<Duration msecs="0"/>

43
tests/auto/testlib/selftests/expected_blacklisted.tap Normal file
View File

@ -0,0 +1,43 @@
TAP version 13
# tst_Blacklisted
ok 1 - initTestCase()
ok 2 - pass() # TODO
ok 3 - skip() # SKIP This test should SKIP
not ok 4 - fail() # TODO 'false' returned FALSE. (This test should BFAIL)
---
type: QVERIFY
message: This test should BFAIL
wanted: true (false)
found: false (false)
expected: true (false)
actual: false (false)
at: tst_Blacklisted::fail() (qtbase/tests/auto/testlib/selftests/blacklisted/tst_blacklisted.cpp:63)
file: qtbase/tests/auto/testlib/selftests/blacklisted/tst_blacklisted.cpp
line: 63
...
not ok 5 - xfail() # TODO This test should XFAIL then BPASS
---
at: tst_Blacklisted::xfail() (qtbase/tests/auto/testlib/selftests/blacklisted/tst_blacklisted.cpp:69)
file: qtbase/tests/auto/testlib/selftests/blacklisted/tst_blacklisted.cpp
line: 69
...
ok 5 - xfail() # TODO
ok 6 - xpass() # TODO 'true' returned TRUE unexpectedly. (This test should XPASS, blacklist ignored for XPASS)
# This is a warning that should not appear in silent test output
# This is an internal testlib warning that should not appear in silent test output
# This is a debug message that should not appear in silent test output
# This is a critical message that should not appear in silent test output
# This is an info message that should not appear in silent test output
# This is an internal testlib info message that should not appear in silent test output
# This is a fatal error message that should still appear in silent test output
not ok 7 - messages() # TODO Received a fatal error.
---
# Received a fatal error.
at: tst_Blacklisted::messages() (Unknown file:0)
file: Unknown file
line: 0
...
1..7
# tests 7
# pass 1
# fail 1

2
tests/auto/testlib/selftests/expected_blacklisted.teamcity
View File

@ -7,7 +7,7 @@
##teamcity[testStarted name='fail()' flowId='tst_Blacklisted']
##teamcity[testFinished name='fail()' flowId='tst_Blacklisted']
##teamcity[testStarted name='xfail()' flowId='tst_Blacklisted']
##teamcity[testStdOut name='xfail()' out='XFAIL |[Loc: qtbase/tests/auto/testlib/selftests/blacklisted/tst_blacklisted.cpp(0)|]: This test should XFAIL then BFAIL' flowId='tst_Blacklisted']
##teamcity[testStdOut name='xfail()' out='XFAIL |[Loc: qtbase/tests/auto/testlib/selftests/blacklisted/tst_blacklisted.cpp(0)|]: This test should XFAIL then BPASS' flowId='tst_Blacklisted']
##teamcity[testFinished name='xfail()' flowId='tst_Blacklisted']
##teamcity[testStarted name='xpass()' flowId='tst_Blacklisted']
##teamcity[testFailed name='xpass()' message='Failure! |[Loc: qtbase/tests/auto/testlib/selftests/blacklisted/tst_blacklisted.cpp(0)|]' details='|'true|' returned TRUE unexpectedly. (This test should XPASS, blacklist ignored for XPASS)' flowId='tst_Blacklisted']

2
tests/auto/testlib/selftests/expected_blacklisted.txt
View File

@ -6,7 +6,7 @@ SKIP : tst_Blacklisted::skip() This test should SKIP
Loc: [qtbase/tests/auto/testlib/selftests/blacklisted/tst_blacklisted.cpp(0)]
BFAIL : tst_Blacklisted::fail() 'false' returned FALSE. (This test should BFAIL)
Loc: [qtbase/tests/auto/testlib/selftests/blacklisted/tst_blacklisted.cpp(0)]
XFAIL : tst_Blacklisted::xfail() This test should XFAIL then BFAIL
XFAIL : tst_Blacklisted::xfail() This test should XFAIL then BPASS
Loc: [qtbase/tests/auto/testlib/selftests/blacklisted/tst_blacklisted.cpp(0)]
BPASS : tst_Blacklisted::xfail()
XPASS : tst_Blacklisted::xpass() 'true' returned TRUE unexpectedly. (This test should XPASS, blacklist ignored for XPASS)

2
tests/auto/testlib/selftests/expected_blacklisted.xml
View File

@ -27,7 +27,7 @@
</TestFunction>
<TestFunction name="xfail">
<Incident type="xfail" file="qtbase/tests/auto/testlib/selftests/blacklisted/tst_blacklisted.cpp" line="0">
<Description><![CDATA[This test should XFAIL then BFAIL]]></Description>
<Description><![CDATA[This test should XFAIL then BPASS]]></Description>
</Incident>
<Incident type="bpass" file="" line="0" />
<Duration msecs="0"/>

4
tests/auto/testlib/selftests/expected_blacklisted.xunitxml
View File

@ -12,7 +12,7 @@
</testcase>
<testcase result="bfail" name="fail"/>
<testcase result="xfail" name="xfail">
<!-- message="This test should XFAIL then BFAIL" type="info" -->
<!-- message="This test should XFAIL then BPASS" type="info" -->
</testcase>
<testcase result="xpass" name="xpass">
<failure message="&apos;true&apos; returned TRUE unexpectedly. (This test should XPASS, blacklist ignored for XPASS)" result="xpass"/>
@ -28,7 +28,7 @@
</testcase>
<system-err>
<![CDATA[This test should SKIP]]>
<![CDATA[This test should XFAIL then BFAIL]]>
<![CDATA[This test should XFAIL then BPASS]]>
<![CDATA[This is a warning that should not appear in silent test output]]>
<![CDATA[This is an internal testlib warning that should not appear in silent test output]]>
<![CDATA[This is a debug message that should not appear in silent test output]]>