1berry/qtbase
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Rewrite the QtTest best practice guide on testing for warnings

Browse Source 
It previously only covered ignoring them; we can now turn them into
errors. Expand the section to include some motivation for why it is
best practice to purge warnings from test output. That prepares the
scene for how to suppress what's expected and make errors of the rest.

Change-Id: Ieb4b14b2b048d1db2fead004ee7471b82507722f
Reviewed-by: Mitch Curtis <mitch.curtis@qt.io>
(cherry picked from commit 2d73cc2b1df03df716e6a76f9dca7761c1363368)
Reviewed-by: Qt Cherry-pick Bot <cherrypick_bot@qt-project.org>
This commit is contained in:
Edward Welbourne 2022-10-14 13:54:05 +02:00 committed by Qt Cherry-pick Bot
parent ddde19129c
commit 755d1b2948
2 changed files with 38 additions and 13 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

47
src/testlib/doc/src/qttest-best-practices.qdoc
View File

@ -355,24 +355,47 @@
helpful test output:
\list
\li \l {Explicitly Ignore Expected Warnings}
\li \l {Test for Warnings}
\li \l {Avoid Printing Debug Messages from Autotests}
\li \l {Write Well-structured Diagnostic Code}
\endlist
\section2 Explicitly Ignore Expected Warnings
\section2 Test for Warnings
If a test is expected to cause Qt to output a warning or debug message
on the console, it should call \l QTest::ignoreMessage() to filter that
message out of the test output and to fail the test if the message is
not output.
Just as when building your software, if test output is cluttered with
warnings you will find it harder to notice a warning that really is a clue
to the emergence of a bug. It is thus prudent to regularly check your test
logs for warnings, and other extraneous output, and investigate the
causes. When they are signs of a bug, you can make warnings trigger test
failure.
If such a message is only output when Qt is built in debug mode, use
\l QLibraryInfo::isDebugBuild() to determine whether the Qt libraries
were built in debug mode. Using \c{#ifdef QT_DEBUG} is not enough, as
it will only tell you whether the test was built in debug mode, and
that does not guarantee that the Qt libraries were also built in debug
mode.
When the code under test \e should produce messages, such as warnings
about misguided use, it is also important to test that it \e does produce
them when so used. You can test for expected messages from the code under
test, produced by \l qWarning(), \l qDebug(), \l qInfo() and friends,
using \l QTest::ignoreMessage(). This will verify that the message is
produced and filter it out of the output of the test run. If the message
is not produced, the test will fail.
If an expected message is only output when Qt is built in debug mode, use
\l QLibraryInfo::isDebugBuild() to determine whether the Qt libraries were
built in debug mode. Using \c{#ifdef QT_DEBUG} is not enough, as it will
only tell you whether \e{the test} was built in debug mode, and that does
not guarantee that the \e{Qt libraries} were also built in debug mode.
Your tests can (since Qt 6.3) verify that they do not trigger calls to \l
qWarning() by calling \l QTest::failOnWarnings(). This takes the warning
message to test for or a \l QRegularExpression to match against warnings; if
a matching warning is produced, it will be reported and cause the test to
fail. For example, a test that should produce no warnings at all can
\c{QTest::failOnWarnings(QRegularExpression(u".*"_s))}, which will match any
warning at all.
You can also set the environment variable \c QT_FATAL_WARNINGS to cause
warnings to be treated as fatal errors. See \l qWarning() for details; this
is not specific to autotests. If warnings would otherwise be lost in vast
test logs, the occasional run with this environment variable set can help
you to find and eliminate any that do arise.
\section2 Avoid Printing Debug Messages from Autotests

4
src/testlib/doc/src/qttestlib-manual.qdoc
View File

@ -733,7 +733,9 @@
The names of rows and columns, in a given test function's data table, should
be unique: if two rows share a name, or two columns share a name, a warning
will (since Qt 6.5) be produced.
will (since Qt 6.5) be produced. See \l qWarning() for how you can cause
warnings to be treated as errors and \l {Test for Warnings} for how to get
your tests clear of other warnings.
\section1 Rewriting the Test Function