This option does nothing -- the only function that checks if the option
has been set is never called. This appears to have been the case ever
since testlib was imported into the Qt repository in October 2005.
Change-Id: I837aa957e2d8bd47c3d1c551f1b94d4374daa25e
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
After the previous commit, QTest::qt_snprintf() is equivalent to
qsnprintf(), so just use that instead.
Change-Id: I89ad6e3749ba5efb1926b0b618a904e8baca9f52
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Previously, unprintable characters were filtered out of test output
while the output strings were being formatted by either qt_snprintf() or
qt_asprintf(). Any strings not formatted by one of those functions
weren't filtered at all, and any strings passed more than once would be
filtered more than once.
This commit separates the filtering of output strings from their
formatting, leaving the filtering until just before the strings are
written to the output stream. For now, the filtering is done by a
protected method of QAbstractTestLogger, but this could easily be
changed to a virtual method in future to allow different filtering
for loggers with different output character sets.
Change-Id: Ia4bb49cd10d37c84af75d2cf58325d27f0e16d99
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
The overload of QTest::qExec() that takes a QStringList is not used
anywhere in Qt's autotests, despite having been in the qtestlib API
since Qt 4.4.
This lack of use most likely derives from the fact that none of the
QTEST_MAIN macros use the overload, and more than 99% of Qt's tests
use those macros to avoid explicitly calling QTest::qExec().
Change-Id: I264b21d7fe1a9f2d565f748cf8bbe32414a73bb0
Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
Previously, autotests silently ignored the -graphicssystem and -qws
parameters so that test-runners did not need to include logic to decide
whether to pass those parameters depending on whether a particular test
was a GUI test or a non-GUI test.
In Qt 5, both of those parameters have been removed from QApplication,
so tests don't need to ignore them anymore as they should no longer be
supplied to any test.
Change-Id: Ic59e38ddfa3303e760645ed1782a211340ce70f2
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Update the documentation to make it clear that regular test applications
should not call QTest::qExec() more than once. Also minor rewording of
description of return value.
Change-Id: I45bdf520ed10fd3c9232847a0ec0bc2b32d4caf3
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Automated tests often need to load some data from external files.
Currently, a wide variety of approaches for this have been used in Qt
autotests, including:
- embed the source directory into the test binary at compile time, and
find the testdata relative to that; this fails when the source tree
is no longer available (e.g. when the tests are deployed to a device).
- use a path relative to the current working directory, and trust that
the caller always sets the current working directory such that the
testdata can be found; this fails when the caller uses a different
working directory than expected.
- use a path relative to QCoreApplication::applicationDirPath();
this fails when source tree != build tree (since testdata is not
automatically copied into the build tree).
- compile the files into the binary using the Qt resource system; this
should work, but does not allow for testing of code which genuinely
needs external files.
It seems that there is not a simple method for determining the testdata
path which can be reliably used in all circumstances, so various tests
have reinvented the testdata location method in different ways.
Therefore, this is a good candidate for an addition to the testlib API.
The current implementation of QFINDTESTDATA is able to find testdata
in all three of (build tree, install tree, source tree), in that order.
Change-Id: Ib2fed860723ccf437240da3b00db22dfe1a6b56c
Reviewed-by: Jason McDonald <jason.mcdonald@nokia.com>
Currently QTest::newRow function expects to have correct tag string for
each test data. This was enforced by a simple assert. By adding a
an assert with message we might simplify debugging of a test that
generates test cases.
Change-Id: I84ebb397264c99e7bb46d281fd38583dd2d6964c
Reviewed-by: Jason McDonald <jason.mcdonald@nokia.com>
To instrument a Qt application or library with the TestCocoon coverage
tool, do `CONFIG+=testcocoon' in the application .pro file.
To instrument Qt itself with testcocoon, use the `-testcocoon' configure
option.
Change-Id: Ie77109a078d11ea51f7a073621e0df9c752c44ae
Reviewed-by: Oswald Buddenhagen <oswald.buddenhagen@nokia.com>
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
(Note: This feature is ported from Qt 4.8.
See the following commits:
01575deafb7d26ca2431374e92c6d71de96547c7
4866d1ba8afbab61e102942d1ea93b81fea053d6
)
Passing the -datatags option to a QTestLib program prints the
available data tags to standard output.
For completeness, the test case name is also printed
at the start of each output line. (Although the file name
is supposed to match the lower-case version of the test case
name, this is currently not true in all cases (particularly not
under tests/benchmarks/). Even if there was a script to enforce this
convention, the -datatags option provides this information in a
reliable way.)
Data tags for each test function (f() in this case) are printed in
four different ways depending on the presence of local and global
data tags:
Case 1: No tags:
tst_MyTestCasetst_MyTestCase f
Case 2: Local tags only:
tst_MyTestCase f local tag 1
tst_MyTestCase f local tag 2
...
Case 3: Global tags only:
tst_MyTestCase f __global__ global tag 1
tst_MyTestCase f __global__ global tag 2
...
Case 4: Local and global tags:
tst_MyTestCase f local tag 1 __global__ global tag 1
tst_MyTestCase f local tag 2 __global__ global tag 1
...
tst_MyTestCase f local tag 1 __global__ global tag 2
tst_MyTestCase f local tag 2 __global__ global tag 2
...
...
Note that the string __global__ is assumed to be highly unlikely to occur
in a data tag (if it does, an ambiguity results).
Change-Id: Ib51aa0c3c32ad52e52ce519729292cf8f0ec5d50
Reviewed-by: Jason McDonald <jason.mcdonald@nokia.com>
Reviewed-by: Sergio Ahumada <sergio.ahumada@nokia.com>
The standard C getenv() returns NULL if the requested environment
variable is not found.
In Qt4 and later, qgetenv() does not return a null pointer if the
requested environment string is not defined. Instead it returns a
QByteArray containing an empty string. If using qgetenv(), there is no
way to tell the difference between an undefined environment variable
and one which is defined to be the empty string.
In testlib, all calls to qgetenv() were checking whether the returned
QByteArray's constData() returned a null pointer, but that would never
happen. These calls must instead check whether the QByteArray contains
a non-empty string.
Change-Id: I342f0e8b196896c26cccce3ff169fa1b9669b5ff
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
When executing a data-driven test, testlib executes the _data function
once, then repeatedly executes init(), then the test function, then
cleanup() for each row of test data.
Change-Id: Icfa1dd19a52fb1debbc92b7cbe13d85bfb0418c7
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
When the SkipAll mode is used, tests only report a SKIP for the first
line of test data and subsequent lines are not reported at all. This
behaviour makes it impossible for anything post-processing test results
to accurately report test pass- and run- rates because they cannot see
how many lines of test data were skipped.
This commit removes SkipMode. QSKIPs in regular test functions and data
functions are treated the same as SkipSingle, so that every skipped line
of local or global test data is reported in the test log. QSKIPs
elsewhere are treated the same as SkipAll -- skipping in init() causes
the next test function to be skipped entirely, and skipping in
initTestCase() or initTestCase_data() causes all test functions to be
skipped.
This commit only changes qtestlib and the selftests. A further commit
will change the autotests to remove the SkipMode parameter from QSKIP
calls.
Note that the change in expected output for the globaldata selftest is
deliberate, as the QSKIP in the skipLocal test function has effectively
changed from SkipAll to SkipSingle.
Task-number: QTBUG-21851, QTBUG-21652
Change-Id: I7b1c53fe7ca9dde032810b789d967e2a402bbe5d
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Reviewed-by: Alex <alex.blasche@nokia.com>
These functions have lived in tests/shared/util.h for a long time, but
they really belong in qtestlib.
Change-Id: I60d569d002dea220b51563931d8b7aa77a20b98b
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
This macro is no longer used in Qt's tests and encourages writing tests
in a way that makes test reporting less accurate -- remove it to prevent
further misuse. If a test can be determined at compile-time to be
inapplicable, it should be omitted from the build via .pro file logic.
If that is not possible (e.g. there is no suitable qmake variable), the
test's initTestCase() function should call QSKIP to skip the entire test
with a meaningful explanation.
Task-number: QTBUG-21851
Change-Id: Icacc8c5567a700191b6ef3fa94ee52ede94c5b34
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Remove the usage of Q_MAC_USE_COCOA and Carbon code paths.
Change-Id: Ib569ad8c6d9ffe258f454b3c3b06e95294a10112
Reviewed-on: http://codereview.qt-project.org/5100
Reviewed-by: Morten Johan Sørvig <morten.sorvig@nokia.com>
Sanity-Review: Morten Johan Sørvig <morten.sorvig@nokia.com>
This change makes QTest::qWarn() consistent with similar testlib
functions, which take __FILE__ and __LINE__.
Change-Id: I12977cb30672899ab38877b4a656f0cc0c7ea02c
Reviewed-on: http://codereview.qt-project.org/3559
Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com>
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Add support for test cases which require an event loop, but do not need
GUI up and running. Such cases are best accommodated by code similar to
QTEST_MAIN, but using QCoreApplication instead of QApplication.
Change-Id: I04ce82d26b80a4edeba6bf7604a7f8d974232a11
Merge-request: 919
Reviewed-on: http://codereview.qt-project.org/5427
Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com>
Reviewed-by: Jason McDonald <jason.mcdonald@nokia.com>
Each destination and the format of output to write there is specified by
adding "-o filename,format" to the command-line. The special filename
"-" indicates that the log output is written to the standard output
stream, though standard output can be used as a destination at most
once.
The old-style testlib output options are still supported, but can only
be used to specify one logging destination, as before.
If no logging options are given on the command-line, a plain text log
will go to the console, as before.
To log to the console in plain text and to the file "test_output" in
xunit format, one would invoke a test in the following way:
tst_foo -o test_output,xunitxml -o -,txt
This commit also enhances the selftests to test with multiple loggers,
but negative tests (e.g. bad combinations of command-line options) are
left for future task QTBUG-21567.
Task-number: QTBUG-20615
Change-Id: If91e752bc7001657e15e427aba9d25ab0a29a0b0
Reviewed-on: http://codereview.qt-project.org/4125
Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com>
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Pass the filename when creating the logger rather than when commencing
output. This will simplify forthcoming changes for running multiple
loggers.
Change-Id: I563aa97661caf279a5dcccb1321ff3ce4725b332
Reviewed-on: http://codereview.qt.nokia.com/4116
Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com>
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Both the old and new XML loggers produce the Complete and Light XML log
formats, while only the new logger produces Xunit XML logs. The
disadvantage of the new logger is that it is more complex and doesn't
produce a partial log if the test fails to terminate gracefully. This
behaviour arises because Xunit format output cannot be written correctly
until all tests have been executed.
This commit removes the Complete and Light XML formats from the new
logger, using the old logger to produce those formats and the new logger
to produce only Xunit XML. Prior to this commit, the qtestlib selftests
demonstrate that the old and new loggers produce identical output for
Complete and Light XML.
This commit also removes the undocumented -flush command-line option,
which was used rather obscurely to select between the old and new
loggers.
The newer logger will be renamed to QXunitTestLogger in a subsequent
commit.
Change-Id: Id304f5b411bdd520409ee233f6bc34e8917942ab
Reviewed-on: http://codereview.qt.nokia.com/3923
Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com>
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Remove the undocumented feature that allows test functions to be
executed in random order. The feature was designed to expose unintended
dependencies between test functions -- test functions are only supposed
to depend on the initTestCase() and init() functions.
Aside from the lack of documentation, there are a number of problems
with this feature. Most importantly, running the tests in random order
has only a 50% chance of exposing dependencies between test functions.
A better strategy would be to run the test functions in reverse order
and complain if that produces different results to running the tests in
the normal order.
Additionally, the random order is not deterministic, so even if a
dependency is exposed during a test run, there's no guarantee that it
will be exposed again. The feature allows the user to optionally
supply a random seed to make the "random" order deterministic, but as
rand() implementations are not identical across platforms, even that
does not guarantee that dependencies between test functions will be
exposed deterministically.
Change-Id: I39eac34c532ccb988116778bbc5ab05d835874c5
Reviewed-on: http://codereview.qt.nokia.com/3720
Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com>
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Tests that are expected to return a non-zero exitcode should be marked
with "CONFIG+=insignificant_test" in their .pro file.
Change-Id: Iebb9c7129c08833ed517115f569086d6fcfe827b
Reviewed-on: http://codereview.qt.nokia.com/3689
Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com>
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
The charting functionality was removed in Qt 4.6, though the
command-line option was retained to print a warning that the feature had
been removed. Sufficient time has passed for this to be removed
completely.
Change-Id: I2adf2818c6a6e57e765104de97b28dbf6914e3fa
Reviewed-on: http://codereview.qt.nokia.com/3638
Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com>
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
The commit changes printf's that output error and warning messages to
send their text to the stderr stream. Non-error output, such as that
produced by passing the -help option to a test, still goes to stdout.
Change-Id: Iea4d62451e3e7e84c654859cb09ea7e717511d13
Reviewed-on: http://codereview.qt.nokia.com/3636
Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com>
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
The mode parameter is never used by the logging code, so there's little
value in passing it there.
Change-Id: Ibe2cbe5eaf457a7e3ffd3aea3a4be7c8278c91b6
Reviewed-on: http://codereview.qt.nokia.com/3547
Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com>
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
This is the beginning of revision history for this module. If you
want to look at revision history older than this, please refer to the
Qt Git wiki for how to use Git history grafting. At the time of
writing, this wiki is located here:
http://qt.gitorious.org/qt/pages/GitIntroductionWithQt
If you have already performed the grafting and you don't see any
history beyond this commit, try running "git log" with the "--follow"
argument.
Branched from the monolithic repo, Qt master branch, at commit
896db169ea224deb96c59ce8af800d019de63f12
