Similar `*_check` targets are created in `qt_internal_add_test`.
This change extends that implementation to auto cmake tests.
Note: These generated targets do not have additional dependencies, on
the build targets, unlike those generated by `qt_internal_add_test`.
You would need to (re-)run the build first separately and then run the
`*_check` target.
Fixes: QTBUG-98640
Pick-to: 6.8
Change-Id: I387f09570b6a4a345756133870c2eb9ef1cf3cf8
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
(cherry picked from commit 8f2f4ad4688df4963982c489c80b84e115cda29d)
Reviewed-by: Qt Cherry-pick Bot <cherrypick_bot@qt-project.org>
Running Chrome in headless mode helps to circumvent
a lot of issues with GPU driver/Wayland interactions
that we encounter in CI.
Run Chrome in headless mode if corresponding environment
variable is set.
Change-Id: I1cf290f484054766e2b99d99045e7b39d5662210
Reviewed-by: Morten Johan Sørvig <morten.sorvig@qt.io>
Reviewed-by: Even Oscar Andersen <even.oscar.andersen@qt.io>
(cherry picked from commit 213a2610d821b93429d6519dda11ab0e7b5cae84)
Reviewed-by: Qt Cherry-pick Bot <cherrypick_bot@qt-project.org>
Swift 5.9 includes built in crash reporting, printing stack traces,
libraries, and registers to stdout/err.
https://www.swift.org/blog/swift-5.9-backtraces/
As (Core)Foundation is written in Swift nowadays, we get this feature
for free even in our "C++" apps, as we always link to CoreFoundation.
To enable the feature the binary needs the com.apple.security.get-task-allow
entitlement, so we add it for all our tests automatically.
The final piece is to run the tests with SWIFT_BACKTRACE=enable=yes,
but we'll do this in our CI provisioning, as setting it from within
testlib doesn't seem to work.
Pick-to: 6.8
Change-Id: I31090efee06460f45522093e17f900e76590b282
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
(cherry picked from commit 9bebdc97f161cc58461530fa0171e0defc6cc1ee)
Reviewed-by: Qt Cherry-pick Bot <cherrypick_bot@qt-project.org>
Prepare for upgrade to Emscripten 3.1.70. Newer Emscripten run out of
memory in CI when linking batched tests. There is no runtime performance
penalty while not having those optimizations.
Disable optimization for that target.
Task-number: QTBUG-131745
Change-Id: I479d2cd38bd6882245d80082d01102e67884413d
Reviewed-by: Morten Johan Sørvig <morten.sorvig@qt.io>
If the QT_BUILD_ENVIRONMENT is set to ci, add --verbose
by default.
Pick-to: 6.8
Change-Id: Ia9f2ba4c0191649d43a17bd76f61ca46180345d7
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
Previously we ran some finalizer functions for tests and manual
tests, only for specific platforms, and via different code paths.
This change introduces a new unified way of running all finalizers for
all test-like executables, including benchmarks.
To ensure a smoother transition, the new way is opt-out, and the old
way can be enabled by setting the QT_INTERNAL_SKIP_TEST_FINALIZERS_V2
variable to true in case we encounter some issues in the CI.
The finalizers are only run for test-like executables, and not all
internal executables, because there are some unsolved issues there.
One particular case is in qtdeclarative where that will create a cycle
for qmlimportscanner to depend on itself.
A proper solution here would be to have some kind of mapping or
mechanism to exclude finalizers for targets where they would try to
run themselves.
Pick-to: 6.8
Task-number: QTBUG-93625
Task-number: QTBUG-112212
Change-Id: I52b3a1c02c298c4a18ce2c75d7e491ae79d191a0
Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
This will be used in a follow up patch to identify them.
Pick-to: 6.8
Task-number: QTBUG-93625
Task-number: QTBUG-112212
Change-Id: I39d5eb9f79ac67af0808efeda4d7f4e0a6908cc9
Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
The _qt_internal_finalize_batch function had a bunch of problems,
style-wise and function-wise:
- it called find_package(Qt6) in its body, which broke when
configuring qtbase with in-tree tests
- it constantly rewrote the merged blacklist file on each
reconfiguration
- it used file(WRITE) to concatenate content
- it was in the public API file
The changes are:
- Move the function to the internal test helpers file.
- Change it to use qt_configure_file instead of file(WRITE).
- Call it at the end of the qt_build_tests for a repo, or at the end
of configuring a super build, instead of relying on a finalizer.
- Remove the find_package call. The reason this was added in the first
place, was to populate the __qt_core_macros_module_base_dir variable
so that qt_internal_add_resource configure_file call doesn't fail in
standalone tests build.
The variable was unset because the finalized function was called in
the top level directory scope, whereas the very first find_package
call was in the tests/ scope, meaning the variable was empty.
This is still a problem in the top-level build where the tests scope
and the top-level scope are different. Work around it by relying on
a global property to reset the value if it's empty.
Amends 6a9e89121d7766a34c4281d298057bfbe8af36b3
Pick-to: 6.8
Change-Id: Id6fe41ee86d09b8118bea52cac8a59965d7ecb9e
Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
... only when QT_ENABLE_VERBOSE_DEPLOYMENT is set.
Change-Id: Icca4b77c9bbe1c4fd3481b3c0925bff627837929
Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
Create a script that allows running an app or a test easily similar
to running on host. This improves development workflow and time by
allowing quick runs without having to manually call the various adb
commands to launch, get a pid of the app and then print the logcat,
let alone passing parameters or environment variables.
For normal apps, the app package name is retrieved by the script, run
and live logcat is printed as long as the app is still running.
For tests, the script calls androidtestrunner, allowing test parameters
to be passed to the test.
For CI debugging, this would save quite a lot of hussle and frustration
trying to run or debug a test app.
One other benefit for this is enabling running Android tests from Qt
Creator's testlib plugin without big changes to Qt Creator to support
androidtestrunner explicitly.
Because file(GENERATE) would fail if called twice for the same file,
I use file(WRITE). This is used because at the time of calling the
target executable finalizer, we don't know if the target is a test
or not, so we rely on writing the script first as a normal target,
then call it if the target is a test where it overrides the script.
For this also, parameters passed to the runner or androidtestrunner
can't handle generator expressions.
[ChangeLog][CMake][Android] Add wrapper scripts to run Android apps and
tests with ease from the host.
Task-number: QTBUG-129889
Change-Id: I84e85ce2bbf6944c8aa20bdc2c2b6d7b956bc748
Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
Qt autotests now require exceptions due to usage of
QTEST_THROW_ON_FAIL and QTEST_THROW_ON_SKIP.
Task-number: QTBUG-121822
Change-Id: I0fa95d61116a2cda1d2955c9d42cac9667ce1b72
Reviewed-by: Lorn Potter <lorn.potter@qt.io>
Now that the ctest is not run as verbose the logging is significantly
decreased and VxWorks will hit timeout on some tests.
There are plenty of other timeouts that will be hit instead
before the total timeout specified for the command e.g.
- Ctest timeout 25min default per test case.
- Testlib timeout per test function 10min default.
Change-Id: Ic02e812d4b059609553d2ac8d44ec91f358d76c5
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Reviewed-by: Assam Boudjelthia <assam.boudjelthia@qt.io>
Use the quote-based argument wrapping when generating the command line
scripts for the tests. Apply the wrapping and escaping of the arguments
right before writing the script with the following advanced rules:
- if argument is already wrapped using quotes, skip wrapping
- if argument is a CMake variable, skip wrapping
- wrap the argument with qoutes otherwise
The above should cover the most common usecases in centralized place
when we generate the test wrapper scripts.
Remove other ways we wrap arguments in other places, like square bracket
wrapping in qt_internal_create_test_script.
Change-Id: If287dd75d6fb36260b5cf8a687215bda9c9fc1c0
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
Using Chrome provided by environment variable will allow
to pin the version of Chrome on CI for better tests stability.
Change-Id: Ic6f3026eb5694b628d3a4dd1dcc49c53270e2b86
Reviewed-by: Lorn Potter <lorn.potter@qt.io>
There is a default tst_qwasmwindow_harness.html created, and we have
our own in addition. This creates problems if one runs configure
and do not perform a clean build afterwards as we would be
left with the autogenerated file.
The solution is to rename our custom file so that there is
no conflict.
Change-Id: I1f0436eea752059d4dcf22407c760339eed46c09
Reviewed-by: Piotr Wierciński <piotr.wiercinski@qt.io>
VxWorks uses emulator like QNX and requires the different setup.
Change-Id: I2225f697980265b3d419c6cbc40f03450e86c577
Reviewed-by: Pasi Petäjäjärvi <pasi.petajajarvi@qt.io>
This reverts commit 85020edf028621cec3100e81ec726b3bda777f98.
Reason for revert: unstable, breaks CI
Change-Id: I58fb4652d777f4d551fe03d5ed7fad3fe07170b5
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Enable JSPI (aka asyncify 2). This allows building
and running tests in that mode, which should faster
and and is also compatible with wasm-exceptions.
Change-Id: I76648de7dabd2297ab271385b45ff095f08b848e
Reviewed-by: Piotr Wierciński <piotr.wiercinski@qt.io>
Reviewed-by: Lorn Potter <lorn.potter@gmail.com>
The new approach allows to imply the macro definitions for every Qt
submodule and user project without the need of setting it explicitly
from CMake. This also prevent users from introducing the
incompatibility between Qt modules due to defining
the QT_DISABLE_DEPRECATED_UP_TO version lower than qtbase was built
with.
Task-number: QTBUG-124765
Change-Id: I7ba481f62cb9073ae0343c400ffc26f239f080f1
Reviewed-by: Ivan Solovev <ivan.solovev@qt.io>
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
Builtin testdata packages all test files into resources. This is used
on mobile platforms by default, but it can be useful on other platforms
as well. Add CMake option to force this on other platforms.
Change-Id: I67c82bc735960be53b351ed16526cbfacfbe9bb6
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
Not all tests have targets that we can attach the BLACKLIST files to.
Pick-to: 6.7
Change-Id: Ie0a2d72859877a8803802a4f7dc996944f002656
Reviewed-by: Ulf Hermann <ulf.hermann@qt.io>
Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
Tests can be blacklisted via BLACKLIST files. This patch adds these
files to the generated IDE projects automatically when available.
Pick-to: 6.7
Change-Id: Icef2c397ada823e7b1c380dbb224694f059891ff
Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
The test system tried to run emrun (No .bat). This failed with a
file-not-found error on windows.
Also there was a test for (WIN32) when deciding to use cmd /c.
This test has been updated to (CMAKE_HOST_WIN32)
Also the aforementioned cmd /c was not present in the test run
output. It has been added.
Fixes: QTBUG-121996
Change-Id: Ib3c053949865038ad43abd479402f5e8e3c015ac
Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
Reviewed-by: Piotr Wierciński <piotr.wiercinski@qt.io>
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
... from CMake so that Coin doesn't end up killing the whole VM if aRows
test gets stuck and instead allow androidtestrunner to cleanup and fetch
the logs so we know what happened, and even potentially ending up re-run
the test and succeeding if it was flaky.
Also, since CMake sets the timeout during configuration time, coin needs
to set COIN_COMMAND_OUTPUT_TIMEOUT under the build target where tests
are configured, so this moves the setting of that env var from the test
target to the build target.
Pick-to: 6.7 6.6 6.5
Change-Id: I9883ea1e98c93f79a088067518d09ca8acd5fdfd
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
If the test runner was interrupted, stop waiting for the test
to start or to finish, go fetch the available test results
and logcat and uninstall the test app.
Also, set CMake TIMEOUT_SIGNAL_NAME to SIGINT and
TIMEOUT_SIGNAL_GRACE_PERIOD to 10 seconds to allow enough
time to fetch the logs and uninstall the test app.
Task-number: QTBUG-106479
Pick-to: 6.7 6.6 6.5
Change-Id: I4820cfe58f05d15179b4af819caa92e475881634
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
Pass a CMake test TIMEOUT argument to androidtestrunner, using
a value of 95% of that timeout to allow time for the test runner
to do any cleanup before being killed.
If no test argument is provided, use the value from CMake property
DART_TESTING_TIMEOUT or CTEST_TEST_TIMEOUT. If that's not provided
default to 25 minutes which is the default for DART_TESTING_TIMEOUT.
Along the way set the default androidtestrunner timeout to 10 minutes
and fix the wrong timeout in the help menu.
Fixes: QTBUG-106479
Pick-to: 6.6 6.5
Change-Id: I12cd531583dd94954caf8044c37c22382d53d43c
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
Reviewed-by: Dimitrios Apostolou <jimis@qt.io>
Reviewed-by: Axel Spoerl <axel.spoerl@qt.io>
Consider CMAKE_RUNTIME_OUTPUT_DIRECTORY when setting the default
OUTPUT_DIRECTORY for tests and benchmarks.
Change-Id: I93d85c77f30fec186e13e8cd3e902027e60c588f
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
The currently provisioned OpenSSL for Android, i.e.,
`prebuilt-openssl-3.0.7-for-android-ndk-r25b.zip` has a different
structure compared to what Vcpkg installed directory has. This patch
makes sure that we are considering the Vcpkg structure as well.
CMAKE_ANDROID_ARCH_ABI should be translated to Vcpkg's triplet target
name, and we add a `-dynamic` suffix to them because we are building
android artifacts with a custom triplets with similar names.
Change-Id: Id48a8ad351a64ab18001f729d948fbf1b7e47b29
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
The qt_install() calls in qt_internal_add_test() did not re-create
subdirectories that are part of the input test data.
Make sure that the subdirectories are created upon installation by
ensuring we specify a relative installation path that includes all the
path parts except for the file name.
That works the same for directories.
Amends ec1546afc4a5e417d37c6a14e2909b063045bf39
Amends 540bd6cf203969363e641027b66fb044d9ccb1f6
Amends 1307736c7db1ff24e3b8282f4a7b14d24866feba
Amends 0a1256a52d8c6c1b85a10f7dc94dfc34e3540040
Task-number: QTBUG-117098
Change-Id: Ia80f4e7e1ec531264864bcde3ac192ce79b65746
Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
Revert incidental change which leaked from dev environment.
It's better to save CI resources by only having one tab
opened in browser at a given time.
Change-Id: I7f8d1af546b749b2fec4f49a6751545f52c16414
Reviewed-by: Morten Johan Sørvig <morten.sorvig@qt.io>
This is part of our testing effort where we try enabling more tests for
Web Assembly platform on CI. Not all tests work out of box, so some of
them will require followup work.
This commmit also introduces a new mechanism of automatically renaming
files when they are added many times with the same filename to single
translation unit.
Change-Id: I620536494ea83aeb9b294c4a35ef72b51e85a38b
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Morten Johan Sørvig <morten.sorvig@qt.io>
VCPKG by default does static builds when building for Android.
This is at odds with the bundling-concept, so it should not be done
unconditionally.
Since we don't necessarily have the WrapOpenSSL target on-hand, let's
just do a file-exists test for the one of the paths we would include.
Pick-to: 6.6 6.5
Change-Id: I3693354308d5168d8a9c3d1659bfa51540114b7f
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
Reviewed-by: Assam Boudjelthia <assam.boudjelthia@qt.io>
Amend 9d5a8162a5a168972c7aaf39b130da6a72dc13e2, after which building Qt
for Android locally required an OPENSSL_ROOT_DIR variable to be set.
Instead of making it harder for everyone to build Qt for Android (even
if no intention to work on or test network code locally), downgrade the
fatal error to a warning.
Pick-to: 6.6 6.5
Change-Id: I6f2176a40663cc9f55aa3066af78499af1f20894
Reviewed-by: Assam Boudjelthia <assam.boudjelthia@qt.io>
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
It should be possible to build a test in any configuration
by building Qt in that configuration, and then building
the test with qt-standalone-test.
Not all Qt configurations will be able to run all tests,
(due to exec() calls and similar) but that's OK - some
tests don't have exec() calls, and we want to be able
to run tests for a given configuration to figure out
how well it works.
On CI we want to use batching and asyncify, so it makes
sense to tie usage of asyncify to batching.
Pick-to: 6.6
Change-Id: I05553d250a45c1831f43dc71a43ef02d01d70535
Reviewed-by: Mikołaj Boc <Mikolaj.Boc@qt.io>
The new argument would allow bundling the prebuilt OpenSSL libs
into the test apk so that SSL can actually run. It expects the CMake
argument OPENSSL_ROOT_DIR to be set ( we set that in Coin configs).
Task-number: QTBUG-110025
Pick-to: 6.6 6.5
Change-Id: I4c82796635ca89f5511255ae26182f41a504b026
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
- Removed the NO_UNITY_BUILD argument from commands that disable it by
default.
- Add a warning in case NO_UNITY_BUILD or NO_UNITY_BUILD_SOURCES is
being used where it is already disabled, e.g., qt_internal_add_test
- Exclude all sources of a target from unity build if NO_UNITY_BUILD
is set on the target. This sounds a bit harsh, but I have noticed that
sometimes the same source file can be included somewhere else, and
some unexpected collision may occur.
- qt_examples_build_end excludes all its examples from the unity build.
- qt_build_test now sets the CMAKE_UNITY_BUILD to OFF before configuring
the tests, and restore its value when done.
Pick-to: 6.5
Task-number: QTBUG-109394
Change-Id: Ia42e7dd5a5bfb151db241deb639325720fd91eec
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
This should be handled later ideally, for now we should skip and
QML tests targets are still being produced, which leads to errors
on WASM.
Task-number: QTBUG-109786
Change-Id: I3d0d1f3115e324c7a0cb036e972226310294b216
Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
Tests generated with qt-cmake-standalone-test will now not be
batched by default. Defining the QT_BATCH_STANDALONE_TESTS
environment variable will make the build system batch them.
Fixes: QTBUG-111226
Change-Id: I9f01c662e22f8ffdd33e1c4d82619db0689fecc7
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
qt_internal_add_test_helper will fail if a test batch is missing and
batching is enabled. It will now copy to the parent binary directory
as without batching for simplicity, as this happens when batched tests
are skipped.
One consequence could be that when the first test specified has
the NO_BATCH argument, this will incorrectly install the helper, but
helpers still need extra work to function properly, so for now we just
make it compile.
Task-number: QTBUG-109786
Change-Id: Ib307ae79799422c2a4102885aa007ef043835e50
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
Manual tests are supposed to display UI and be assessed manually, but
currently they use the auto test runner by mistake.
Use the normal wasm shell to fix this and make them work like usual
applications.
Fixes: QTBUG-111753
Change-Id: I9d3c0ad56e913b73737c5b72087e82980989d8b8
Reviewed-by: Morten Johan Sørvig <morten.sorvig@qt.io>
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
They now share their implementation. Manual tests can be created by
passing the MANUAL arguments to the qt_internal_add_test as well.
Pick-to: 6.5
Fixes: QTBUG-111382
Change-Id: I1c207b7e4a67526554df1ee43fe032bb20fc92fa
Reviewed-by: Mitch Curtis <mitch.curtis@qt.io>
Reviewed-by: Matthias Rauter <matthias.rauter@qt.io>
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
The previous attempt failed as target coin configuration doesn't
use superbuild. Skip the tests based on the TESTED_MODULE_COIN
environment variable
Task-number: QTBUG-109786
Change-Id: I0dbe6ff64ca4a2e81fef377865ef4e99b58c5eb2
Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
- The following commands accepts NO_UNITY_BUILD, and
NO_UNITY_BUILD_SOURCES arguments to opt out of the unity build, and
to exclude some source files from unity build, respectively.
- qt_internal_add_executable
- qt_internal_add_module
- qt_internal_add_plugin
- qt_internal_add_tool
- qt_internal_extend_target
- qt_internal_add_common_qt_library_helper
- qt_internal_add_cmake_library
- qt_internal_add_simd_part
- Unity build is disabled by default in these:
- qt_internal_add_test
- qt_internal_add_test_helper
- qt_internal_add_benchmark
- qt_internal_add_3rdparty_library
- qt_update_ignore_pch_source also excludes the files from unity_build
Pick-to: 6.5
Task-number: QTBUG-109394
Change-Id: I5d0e7df633738310a015142a6c73fbb78b6c3467
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
The batcher does not currently work properly with non-qtbase
submodules. Disable them temporarily so that at least the
tests in qtbase are batched. Do this from qtbase so that changes
to each and every submodule are not necessary.
Also, maintain a list of tests that were thus skipped. On any
qt cmake internal function call that refers to such a target,
identify the target as skipped and make the call a no-op.
Fixes: QTBUG-109785
Change-Id: Ib0aa5d39eee8315ffd4ac62f6d1f44fe9bbf7a2f
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
This replaces the qt_parse_all_arguments macro with the built-in
`cmake_parse_arguments(PARSE_ARGV`. In addition, a new function,
_qt_internal_validate_all_args_are_parsed, can be used to check whether
any _UNPARSED_ARGUMENTS have been passed to the function.
Fixes: QTBUG-99238
Change-Id: I8cee83dc92dc6acdaaf747ea6ff9084c11dc649b
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
This works by collecting the paths of all blacklist files
and deferring a call which ultimately reads all of the files and
glues them together to form a master blacklist file for the batch.
There might be conflicting function names inside the batch. For now
we ignore the problem, while keeping in mind that it exists.
Fixes: QTBUG-110016
Change-Id: I9c8412097418c6e93297ab89af718d7466e2e451
Reviewed-by: Jörg Bornemann <joerg.bornemann@qt.io>
Otherwise CI hangs if no existing browser can be reused (typical
case)
Fixes: QTBUG-109876
Change-Id: I63436a23c23c4b74c27c18effafde53bb6a4a34e
Reviewed-by: Morten Johan Sørvig <morten.sorvig@qt.io>
