A new QtBuildStaticDocToolsScript.cmake script is added which:
- clones qt5.git
- initializes required submodules for building doc tools
- checks out qttools/dev/HEAD
- syncs the dependency SHA1s as specified in qttools'
dependencies.yaml file
- configures qt5 as a top-level static build
- builds qdoc and other tools required for documentation generation
- installs them into a custom prefix under a subdirectory of the
repo's root build dir.
The script is intended to be used as part of qtbase's CI build
instructions. See the follow-up coin change for reasons on why we want
to do that and how to configure optional ref pinning, instead of using
dev branch always.
Task-number: QTBUG-128730
Change-Id: I836fc2f1b47c07171d2a2bd54a85bc2145212e46
Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
(cherry picked from commit f7f02c791bcbee44597f1fe24570ebdf352ec648)
Reviewed-by: Qt Cherry-pick Bot <cherrypick_bot@qt-project.org>
Previously SBOM generation was opt-in.
This patch changes the generation of the plain-text tag:value SBOM
to be enabled by default, except for:
- developer builds
- no-prefix builds
- standalone tests or examples
- cmake build tests
The JSON SBOM generation and the verification steps have also been
changed to be enabled by default, but only if the Python dependencies
can be found. If the dependencies are not found, the build will
skip the generation and verification steps.
Four new configure options have been added to control these aspects:
-(no-)sbom-json: Allows explicitly enabling or disabling JSON SBOM
generation
-(no-)sbom-json-required: Fails the build if JSON SBOM generation
Python dependencies are not found
-(no-)sbom-verify: Allows explicitly enabling or disabling SBOM
verification
-(no-)sbom-verify-required: Fails the build if SBOM verification
Python dependencies are not found
There are corresponding CMake variables for each of the configure
options, see the cmake mapping document.
[ChangeLog][Build Systems] SBOM generation is now enabled by default,
when building Qt, except for developer builds and no-prefix builds.
JSON SBOM generation is enabled by default if the required Python
dependencies are available.
Task-number: QTBUG-122899
Change-Id: I6dbe1869f8342154a89ff2ab84ac53c9ef1b2eb7
Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
(cherry picked from commit 96c5e55c111d957bd7b5294d2c2eb1d919cce871)
Reviewed-by: Qt Cherry-pick Bot <cherrypick_bot@qt-project.org>
Can be useful for non-Makefile and Ninja generators, even if they
are unsupported for building Qt.
Change-Id: I02a622c89f725c3ae5b51ea345a530eaa5529976
Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
(cherry picked from commit 08d1d113f140b1b4be5a387696a8a35934905d64)
Reviewed-by: Qt Cherry-pick Bot <cherrypick_bot@qt-project.org>
The file was missing from both installation and inclusion.
Pick-to: 6.7 6.5
Change-Id: I3e904071cc28f674750aca38050fbe89d75a585d
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
(cherry picked from commit 084911a67bde29e49bcb001d593e96fca7836142)
Reviewed-by: Qt Cherry-pick Bot <cherrypick_bot@qt-project.org>
This change adds a new -sbom configure option to allow generating and
installing an SPDX v2.3 SBOM file when building a qt repo.
The -sbom-dir option can be used to configure the location where
each repo sbom file will be installed.
By default it is installed into
$prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx
which is basically ~/Qt/sbom/qtbase-6.8.0.spdx
The file is installed as part of the default installation rules, but
it can also be installed manually using the "sbom" installation
component, or "sbom_$lower_project_name" in a top-level build. For
example: cmake install . --component sbom_qtbase
CMake 3.19+ is needed to read the qt_attribution.json files for
copyrights, license info, etc. When using an older cmake version,
configuration will error out. It is possible to opt into using an
older cmake version, but the generated sbom will lack all the
attribution file information.
Using an older cmake version is untested and not officially supported.
Implementation notes.
The bulk of the implementation is split into 4 new files:
- QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing
and dispatching the generation of various pieces of the SBOM document
e.g. a SDPX package associated with a target like Core, a SDPX
file entry for each target binary file (per-config shared library,
archive, executable, etc)
- QtPublicSbomGenerationHelpers.cmake - for non-Qt specific
implementation of SPDX generation. This also has some code that was
taken from the cmake-sbom 3rd party project, so it is dual licensed
under the usual Qt build system BSD license, as well as the MIT
license of the 3rd party project
- QtPublicGitHelpers.cmake - for git related features, mainly to embed
queried hashes or tags into version strings, is dual-licensed for
the same reasons as QtPublicSbomGenerationHelpers.cmake
- QtSbomHelpers.cmake - Qt-specific functions that just forward
arguments to the public functions. These are meant to be used in our
Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones
for naming consistency. These function would mostly be used to
annotate 3rd party libraries with sbom info and to add sbom info
for unusual target setups (like the Bootstrap library), because most
of the handling is already done automatically via
qt_internal_add_module/plugin/etc.
The files are put into Public cmake files, with the future hope of
making this available to user projects in some capacity.
The distinction of Qt-specific and non-Qt specific code might blur a
bit, and thus the separation across files might not always be
consistent, but it was best effort.
The main purpose of the code is to collect various information about
targets and their relationships and generate equivalent SPDX info.
Collection is currently done for the following targets: Qt modules,
plugins, apps, tools, system libraries, bundled 3rd party libraries
and partial 3rd party sources compiled directly as part of Qt targets.
Each target has an equivalent SPDX package generated with information
like version, license, copyright, CPE (common vulnerability
identifier), files that belong to the package, and relationships on
other SPDX packages (associated cmake targets), mostly gathered from
direct linking dependencies.
Each package might also contain files, e.g. libQt6Core.so for the Core
target. Each file also has info like license id, copyrights, but also
the list of source files that were used to generate the file and a
sha1 checksum.
SPDX documents can also refer to packages in other SPDX documents, and
those are referred to via external document references. This is the
case when building qtdeclarative and we refer to Core.
For qt provided targets, we have complete information regarding
licenses, and copyrights.
For bundled 3rd party libraries, we should also have most information,
which is usually parsed from the
src/3rdparty/libfoo/qt_attribution.json files.
If there are multiple attribution files, or if the files have multiple
entries, we create a separate SBOM package for each of those entries,
because each might have a separate copyright or version, and an sbom
package can have only one version (although many copyrights).
For system libraries we usually lack the information because we don't
have attribution files for Find scripts. So the info needs to be
manually annotated via arguments to the sbom function calls, or the
FindFoo.cmake scripts expose that information in some form and we
can query it.
There are also corner cases like 3rdparty sources being directly
included in a Qt library, like the m4dc files for Gui, or PCRE2 for
Bootstrap.
Or QtWebEngine libraries (either Qt bundled or Chromium bundled or
system libraries) which get linked in by GN instead of CMake, so there
are no direct targets for them.
The information for these need to be annotated manually as well.
There is also a distinction to be made for static Qt builds (or any
static Qt library in a shared build), where the system libraries found
during the Qt build might not be the same that are linked into the
final user application or library.
The actual generation of the SBOM is done by file(GENERATE)-ing one
.cmake file for each target, file, external ref, etc, which will be
included in a top-level cmake script.
The top-level cmake script will run through each included file, to
append to a "staging" spdx file, which will then be used in a
configure_file() call to replace some final
variables, like embedding a file checksum.
There are install rules to generate a complete SBOM during
installation, and an optional 'sbom' custom target that allows
building an incomplete SBOM during the build step.
The build target is just for convenience and faster development
iteration time. It is incomplete because it is missing the installed
file SHA1 checksums and the document verification code (the sha1 of
all sha1s). We can't compute those during the build before the files
are actually installed.
A complete SBOM can only be achieved at installation time. The install
script will include all the generated helper files, but also set some
additional variables to ensure checksumming happens, and also handle
multi-config installation, among other small things.
For multi-config builds, CMake doesn't offer a way to run code after
all configs are installed, because they might not always be installed,
someone might choose to install just Release.
To handle that, we rely on ninja installing each config sequentially
(because ninja places the install rules into the 'console' pool which
runs one task at a time).
For each installed config we create a config-specific marker file.
Once all marker files are present, whichever config ends up being
installed as the last one, we run the sbom generation once, and then
delete all marker files.
There are a few internal variables that can be set during
configuration to enable various checks (and other features) on the
generated spdx files:
- QT_INTERNAL_SBOM_VERIFY
- QT_INTERNAL_SBOM_AUDIT
- QT_INTERNAL_SBOM_AUDIT_NO_ERROR
- QT_INTERNAL_SBOM_GENERATE_JSON
- QT_INTERNAL_SBOM_SHOW_TABLE
- QT_INTERNAL_SBOM_DEFAULT_CHECKS
These use 3rd party python tools, so they are not enabled by default.
If enabled, they run at installation time after the sbom is installed.
We will hopefully enable them in CI.
Overall, the code is still a bit messy in a few places, due to time
constraints, but can be improved later.
Some possible TODOs for the future:
- Do we need to handle 3rd party libs linked into a Qt static library
in a Qt shared build, where the Qt static lib is not installed, but
linked into a Qt shared library, somehow specially?
We can record a package for it, but we can't
create a spdx file record for it (and associated source
relationships) because we don't install the file, and spdx requires
the file to be installed and checksummed. Perhaps we can consider
adding some free-form text snippet to the package itself?
- Do we want to add parsing of .cpp source files for Copyrights, to
embed them into the packages? This will likely slow down
configuration quite a bit.
- Currently sbom info attached to WrapFoo packages in one repo is
not exported / available in other repos. E.g. If we annotate
WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be
available when looking up WrapZLIB in qtimageformats.
This is because they are IMPORTED libraries, and are not
exported. We might want to record this info in the future.
[ChangeLog][Build System] A new -sbom configure option can be used
to generate and install a SPDX SBOM (Software Bill of Materials) file
for each built Qt repository.
Task-number: QTBUG-122899
Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4
Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
(cherry picked from commit 37a5e001277db9e1392a242171ab2b88cb6c3049)
Reviewed-by: Qt Cherry-pick Bot <cherrypick_bot@qt-project.org>
Versionless targets in Qt6 are interface libraries that link the
versioned libraries using the INTERFACE link type. This makes the
linking chain more complicated than it can be. Also we miss some
significant interface properties in the versionless targets comparing
to the versioned targets.
The new approach manually generates the versionless targets, instead
of using CMake exports.
For CMake versions < 3.18 we now create a copy of the versioned
targets. The copy includes all the relevant INTERFACE properties from
the versioned targets and imported locations for all configs.
For CMake versions >= 3.18 we now create the versionless target ALIASes
which should behave give the transparent access to the versioned
targets.
Using the QT_USE_OLD_VERSION_LESS_TARGETS flag you may force the
behavor of the CMake versions <= 3.18
The change is partial workaround for QTBUG-86533.
Task-number: QTBUG-114706
Change-Id: Iafadf6154eb4912df0697648c031fcc1cbde04e0
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
The default manifest is a minimal file that claims NSPrivacyTracking
false, along with an empty list of NSPrivacyTrackingDomains and
NSPrivacyCollectedDataTypes, as Qt does not generally do user tracking.
Modules can override the default manifest by setting the
PRIVACY_MANIFEST target property, specifying a custom privacy
manifest.
The NSPrivacyAccessedAPITypes key is only required for iOS for now.
Even though we don't build Qt for iOS as frameworks yet, which is
required to embed a privacy manifest, we include the keys for the
APIs we known we use.
Task-number: QTBUG-114319
Change-Id: I654bb52b98ee963adeeb744b35f3a1c2a1270969
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
This function calculated the values of the features 'no-prefix' and
'developer-build' from INPUT_* values. Since configure directly
translates -no-prefix and -developer-build to FEATURE_no_prefix and
FEATURE_developer_build, we can remove the function.
Task-number: QTBUG-120529
Change-Id: Ide1fa61af175d8f6a6aa6363dfdfa94912836345
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
Previously we had four-ish locations where the CMAKE_BUILD_TYPE was
force set.
Twice in QtBuildInternalsExtra.cmake via
qt_internal_force_set_cmake_build_type_conditionally(), depending on
some conditions. This was executed right at
find_package(Qt6 COMPONENTS BuildInternals)
time.
And twice in qt_internal_set_default_build_type() via
qt_build_repo_begin() / qt_prepare_standalone_project() that goes
through QtSetup.cmake. This was executed only if the relevant functions
were called, rather than directly at find_package() time.
The exact logic of which build type ended up being set was very
confusing.
Refactor the code to decide the build type in one single location
when qt_build_repo_begin() / qt_prepare_standalone_project() are
called, rather than directly at find_package() time.
The actual logic when we override the build type depends on many
factors:
- when an explicit CMAKE_BUILD_TYPE is given, honor it, unless it's
a multi-config build
- when it's a multi-config build, don't set any CMAKE_BUILD_TYPE,
use the value of CMAKE_CONFIGURATION_TYPES
- when it's a qtbase build, compute a default unless an explicit value
was given
- the default is Debug if FEATURE_developer_build is ON
- otherwise the default is Release
- when it's a top-level build, only choose a build type for qtbase
- when it's another repo build, use the original build type unless
another was given explicitly (including in a top-level build)
- when it's a standalone tests build
- if qt is multi-config, the tests will be single config, due to
various CI failure reasons, this hasn't changed
- if qt is single config, use the original unless an explicit
value was given
- when it's a single standalone test build, use the original unless
an explicit value was given
To determine when an explicit CMAKE_BUILD_TYPE was given in contrast
to when it was default initialized, we now have one single function
that uses a few heuristics.
The heuristics are needed because we can't reliably determine an
explicitly given 'Debug' build on Windows, because CMake default
initializes to that.
The heuristics include:
- checking whether CMAKE_BUILD_TYPE_INIT is different from
CMAKE_BUILD_TYPE
- checking what the CMAKE_BUILD_TYPE was before the first project()
call when CMake default initializes
- we save the previous value in the qt.toolchain.cmake file
- also in QtAutoDetect during qtbase configuration
- also when building the sqldrivers project
- honoring the value of QT_NO_FORCE_SET_CMAKE_BUILD_TYPE
As a result of the above changes, the build type will be set exactly
zero or one times, for a particular build directory.
Note that the configure script also has some logic on which
CMAKE_BUILD_TYPE / CMAKE_CONFIGURATION_TYPES to pass to CMake
depending on whether -debug / -release / -debug-and-release /
-force-debug-info were passed. But once the values are passed,
CMake will honor them.
Amends 48841c34d2e86a741ec9992b9704c0fa5973503c
Amends 8c912cddebe544010e7da3f87af5b21f3328d7ec
Pick-to: 6.7
Task-number: QTBUG-114958
Task-number: QTBUG-120436
Change-Id: I30db14d1e8e9ff9bd2d7ea1d2256cdeb9493ca0d
Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Previously we only warned about unsupported cmake generators when
configuring qtbase.
Now we do it for other submodules as well.
Pick-to: 6.7 6.6 6.5
Task-number: QTBUG-120602
Change-Id: I9d78db546bcf1238604362b248d41d4516b60b2a
Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
The warnings are shown when configuring any Qt submodule or top-level.
The warnings are NOT shown when configuring a user project with CMake.
Opt out CMake cache variables can be set to silence any of the
warnings:
- QT_NO_APPLE_SDK_AND_XCODE_CHECK
- QT_NO_APPLE_SDK_MIN_VERSION_CHECK
- QT_NO_XCODE_MIN_VERSION_CHECK
- QT_NO_APPLE_SDK_MAX_VERSION_CHECK
The warnings can be upgraded into errors by configuring with
-DQT_FORCE_FATAL_APPLE_SDK_AND_XCODE_CHECK=ON
The platform version requirements that qtbase specifies in .cmake.conf
are saved in Qt6ConfigExtras.cmake so that they can be used when
configuring other non-qtbase submodules.
The code is added to the public CMake files, so that in the future we
don't need to move code around if we enable the checks for public
CMake projects as well.
The version extraction helpers were moved out of QtAutoDetectHelpers
into QtPublicAppleHelpers.
Task-number: QTBUG-119490
Change-Id: Ic840e1013aeb607bf23247a9cb43471dde802e9d
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Amir Masoud Abdol <amir.abdol@qt.io>
And automatically include and install it in all qt builds.
We will use some of its functions for enforcing apple sdk
requirements when building Qt.
Change-Id: I46383bc857430d35314dfa2ebef9eb342fb63560
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Reviewed-by: Amir Masoud Abdol <amir.abdol@qt.io>
Instead of duplicating file names that need to be include()'ed
in two different places if the files are meant to be used
in both Qt builds and public user projects and then also mention the
name when installing, extract the file names into lists returned by
functions.
Call these functions where needed to iteratively include() the files
as well as install them, without having to remember updating multiple
code locations.
The new functions return the following list of file names:
- upstream cmake modules that need to be included during a Qt build
- public (Qt build and user project) cmake helpers that need to be
included and installed
- private (Qt build only) cmake helpers that need to be included and
installed
- public files that need to be installed, but not included
- private files that need to be installed, but not included
We also generate the list of public files to include in
Qt6Config.cmake.
Task-number: QTBUG-86035
Change-Id: I1e7287f4e1d041c723f144ba9626b34f873c4891
Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
Make the QtBuildRepoHelpers and QtBuildRepoExamplesHelpers files
that were previously loaded as part of BuildInternals package instead
be loaded when qt_internal_include_all_helpers is called.
Load all the helpers as soon as find_package(QtBuildInternals) is
called rather than when qt_build_repo() is called.
This is a behavior change, but because including the Qt's Helpers
should have no side-effects aside from defining functions,
it should be fine.
This lets us have a unified location where to include Helpers files,
instead of thinking whether it needs to be done in QtBuildInternals or
in QtBuildHelpers or some other place.
Move also some additional inclusions into the same function.
Note that including some upstream CMake files like CMakeFindBinUtils
does have side-effects, but we've been doing it already anyway,
so moving it to the top should not make a difference because any
modifications we would do to the globally assigned variables would
have come later when we actually called our own functions.
Task-number: QTBUG-86035
Change-Id: I33f36f7e8db69d504c34a4d4a094b98f6fa50ee4
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
They don't have side-effects, so no need to keep the checks.
Task-number: QTBUG-86035
Change-Id: Ic2c3aee1b19d8b1727936582bfe366c8277d11c2
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
Split all code in QtSetup into separate functions and macros, put them
in more appropriate files, and call them in
qt_internal_setup_build_and_global_variables.
A new QtBuildOptionsHelpers.cmake is created which takes care of
computing the default values of user-customizable options that are not
pure configure features, like the cmake build type, whether to build
tests / examples, whether to enable ccache, etc.
The new function calls added in
qt_internal_setup_build_and_global_variables
try to preserve the previous code flow when QtBuild was included
in-between the code that was run in QtSetup.
Macros that have dependencies on various global variables were marked
as such with inline comments for easier navigation and comprehension.
After this change, QtSetup.cmake just includes QtBuild.cmake. We leave
it to exist for easier git blaming, but new code should not be added
to it unless really necessary.
The intent is to merge a variant of this change to 6.6 and 6.5 as
well.
Task-number: QTBUG-86035
Change-Id: I3409c2d3ea8ee19a69104b12ab2692966ba5f9cf
Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
My motivation to do this:
- it got big and tangled again
- sometimes functions need to be added to QtBuild.cmake rather than
to a separate file because they need to be called before some of the
global variables are set, to determine the value of those global
variables (in my case install paths needed to be modified when
building with xcframework support)
- some of the global variable assignments have dependencies on other
variables already being set and it's hard to keep track where that
happens
Split the contents of the file into smaller functions and macros
and place them into pre-existing files when appropriate, or
into new files. The new files are:
- QtBuildHelpers.cmake
- QtBuildPathsHelpers.cmake
- QtMkspecHelpers.cmake
The idea is to have Helpers file only define functions and never call
them, so it's easy to include the file where needed without being
scared of side effects.
QtBuild.cmake will just include the helpers and call one entry point
function to set up everything that was done by the file before.
QtBuild.cmake is not merged into QtSetup, to make it easier to git
blame (it's hard to blame a removed file).
No new features were added as part of the refactoring.
Some function names were renamed (but not all of them) to include
the qt_internal prefix.
Some lines were reformatted so they don't pass 100 chars limit after
the code was placed into a function / macro.
The Helpers includes were re-sorted.
Some function calls were re-ordered where the order call didn't
matter.
Some of the code in QtAndroidHelpers.cmake was wrapped into a macro
so that including the file does not cause side-effects by default.
I'd like to follow up with similar changes for QtSetup.cmake and
QtBuildInternalsConfig.cmake where possible, because having a few
"entry points" into building a Qt submodule is also confusing,
especially for those that aren't familiar with the build system and
why certain things go into certain places.
The intent is to cherry-pick this also to 6.5 and 6.6.
Amends 44cce1a2ea9dadd8b2de93f40de34269dda703c0
Task-number: QTBUG-86035
Change-Id: I02ceff8ceb9b6e9c78bc85d6a42deb02fca3e46b
Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
Reviewed-by: Orkun Tokdemir <orkun.tokdemir@qt.io>
