Support global data tags in blacklisting identification of test-cases

Previously the blacklisting file format only worked with slot:data or
plain slot names for the items to blacklist. However, tests with
global data report themselves with the global data-row tag in the same
way as function-specific ones do; and tests which have both join the
two as slot(global:data) in the test output name, so the reader is apt
to mistake global:data for a data tag. In any case, it is potentially
desirable to be able to blacklist a function with either or both of
global and local data-row tags specified. Add support for that and
remove a blacklisting that was only needed due to the lack of this
support.

For now, make the new parameter to checkBlackLists() optional, so
that qtdeclarative's qmltest framework can adapt to this change.

Fixes: QTBUG-100870
Change-Id: I9125811ebdab75d3fb462ba8b60561f003426502
Reviewed-by: Pasi Petäjäjärvi <pasi.petajajarvi@qt.io>
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>

src/testlib/qtestblacklist.cpp

@@ -1,6 +1,6 @@
 /****************************************************************************
 **
-** Copyright (C) 2016 The Qt Company Ltd.
+** Copyright (C) 2022 The Qt Company Ltd.
 ** Contact: https://www.qt.io/licensing/
 **
 ** This file is part of the QtTest module of the Qt Toolkit.
@@ -59,17 +59,25 @@ QT_BEGIN_NAMESPACE
   referring to this documentation is kind to readers.  Comments can also be used
   to indicate the reasons for ignoring particular cases.
 
-  The key "ci" applies only when run by COIN.
-  The key "cmake" applies when Qt is built using CMake. Other keys name platforms,
-  operating systems, distributions, tool-chains or architectures; a !  prefix
-  reverses what it checks.  A version, joined to a key (at present, only for
-  distributions and for msvc) with a hyphen, limits the key to the specific
-  version.  A keyword line matches if every key on it applies to the present
-  run.  Successive lines are alternate conditions for ignoring a test.
+  The key "ci" applies only when run by COIN. The key "cmake" applies when Qt
+  is built using CMake. Other keys name platforms, operating systems,
+  distributions, tool-chains or architectures; a ! prefix reverses what it
+  checks. A version, joined to a key (at present, only for distributions and
+  for msvc) with a hyphen, limits the key to the specific version. A keyword
+  line matches if every key on it applies to the present run. Successive lines
+  are alternate conditions for ignoring a test.
+
+  Ungrouped lines at the beginning of a file apply to the whole testcase. A
+  group starts with a [square-bracketed] identification of a test function to
+  ignore. For data-driven tests, this identification can be narrowed by the
+  inclusion of global and local data row tags, separated from the function name
+  and each other by colons. If both global and function-specific data rows tags
+  are supplied, the global one comes first (as in the tag reported in test
+  output, albeit in parentheses after the function name). Even when a test does
+  have global and local data tags, you can omit either or both. (If a global
+  data row's name coincides with that of a local data row, some unintended
+  matches may result; try to keep your data-row tags distinct.)
 
-  Ungrouped lines at the beginning of a file apply to the whole testcase.
-  A group starts with a [square-bracketed] identification of a test function,
-  optionally with (after a colon, the name of) a specific data set, to ignore.
   Subsequent lines give conditions for ignoring this test.
 
         # See qtbase/src/testlib/qtestblacklist.cpp for format
@@ -89,6 +97,9 @@ QT_BEGIN_NAMESPACE
         [testfunction2:testData]
         msvc-2010
 
+        [getFile:withProxy SSL:localhost]
+        android
+
   QML test functions are identified using the following format:
 
         <TestCase name>::<function name>:<data tag>
@@ -296,17 +307,25 @@ void parseBlackList()
     }
 }
 
-void checkBlackLists(const char *slot, const char *data)
+void checkBlackLists(const char *slot, const char *data, const char *global)
 {
     bool ignore = ignoreAll;
 
     if (!ignore && ignoredTests) {
         QByteArray s = slot;
-        ignore = (ignoredTests->find(s) != ignoredTests->end());
+        ignore = ignoredTests->find(s) != ignoredTests->end();
         if (!ignore && data) {
-            s += ':';
-            s += data;
-            ignore = (ignoredTests->find(s) != ignoredTests->end());
+            s = (s + ':') + data;
+            ignore = ignoredTests->find(s) != ignoredTests->end();
+        }
+
+        if (!ignore && global) {
+            s = slot + ":"_qba + global;
+            ignore = ignoredTests->find(s) != ignoredTests->end();
+            if (!ignore && data) {
+                s = (s + ':') + data;
+                ignore = ignoredTests->find(s) != ignoredTests->end();
+            }
         }
     }
 

src/testlib/qtestblacklist_p.h

@@ -1,6 +1,6 @@
 /****************************************************************************
 **
-** Copyright (C) 2018 The Qt Company Ltd.
+** Copyright (C) 2022 The Qt Company Ltd.
 ** Contact: https://www.qt.io/licensing/
 **
 ** This file is part of the QtTest module of the Qt Toolkit.
@@ -59,7 +59,8 @@ QT_BEGIN_NAMESPACE
 namespace QTestPrivate {
     // Export functions so they can also be used by QQuickTest
     Q_TESTLIB_EXPORT void parseBlackList();
-    Q_TESTLIB_EXPORT void checkBlackLists(const char *slot, const char *data);
+    Q_TESTLIB_EXPORT void checkBlackLists(const char *slot, const char *data,
+                                          const char *global = nullptr);
 }
 
 QT_END_NAMESPACE

src/testlib/qtestcase.cpp

@@ -1164,6 +1164,9 @@ bool TestMethods::invokeTest(int index, const char *data, WatchDog *watchDog) co
     const QTestTable *gTable = QTestTable::globalTestTable();
     const int globalDataCount = gTable->dataCount();
     int curGlobalDataIndex = 0;
+    const auto globalDataTag = [gTable, globalDataCount](int index) {
+        return globalDataCount ? gTable->testData(index)->dataTag() : nullptr;
+    };
 
     /* For each entry in the global data table, do: */
     do {
@@ -1180,6 +1183,9 @@ bool TestMethods::invokeTest(int index, const char *data, WatchDog *watchDog) co
         bool foundFunction = false;
         int curDataIndex = 0;
         const int dataCount = table.dataCount();
+        const auto dataTag = [&table, dataCount](int index) {
+            return dataCount ? table.testData(index)->dataTag() : nullptr;
+        };
 
         // Data tag requested but none available?
         if (data && !dataCount) {
@@ -1200,7 +1206,8 @@ bool TestMethods::invokeTest(int index, const char *data, WatchDog *watchDog) co
             if (!data || !qstrcmp(data, table.testData(curDataIndex)->dataTag())) {
                 foundFunction = true;
 
-                QTestPrivate::checkBlackLists(name.constData(), dataCount ? table.testData(curDataIndex)->dataTag() : nullptr);
+                QTestPrivate::checkBlackLists(name.constData(), dataTag(curDataIndex),
+                                              globalDataTag(curGlobalDataIndex));
 
                 QTestDataSetter s(curDataIndex >= dataCount ? nullptr : table.testData(curDataIndex));
 

tests/auto/network/socket/qtcpsocket/BLACKLIST

@@ -1,7 +1,5 @@
 [timeoutConnect:ip]
 windows
-[connectToHostError]
-windows-10 gcc developer-build
 # QTBUG-66247
 [delayedClose:WithSocks5Proxy]
 windows-10 gcc developer-build