macOS: Compute NSWindow background color without checking styleMask

The check for styleMask == NSWindowStyleMaskBorderless to decide whether
to clear the NSWindow background was broken, as NSWindowStyleMaskBorderless
has the value 0, but is only supposed to be compared to its companion
NSWindowStyleMaskTitled (with value 1). A window can perfectly well be
NSWindowStyleMaskBorderless and NSWindowStyleMaskMiniaturizable e.g.,
so by comparing directly to NSWindowStyleMaskBorderless instead of
masking to the first bit first we ended up making miniaturizable
windows non-translucent.

We now check the Qt::FramelessWindowHint directly, and also whether
the window is opaque. Ideally we'd have QWindow flags that could
plumb WA_NoSystemBackground from Qt Widgets, as well as a background
color property on QWindow to control the system background, but
in the meantime we'll have to use the FramelessWindowHint heuristic.

The QWidget docs have been updated to reflect this.

Task-number: QTBUG-95042
Change-Id: I0d40eecace60883c205ebb8c76cef1092cdf1144
(cherry picked from commit 2f6d572dad031d2757a0f307cba56ae7b01c390a)
Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>

src/plugins/platforms/cocoa/qnswindow.mm

@@ -253,8 +253,18 @@ static bool isMouseEvent(NSEvent *ev)
 
 - (NSColor *)backgroundColor
 {
-    return self.styleMask == NSWindowStyleMaskBorderless ?
-        [NSColor clearColor] : [super backgroundColor];
+    // FIXME: Plumb to a WA_NoSystemBackground-like window flag,
+    // or a QWindow::backgroundColor() property. In the meantime
+    // we assume that if you have translucent content, without a
+    // frame then you intend to do all background drawing yourself.
+    const QWindow *window = m_platformWindow ? m_platformWindow->window() : nullptr;
+    if (!self.opaque && window && window->flags().testFlag(Qt::FramelessWindowHint))
+        return [NSColor clearColor];
+
+    // This still allows you to have translucent content with a frame,
+    // where the system background (or color set via NSWindow) will
+    // shine through.
+    return [super backgroundColor];
 }
 
 - (void)sendEvent:(NSEvent*)theEvent

src/widgets/kernel/qwidget.cpp

@@ -910,6 +910,8 @@ void QWidget::setAutoFillBackground(bool enabled)
     and a compositing window manager.
     \li Windows: The widget needs to have the Qt::FramelessWindowHint window flag set
     for the translucency to work.
+    \li \macos: The widget needs to have the Qt::FramelessWindowHint window flag set
+    for the translucency to work.
     \endlist