macOS: Use CALayer backingstore when widgets request non-raster surface

When a non-raster widget is inserted into the widget hierarchy, widgets
switches into using RHI for flushing its backingstore, matching the
window's surface type with the non-raster widget that was inserted.

This happens for example when inserting an QOpenGLWidget, which switches
the surface type to QWindow::OpenGLSurface, or a QQuickWidget, which
switches the surface type to that used for Qt Quick.

In these cases, there might still be child widgets with their own
window handle, and as long as these child windows have the same
surface type as the window owning the backing store these child
windows are flushed through the same rhiFlush() code path as the
top level window.

For child windows with a different surface type, QWidgetRepaintManager
chooses the pain QPlatformBackingStore::flush(). Unfortunately, the
implementation of QRhiBackingStore::flush() bails out when flushing
child windows, so when we were using QRhiBackingStore on macOS for
non-raster surfaces we would end up not flushing these child windows.

The way to fix this is to use QCALayerBackingStore for the case
where we know the client of the backing store (QWidgetWindow) is
going to call rhiFlush() explicitly for non-raster surfaces. That
will lead us into QCALayerBackingStore::flush() for the child windows
with a non-matching surface type, typically QWindow::RasterSurface,
and we'll flush those child windows by setting the layer's contents
property.

Fixes: QTBUG-119309
Pick-to: 6.5
Change-Id: Ia7052b9e651d575d8e34942a25e17ab5a1f5ce05
Reviewed-by: Timur Pocheptsov <timur.pocheptsov@qt.io>
(cherry picked from commit f16f31ba3093f35134ceff6a784ad99c1867bfee)
Reviewed-by: Qt Cherry-pick Bot <cherrypick_bot@qt-project.org>
(cherry picked from commit d26be2a2062e4b24a5177a7ee92403c58ffa718e)

src/plugins/platforms/cocoa/qcocoaintegration.mm

@@ -300,6 +300,17 @@ QPlatformBackingStore *QCocoaIntegration::createPlatformBackingStore(QWindow *wi
     case QSurface::MetalSurface:
     case QSurface::OpenGLSurface:
     case QSurface::VulkanSurface:
+        // If the window is a widget window, we know that the QWidgetRepaintManager
+        // will explicitly use rhiFlush() for the window owning the backingstore,
+        // and any child window with the same surface format. This means we can
+        // safely return a QCALayerBackingStore here, to ensure that any plain
+        // flush() for child windows that don't have a matching surface format
+        // will still work, by setting the layer's contents property.
+        if (window->inherits("QWidgetWindow"))
+            return new QCALayerBackingStore(window);
+
+        // Otherwise we return a QRhiBackingStore, that implements flush() in
+        // terms of rhiFlush().
         return new QRhiBackingStore(window);
     default:
         return nullptr;