rhi: Add a way to communicate back the native image layout for a QRhiTexture

Relevant when doing custom rendering combined with QRhi, and only for
APIs like Vulkan, where image layouts are a thing.

As shown by demo apps, it is not currently possible to implement a
correct application that renders or raytraces into a QRhiTexture's backing
VkImage, and then uses that QRhiTexture in a QRhi-based render pass.
This is because QRhi has no knowledge of the image layout if it changes
due to commands recorded by direct Vulkan calls, and not via QRhi
itself. So, except for certain simple cases, one will end up with
incorrect image layout transitions in the barriers. (at minimum this
will be caught by the validation layer)

To remedy this, add a simple function taking the layout as int (we already
do the opposite in nativeTexture()).

Task-number: QTBUG-82435
Change-Id: Ic9e9c1b820b018f3b236742f99fe99fa6de63d36
Reviewed-by: Eirik Aavitsland <eirik.aavitsland@qt.io>

src/gui/rhi/qrhi.cpp

@@ -2339,6 +2339,31 @@ bool QRhiTexture::buildFrom(QRhiTexture::NativeTexture src)
     return false;
 }
 
+/*!
+    With some graphics APIs, such as Vulkan, integrating custom rendering code
+    that uses the graphics API directly needs special care when it comes to
+    image layouts. This function allows communicating the expected layout the
+    image backing the QRhiTexture is in after the native rendering commands.
+
+    For example, consider rendering into a QRhiTexture's VkImage directly with
+    Vulkan in a code block enclosed by QRhiCommandBuffer::beginExternal() and
+    QRhiCommandBuffer::endExternal(), followed by using the image for texture
+    sampling in a QRhi-based render pass. To avoid potentially incorrect image
+    layout transitions, this function can be used to indicate what the image
+    layout will be once the commands recorded in said code block complete.
+
+    Calling this function makes sense only after
+    QRhiCommandBuffer::endExternal() and before a subsequent
+    QRhiCommandBuffer::beginPass().
+
+    This function has no effect with QRhi backends where the underlying
+    graphics API does not expose a concept of image layouts.
+ */
+void QRhiTexture::setNativeLayout(int layout)
+{
+    Q_UNUSED(layout);
+}
+
 /*!
     \class QRhiSampler
     \internal

src/gui/rhi/qrhi_p.h

@@ -791,6 +791,7 @@ public:
     virtual bool build() = 0;
     virtual NativeTexture nativeTexture();
     virtual bool buildFrom(NativeTexture src);
+    virtual void setNativeLayout(int layout);
 
 protected:
     QRhiTexture(QRhiImplementation *rhi, Format format_, const QSize &pixelSize_,

src/gui/rhi/qrhivulkan.cpp

@@ -5563,6 +5563,11 @@ QRhiTexture::NativeTexture QVkTexture::nativeTexture()
     return {&image, usageState.layout};
 }
 
+void QVkTexture::setNativeLayout(int layout)
+{
+    usageState.layout = VkImageLayout(layout);
+}
+
 VkImageView QVkTexture::imageViewForLevel(int level)
 {
     Q_ASSERT(level >= 0 && level < int(mipLevelCount));

src/gui/rhi/qrhivulkan_p_p.h

@@ -123,6 +123,7 @@ struct QVkTexture : public QRhiTexture
     bool build() override;
     bool buildFrom(NativeTexture src) override;
     NativeTexture nativeTexture() override;
+    void setNativeLayout(int layout) override;
 
     bool prepareBuild(QSize *adjustedSize = nullptr);
     bool finishBuild();