HZB Occlusion causes large stalls on the render thread

HZB Occlusion causes large stalls (greater than 5ms seen) on the render thread due to an RHI flush. This is caused by the following code in FDynamicRHI::RHIMapStagingSurface_RenderThread which forces an RHI thread flush if the HZB occlusion readback fence has not yet been signaled.

if (Fence == nullptr || !Fence->Poll() || Fence->NumPendingWriteCommands.GetValue() != 0)
{
    QUICK_SCOPE_CYCLE_COUNTER(STAT_RHIMETHOD_MapStagingSurface_Flush);
    RHICmdList.ImmediateFlush(EImmediateFlushType::FlushRHIThread);
}

The HZB occlusion readback fence is placed on the GPU after the previous frame’s HZB has finished rendering, and has been copied back to a staging buffer for CPU readback. However, when the render thread calls FHZBOcclusionTester::MapResults, due to parallel rendering and frame timing, the previous frame is often still in flight, and may not yet have completed the HZB occlusion pass. Instead of waiting on the Fence, the above code then forces an RHI thread flush and wait. This often takes significantly longer than simply waiting on the fence to signal.

On the D3D12 RHI, if the above code is removed, then FD3D12DynamicRHI::RHIMapStagingSurface, which is called immediately after, handles the fence wait, and the stall is reduced from ~5ms to ~1.4ms on the test device. This prevents the render thread from bottlenecking the frame time due to this stall. However, doing this is likely to break on other RHIs. A stall is also still present after this modification due to fence wait.

A potentially better, or additional, solution would be to buffer the HZB occlusion readbacks, so that we can readback slightly older results, if the previous frames results are not yet ready.

• Create blank project
• Add r.HZBOcclusion=1 to defaultEngine.ini
• Package project for Windows, in development configuration
• Launch packaged project
• Run `stat namedevents` command in console
• Run insights trace with `trace.send <ip> -trace=default,task`
• Open insights trace, look at render thread.
• Observe large stalls on MapOcclusionResults, waiting on RHI flush.