Possible shader preload deadlock when using Zen streaming

The deadlock happens when enough shader preload are launch in parallel on a slower IO backend (like zen). When that happens, the preload starts a waiting period that never ends because it cannot be triggered anymore.

When the conditions mentioned in the Steps to Reproduce happens, the following happens:

• GameThread eventually get stuck on FlushRendering

• Workers (even including standbys when option is activated) and renderthread get stuck on FShaderCodeArchive::WaitForPreload

When going for the preload, an extra task that monitor the output of the IO completion event is created. The problem is that when the workers are oversubscribed, the extra tasks are never executed which cause the deadlock even though the IO for those preload requests are probably done.

See FShaderCodeArchive::WaitForPreload and FShaderCodeArchive::PreloadShader, the latter which create a FPreloadShaderTask with the read completion events from the storage as prerequisites.

Increasing the number of standby workers only delay the problem or in the case I studied did nothing at all. Disabling preload didn't seem like a great idea considering the slowness of zen storage (in comparison to straight SSD).

In the end, the solution seemed to be to bypass the prerequisites entirely and wait on the completion event directly (and the FPreloadShaderTask afterward). This enabled the tasks to work as intended possibly due to the task being able to run directly on the same thread during the wait period without having extra coordination through the prerequisite.

In a large project with a high number of shaders:

• Cook using zen storage

• Deploy and run using zen on a platform with relatively low core count (or limit core count).

• Launch a map containing actors that request large number of shaders to load

 

Depending on timing and other actions that influences preload and loading of shaders, this could result in all workers waiting on streaming while the gamethread waits on the rendering thread which itself is waiting on other tasks.