Summary
When UnrealTestNode.StartTest returns false (or throws), TestExecutor.ExecuteTests moves the test straight to CompletedTests. It never calls CleanupTest or StopTest on the node, so the devices reserved for that test are never released until DevicePool is disposed at the end of the whole run.
- In 5.6 / 5.7 the devices are reserved in UnrealTestNode.IsReadyToStart (UnrealApp.TryReserveDevices()).
- In 5.8 / Main they are reserved by base.StartTest → UnrealSession.LaunchSession.
Tests waiting for those devices fail with "Insufficient devices" after -wait. With -NoTimeout they hang forever while any other test keeps running (the MaxDuration check is disabled, and the insufficient-devices check only runs when no test is running or starting): a soft deadlock.
Reported by a licensee (SF 27104158).
Code (5.6, Engine/Source/Programs/AutomationTool/Gauntlet)
- Framework/Gauntlet.TestExecutor.cs — the Started == false branch in ExecuteTests (~:333-339 in 5.6, ~:384 in Main) only sets timestamps, sets Result = Interrupted, adds the test to CompletedTests and logs "failed to start". CleanupTest is only reached from TickTest (running tests) and the cancel path (starting tests). TestExecutor.Dispose is empty.
- Unreal/RunUnreal.cs — DevicePool.Instance.Dispose() in the finally of ExecuteTests is the only place the leaked devices are released.
- The branch is the same in //UE5/Release-5.6, //UE5/Main and //UE6/Main; its base logic dates from CL 13149436.
CLs 51897540 and 52028138 (5.8+) only partly mitigate it, by moving reservation out of IsReadyToStart. It still leaks:
- when a derived StartTest returns false after base.StartTest succeeded;
- when LaunchSession returns null after exhausting its attempts (only problem devices are released).
Suggested fix
- In the Started == false branch, call TestToStart.TestNode.CleanupTest() in a try/catch, and force the device release. This is needed for multi-pass tests, because CleanupTest only disposes UnrealApp on the last pass.
- Release devices in the failure paths of UnrealTestNode.StartTest.
- Don't call StopTest there: TestInstance is null after a failed launch.
Workaround
In the UnrealTestNode subclass, call UnrealApp?.ReleaseSessionDevices() before returning false from StartTest (e.g. in a try/finally around base.StartTest). Throwing instead of returning false doesn't help; the executor catches the exception and takes the same path.
Affects
5.6, 5.7, 5.8, Main