Navigation: Exiting game while async nav building tasks are running can cause null pointer access of OctreeController.NavOctree from worker thread

Context

Projects can turn on the project setting bDoFullyAsyncNavDataGathering to execute navigation mesh building tasks on worker threads. Some of these tasks will access the NavOctree, whose lifetime is managed by UNavigationSystemV1.

Problem

When the user exits the application, UNavigationSystemV1::CleanUp() will immediately execute on the GameThread which destroys the NavOctree and null its reference to it. Navigation tasks may still be executing on other threads and they are not prepared for the NavOctree to be destroyed while the task is running. This is a race condition that can result in null pointer access from the worker thread running the nav task. A licensee has provided one worker thread callstack where this happened.

• Create a UE 5.4 project from the ThirdPersonTemplate
• Open the project settings and
  • set Navigation Mesh > bDoFullyAsyncNavDataGathering to true
  • set Navigation Mesh > Runtime Generation to Dynamic (causes nav mesh build at runtime)
• Drag a NavMeshBoundsVolume into the level. Press P to confirm a navigation mesh is being built.
• Put a breakpoint on FRecastTileGenerator::GatherNavigationDataGeometry
• Start playing in editor and confirm that FRecastTileGenerator::GatherNavigationDataGeometry is executed at runtime from a worker thread and not the GameThread
• The following is a manual way to reproduce the race condition. Modify FRecastTileGenerator::GatherNavigationDataGeometry and add the following code.

check(NavSys.GetNavOctree());
FPlatformProcess::Sleep(5.0f);
check(NavSys.GetNavOctree()); 

• Start PIE and stop immediately.
• Observe: The editor will freeze while waiting for the worker thread to finish, so about 5 seconds. Then check(NavSys.GetNavOctree()) fails because the NavOctree has been destroyed immediately on stopping by UNavigationSystemV1::CleanUp().
• Expected: If the task started with the octree valid, it should finish while the octree is valid.

Callstack of null pointer access (non-GameThread, fatal):

FNavigationOctree::DemandLazyDataGathering (NavigationOctree.cpp:76)
FNavigationDataHandler::DemandLazyDataGathering (NavigationDataHandler.cpp:585)
UNavigationSystemV1::DemandLazyDataGathering (NavigationSystem.cpp:3352)
FRecastTileGenerator::GatherNavigationDataGeometry (RecastNavMeshGenerator.cpp:2092)
FRecastTileGenerator::GatherGeometryFromSources (RecastNavMeshGenerator.cpp:1945)
FRecastTileGenerator::DoWork (RecastNavMeshGenerator.cpp:1875)
FAsyncTaskBase::DoWork (AsyncWork.h:288)
FAsyncTaskBase::DoThreadedWork (AsyncWork.h:312)

Example callstack where NavOctree got nulled before (GameThread, non-fatal):

UnrealEditor-NavigationSystem.dll!UNavigationSystemV1::CleanUp(const FNavigationSystem::ECleanupMode Mode) Line 5054    C++
UnrealEditor-NavigationSystem.dll!UNavigationSystemV1::OnBeginTearingDown(UWorld * World) Line 1271    C++
[External Code]    
UnrealEditor-UnrealEd.dll!UEditorEngine::TeardownPlaySession(FWorldContext & PieWorldContext) Line 690    C++
UnrealEditor-UnrealEd.dll!UEditorEngine::EndPlayMap() Line 344    C++
UnrealEditor-UnrealEd.dll!UEditorEngine::Tick(float DeltaSeconds, bool bIdleMode) Line 2475    C++
UnrealEditor-UnrealEd.dll!UUnrealEdEngine::Tick(float DeltaSeconds, bool bIdleMode) Line 546    C++
UnrealEditor.exe!FEngineLoop::Tick() Line 5872    C++
[Inline Frame] UnrealEditor.exe!EngineTick() Line 69    C++