Crash in GC - FSharedStruct reachability race condition

This Jira was created from the linked EPS case:

===============================================

This issue is caused by FSharedStruct batching reachability requests to the same underlying FInstancedStruct multiple times. This is not thread safe as nothing prevents the reachability processor from earlying out, so the hierarchy is processed multiple times across all GC workers.

This can lead to some race conditions where garbage exclusion kills the reference to the UObject** on the struct on one worker as it's currently being batched on another worker. This can lead the reachability to reference nullptrs all over the place.

In essence we need the following 4 criteria:

1. Lots of root UObjects with a reference to a FSharedStruct
2. The underlying FInstancedStruct must be shared between all FSharedStruct. We can do this by copying the FSharedStruct across all root UObjects so the FSharedStruct instance is unique, but the underlying TSharedPtr<FInstancedStruct> just bumps up the ref count.
3. The base struct type for the FInstancedStruct must have a TObjectPtr on it.
4. We must explicitly mark the TObjectPtr as garbage so it goes through garbage elimination during reachability.

If you have a look in the GCStressTestSubsystem in my repro sample for an example.

So the root cause is that FSharedStruct::AddStructReferencedObjects is invoked for each unique FSharedStruct instance, which in our case matches the number of root UObjects we created (10,000 by default in the attached sample). This will invoke Collector.AddReferencedObjects on the underlying UScriptStruct* which is simply the underlying shared single instance shared across all 10,000 instances. This means that the UScriptStruct* reachability is done 10,000 times in this example. This then ends up batching all its own properties this many times, including the TObjectPtr. This gets distributed between all the worker threads and performed in parallel. It's a race to see which route eliminates the UObject** address against all the other workers which are currently reading the same UObject*. We can end up crashing in lots of places in the batching, but it typically goes wrong in DrainValidated when we invoke a ObjectToIndex call, e.g:

ObjectIndices[Idx] = GUObjectArray.ObjectToIndex(ValidatedReferences[Idx].Object);

This reads offset 0xc on a nullptr because it's been eliminated on the other thread. This is checked when we DrainUnvalidated, but as soon as we pass the valid checks we assume that the ValidatedReferences only contains non-null references, which now causes issues because it was valid when the checks were first performed but has been invalidated between the DrainUnvalidated and the DrainValidated calls.

I have created a modified version of FSharedStruct called FFixedSharedStruct. This is effectively an identical version of the engine's version, but I've added an atomic int to the shared FFixedStructSharedMemory which contains the reachability flag. We mark this as reachable when the first FSharedStruct reaches the AddReferencedObject call which causes all future calls to the same underlying TSharedPtr to early out. This ensures that we only batch a single instance of the UScriptStruct.

You can toggle between the 2 FSharedStruct versions with the define in GCStressTestSubsystem.cpp

#define TEST_VANILLA_SHARED_STRUCT 1

#define TEST_FIXED_SHARED_STRUCT 1

This seems to work and follows a similar way to the UObject reachability flag, but I'm not sure if you have a better suggestion for a fix.

Steps to Reproduce

• Unzip the project
• Simply boot into PIE. It should automatically stress test as soon as the world is created. It should crash after a while.
• To test a standalone game, cook a win64 and boot a Shipping build (You need shipping so it doesn't load the TDebugReachabilityProcessor and disables GC debug).

LoginId:7c4aee5540a177fa8d7886a5b27125d5
EpicAccountId:3c9afe0965794ee3b82938927ba730c9

Unhandled Exception: EXCEPTION_ACCESS_VIOLATION reading address 0x000000000000000c

UnrealEditor_CoreUObject!UE::GC::TBatchDispatcher >::HandleReferenceDirectly() [D:\build\++UE5\Sync\Engine\Source\Runtime\CoreUObject\Private\UObject\GarbageCollection.cpp:3104]
UnrealEditor_CoreUObject!UE::GC::TReachabilityCollector<5>::HandleObjectReference() [D:\build\++UE5\Sync\Engine\Source\Runtime\CoreUObject\Private\UObject\GarbageCollection.cpp:3770]
UnrealEditor_CoreUObject!CollectPropertyReferences<2>() [D:\build\++UE5\Sync\Engine\Source\Runtime\CoreUObject\Private\UObject\UObjectGlobals.cpp:5548]
UnrealEditor_CoreUObject!CollectStructReferences<2,UStruct>() [D:\build\++UE5\Sync\Engine\Source\Runtime\CoreUObject\Private\UObject\UObjectGlobals.cpp:5364]
UnrealEditor_CoreUObject!AddStructReferencedObjectsOrNot() [D:\build\++UE5\Sync\Engine\Source\Runtime\CoreUObject\Public\UObject\Class.h:1124]
UnrealEditor_CoreUObject!UE::GC::TFastReferenceCollector,UE::GC::TReachabilityCollector<5> >::ProcessObjectArray() [D:\build\++UE5\Sync\Engine\Source\Runtime\CoreUObject\Public\UObject\FastReferenceCollector.h:938]
UnrealEditor_CoreUObject!`UE::GC::FRealtimeGC::CollectReferencesForGC,UE::GC::TReachabilityProcessor<5> >'::`3':::[Image Removed]) [D:\build\++UE5\Sync\Engine\Source\Runtime\CoreUObject\Private\UObject\GarbageCollection.cpp:4075]
UnrealEditor_CoreUObject!UE::Tasks::Private::TExecutableTaskBase<`UE::GC::ProcessAsync'::`8'::,void,void>::ExecuteTask() [D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Public\Tasks\TaskPrivate.h:898]
UnrealEditor_CoreUObject!UE::Tasks::Private::FTaskBase::TryExecuteTask() [D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Public\Tasks\TaskPrivate.h:518]
UnrealEditor_CoreUObject!LowLevelTasks::TTaskDelegate::TTaskDelegateImpl<`LowLevelTasks::FTask::Init<`UE::Tasks::Private::FTaskBase::Init'::`2':: >'::`13'::,0>::CallAndMove() [D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Public\Async\Fundamental\TaskDelegate.h:171]
UnrealEditor_Core!LowLevelTasks::FTask::ExecuteTask() [D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Public\Async\Fundamental\Task.h:627]
UnrealEditor_Core!LowLevelTasks::FScheduler::ExecuteTask() [D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Private\Async\Fundamental\Scheduler.cpp:397]
UnrealEditor_Core!LowLevelTasks::FScheduler::WorkerLoop() [D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Private\Async\Fundamental\Scheduler.cpp:757]
UnrealEditor_Core!`LowLevelTasks::FScheduler::CreateWorker'::`2'::::operator()() [D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Private\Async\Fundamental\Scheduler.cpp:220]
UnrealEditor_Core!FThreadImpl::Run() [D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Private\HAL\Thread.cpp:69]
UnrealEditor_Core!FRunnableThreadWin::Run() [D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Private\Windows\WindowsRunnableThread.cpp:159]