GAS: Async loaded GameplayCueNotify can execute cue events out of order

Licensee reported problem via UDN.

Context

GameplayCues ('cue') are events that can be fired by game code, that are identified via a GameplayTag. Cues can be instantaneous or can be a state on any actor with an AbilitySystemComponent. GameplayCueNotifies (GCN) are (blueprint) actor classes that respond to fired cues. The GameplayCueManager knows which GCN responds to which tag. It can preload GCN blueprint classes, or load them on-demand if the class isn't loaded yet. Loading them on-demand can happen synchronously or async, depending on how the GameplayCueManager subclass overrides certain virtual functions.

Problem

When game code adds and removes a stateful cue on an actor, but the GCN class isn't loaded yet and is async loaded, the cue events can be executed out of order on the GCN depending on timing. Specifically if the game code calls AddGameplayCue and then RemoveGameplayCue before the GCN class has finished loading, then the removal and add events executing out of order can lead to GCNs staying active indefinitely. In practice this can have unintended effects like particles or looping sounds staying active.

When AddGameplayCue followed by RemoveGameplayCue is called during the async loading of a GCN class, one of two scenarios can happen timing wise:

1. Either two async load requests are started and the events are executed in order, because async loading fires the streaming delegates (async load finished delegates) in order. This results in the events being executed in order (good), but having started two async load requests for the same GCN class is inefficient (could be better).
2. If the async loading thread has finished loading the GCN class (at least, the class can be statically found) but not fired the callback yet and then RemoveGameplayCue is called, that removal event is executed immediately. The async load callback will then execute the add event later, resulting in Remove-Add happening out of order (bad).

The second case happens because GameplayCueSet performs a just-in-time class lookup to decide whether to (a) async load and defer the event or (b) execute immediately. The problem happens because the class can be found before a prior cue's async load callback has been executed. The likelihood of this problem is increased by the fact that async load callbacks are deferred until the end of the frame: see

/** Helper class that defers streamable manager delegates until the next frame */
class FStreamableDelegateDelayHelper : public FTickableGameObject 

There are many opportunities for game code to execute between a GCN class being findable, but before the related async load callback is executed, so scenario (2) must be addressed to execute events in order.

Suggested Fix

When async loaded GCNs finish loading and multiple events (such as add-remove) have been requested in the meantime, there are multiple ways to deal with this:

• Execute all of them in order
• Have add-remove cancel each other out, fire no events
• Execute them if within some deadline (i.e. up to 1 second after the async load request)

Which one is preferred by developers will be use-case specific based on what the GCN is intended to do. For example: if it's to activate/deactivate particle systems or looping sounds then having nothing happen may be better. If it's to play one-shot sounds, particles or spawn combat text then executing the events may be better. Because GCN can contain any game code, I'm opting to execute all events in order. This results in consistency: executing cues will result in the associated GCN being played regardless of preloaded/sync loaded/async loaded. If immediate playback is important, then Epic's recommendation will be to update your configuration to preload the GCN classes. See Lyra for examples.

Repro steps in Lyra:

• Modify ULyraGameplayCueManager::ShouldAsyncLoadRuntimeObjectLibraries() to return false (no GCNs will be preloaded).
• Create a GameplayCueNotify_Static blueprint 'GCN_MyReproNotify' and configure its default to listen for GameplayCueTag = GameplayCue.MyRepro (create that tag). Override its HandleGameplayCue function to print out the event type.
• Hacky race condition repro:
  • Modify UGameplayCueManager::OnMissingCueAsyncLoadComplete() to broadcast a multicast delegate.
  • Let LyraCharacter::BeginPlay subscribe to that manager delegate to remove a GameplayCue from itself with tag 'GameplayCue.MyRepro' by calling UGameplayCueFunctionLibrary::RemoveGameplayCueOnActor
  • Although it seems obvious that remove/add is now executed out of order, even without the delegate that is purposefully executed right before executing the deferred add events, game code can call Remove at any time between async load thread finishing loading the GCN class and GameThread firing the streaming callback.
• In-game, add a Debug Key Z event in blueprint Character_Default  that adds a Gameplay Cue with tag 'GameplayCue.MyRepro' to the controlled character by calling UGameplayCueFunctionLibrary::AddGameplayCueOnActor
• Start PIE. Execute Lyra.DumpGameplayCues to confirm that GCN_MyReproNotify is not preloaded yet.
• Put a breakpoint in UGameplayCueSet::HandleGameplayCueNotify_Internal().
• Press the debug key Z, which adds the cue.
• Observe:
  • This triggers an async load of GCN_MyReproNotify. It sets up a streaming callback to execute a EGameplayCueEvent::Type::OnActive and EGameplayCueEvent::Type::WhileActive event on the GCN once done loading.
  • Once those loads are finished, it executes the delegate we setup in GameplayCueManager, which tries to remove the gameplay cue. This is to emulate other game code that may have done the same logic earlier in this tick.
  • The Removed event executed immediately because UGameplayCueSet::HandleGameplayCueNotify_Internal finds the class just in time, while the OnActive/WhileActive events haven't fired yet.
• Expected:
  • When game code calls AddGameplayCueOnActor and then RemoveGameplayCueOnActor that requires a GCN class to be async loaded, the GCN class will execute the OnActive-WhileActive-Removed events in that order.

Non-fatal, callstack for reference for when deferred gameplay cues are executed:

     UnrealEditor-GameplayAbilities.dll!UGameplayCueManager::OnMissingCueAsyncLoadComplete(FSoftObjectPath LoadedNotifyClass) Line 380    C++
     [External Code]    
     [Inline Frame] UnrealEditor-GameplayAbilities.dll!TDelegate<void __cdecl(void),FDefaultDelegateUserPolicy>::ExecuteIfBound() Line 635    C++
     UnrealEditor-GameplayAbilities.dll!UE::StreamableManager::Private::WrapDelegate::__l5::<lambda_1>::operator()(TSharedPtr<FStreamableHandle,1> __formal) Line 99    C++
     [External Code]    
     [Inline Frame] UnrealEditor-Engine.dll!TDelegate<void __cdecl(TSharedPtr<FStreamableHandle,1>),FDefaultDelegateUserPolicy>::ExecuteIfBound(TSharedPtr<FStreamableHandle,1>) Line 635    C++
>    UnrealEditor-Engine.dll!FStreamableDelegateDelayHelper::Tick(float DeltaTime) Line 214    C++
     UnrealEditor-Engine.dll!FTickableGameObject::TickObjects(UWorld * World, ELevelTick LevelTickType, bool bIsPaused, float DeltaSeconds) Line 196    C++
     UnrealEditor-UnrealEd.dll!UEditorEngine::Tick(float DeltaSeconds, bool bIdleMode) Line 2199    C++
     UnrealEditor-UnrealEd.dll!UUnrealEdEngine::Tick(float DeltaSeconds, bool bIdleMode) Line 533    C++
     UnrealEditor-LyraEditor.dll!ULyraEditorEngine::Tick(float DeltaSeconds, bool bIdleMode) Line 39    C++
     UnrealEditor.exe!FEngineLoop::Tick() Line 5933    C++