When using an InstancedStaticMeshComponent with PerInstanceCustomData, the number of float elements in the "Custom Data" array should always be equal to NumCustomDataFloats multiplied by the number of instances. However, when removing individual elements from the "Instances" array, the "PerInstanceCustomData" array is not properly updated, meaning that its size no longer matches the expected number of elements. Later, when re-adding more instances, a check() fails inside function Scatter() in file "InstanceDataUpdateUtils.h", causing the Editor to crash.
An initial investigation tracked the problem to function UInstancedStaticMeshComponent::PostEditChangeChainProperty(), which calls UInstancedStaticMeshComponent::RemoveInstanceInternal() with "InstanceAlreadyRemoved = true". This makes RemoveInstanceInternal() behave as if both SM and CustomData arrays have already been updated externally, but in reality only the SM array has been updated at this point (by the user action), so the CustomData array ends up not being updated anywhere. It is interesting to note that UInstancedStaticMeshComponent::AddInstanceInternal() works differently, and seemingly correctly, despite not having an "InstanceAlreadyAdded" parameter. Also note that both RemoveInstanceInternal() and AddInstanceInternal() functions can also be called directly by BP-exposed functions.
Note: This repros in UE 5.4+. Even prior to that, the "Custom Data" array seems to become incorrectly sized on instance removal, but no crash is observed.
Note: The linked EPS case contains a quick experimental fix.
1. In the Content Browser,create a new Material "M_ISM"
2. Edit the new material, add node "PerInstanceCustomData[0]", link it to any attribute
3. Save and close
4. Place an empty actor on any level
5. In the Details Panel, add an "Instanced Static Mesh" Component
6. Select the InstancedStaticMeshComponent. In the details panel:
6.1. Set "Instances – NumCustomDataFloats" to 1 (or any non-zero value).
6.2. Set "Static Mesh" to "/Engine/BasicShapes/Sphere" (or any other).
6.3. Set "Materials – Element 0" to "M_ISM" (minor bug note: it may be necessary to deselect and reselect the InstancedStaticMeshComponent for this property to appear)
7. In the details panel, under category "Instances", edit array "Instances":
7.1. Add an instance - Note that "Instances – Custom Data" correctly gets one or more entries as appropriate
7.2. Remove all instances by clicking the trash can icon - Note that "Instances – Custom Data" is correctly emptied as well
7.3. Repeat step 7.1.
7.4. Remove an instance by clicking on the arrow next to it and selecting "Delete", or by resetting the array property to its default value - Note that "Instances – Custom Data" is NOT correctly updated
7.5. Repeat step 7.1. This time, the engine will crash on a check() inside function Scatter() in file "InstanceDataUpdateUtils.h", and on several other checks afterwards.
Notes:
There's no existing public thread on this issue, so head over to Questions & Answers just mention UE-361676 in the post.