We're trying to reduce component count on our landscape using the resize tool. When modifying the landscape execution never completes. When I hit pause and get a breakpoint to see what it's doing, it appears to be looping through AInstancedFoliageActor::MoveInstancesToNewComponent running GetInstancesOverlappingBox (InstancedPlacementHash.h line 98) seemingly forever. I have tried this on multiple landscapes and this problem exists on all of them. I never had this issue resizing landscapes in 4.21, even on a fairly large and complex landscape.
Another licensee commented mentioning they're experiencing the same issue:
[...] we have pretty much the same issue on our project. We are currently on 5.2. Our landscapes have 1024 components (it's not the best setup, since they're really not that big, or at least the gameplay part isn't) and since Nanite in 5.2 requires 64 per landscape proxy actor, I wanted to take a look at what could be done. Unfortunately, I have little to no experience with the landscape editor, so I was just looking into things and found the Resize option in the editor. With the resize mode set to Expand or Clip it seemed that we could resize it to be within Nanite's limits, but upon trying it for the first time it just stalled for a long time exactly as OP described. Even after 20-30 minutes it was still not done, so I didn't wait any longer.
Pausing the execution breaks into the same functions mentioned by OP, `FInstancedPlacementHash::GetInstancesOverlappingBox` being top of the callstack. The outer z-dimension `for` loop iterates from -17179869184 to 17179869184 because the `Z` of the bounding box is (-WORLD_MAX, WORLD_MAX) passed along all the way from `FEdModeLandscape::ChangeComponentSetting`. And this happens for every foliage type of every foliage actor, for every component of the new landscape paired with every component of the old one, so it's probably the first thing to investigate most likely. I've only looked superficially through the code, so I'm not sure why we can't limit the z-axis to the common height of both the old and new landscapes, but maybe it can at least be constrained to the bounds of the level/world? I'm out on vacation the next week, but I'll try to look at it more in-depth and if I find some new info I'll be sure to update the thread.
Create a landscape
Add some foliage to it via the foliage tool (Ctrl + 3)
Resize the landscape (Ctrl+3, Manage -> Resize)
There's no existing public thread on this issue, so head over to Questions & Answers just mention UE-197004 in the post.