This is a regression from 4.22.3 (CL-7053642) where this setup executed as expected. Toggling Collision Profiles in 4.23.1 (CL-9631420) and onward do not affect the Static Mesh and it acts as if it has no collision even if the profile is set to have it. The NavMesh is just used as an example to showcase the collision swaps.
This was reported and tested in 4.23.1 (CL-9631420). This was reproduced in 4.24 Preview 4 (CL-10387632) and Main 4.25 (CL-10520629)
Results: Collision Profile changes visually (PrintString) but doesn't actually affect the NavMesh or the Static Mesh. When in PIE and BlockAll is set the Cylinder doesn't have collision
Expected: Physics state changes and navigation updates every time when switching between collision profiles
There's no existing public thread on this issue, so head over toAnswerHub just mention UE-85449 in the post.