The collision response of an actor can be changed through the Details panel of the instanced actor and this overrides the collision response settings that were set through the Construction Script.
Moving the actor around in the viewport so that the construction script is re-ran does not change the actor's collision response.
I expected that the collision response cannot be changed if it is set through the Construction Script similar to the way a point light component's light color cannot be changed in the details panel if it is already set through the construction script.
The player collides with the Cube
The player flies through the Cube
There's no existing public thread on this issue, so head over to Questions & Answers just mention UE-94789 in the post.