Reproduced 3/3 times, issue does not occur in //UE5/Release-5.0 - CL 20979098 Binary. Confirmed Regression.
It's only through the Details panel that this happens; Geometry Collection visibility can be toggled through blueprint events as expected. When GC is by itself, it resets visibility on PIE but it can still be toggled as expected.
Actual Result: Geometry Collection component becomes visible while PIE, and stays visible even when user attempts to toggle visibility through Detail Panel
Expected Result: Geometry Collection component is not visible at start of PIE, and has appropriate visibility when toggled through Detail Panel
There's no existing public thread on this issue, so head over to Questions & Answers just mention UE-171590 in the post.