In UE4 (using DX11), WmfMedia was able to play .mp4 files by default. However, WmfMedia doesn't support playing .mp4s in DX12, which is what is used in UE5. Electra can handle playing .mp4s in UE5, but it's not a plugin that's enabled by default.
All of the above means that, if someone creates a new project and attempts to play an .mp4 without first enabling Electra, it'll error out and prevent playback.
The media player remains black; an error from WmfMedia indicates "Format is not supported in D3D12."
The .mp4 should begin playing in the media player.
There's no existing public thread on this issue, so head over to Questions & Answers just mention UE-148398 in the post.