I'm seeing it as an issue in at least 4.25.4, maybe prior. I'll update after checking 4.24.
From the tests at my desk I could see that the HotReload.State file that DeleteTemporaryFiles (HotReload.cs) reads doesn't seem to have any references to temporary files inside of it, so this could be a problem with how that file is written out, or how the temporary files are collected before they're written out.
This seems to be tied to having the HotReloadState set to disabled when running, but that seems to be the case in a normal build of Dev Editor, so this is why I'm suspecting the state file creation/reading instead of the checks in BuildMode.cs.
Expectation: These are cleaned up after each run or before every new run.
There's no existing public thread on this issue, so head over to AnswerHub just mention UE-104531 in the post.