This issue is in the operation of World Outliner.
If it attach another actor to the actor, even if you deleted folder that existed before, the folder will exist after restarting the editor.
If it move outside the folder without attaching it, folder can be deleted correctly.
1. Create a Folder on the World Outliner.
2. Move any Actor on the level to Folder.
3. Attach the actor to another actor.
4. Delete Folder created in Step1.
5. Save PersistentLevel.
6. Restart the Editor.
When Editor starts up, deleted Folder exists in Step4.
There's no existing public thread on this issue, so head over to Questions & Answers just mention UE-53812 in the post.