The modules file that is generated when packaging a plugin appends a number to the end of the .dll reference, similar to hot reload modules. This causes the reference to refer to a nonexistent file. Attempting to open a project with this packaged plugin included prompts the user to recompile the plugin, due to the dll not being found.
This did not occur in 4.21
There's no existing public thread on this issue, so head over toAnswerHub just mention UE-72487 in the post.