This video shows how to get around the issue in Virtual Machine Manager where you are constantly prompted for a password when connecting (turning on or creating) a virtual machine on the remote server.
Long story short, use VNC as the display server rather than the default Spice server.
Check out the related post at CyberScribe.org