Remove Identity from SSH

ssh-add -D # to forget all loaded identities
ssh-add -d # to forget primary identity ($HOME/.ssh/id_rsa)

Reference