When is it appropriate to despise someone?

When you have a low opinion of their moral compass. It doesn't matter whether they constitute a threat to your values or not. One's fear and anxiety are not the cause of contempt. It's perfectly fine to despise the mean, petty, weak, or worthless --especially when such traits are present in someone who believes in their own excellence.