Mom always said if you can't say something nice, don't say anything at all.
She also said that as you go through life, you are going to meet people that you just don't care for.
Come on, Mom, you're making it hard on us here. How do you expect us to not say anything when we're going to be around people we can't stand?
Thankfully Mom always said it's polite to send cards to people to let them know how you feel on special occasions.