It would totally be worse for me to give up dairy than meat. I've almost given up most meat as it is. I just don't eat that much beef, and I almost never eat pork anymore other than the insanely seldom ham sandwich. I can't imagine giving up dairy, since chocolate milkshakes are like my favorite thing ever.
Since we started using myfitnesspal.com, I haven't had as many chocolate milkshakes as I did in the past, but I have had some. I have cut down on whole dairy, using 1% milk and such instead, and eating low-fat yogurt for smoothies and such. Like I said earlier on Facebook, I will not go quietly into my 40s! I want to stay active and fit into my 70s for goodness sake!
So, if I had to give up meat, it wouldn't be nearly as big a deal as it would be to give up dairy. I like cereal in the mornings, and a wee bit of milk in my coffee. I can go a whole day without eating meat and really not miss it at all. I try to get protein from beans and such, to make sure that I'm eating enough to carry me through the day.