While having sex I feel I am not much aroused.?

I experience at times a strange feeling when he directly touches my genitalia. I just feel that I do not want to be touched in intimate places. I have no problem at all with penetration and I really like it, but I just cannot handle direct stimulation to my genitalia every time we have sex. This does not happen all the time but it puts him off and he starts complaining that I am cold. He asks me directly why do I refuse being touched and I do not know how to explain what I am feeling. Sometimes it is just the opposite and we have great sex and he would be very happy about it. Is there any way I can increase my libido? Do I need to take any medicine and why do I get such a feeling in bed with him? This is affecting our relationship and I really love him so much and do not want anything to affect our love for one another.