It's not so much the fact that he's cold that bothers me, because I can't blame him on that. It's the fact that he can be downright disrespectful sometimes. I think it was one of the encounters with him early in the series between him and the finders, really gave me a sour impression of him. Also, when the supposed "bad guys" of the series seem to care more about their fellow teammates than he does, it really doesn't say much for him as a person. I realize he does give a tiny fragment of a damn about some of them deep down, but it's his attitude the strikes me the wrong way.
That, and I find him to be the lovechild of Sasuke and Mikagami, so perhaps I'm a tad biased in my dislike of him. =p