I think Matt is generally difficult or at times downright unlikable. His shaky mental and emotional state is what makes him one of Marvel's most fascinating heroes to me.
I get that, the problem is that there are two kinds of unlikable/"difficult" characters - the ones that you can sympathize with and root for despite their assiness, and the ones you want to diaf. Right now, Matt is in the diaf group (for me).
I don't really care about Milla either. I don't know in what universe Matt Refrigerator Murdock marrying a blind civilian was a good idea, especially after less than 6 months of dating, but whatever. He married her.