It's not everything. What's the use of love if the other person won't listen (and also thinks you're not listening)?
"Fuck off!" he yells, because he's sick of being told to do the work when that's what he was fucking trying to do, and he lashes out, aiming his fist at that smug, infuriating face, furious with Spike for apparently having all the answers, for being perfect and instinctively getting everything while he's still floundering in the dark.