You were under his thrall. Basically, he compelled you. Not fun on the turnaround is it. And Damon'll always be easy for you to get to, both of them will. Just with Damon I care less than if you hurt my boy. Not that I hold a grudge but y'know, staked me.
You're doing this cause you want to, you turned it back on because you wanted to. You hold to that and you'll get through.