Your father does a wonderful job of acting like he has everything together all of the time. It's how he copes with not being in control. I don't know how different he is from the Clint Barton who raised you, but I can tell you that he was a mess when I met him. Still is. He's just learned how to hide it. If our life is calmer than yours, it's because we were lucky, not because we're any better than you, or because you don't have just as much value.
Since we are both being truthful, I will tell you this. It would hurt Clint more not to be part of your life than to be pulled into whatever messes you happen to find along the way. So please don't hold back because you feel like you're somehow holding him back. You aren't. I promise.