From someone who has been there? I agree you need to figure out what YOU want and what's best for you. In my case, I went back after Atlanta and really wrecked things both for me and for the team in Sydney because I'd talked myself into this mindset where I A. didn't care anymore, and b, was there to do other stuff. I'd lost the discipline because I already had what I'd wanted and let other people talk me into continuing. It really was a problem because that slot could've been taken by someone else who wanted and deserved to go.
And there's nothing wrong at all with having a life post Olympics. Took me a while to figure that one out too, and, while you shouldn't have to choose, usually it ends up GOING that way. Sounds like you know what you want to go for, which is awesome.