Aww, how old is your daughter? Mine used to be the light of my life, so I know what you mean. These days I do the same for my friends, I don't like to very upset around them. Family is like that sometimes.
Have you ever opened up to your brothers about it?