It was meant to be an insult. Damian knew that, but considering the charges over the years... an orphaned circus boy, a street thief, and a sycophantic do-gooder... it wasn't much of a reputation to live up to.
"It's not his fault that he's had to make do with poor substitutes all these years."