It is a delicate matter, yes. It is not the sort of thing one wants to spring on someone if they have not yet begun to dream. And the condition of life in 1830s Paris is not always easy to find a way of slipping into casual conversation.
Extrapolating from what I know of then and what I know of now, it is possible that he was there to check on Marius. Perhaps he wished to find a way to save him, as unlikely as such way actually existing is.