At the age of ten, there wasn't much life experience to bring about the feeling of the whole sky falling down on oneself – at least, not a normal ten year old's experience. Ben's experience was far more varied and complicated and painful that normal kids had, but all of that was still no comparison to the weight that hit him at the idea that people had forgotten, that they were lost, that his grandma was the only one left of his whole family. For a boy who hadn't had more than a mom before arriving here, then grown quickly use to having a family and friends who were basically finally, that was difficult to accept.
But there was a piece that his brain turned over and over, like a foreign object that it couldn't recognize, because the idea was just that horrifying.
"You can't find Dad? Nobody knows who we are, including Dad?"
No dad, get a dad, lose a dad. It was like a terrifying anti-mantra that forced itself on repeat for Ben and he could only stare at Castiel with eyes that were too wide, suddenly too terrified and several seconds from a hated show of 'girly' weakness.