Sam normally wouldn't have taken Dean's advice to let something go; he didn't like to bury his emotions where they might fester, and he didn't like to leave questions unanswered. But in this case, Dean had a point. Following his usual instincts to brood over it and turn it over in his mind could literally break down his sanity. It went against his every instinct to ignore something that big, but he was going to have to try.
He closed his eyes and ran over the memory once more in his mind, before storing it away. He conjured a mental image of the wall, and the gap that the memory had come through, and imagined himself forcing the stones in the wall back together. He had no concept of how big the gap actually was, or whether his image of the wall was even remotely accurate, but it helped; he pulled himself together, and straightened up completely, releasing his hold on Dean.
"I'm doing my best, Dean," he said. "I want to know, but it's not the same here as it was at home. They're just... sneaking up on me all of a sudden." He knew that didn't really help Dean worry less, but it was instinctive to protest that it wasn't his fault even if he wasn't being blamed. "I don't know how to avoid remembering things if I don't know what could set it off."
Maybe if Dean had lived through it all, they could guess at it, but Dean couldn't protect him from every situation that could trigger a memory. Even if the wall could be reinforced somehow, how long would that hold? There wasn't anyone here who had more power than Death himself. But there were angels, or former angels, who might be able to tell how bad it was, at least. "We could talk to Cas."