Lucifer would have made Sam watch himself beat his own brother to Death. But Sam had remembered, and wrestled back control of his body. He'd thrown himself into that hole, alive, and kicking--and he'd dragged their brother in after him.
If waking up alone in that cornfield was any indication, he hadn't been able to pull Adam back out. That haunted him worse than anything Lucifer or Michael had done to him in the interim.
And Dean was apologizing. That almost made Sam break down then and there.
He squeezed his eyes shut, pinching the bridge of his nose, taking another minute after Dean had spoke again to actually answer. "I need to work. I know you're not supposed to look a gift horse in the mouth, but I need to know what's happening here."