"I've changed my mind if he's getting worse, outside of the hospital." Draco didn't hesitate to say it, though he didn't relish the idea of sending him back to that small room and callous attendants. "If Goyle's too caught in some fantasy of being with him to actually care for him properly, I might consider changing his caretaker." Which would have Goyle wanting to kill him, he was fairly sure, and Draco didn't like the idea of that path, either.
"I see. So rather than a modification charm, we would be removing those memories entirely."