Edward felt frustrated, frustrated with Envy, frustrated with himself, frustrated with the Gate. He could plainly see that Envy was not calmed down enough to play domestic husband again. He was afraid the Gate had put Delilah in danger with Envy.
But, at least Envy was genuinely concerned for her, that was a point in his favor.
But he was frustrated. Part of him had wanted to be friends with his biological older half brother, to take back that from the old bitch, and maybe put his own ghosts to rest. But Envy was too suspicious and Edward probably wasn't trying as hard as he should've, because Envy was right- Edward never wanted him to be there. He would've been happier with Envy being dead.
But he wasn't, so Edward had tried to deal with it as best he could and clearly had failed.