Xel doesn't generally indulge in the extreme hypocrisy of look at people like they're crazy, so the gaze he levels on Joker's back is only thoughtful (his thoughts are highly irreverent). He doesn't particularly want to be interrupted in his work by having his throat cut unexpectedly or anything, but it'd be uncomfortable to sit with his back against a wall, so he sets up a protective circle on the ground before he starts to draw on the cards: the same array, over and over and over.
While Xel doesn't doubt that this is true, they're still the wrong hands. ;)