I think a large part of it comes down to disenfranchised English Nationalism.
With Scotland and Ireland being largely remain, but the majority of England believing that this would close their borders and halt immigration, well, I think the closet racists came out in force.
The economic backlash will undoubtedly fuck them over more than their foolish concerns of 'foreigners'. Especially since, in order to continue trading with other countries... they'll need to strike up immigration agreements anyway, it's a farce.