What would affect physical bodies and environment but leave our consciousness intact? I assume you've tested whether these are just complex illusions of some sort?
That would work - and even if the mushrooms aren't visible, we could test for spores - sometimes that's enough because it's more extensive than we could imagine.