Well I think Alice probably knows that Caleb made a deal to get out of jail and that he works for the government in some capacity...like secret stuff that he can't tell her about.
In regards to Boo, it doesn't have to really be a secret he can tell her. But it's sort of up to Lyddia what he tells her. However Josie was very clear with him how important it is that he tells no one about this. So it would be okay if he said he's working with Caleb now and that would sort of answer everything. All the things she knows about Caleb's job she would know about Boo.