Davina | Inej
You asked for my opinion. Here it is: he may be jealous when he finds out, but you have done nothing wrong by enjoying an activity you enjoy doing with someone who, ultimately, was not him. Should you have refused the opportunity, rare as it is, because he wouldn't get to participate if he had been there? That would be ridiculous. Therefore, you shouldn't feel guilty.
That said, he may attempt to make you feel guilty. Jealous people often do.
I don't know him very well, so I don't know if this would work, but I would tell him about this impossible chance you had to do magic with him while he was temporarily younger. Make him feel as though he were there and as though he had participated. Make him believe *it was him*. This may alleviate some of the jealousy.