SUCCESS. I have actually been dying for this plot ffffff---. I'm assuming you'll want it not following the plot I had in mind (because honestly, I can have this happen any way and will be great with it) as that's a lot more dark and leaves little room for the majesty and initial excitement of traveling in the TARDIS and such crazy things.
We would have to figure out a reason of course as to why Ianto gets dragged onto the TARDIS if we don't use the original plot and of course why he's not in Torchwood at the time/where in the world Rory and Amy went and the such, but a lot of that can happen in many different ways. Such as it could just be a typical Ianto runs into Eleven while he's investigating some alien phenomenon, gets dragged into it all, and Eleven offers Ianto the whole join me on adventures routine while mentioning "hey no worries we can be back by tomorrow whenever you want!" Because Eleven is obviously so good at doing this. >_____> I'm not sure if that would be enough to entice Ianto to go though, what with Jack and Torchwood and all; but again, working out plots is the best part of such madness!
For Rory and Amy we'd have to probably figure out a reason why they're not aboard any longer, which could be somewhat more challenging thinking on it....