With Q safely returned to him, it was true that James felt no personal obligation to remove anyone in particular from the jail. But that was just it- someone or something had taken Q from him, and locked him up, with God knows what intentions. Call it finishing the job. James was less interested in saving the remaining inmates (although he would, of course), and more interested in meeting the person responsible and teaching them a lesson. You don't mess with those close to James Bond.
He headed over to the meeting point with an air of nonchalance. It was all in a days work for him, and there was really no reason to feel any anxiety just yet. He simply had his gun on him- he hadn't had a need to actually fire it since arriving on the island, so limited though it was, he did have ammunition. And if that wasn't enough, hand-to-hand combat would have to suffice.
For now, he casually lit a cigarette, and waited for their departure to be announced.