Unfortunately, it's not something I get a say in, or yes, Victor's powers would be off except in training circumstances.
Handlers aren't involved in the inhibitor decision process, we're here to help you learn and work together. I'd advise discussing the frequency in which your powers are off with your handler.