In my mind, rape as a victim is being made to have sex without truly informed, truly voluntary consent. If you are scared to say no, it's not voluntary.

Rape as a rapist is having sex with someone when you do not have good reason to believe they are giving informed, voluntary consent, and in fact, they are not giving informed, voluntary consent.

You can have the first without the second. In other words, you can be raped when your rapist has no moral or criminal culpability because he had no reason to believe it wasn't truly consensual. If you are scared to say no and it's not the fault of the rapist that you are scared then you are a victim of circumstances, or perhaps a victim of a third party who made you scared.