There is still a large social stigma attached to being raped. Not only are there physical, emotional, and psychological costs to being raped (as I hope it would go without saying), there are high social costs which are as yet underexamined.
After being raped, the victim is blamed for the crime: they had to much to drink; they consented to other sexual acts and so further consent was not required; and/or their clothing implied consent. Likewise victims are blamed for knowing their rapists: they should have known the rapist was malicious; they shouldn't have stayed in an abusive relationship; or they generally "should have known better". Because victim-blaming is well understood (though still rampant), I am going to focus on two other lesser-documented social costs of being raped.
The rapist is usually someone the victim knows. If you accuse them of rape, people you both know usually ally themselves with the rapist, either explicitly or implicitly. This leads to the loss of relationships, and potentially entire social circles, for the victim. In explicitly allying with the rapist, people will tell victims that they are lying, or that they weren't "really" raped because the crime doesn't fit their false narrative of what rape is. The victim is then socially isolated.
But more often, people implicitly ally with the rapist, willfully ignoring the accusation and continuing to associate with the rapist under the guise of social neutrality, using dismissive (cowardly) excuses like "I can't know since I wasn't there". This reaction is common because it is far easier to assume that the victim is lying or "confused" than it is to acknowledge that someone you know is a rapist. Rapists are shadowy figures in dark alleyways, or evil powerful men who must be castrated; both of these popular images of rapists make it difficult to engage with the fact that you probably know someone (or worse, are someone) who has raped. Because people side with rapists instead of victims, victims either lose their friends or have friends they don't fully trust or feel safe around. One way to deal with this is for the victim themselves to continue to associate with the rapist, but this (unfairly) devalues their accusation and undermines their sense of self and legitimacy.
Language is very powerful in constructing the social view of people who are raped. People who have been raped are either branded "victims" or "survivors". The word "victim" strips people of their agency and implies that they are broken, and literally victimizes them. When people suffer other violent crimes they are only temporarily the victim, or only within the context of the crime and its immediate aftermath; with rape, people are victimized indefinitely and their rape comes to define them socially. The term "survivor" is less disempowering than "victim", but still requires the survivor to indefinitely carry the weight of their rape.
Both of these labels can be empowering if they are claimed by the person who was raped: both validate the victim's rape as a violent crime and are far better alternatives to "liar". But often these terms are thrust upon people. According to my acecdata (anecdotal "data"), people who are raped reject these labels fairly quickly but socially they are difficult to shake. Instead victims prefer to simply say they were raped—to affirm that that's something which has happened to them—which legitimizes their experience without implying that it defines them.
Rape has such strong connotations as a horrific and traumatizing event that if you tell someone that you were raped, you risk changing the way that they see you. While it is horrific and can be traumatizing, rape is not totalizing: victims and survivors continue to have humanity beyond their victimhood. In the dominant media, rape is a characterization which alone defines a victim—often the motivation for a different character (the male "hero") to seek vengeance and retribution. In mainstream psychology rape is the root "cause" of any given psychological problem, the answer to the question, "What's wrong with you?"
These are contributing factors to why people don't report rape or publicly accuse their rapists; they also stop people from privately confiding in their loved ones. As feminists intent on destroying rape culture, we need to not only listen to and believe victims but to allow them to write their own narrative of their rape.