I have never felt threatened or even uneasy about walking alone. Just use some common sense, like you would anywhere.
It doesn't seem like a crime riddled place to me. It has a small population, relatively speaking, and so any incident gets around. This may make one think more is happening there than actually does.
Most of the crime is petty theft. Don't leave your belongings unattended. Lock your hotel room door, if you have alot of cash leave it with the hotel office. Drink responsibly. Just the normal precautions that one would take anywhere.
There seems to be alot of talk, gossip if you will, about the men. Seems some have roving eyes and more if the situation presents itself. There also seem to be those who would take advantage. The locals even have called them gigilos. Of course none of this can happen unless you let it.
There was a post about a possible rapist. It has since been removed. Not sure if the party was caught or left the island.
You will probably find other travelers to hang out with anyway. Just go and enjoy.