Gawd, I've read all your posts and you're waaaay too high-strung for San Pedro. Crime is minimal, you won't find drunks on every corner and the people are as kind as you'll find anywhere.
Banana Beach should be fine for you--except I might be there with my (3) kids. We're taking them along...just like we did when we stayed at Xanadu last year. Both are quiet and very nice. Tim's staff at Banana's is wonderful and Ivan and Co. at Xanadu make a smaller but equally accomodating bunch.
We went WITHOUT our kids the first time we went to BB and swore we'd never go without them again.
Relax. Go to any of the places you've selected and you'll be just fine. No one will hassle you and, in fact, you'll be shocked by the warmth you'll experience.
Did I mention that you should relax?