Just returned from an 11 night stay at Banana Beach. We walked into town and back a couple of times, but with a 6 year old that walk is a bit too long. We grabbed a taxi a couple of times BZ$10 and then for the 2nd half of the trip rented bikes from Joe's bike rental. Having the bikes worked well for us (1 with a child seat on the back) and at BZ$50 each for a week was the most economical.
As for the beach at Banana Beach, there is an area between Banana Beach and Banyan Bay with no sea grass which you can swim in from the shore. Also, there is a VERY large sandbar off the end of the dock which has no sea grass, soft white sand and crystal clear water and ranges from 2 to 4 feet deep, we spent a lot of time there!
Having seen the location of The Palms I would agree that this is probably a better location if you want to be closer to town, but then I did notice that as you get closer to town the boat traffic increases and swimming from the beach might not be as easy.
The staff at Banana Beach is great, helpful and friendly and at the price for a 1 bedroom apartment the value is great, the grounds and rooms are clean and the small size of the resort means that you get to know the other guests very quickly. There were a couple of things at Banana Beach that could have been better. 1)We had a 3rd floor apartment and most mornings between about 7:30am and 8:30am we had no water. Once we figure this out, we showered earlier! 2) The large pool became very cloudy about 5 days into our stay and you couldn't even see the bottom of the pool. It was closed for over a day while they attempted to fix the problem, but never returned to "normal" the rest of our trip. So...we swam out on the sandbar instead.