Define "non-swimability". I swim off the beach all the time, walk on sea grass in bare feet when I'm fly-fishing or when I'm walking a windsurfer out off the beach. In the 2.5 years I've been on the island I've only cut my foot twice, both times in sand and both times while stepping off a windsurfer onto a broken piece of Conch.
The only thing wrong with sea grass is it feels "yucky" to some people, mainly those who worry about "nasty" things biting their feet.
There are those out there who are going to say it's too shallow to swim over the sea grass, if you find that to be the case walk up or down the beach a few hundred yards and you'll find a spot deep enough to swim.
The only caution about swimming is to not swim past the end of the dock in high boat traffic areas and even in low traffic areas keep a close watch out and don't stray too far from the end of the dock. A personal dive flag float towed behind is a great idea when snorkeling off the beach.
Sea lice or Pica Pica as it's called here is uncomfortable during the season but it won't kill you so for me it comes under the "suck it up category". I have about 5 spots of it right now and you just live with it. Even my friends who are DM's don't stop diving just because they itch a bit, and I've seen some good cases on them.
Bugs in your drink? Protein! (and occasionally tasty.) Throw caution to the wind, be adventurous and don't let a case of the willys interfere with enjoying the greatest place on earth.
Just my two cents...