My wife and I were there 11/8 - 11/15 last year. Weather wasn't too bad. It rained every night (sometimes POURED during the night), and rain showers blew through everyday. Daytime showers lasted 5-15 minutes, you'll see them coming, so you have plenty of time to find a place indoors and drink a Belikin.
Activity wise we fished & snorkled - no problem with either. If the rain was coming the boat moved around it with no problems.
Crowd wise, there were not a lot of tourist. No problems getting into restaurants, etc.
In a nutshell, we wouldn't hesitate to go back that time of year again. Plus it's lobster and shrimp season (perhaps conch too - but you'd need to check on that).