You will have to be very comfortable with open water kayaking. Power boats will not give you any quarter and at busy tour times, from about 9am to 5pm, you will be constantly riding (sometimes big) wake.
Once you are on the reef, this is not a big issue for swimming/snorkeling as the water is too shallow for boats to use a lot of speed.
Due to time constraints, we didn't try to kayak to the other cayes but again, if you're good in open water, it can be done.
I saw some pretty close calls w/boats and swimmers at piers and at the beach so I would only swim in designated roped off areas.