I like Caulker, have enjoyed my brief stay there. But, I love Ambergris. More choices as far as food, drink and places to stay. Easier to find live music at night.
I've also been to Isla several times, will return again in March. My last visit was January. I found it anything but quiet, Hidalgo was a bit of a zoo at night. Defnitely more going on in a small space than on Ambergris.