Not sure about US bank accounts, but we (Canadian bank account holders) do pay a service charge for using a debit card to withdraw funds from a Belize bank. Of course, we also pay the conversion rate to exchange the Belize monies to US funds BUT soon it appears that we will get a better rate.. smile
As you say, you don't want to carry around too much cash, so a better option.