I started bringing my son to Belize to visit family and friends when he was two and it went ok as I had someone to help out. I travelled a lot when he was one but mostly he stayed at home. It was difficult being away from him and I was always in a hurry to get back to him so when he got a bit older I started travelling with him and it was a big weight off my mind and I could enjoy myself more. He even lived in Belize with me for six months while I worked at the University for a year (the other six months he stayed in the US with his dad).
I fully understand why you want to bring him and mom along. CONGRATS on your upcoming anniv.