The road is long, bumpy and dark at night as there aren't street lights. I would say that during the day the road is safe and you shouldn't have concerns expect for, sometimes, mosquitos! But ideally I would say that it is wiser to not travel in a cart at night - if you break down you may not be able to flag down assistance. If you go in a cab you will be fine at night, but it is not a cheap ride. If it were me, I would take the water taxi to town and walk around, have dinner and return by water taxi for those nights that you don't want to eat at the resort. Also note that Portofino and Mata Chica will pick you up and drop you off by boat if you dine with them.
All in the all, the road is fairly safe, but it isn't a lot of fun to do it every night unless you are with an adventurous crowd that doesn't mind some jostling in the pot holes.
Yes, crime happens and from time to time there will be a punk trying to prey on passerby - but that is definitely not a normal, every day occurrence.
I always say better to be safe than sorry so I do my best to travel with a group or take the water taxi or a cab most of the time when I am out at night - and that is true of wherever I am in the world.