You shouldn't fear them or worry about them at all unless they are swarming.
Actually swarming bees are less likely to be agressive than those protecting a hive. Swarming is the natural method of honeybee reproduction. The current queen leaves the existing hive along with many workers (and your cash for the season)they will often land in trees or out of the way places and you see this mass of bees they are basically pulled over to the side of the road till scouts return with information of a good location for the new hive. This is where they will go to set up shop. If you are lucky you can grab the whole mass and have an instant hive, a bee keepers version of hitting the lottery.
Obviously stings can occur during the swarm or anytime, but it's not like the defensive attack when the hive is threatened.
Very often local police will keep a bee keepers contact info at hand so swarms can be easily dealt with. Extermination should be a last resort.