Not that I'm defending the airlines, because I most certainly am not after my last horrible nightmare travel with my dog, but you take care of yourself when you get on the plane, off the plane, etc. There's a bit (yes, not much) more work for them storing your dog prior to loading, putting the dog in the appropriate place on the plane (which by the way it not exactly with the luggage), and handling your kennel with care when deplaning and getting it to the appropriate location since they don't ride down the luggage shoot at the baggage claim.
Also, it is a huge hastle for the airlines and a liability so they tend to keep prices high for animals to discourage us from traveling with them.
I agree it is extremely expensive, but if you don't want to pay it maybe you should consider traveling without your loved ones. I hate to pay it but my "little one" who is not so little is worth every penny.
By the way, you mention that dogs need special papers but so do we...it's our passport. You can now apply with BAHA for a doggie passport if you are traveling on multiple occasions with your animal and you just do that once like you do your passport. Same thing.