A good friend of mine works for a consulting company, who has as a client one of the biggest telco's in Germany. The combination of the corporate culture, telco culture, and German culture mean a whole lot of bookkeeping, estimating, and bean-counting. My friend is a scrum master in charge of a few teams of programmers, and he reported to a manager-type person whose job consisted mostly of number crunching and coloring between the lines. Scrum was kryptonite in her world, but since it consistently produced results, management tolerated it.
One thing that they could not stand, however, was the nature of the estimates generated by the scrum teams. Every day she would run the numbers, and discover that the actual number of man-hours required for features and bug fixes was between 20-30% more than the initial estimate. This resulted in many tense confrontations between the two, where she demanded to know why the estimates never reflected the actual time taken for the job.
My friend would simply say, "This is a good thing. If you want to know the actual number of hours, just add 30% to my estimate."
Ok, my story is a bit belaboured here, so I'll get to the point: estimates aren't meant to be trusted. The problem comes when actual important things are scheduled around estimates (which is at times a necessary evil), but it's important to realize that an estimate isn't a piece of hard data, just like how you wouldn't use numbers from the hypothesis in a scientific experiment to prove a related theory -- you'd use the results instead.
But to answer the original question, I tend to think that that estimates are generally taken for granted because people want to believe them. It's like a horoscope... when the predicted outcome is good, people will take it at face value; but when it's bad, they will panic. Even worse, in an environment where estimates are never compared to the outcomes, then people don't have a chance to see the bigger picture, which often re-enforces the psychological impact of the estimate.
So in other words, base your estimates on previous data, not a "gut feeling". Otherwise, you risk entering a vicious cycle of everything being taken for granted, but without good cause to do so.