Saturday, June 02, 2007

Hofstadter's law

An interesting post by Eric Falkenstein on the Overcoming Bias blog about Hofstadter's law. The law goes like this :
It always takes longer than you expect, even when you take into account Hofstadter's law.
Applying this law to IT projects for instance which regularly take much more time than initially estimated, the author believes that this is because the goals become self fulfilling for people involved in sufficiently complicated tasks.
As they say, Hofstadter's Law is funny because it rings true to many programmers, who often work on complex projects that take years to complete. Clearly an alternative to the Law of Iterated expectations. Why might people involved in sufficiently complicated tasks--writing a paper, a book, building a deck--generally underestimate their length? I think the main reason is that goals become self-fulfilling, so any lengthening of a goal time would add to the total time the way bureaucracies spend the limit of their budget whatever it is. Just like a group of people, people themselves have multiple goals; to watch tv, to get a project done, to be a better golfer. A successful goal needs a bias to compete with your other goals, who probably also have biased homunculus advocating for them in your mind.
I partially agree with his conclusion, or I should say that I partially disagree with it:) I mean that as far as IT projects are concerned what he says is true only for projects on which there isn't a strong top management pressure for results in a rigid frame of time right from the beginning, projects in which the result is much more important than the deadline. In those cases the goal can become self fulfilling. Of course, even when IT projects are done under strong top management pressure they end up taking longer than initially estimated, but from completely different reasons. Mainly because of bad management of the relationship with the client or because the entire complexity of the project is insufficiently grasped from the start. In these cases the goal becomes a pain you want to get rid off as fast as possible. And once more, managing IT projects is far from being an exact science.

But, as the author says, biased expectations can sometimes be useful :
Think about the guy who thinks he is a better dancer than he really his confidence actually makes him a better dancer, because part of good dancing is not being self-conscious. Robert Trivers has pointed out that self-deception is, in moderation, an evolutionary advantage, in that a liar who believes his own lies is a more effective persuader than a lie who knows he is lying, and fundamentally we are social animals trying to convince others to do this or think that.