Robert Shiller's book Irrational Exuberance is a well-researched effort to put recent stock market fluctuations in the appropriate historical perspective. The book contains many minor flaws, but most of his arguments are robust enough to be little affected by his mistakes.
He provides strong arguments against the most popular versions of the efficient market hypothesis, shows that the rationalizations that have been given recently for buying stocks at high prices are quite similar to those given in the 1920s, is moderately convincing in his forecast of well below normal returns for the S&P 500 over the next decade, and does a fairly good job of explaining why bubbles happen.
The rest of this review will focus on some of the book's flaws, but I hope nobody takes these criticisms as excuses to buy the S&P 500 and hold it for a long time. Nor should my comments be taken as advice to sell all stocks today. The unusually low real short-term interest rates that we have today make the outlook for 2002 fairly good. Also, stocks that have had very small market capitalizations haven't been affected much by the bubble, and might be safe to buy and hold.
The book's first and most important graph is somewhat misleading because Shiller uses a linear vertical axis to plot prices and earnings over time. This makes recent price/earnings ratios look higher relative to older ratios than is actually the case. For instance, p/e ratios of the 1960s look much higher than the 1929 peak, when they are actually about the same. (The 2000 peak looks unusual enough either way that this has little effect on his main point). He is aware of criticism that the linear axis is misleading, but shows no sign of understanding the criticism.
Shiller relies a bit too much on the assumption that what the average person is saying reflects the reasons why stock prices are going up. That assumption is often adequate, but sometimes produces confusing results. For example, I don't recall hearing anyone question the widely held myth that Amazon was losing money on each book but trying to make it up on volume around the time that Amazon's stock was selling at internet bubble levels.
Demonstrating that most investors are thinking like investors have in past bubbles, and don't have strong arguments for higher stock p/e's, does not prove the absence of a good reason for higher p/e's. I have one such reason to propose that works even if nobody understands it. In most previous bull markets, the dominant companies were very dependent on investment in factories and equipment to enable them to handle increased demand. In contrast, many of the companies that led the bull market of the 1990s depended much more on forms of capital such as reputation and human minds, and can expand their capacity with with only small additional purchases of factories and equipment.
Increased prices for human capital doesn't cause much overinvestment in human capital (i.e. having more babies - no company owns that new capital). The internet bubble caused some excessive spending on reputation capital (as evidenced by the fluctuations in internet advertising payments), but many large companies like Coca-Cola weren't influenced by the optimism created by their high stock prices to invest in more reputation capital, because the connection between their industry's sales and their need for reputation is quite weak.
This implies less resistance to bubble-level stock prices from companies selling stock to fund capital investments, but also a long-term slowdown in the growth of demand for capital. This may well mean that capital is becoming permanently more abundant relative to demand, implying that in the long run price/earnings ratios will be higher, and returns will be lower, than in the past. Unfortunately, this hypothesis does not provide any easy way to predict how much higher p/e ratios will be in the future than they have in the past, so it is nearly useless at determining whether current p/e ratios are reasonable. It's main implication is that there is much room for doubt.
Shiller points out that the similar (but temporary) effects of the baby boom demographics on the supply of capital don't explain why efficient markets would have reacted to the capital surplus with a more uniform rise in prices of all investment vehicles (including real estate and bonds) than we have actually seen, and that comment applies to my proposed cause as well. I don't think the markets have been reacting in a very efficient way to the capital surplus - they have been acting like people have very imperfect guesses about why stocks have been rising.
The word irrational in the title suggests that he demonstrates that investors are irrational during bubbles, but much of his explanations are at least as consistent with the hypothesis that investors are pursuing multiple goals, and that the other goals sometimes interfere with the goal of maximizing the return from their investments. Fitting in with one's social group is an important goal for many people, using the same investment strategy as one's peers are using appears to be a means towards accomplishing that, which implies that people who give up some financial returns in order to be like others aren't being obviously foolish (although they usually have to mislead themselves about what they are doing). Also, the degree to which people overestimate their own competence is a persistent enough phenomenon that it must have some advantages which offset the problems it creates (e.g. being confident may make one more influential).