data mining

All posts tagged data mining

Book review: Error and the Growth of Experimental Knowledge by Deborah Mayo.

This book provides a fairly thoughtful theory of how scientists work, drawing on
Popper and Kuhn while improving on them. It also tries to describe a quasi-frequentist philosophy (called Error Statistics, abbreviated as ES) which poses a more serious challenge to the Bayesian Way than I’d seen before.

Mayo’s attacks on Bayesians are focused more on subjective Bayesians than objective Bayesians, and they show some real problems with the subjectivists willingness to treat arbitrary priors as valid. The criticisms that apply to objective Bayesians (such as E.T. Jaynes) helped me understand why frequentism is taken seriously, but didn’t convince me to change my view that the Bayesian interpretation is more rigorous than the alternatives.

Mayo shows that much of the disagreement stems from differing goals. ES is designed for scientists whose main job is generating better evidence via new experiments. ES uses statistics for generating severe tests of hypotheses. Bayesians take evidence as a given and don’t think experiments deserve special status within probability theory.

The most important difference between these two philosophies is how they treat experiments with “stopping rules” (e.g. tossing a coin until it produces a pre-specified pattern instead of doing a pre-specified number of tosses). Each philosophy tells us to analyze the results in ways that seem bizarre to people who only understand the other philosophy. This subject is sufficiently confusing that I’ll write a separate post about it after reading other discussions of it.

She constructs a superficially serious disagreement where Bayesians say that evidence increases the probability of a hypothesis while ES says the evidence provides no support for the (Gellerized) hypothesis. Objective Bayesians seem to handle this via priors which reflect the use of old evidence. Marcus Hutter has a description of a general solution in his paper On Universal Prediction and Bayesian Confirmation, but I’m concerned that Bayesians may be more prone to mistakes in implementing such an approach than people who use ES.

Mayo occasionally dismisses the Bayesian Way as wrong due to what look to me like differing uses of concepts such as evidence. The Bayesian notion of very weak evidence seems wrong given her assumption that concept scientific evidence is the “right” concept. This kind of confusion makes me wish Bayesians had invented a different word for the non-prior information that gets fed into Bayes Theorem.

One interesting and apparently valid criticism Mayo makes is that Bayesians treat the evidence that they feed into Bayes Theorem as if it had a probability of one, contrary to the usual Bayesian mantra that all data have a probability and the use of zero or one as a probability is suspect. This is clearly just an approximation for ease of use. Does it cause problems in practice? I haven’t seen a good answer to this.

Mayo claims that ES can apportion blame for an anomalous test result (does it disprove the hypothesis? or did an instrument malfunction?) without dealing with prior probabilities. For example, in the classic 1919 eclipse test of relativity, supporters of Newton’s theory agreed with supporters of relativity about which data to accept and which to reject, whereas Bayesians would have disagreed about the probabilities to assign to the evidence. If I understand her correctly, this also means that if the data had shown light being deflected at a 90 degree angle to what both theories predict, ES scientists wouldn’t look any harder for instrument malfunctions.

Mayo complains that when different experimenters reach different conclusions (due to differing experimental results) “Lindley says all the information resides in an agent’s posterior probability”. This may be true in the unrealistic case where each one perfectly incorporates all relevant evidence into their priors. But a much better Bayesian way to handle differing experimental results is to find all the information created by experiments in the likelihood ratios that they produce.

Many of the disagreements could be resolved by observing which approach to statistics produced better results. The best Mayo can do seems to be when she mentions an obscure claim by Pierce that Bayesian methods had a consistently poor track record in (19th century?) archaeology. I’m disappointed that I haven’t seen a good comparison of more recent uses of the competing approaches.

Book review: The Signal and the Noise: Why So Many Predictions Fail-but Some Don’t by Nate Silver.

This is a well-written book about the challenges associated with making predictions. But nearly all the ideas in it were ones I was already familiar with.

I agree with nearly everything the book says. But I’ll mention two small disagreements.

He claims that 0 and 100 percent are probabilities. Many Bayesians dispute that. He has a logically consistent interpretation and doesn’t claim it’s ever sane to believe something with probability 0 or 100 percent, so I’m not sure the difference matters, but rejecting the idea that those can represent probabilities seems at least like a simpler way of avoiding mistakes.

When pointing out the weak correlation between calorie consumption and obesity, he says he doesn’t know of an “obesity skeptics” community that would be comparable to the global warming skeptics. In fact there are people (e.g. Dave Asprey) who deny that excess calories cause obesity (with better tests than the global warming skeptics).

It would make sense to read this book instead of alternatives such as Moneyball and Tetlock’s Expert Political Judgment, but if you’ve been reading books in this area already this one won’t seem important.

Book review: Expected Returns: An Investor’s Guide to Harvesting Market Rewards, by Antti Ilmanen.
This book is causing me to change my approach to investing much more than any other book has. It is essential reading for any professional investor.

The foreword starts by describing Ilmanen as insane, and that sounds like a good description of how much effort was needed to write it.

Amateur investors will have trouble understanding it – if you’re not familiar with Sharpe ratios, you should expect to spend a lot of time looking elsewhere for descriptions of many concepts that the book uses. I had a few problems understanding the book – he uses the term information ratio on page 188, but doesn’t explain it until page 491 (and it’s not indexed). I was also somewhat suspicious about how he handled data mining (overfitting) concerns in momentum strategies until I found a decent answer in a non-obvious place (page 404).

The most important benefit of this book is that he has put a lot of thought into identifying which questions investors should be trying to answer. Questions such as whether past performance is a good indicator of future returns, and what would cause a pattern of superior returns to persist or vanish.

Some other interesting topics:

  • why it’s important to distinguish between different types of undiversifiable risk, and how to diversify your strategies so that the timing of losses aren’t highly correlated across those strategies.
  • why earnings per share growth has been and probably will continue to be below GDP growth, contrary to what most forecasts suggest.
  • how to estimate the premium associated with illiquidity
  • why it’s useful to look at changes in correlations between equities

It’s really strange that I ordered this a few weeks after what Amazon lists as the publication date, but it took them nearly 7 weeks to find a copy of it.

Some quotes:

overfitting bias is so insidious that we cannot eliminate it (we cannot “become virgins again” and forget our knowledge)

the leverage of banks will soon be more tightly restricted by new regulations. The practical impact will be more pronounced risk premia for low-volatility assets, more sustained mispricings, and greater opportunities for those who can still apply leverage

Book review: Evidence-Based Technical Analysis: Applying the Scientific Method and Statistical Inference to Trading Signals, by David Aronson.

This is by far the best book I’ve seen that is written for professional stock market traders. That says more about the wishful thinking that went into other books that attempt to analyze trading rules than it does about this author’s brilliance. There are probably books about general data mining that would provide more rigorous descriptions of the relevant ideas, but they would require more effort to find the ideas that matter most to traders.

There hasn’t been much demand for rigorous analysis of trading systems because people who understand how hard it is to do it well typically pick a different career, leaving the field populated with people who overestimate their ability to develop trading systems. That means many traders won’t like the message this book sends because it doesn’t come close to fitting their preconceptions about how to make money. It is mostly devoted to explaining how to avoid popular and tempting mistakes.

Although the book only talks specifically about technical analysis, the ideas in it can be applied with little change to a wide variety of financial and political forecasting problems.

He is occasionally careless. For example: “All other things being equal, a TA rule that is successful 52 percent of the time is less valuable than one that works 70 percent of the time.” There might be a way of interpreting this that is true, but it’s easy for people to mistake this for a useful metric, when it has little correlation with good returns on investment. It’s quite common for a system’s returns to be dominated by a few large gains or losses rather than the frequency of success.

The book manages to spell Occam three different ways!