Memory tests appear to exhibit a Flynn Effect.
Archives
All posts by Peter
The most interesting talk at the Singularity Summit 2010 was Shane Legg‘s description of an Algorithmic Intelligence Quotient (AIQ) test that measures something intelligence-like automatically in a way that can test AI programs (or at least the Monte-Carlo AIXI that he uses) on 1000+ environments.
He had a mathematical formula which he thinks rigorously defines intelligence. But he didn’t specify what he meant by the set of possible environments, saying that would be a 50 page paper (he said a good deal of the work on the test had been done last week, so presumably he’s still working on the project). He also included a term that applies Occam’s razor which I didn’t completely understand, but it seems likely that that should have a fairly non-controversial effect.
The environments sound like they imitate individual questions on an IQ test, but with a much wider range of difficulties. We need a more complete description of the set of environments he uses in order to evaluate whether they’re heavily biased toward what Monte-Carlo AIXI does well or whether they closely resemble the environments an AI will find in the real world. He described two reasons for having some confidence in his set of environments: different subsets provided roughly similar results, and a human taking a small subset of the test found some environments easy, some very challenging, and some too hard to understand.
It sounds like with a few more months worth of effort, he could generate a series of results that show a trend in the AIQ of the best AI program in any given year, and also the AIQ of some smart humans (although he implied it would take a long time for a human to complete a test). That would give us some idea of whether AI workers have been making steady progress, and if so when the trend is likely to cross human AIQ levels. An educated guess about when AI will have a major impact on the world should help a bit in preparing for it.
A more disturbing possibility is that this test will be used as a fitness function for genetic programming. Given sufficient computing power, that looks likely to generate superhuman intelligence that is almost certainly unfriendly to humans. I’m confident that sufficient computing power is not available yet, but my confidence will decline over time.
Brian Wang has a few more notes on this talk
Tyler Cowen has a good video describing why we shouldn’t be too influenced by stories. He exaggerates a bit when he says
There are only a few basic stories. If you think in stories, that means you are telling yourself the same thing over and over
but his point that stories allow storytellers to manipulate our minds deserves more emphasis. For me, one of the hardest parts of learning how to beat the stock market was to admit that I did poorly when I was influenced by stories, and did well mainly when I relied on numbers that are available and standardized for most companies, and on mechanical rules which varied little between companies (I sometimes use different rules for different industries, but beyond that I try to avoid adapting my approach to different circumstances).
For example, The stories I heard about Enron’s innovative management style gave me a gut feeling that it was a promising investment. But its numbers showed an uninteresting company, and persuaded me to postpone any investment.
But I’ve only told you a story here (it’s so much easier to do than provide rigorous evidence). If you really want good reasons, try testing for yourself story versus non-story approaches to something like the stock market.
(HT Patri).
Book review: Probability Theory: The Logic of Science, by E. T. Jaynes.
This book does an impressive job of replacing ad hoc rules of statistics with rigorous logic, but it is difficult enough to fully understand that most people will only use small parts of it.
He emphasizes that probability theory consists of logical reasoning about the imperfect information we have, and repeatedly rants against the belief that probabilities or randomness represent features of nature that exist independent of our knowledge. Even something seemingly simple such as a toss of an ordinary coin cannot have some objectively fixed frequency unless concepts such as “toss” are specified in unreasonable detail. What we think of as randomness is best thought of as a procedure for generating results of which we are ignorant.
He derives his methods from a few simple axioms which appear close to common sense, and don’t look much like they are specifically designed to produce statistical rules.
He is careful to advocate Bayesian methods for an idealized robot, and avoids addressing questions of whether fallible humans should sometimes do something else. In particular, his axiom that the robot should never ignore information is a goal that will probably reduce the quality of human reasoning in some cases where there’s too much information for humans to handle well.
I’m convinced that when his methods can be properly applied and produce different results than frequentist methods do, we should reject the frequentist results. But it’s not obvious how easy it is to apply his methods properly, nor is it obvious whether he has accurately represented the beliefs of frequentists (who I suspect often don’t think clearly enough about the issues he raises to be clearly pinned down).
He does a good job of clarifying the concept of “induction”, showing that we shouldn’t try to make it refer to some simple and clearly specified rule, but rather we should think of it as a large set of rules for logical reasoning, much like the concept of “science”.
Book review: Hierarchy in the Forest: The Evolution of Egalitarian Behavior, by Christopher Boehm.
This book makes a good argument that a major change from strongly hierarchical societies to fairly egalitarian societies happened to the human race sometime after it diverged from Chimpanzees and Bonobos. Not due to any changes in attitudes toward status, but because language enabled low-status individuals to cooperate more effectively to restrain high-status individuals, and because of he equalizing effects of weapons. Hunter-gatherer societies seem rather consistently egalitarian, and the partial reversion to hierarchy in modern times may be due to the ability to accumulate wealth or the larger size of our societies.
He provides a plausible hypothesis that this change enabled group selection to become more powerful than in a typical species, but that doesn’t imply that group selection became as important as within-group selection, and he doesn’t have a good way of figuring out how important the effect was.
He demonstrates that humans became more altruistic, using a narrow biological definition of altruism, but it’s important to note that this only means agreeing to follow altruistic rules. He isn’t able to say much about how well people follow those rules when nobody notices what they’re doing.
Much of the middle of the book recounting anthropological evidence can be skipped without much loss – the most important parts are chapters 8 and 9.
Book review: Breakdown of Will, by George Ainslie.
This book analyzes will, mainly problems connected with willpower, as a form of intertemporal bargaining between a current self that highly values immediate temptation and future selves who prefer that current choices be more far-sighted. He contrasts simple models of rational agents who exponentially discount future utility with his more sophisticated and complex model of people whose natural discount curve is hyperbolic. Hyperbolic discounting causes time-inconsistent preferences, resulting in problems such as addiction. Intertemporal bargains can generate rules which bundle rewards to produce behavior more closely approximating the more consistent exponential discount model.
He also discusses problems associated with habituation to rewards, and strategies that can be used to preserve an appetite for common rewards. For example, gambling might sometimes be rational if losing money that way restores an appetite for acquiring wealth.
Some interesting ideas mentioned are that timidity can be an addiction, and that pain involves some immediate short-lived reward (to draw attention) in addition to the more obvious negative effects.
For someone who already knows a fair amount about psychology, only small parts of the book will be surprising, but most parts will help you think a bit clearer about a broad range of problems.
[See here and here for some context.]
John Salvatier has drawn my attention to a paper describing A Practical Liquidity-Sensitive Automated Market Maker [pdf] which fixes some of the drawbacks of the Automated Market Maker that Robin Hanson proposed.
Most importantly, it provides a good chance that the market maker makes money in roughly the manner that a profit-oriented human market maker would.
It starts out by providing a small amount of liquidity, and increases the amount of liquidity it provides as it profits from providing liquidity. This allows markets to initially make large moves in response to a small amount of trading volume, and then as a trading range develops that reflects agreement among traders, it takes increasingly large amounts of money to move the price.
A disadvantage of following this approach is that it provides little reward to being one of the first traders. If traders need to do a fair amount of research to evaluate the contract being traded, it may be that nobody is willing to inform himself without an expectation that trading volume will become significant. Robin Hanson’s version of the market maker is designed to subsidize this research. If we can predict that several traders will actively trade the contract without a clear-cut subsidy, then the liquidity-sensitive version of the market maker is likely to be appropriate. If we can predict that a subsidy is needed to generate trading activity, then the best approach is likely to be some combination of the two versions. The difficulty of predicting how much subsidy is needed to generate trading volume leaves much uncertainty.
[Updated 2010-07-01:
I’ve reread the paper more carefully in response to John’s question, and I see I was confused by the reference to “a variable b(q) that increases with market volume”. It seems that it is almost unrelated to what I think of as market volume, and is probably better described as related to the market maker’s holdings.
That means that the subsidy is less concentrated on later trading than I originally thought. If the first trader moves the price most of the way to the final price, he gets most of the subsidy. If the first trader is hesitant and wants to see that other traders don’t quickly find information that causes them to bet much against the first trader, then the first trader probably gets a good deal less subsidy under the new algorithm. The latter comes closer to describing how I approach trading on an Intrade contract where I’m the first to place orders.
I also wonder about the paper’s goal of preserving path independence. It seems to provide some mathematical elegance, but I suspect the market maker can do better if it is allowed to make a profit if the market cycles back to a prior state.
]
Book review: Awakening Giants, Feet of Clay: Assessing the Economic Rise of China and India by Pranab Bardhan.
This short book has a few interesting ideas.
The most surprising ones involve favorable claims about China’s collectivist period (but without any claim that that period was better overall).
China under Mao apparently had a fairly decentralized economic system, with reasonable performance-based incentives for local officials, which meant that switching to functioning capitalism required less change than in Russia.
Chinese health apparently improved under Mao (in spite of famine), possibly more than it has since, at least by important measures such as life expectancy. This is reportedly due to more organized and widespread measures against ordinary communicable diseases under collectivism.
One simple way to prevent fluctuations like those of last Thursday would be for stock exchanges to prohibit orders to buy or sell at the market.
That wouldn’t mean prohibiting orders that act a lot like market orders. People could still be allowed to place an order to sell at a limit of a penny. But having an explicit limit price would discourage people from entering orders that under rare conditions end up being executed at a price 99 percent lower than expected.
It wouldn’t even require that people take the time to type in a limit price. Systems could be designed to have a pseudo-market order that behaves a lot like existing market orders, but which has a default limit price that is, say, 5 percent worse than the last reported price.
However, it’s not obvious to me that those of us who didn’t sell at ridiculously low prices should want any changes in the system. Moderate amounts of money were transferred mainly from people who mistakenly thought they were sophisticated traders to people who actually were. People who are aware that they are amateurs rarely react fast enough to declines to have done anything before prices recovered. The decline looked like it was primarily the result of stop-loss strategies, and it’s hard to implement those without at least superficially imitating an expert investor.
Book Review: Simple Heuristics That Make Us Smart by Gerd Gigerenzer and Peter M. Todd.
This book presents serious arguments in favor of using simple rules to make most decisions. They present many examples where getting a quick answer by evaluating a minimal amount of data produces almost as accurate a result as highly sophisticated models. They point out that ignoring information can minimize some biases:
people seldom consider more than one or two factors at any one time, although they feel that they can take a host of factors into account
(Tetlock makes similar suggestions).
They appear to overstate the extent to which their evidence generalizes. They test their stock market heuristic on a mere six months worth of data. If they knew much about stock markets, they’d realize that there are a lot more bad heuristics which work for a few years at a time than there are good heuristics. I’ll bet that theirs will do worse than random in most decades.
The book’s conclusions can be understood by skimming small parts of the book. Most of the book is devoted to detailed discussions of the evidence. I suggest following the book’s advice when reading it – don’t try to evaluate all the evidence, just pick out a few pieces.