When theory met markets, the results were gold.

This is Part 2 of an exclusive, two-part excerpt from The Myth of the Rational Market: A History of Risk, Reward, and Delusion on Wall Street, which was published this week by Harper Business. You can read Part 1 here.

People in the investment business wanted answers. During the 1970s, young scholars with quantitative training were accorded spectacular power and influence by pension chiefs and other money managers in search of guidance. Among the most influential were two Chicagoans, Rex Sinquefield and Roger Ibbotson, who seemed to find an answer to the biggest question of all-where stock prices were headed. Sinquefield was the former seminarian and 1972 Chicago MBA who declared that the efficient market hypothesis was "order in the universe" and launched one of the first index funds. His roommate Ibbotson had entered the Ph.D. program in the Chicago Business School after getting an MBA from Indiana University in 1967 and struggling a bit in the business world.

While still a grad student, Ibbotson got a part-time job managing the university's bond portfolio. When it came to stocks, the Chicago approach encouraged formerly frenetic traders to buy and hold. Bond investing had long been all about buying and holding, but the high inflation of the 1970s made that untenable. Holding on to a bond paying 5 percent interest when inflation was 10 percent was equivalent to giving away money. Ibbotson, using analytical tools being developed or rediscovered on campus, began buying and selling bonds and running rings around the market.

Because of his bond job, Ibbotson was interested in possessing the sort of historical data on bond returns that his Chicago professors James Lorie and Lawrence Fisher had compiled for stocks. Sinquefield was in the midst of starting a stock index fund at American National Bank, and he wanted updated stock market data. Fisher showed no interest in reprising his great labor of the early 1960s, so Ibbotson and Sinquefield took on the job themselves. Then Fischer Black, who had recently joined the Chicago faculty as a finance professor, came up with a twist. After getting a Ph.D. in applied mathematics (a.k.a. computer science) from Harvard in 1965, Black had gone to work at the Cambridge-based consulting firm Arthur D. Little. There Jack Treynor introduced him to the capital asset pricing model and its simple linkage of market risk and reward. Black soon sought out the other two creators of the theory, paying regular visits to John Lintner at Harvard and getting his employer to fly Bill Sharpe to Chicago for a meeting in a hotel near O'Hare Airport (Michael Jensen was there as well) to discuss CAPM's implications for performance measurement. "The CAPM was ... the final jump he needed to solve the problem of what to do with his life," wrote Black's biographer, economist Perry Mehrling. "From then on he knew what he had to do, and so he did it."

When Black heard of Ibbotson and Sinquefield's plans to gather data on stock and bond returns, he saw a connection to asset pricing theory. He told Ibbotson that with data on risky stocks in one hand and data on risk-free government bonds in the other, he and Sinquefield could calculate the historical premium that investors received for owning stocks. It was wonderfully simple—all they had to do was subtract one from the other. Once they'd figured out this "equity risk premium," Ibbotson and Sinquefield could add it to the prevailing interest rate on government bonds to get what they called "the market's ‘consensus' forecast" of its own future trajectory. It looked an awful lot like extrapolating the future from the past, and some critics at the time said as much. But if you bought the finance professors' understanding of risk, it was something more solid. It was, Ibbotson said later, "the first scientific forecast of the market."

Ibbotson and Sinquefield presented their findings at the May 1974 seminar of Lorie's Center for Research in Security Prices. Their audience consisted mostly of fellow quants, so they presented their forecast probabilistically, following a random walk to see how far actual returns might wander from the expected mean. Just for fun, they also churned out a version geared more to the sensibilities of Wall Street, forecasting that the Dow Jones Industrials average, floundering in the 800s at the time, would hit 9,218 at the end of 1998. They also said it would get to 10,000 by November 1999.