A hedge fund will invest your money for you – if you’ve got enough of it – in exchange for 20% of the profit. So you might wonder, how can they possibly earn a large enough return to make it worth your while to let them lop off that 20%? I’m going to tell you.

(First, a disclaimer. I’m not making any claims about any specific hedge funds that I might once have been or currently am affiliated with in some manner. This is about hedge funds in general.)

Well, maybe first I should explain why you should be skeptical of the very possibility of beating the market. There is a great deal of evidence for what is known as the Efficient Market Hypothesis (EMH). There are multiple version of EMH but to keep matters simple, let’s put it this way: stock prices move for two reasons and you can’t exploit either one of them. If there is news that would justify a price adjustment, the adjustment is instantaneous and discontinuous, so you can’t exploit it. Otherwise, price fluctuations are just random and hence unexploitable by definition.

If EMH were to constitute a perfectly accurate characterization of the market, the situation would indeed be quite hopeless. But there actually is a small amount of wiggle room. First of all, information does not spread instantly and isn’t acted on instantly or uniformly. Second, other fluctuations are not completely random; they are a function of the interdependent actions of multiple agents, at least some of which are at least partially predictable. Third, in the specific case of hedge funds, there is an asymmetry in the compensation structure that allows them to make a great deal of money at the expense of their investors.

Let’s consider each of these three loopholes and how they can be exploited.

Exploiting information. Well, you can make money the old-fashioned way by earning it. Investigate companies in depth, determine their true value, and invest in those the market has undervalued. This isn’t guaranteed to work since the price of a stock doesn’t always converge to its value (however you want to measure value). But Warren Buffett has made a bundle this way for many years.

Most aggressive young hedge fund mangers would regard the old-fashioned approach as rather tedious. There’s a faster way to make a buck from information. While it is illegal to trade on insider information, there is a very fine line between insider and public information that becomes blurred at those sensitive moments when information is just about to undergo a phase transition from insider to public. Investment houses have many departments serving multiple needs: they act as brokers for the sale and purchase of equities, they offer investment analysis and advice, they underwrite public offerings, they broker mergers and acquisitions, etc. Of course, this leaves room for plenty of mischief, the most notorious example being analysts pushing certain stocks in order to curry favor with companies that generate business for other departments. A less well-known breach of the so-called “Chinese Wall” between departments is the one through which brokers obtain information that is about to become public so that they can pass it on to their favored clients. One of the tricks hedge funds use in order to “get the first phone call” is to “churn”, that is, to make lots of meaningless trades in order to generate huge fees for the brokers. These fees are essentially legalized bribery.

Crystal balls. Stock prices are driven by imbalances between how badly and in what quantities people want to buy shares of a stock at a given moment and how badly and in what quantities people want to sell shares of that stock at that moment. People often act predictably; we are programmed by evolution to freeze up under certain circumstances and to act rashly under others. Transactions generated by computer models are predictable in other ways: they must be responsive to the demands of creditors, clients and SEC regulations. If you can figure out how to exploit this, you can make money. The way you do this is not by directly applying psychological theories or reverse engineering programs but simply by mining price movements for regularities that might indirectly result from indeterminate herd behavior or program trading. Such regularities can be found, but they will not be simple.

Many amateurs don’t appreciate the futility of overly simplistic strategies. They inevitably come up with one of two simplistic strategies. The first is trend-following: if a stock’s prices are going up, buy some on the assumption that it will continue going up. (Indeed, this will happen if some gorilla needs to buy a lot of the stock and is parceling out its buy orders.) The second is reversion to the mean: if a stock’s prices are going up, sell some on the assumption that it will soon revert to its “natural” price. (Indeed, this will happen if the gorilla needed to buy some of the stock and has gotten what it wanted.) If you can somehow divine whether the gorilla is starting or finishing, you can make money. Statistical arbitrage (“stat arb”) guys made a ton of money on reversion for a while and some day-traders still make money following trends on a few individual stocks. But the stat arb party ended in 2002 when too many people got in on it and trend-following doesn’t scale up well to the kind of money hedge funds need to put to work.

Of course, both reversion and trend-following strategies come in many flavors. For example, if somebody introduces you to a “Russian genius who has cracked the market”, you can be pretty sure that the strategy is to assume that a stock that has moved x standard deviations from its y-day moving average will revert towards that average (x and y are the secret numbers). The problem usually is that the trigger to buy or sell tends to fire infrequently and therefore the only way to make money on such a strategy is too invest a ton every time the trigger fires. But then you end up with two liabilities: first, you could seriously impact the price (if you want to buy a lot of the stock, you’ll pay extra to get it and when you want to sell it, you’ll have to settle for less to unload it) and second, you risk too much of your stake on too few positions. Amateurs underestimate the significance of both these liabilities because they assume that markets are perfectly liquid and that prices move continuously. They’re not and they don’t and if you don’t take that into account, you either won’t often get into the positions you want or you’re going to eventually get killed on a position you can’t get out of.

But like I said, if you know how to mine the data for subtle regularities without cheating and you know how to manage risk and market impact, you can make good money even with a very cloudy crystal ball.

Selling earthquake insurance. Finally, we get to the main point. Sophisticated investors don’t just buy and sell stocks and commodities. They trade a dizzying array of ever-more-complicated “derivatives”. All of these derivatives amount to the same thing, best explained by analogy. When you buy a car, you pay separately for the car and for insurance on the car. You pay, say, $25,000 for the car because that’s what it’s worth (taking into account its utility, resale value, risk of ownership, etc.) and another, say, $1,000 because you don’t want to assume the risk of losing everything in the event the car is stolen or totaled. Stockowners, like car owners, are often willing to pay a premium to be insured against being left holding the bag if some catastrophe sends the stock price plummeting. Derivatives are devices for separating out value from risk; in short, they are complicated insurance policies.

So, if you were a hedge fund, would you be buying insurance or selling it? You’re probably thinking that hedge funds would be eager to minimize risk by buying insurance. But, no. Buying insurance can be a useful supplement to a crystal ball (see above) if you happen to have one, which most people don’t, but it’s not much of a stand-alone strategy. In the short run, you mostly lose money, and in the long run, your investors have vanished.

But selling insurance, now that’s a good business for a hedge fund. Here’s why. For simplicity, let’s assume earthquake insurance is fairly priced. Then, if you’re investing your own money, in the long run, whether you buy or sell earthquake insurance, your expected profit or loss is zero (by definition of "fairly priced"). Most of the time the insurance seller makes money and every once in a (possibly, very long) while there is an earthquake that transfers the money back from the insurance seller to the insurance buyer. But hedge funds don’t invest their own money; they (usually) invest somebody else’s money. The key point is that they take 20% of the profits when there is a profit but they don’t give back money when they incur a loss. Now, there is a really great chance that ten years will go by without an earthquake, so that hedge funds selling earthquake insurance will make very consistent profits. Solid investors will happily give their money to someone with steady profits in shifting markets, the funds will grow quickly and the managers will get very rich.

Of course, one day there will be an earthquake and the investors will be wiped out. But the fund managers will still be rich because they don’t incur 20% of the losses. (They will have to make back the losses before they can cash in on subsequent profits, which is a bummer. That’s why they will close the underwater fund and start a new fund.)