A few years ago, Brian Uzzi of Northwestern University and Jarrett Spiro of Stanford University set out (pdf) to answer this question: What determines the success of a Broadway musical? Uzzi and Spiro began by poring through a data set that included 2,092 people who worked on 474 musicals from 1945 to 1989. To determine how good each production was they considered metrics such as reviews and financial success. They also controlled for things like talent and economic and geographic conditions to ensure that the big New York City musicals didn’t flub the data.

What they found was that successful productions relied on two components: “The ratio of new blood versus industry veterans and the degree to which incumbents involved their former collaborators and served as brokers for new combinations of production teams.” In other words, productions that worked found a balance between strong social ties and weak ones, rookies and veterans, familiarity and novelty. They weren’t flooded with a group of like-minded people but neither was everyone a stranger to each other. Uzzi and Spiro hypothesized that the reason intellectual diversity was important is because “small-world networks that help to create success or failure in Broadway musicals… face liabilities in the realms of innovation and collaboration that impede their creating new, successful musical hits… too much small-worldliness can undermine the very benefits it creates at more moderate levels, due to a decrease in artists’ ability to innovate and break convention.”

What's alarming about their conclusions is that a plethora of psychological data suggests that most of us balk when we are given the chance to connect with people who might not share similar intellects. Consider a study (pdf) done back in 2007 by Paul Ingram and Michael Morris at Columbia University. The psychologists gathered a group of executives and had them attend a cocktail mixer where the psychologists encouraged the executives to exchange ideas, network and meet new people. Like good behavioral scientists, Ingram and Morris weaseled microphones on all the nametags to record what was said. Prior to the “mixing” the executives stated that they wanted to “meet as many different people as possible” or “expand their social network,” but Ingram and Morris found just the opposite. “Do people mix at mixers?“ they asked in the concluding remarks of their study. “The answer is no… our results show that guests at a mixer tend to spend the time talking to the few other guests whom they already know well.” Or, as Jonah Lehrer somewhat sarcastically puts it in a recent post, “investment bankers chatted with other investment bankers, and marketers talked with other marketers, and accountants interacted with other accountants.”

Ingram and Morris’ study should be taken as a warning: If we want to broaden our intellectual horizons it's important to remember our natural tendency to drift towards and eventually connect with only like-minded people. Stories of innovation and discovery throughout history illustrate how important this point is. My favorite, which doesn’t get told enough, is the discovery of Cosmic Microwave Background Radiation (CMB), a key piece of evidence that changed our understanding of the origin of the universe forever.

The story begins in Holmdel New Jersey at Bell Labs where Arno Penzias and Robert Wilson were experimenting with a horn antenna originally built to detect radio waves that bounced off of echo balloon satellites. After spending some time with the antenna they ran into a problem. It was a mysterious hissing noise – like static on the radio – that persisted all over the sky, day and night. The duo went to great lengths to eliminate the hiss – they even washed bird droppings off of the dish - but it was all to no avail. Meanwhile, at Princeton University just 60 miles down the road, Robert Dicke, Jim Peebles and David Wilkinson were trying to find evidence for the Big Bang in the form of microwave radiation. They predicated that if the Big Bang did, in fact, take place it must have scattered an enormous blast of radiation throughout the universe much like how a rock thrown into a lake creates ripples that broadcast outwards. With the right instrumentation, they believed, this radiation could be all over the sky, day and night.

It was only a matter of time before serendipity set in and a mutual friend at MIT, professor of physics Bernard F. Burke, told Penzias about what the researchers at Princeton were looking for. After that, the two teams exchanged ideas and realized the implications of their work. It turned out that the hiss that Penzias and Wilson were trying so hard to get rid of was precisely the radiation that the Princeton team was looking for. A few calculations and a published paper later landed Penzias and Wilson the 1978 Noble Prize in Physics; the rest of us are still repeating the benefits of a more complete understanding of the universe.

The story of CMB reminds us that when it comes to solving difficult problems, a fresh set of eyes, even one that comes from a different field, is vital. The CMB story shows itself in one form or another, many times throughout history. The world’s great ideas are as much about other people as they are about the individual who makes it into the textbook. As Matt Ridely explains in a TED lecture in a slightly different context, “what’s relevant to a society is how well people are communicating their ideas and how well they are cooperating not how clever the individuals are… it’s the interchange of ideas, the meeting and mating of ideas between that [causes]… innovation.”

There is a wonderful website called InnoCentive.com that facilitates what Ridley calls the meeting and mating of ideas. The framework of InnoCentive is quite simple: “seekers” go to the website to post their problems for “solvers.” Problems range from the “Recovery of Bacillus Spore from Swabs,” to “Blueprints for a Medical Transportation Device for Combat Rescue,” and multi-billion dollar companies like General Electric and Procter and Gamble often post them with cash prizes up to $1 million.

The amazing part is that it’s working. A study (pdf) by researchers at Harvard Business School found that about 33 percent of problems posted on InnoCentive were solved on time. Why does InnoCentive work? The same reason that successful Broadway plays do and CMB was discovered: intellectual diversity. If an organic chemistry problem only attracted organic chemists it tended to be troublesome. However, if a biologist got involved with that same problem then the chances were greater that the problem was solved. The implications of this should make you think: solvers were at their best when they were at the margins of their fields of expertise.

Maybe it sounds obvious to suggest that a proper mixture of minds is important for accomplishing tasks, but remember the lesson from Ingram and Morris' cocktail party study: it’s really hard not to surround yourself with people like you. Don't hang out with too many opposite though; we don't want another Spider Man: Turn Off The Dark.