Looking for the Next Big Thing? Look to the Fringes, Deep-learning Model Shows

Subtle shifts in how people use language can foretell big changes in how we think about the world. For example, when followers of astronomer Copernicus stopped calling the sun a planet, it signaled the beginning of the end of the belief that the earth was the center of the universe.

Getting out ahead on far-sighted ideas can yield big financial or reputational rewards, but these can be difficult to spot, even though signals about the next big idea may be lurking in everyday language. Advances in natural language processing are now making it possible.

Berkeley Haas Professor Sameer Srivastava and co-authors have developed a deep learning model that can identify where and when prescient ideas—those that go against convention but later become widely adopted—first emerge.

By parsing millions of public utterances by senators, judges, and executives, they found that far-sighted ideas tend to emerge not from established leaders but rather in the language first used by those on the fringes.

“This is a way to go back and actually find when somebody first used an idea in a way that became prescient,” says Srivastava, whose research with Stanford’s Paul Vicinanza and Amir Goldberg was published in PNAS Nexus. “Our results suggest that those people were more likely to be on the periphery, rather than center, of their field.”

Upstart or Apple?

Social scientists have long grappled with the question of where paradigm shifts originate, but evidence about where such world-changing ideas originate has often been anecdotal and contradictory. The theory of disruptive innovation, for example, assumes that the ideas behind revolutionary products or business models come from industry upstarts. Yet Apple released its groundbreaking iPhone when it was already a dominant company. And in the law, it’s the Supreme Court—the pinnacle of the establishment—that is generally seen as issuing landmark rulings.

Srivastava and his co-authors used a deep neural network known as Bidirectional Encoders from Transformers (BERT) to unearth the linguistic markers of prescient ideas and trace how they became mainstream. They first defined “contextually novel ideas” as words or phrases that are used for the first time in a new context, and that also reframe the dominant assumptions in a particular field. Those that prove to be prescient ideas are “contextually novel” at the time they are uttered but later become commonplace. In other words, they foreshadow big changes.

Political Outsider

Among nearly five million floor speeches delivered by members of the U.S. Congress from 1961–2017, the model flagged Mississippi Senators John Stennis as the most prescient and James Eastland as the least prescient. Both fiercely opposed civil rights legislation in the 1960s, but the well-connected and powerful Eastland made his case with overtly racist rhetoric. Stennis, meanwhile, was “among the first to base his objections on the principles of ‘color blindness,’ limited government, and individual freedom,” the researchers wrote. This indirect set of arguments proved highly prescient, “laying the groundwork for contemporary conservative talking points on race relations in the U.S.”

Landmark Lower Courts

The same idea held true for the law.

In examining 4.2 million digitized federal and state legal rulings, the researchers found that landmark U.S. Supreme Court decisions, such as legalizing gay marriage or affirming the Affordable Care Act, tend to originate in lower courts. In fact, the most prescient decisions—those with the highest number of citations—were 22 times more likely to come from state appellate courts than the U.S. Supreme Court.

They also found that the likelihood of authoring a highly prescient decision declines as a judge is promoted from district court to appeals court. This lends further credence to the idea that novel ideas tend to come from those outside the center of a field, Srivastava said. “It’s hard to imagine that a judge would get promoted on the basis of becoming less prescient over time.”

Prescient Businesses

In the business world, the researchers had a smaller dataset to work with, analyzing transcripts of the Q&A portion of public quarterly earnings conference calls. In these calls, managers often reveal strategy not found in press releases or official filings. The model flagged smaller firms as more prescient than larger, more established players. Highly prescient firms, according to the model, had above-average stock returns. (The authors are doing more work on business figures to expand the data pool.)

Implications

The findings have implications across many disciplines, says Srivastava, the Ewald T. Grether Professor of Business Administration and Public Policy. Rather than, for example, just examining article or patent citations, which may accrue disproportionately to those who have resources or social status, the language-based approach allows researchers to trace the context in which an idea is first born.

This may pave the way, he says, to more recognition for people who have been historically marginalized—such as women and minorities. “They may be the ones generating a lot of the ideas, even if they aren’t getting credit for all of them.”

Read the full paper:

A deep-learning model of prescient ideas demonstrates that they emerge from the periphery
By Paul Vicinanza, Amir Goldberg, and Sameer B. Srivastava
PNAS Nexus, 2023