Two years later, Mr. James convinced Wall Street’s biggest money managers to back a dramatic change in board oversight at

Exxon Mobil Corp.

, a watershed moment in the effort seeking more environmental, social and governance modifications at the nation’s biggest companies. He contributed nearly $250 million of his own money to support the campaign and launch a new fund called Engine No. 1 LLC. The resulting shareholder vote, tallied last month, secured board seats for three of his handpicked candidates.

Mr. James, 51 years old, was an unlikely catalyst for change at an energy giant. After making a name for himself during the dot-com boom and bust, he operated an Illinois coal mine and built storage facilities used by the oil-and-gas companies. Away from work, however, he supported conservationist causes. The Exxon campaign offered a chance to align his personal values with an investment thesis—that the giants of the oil industry would drop in value unless they embraced a transition to renewable energy.

He benefited from good timing. Exxon had been weakened by a $22 billion loss in 2020, making it more vulnerable to investors’ fears about shrinking profits and concerns about the company’s future.

BlackRock Inc.,

Vanguard Group and State Street Global Advisors—which typically make or break any activist campaign as the market’s biggest shareholders—also wound up in his corner at a time when the big money managers are applying more pressure on companies to change their practices. Together the three managers control more than 20% of Exxon’s shares outstanding for investors, according to data compiled by S&P Global Market Intelligence.

He also proved he could make his case for change without any of the usual fiery tactics activists employ to lift share prices in a hurry. Engine No. 1’s list of demands, instead, read more like a corporate sustainability report. In its December note to Exxon’s board, Mr. James’s fund called on the company to slash expenses on projects that might lose money when oil and gas prices are low, realign management incentives and develop a plan to invest in renewable energy.

“If we’re right on getting Exxon to mitigate these impacts, the stock should go up,” Mr. James said in an interview. “And maybe Exxon does have a future.”

So far, the fund has made money on that investment, he said. It is now seeking to raise outside capital and launch an exchange-traded fund. Exxon’s shares, which Engine No. 1 bought at around $36, have climbed above $62. The stock closed Friday at $62.17. A rally in oil prices has helped boost energy companies’ shares in recent weeks.

“We’re focused on executing a strategy that positions the company for long-term success through the energy transition, and strengthening shareholder returns,” an Exxon spokesman said. The company’s first-quarter results, he said, show that strategies put in place before and during the pandemic are paying off.

Going west

Mr. James grew up in a place that once relied heavily on the energy industry. His hometown, Harrisburg, Ill., thrived as a coal-mining region before experiencing a decline as jobs dried up during the 20th century. Mr. James went to Tulane University in New Orleans, where he studied economics, and knew where he wanted to go after graduation: Wall Street. When he was turned down by many investment banks and securities firms, he taped the rejection letters to the wall of his room as motivation to press forward.

He was able to find work after college in the hedge-fund industry, where he cycled through several New York City firms in the 1990s. He first experienced the pull of tech stocks when he attended the roadshow presentation of Netscape founder

Marc Andreessen

ahead of the internet-search company’s initial public offering. “I walked out of his meeting thinking this internet was really going to be a big deal,” Mr. James said.

He decided to move to California, where he wanted to use his money earned in the hedge-fund world to invest in tech and open a winery with a friend on a patch of farmland in Sonoma County. Dan Benton, a fellow tech investor, was impressed with Mr. James’s industry contacts and his relationships with company executives. “He was passionate, intense and impatient,” said Mr. Benton, who eventually asked Mr. James to work with him at his firm, Pequot Capital.

As the bubble popped in 2000, Messrs. Benton and James reversed course by selling some positions and betting that others would drop in value. The two investors started a new fund together in 2001 before parting ways in 2004. Mr. James then helped launch a new firm, Partner Fund Management LP.

Mr. James pursued investments in other industries, including an ill-fated bet on blood-diagnostics startup Theranos. He turned to energy and created a coal company called White Oak Resources that operated a mining facility near his hometown. The venture brought new jobs to the area, but by the time White Oak Mine No. 1 opened, natural gas had taken over coal’s role as a major U.S. energy source. In 2015, Mr. James sold White Oak to

Alliance Resource Partners.

He invested in storage facilities used by the oil-and-gas industry via Pelican Advisors LLC, a firm that manages some of Mr. James’s assets.

Mr. James has also faced some activist opposition of his own. He controls a company seeking approval to build a grain terminal in Wallace, La., and local residents have complained about potential air pollution from dust emissions. They also raised concerns about construction on former plantation grounds where the remains of former enslaved African-Americans may rest.

A spokesman for Mr. James said the project “provides the cleanest, safest and most-efficient way to support U.S. farmers” and has won support from residents and public officials for bringing industrial development and jobs to the region.

No more ‘compartmentalization’

By 2019 Mr. James was struggling to square his investing career with his other interests. He had served on the board of the National Fish and Wildlife Foundation, a group with a conservationist mission, and his sons had challenged his thinking on energy companies after discussing climate change in school.

“I kept trying to figure out if there was a way for me to wake up in the morning and everything I do is focused on one thing,” he said.

In September of that year he landed on a new investment idea while hiking alone in the Bridger-Teton National Forest in western Wyoming. His hedge fund had done an internal study on retailers, and found that many of their valuations had started to shrink long before they appeared in imminent danger. The same thing, he said, might happen in the energy sector if those companies ignored the transition to renewable power sources. So he would form a socially responsible activist hedge fund and look to force that change at those firms.

“I can get rid of this compartmentalization,” he said. “I could realign my values with an investment thesis.”

He got help from

Charles Penner,

a former partner at Jana Partners LLC who had led an activist campaign pressuring

Apple Inc.

to help address childrens’ addiction to mobile devices. Messrs. Penner and James shared a view—informed by the study of stock-performance data—that when companies are more open about the problems they cause they are also more likely to outperform. They agreed to join forces at Engine No. 1, which was named after a fire station in San Francisco a block away from the firm’s office. It launched in December. It was Mr. Penner’s idea to set their sights on Exxon.

They chose Exxon as their first big target because it had already drawn the ire of a number of other large shareholders for its lackluster performance and refusal to engage. The Engine team also knew that Exxon was a familiar name to investors of all sizes, assuring the campaign would resonate with many of the company’s individual shareholders. Some Exxon investors had also questioned why Exxon hadn’t added more directors with industry expertise.

Engine’s campaign featured ads, a website and a letter to Exxon shareholders listing its demands. The fund hired people from the worlds of activist and index fund investing, including two former BlackRock executives. The firm has 31 employees.

Asking an oil company to reduce its emissions was a test of how far an activist could go to challenge a business model. It also was one of the first significant tests of the unintended consequences of the rise of index fund giants such as BlackRock, Vanguard and

State Street.

For years the theoretical fears about these firms was that they would eventually distort prices or destroy competition. Now, if they voted with Engine, they could show their power to change a company. They didn’t buy these shares to push change the way activists do, but the index funds could assert their power as if they did.

Engine No. 1 hired a search firm, and interviewed 60 candidates before choosing the four nominees on its slate of directors. None of them were close friends of Mr. James, allaying a concern that hedge-fund activists often pitch their associates—or themselves—as the most-qualified. All had lots of energy-industry experience.

Mr. Penner led the calls with shareholders, and his pitch was to steer clear of ideological arguments about climate change. Instead, he said investors should focus on how much value had been lost. The proposed directors on these calls said the company needed to perform better and had allocated capital poorly over a decade.

“We welcome the new directors to the board and look forward to working with them—constructively and collectively on behalf of all shareholders,” Exxon’s spokesman said.

The Engine slate addressed a key concern that BlackRock and other shareholders such as the public pension giant California Public Employees’ Retirement System had with the company: Few on the board had any real energy industry experience. Calpers, the public pension, was typically reluctant to support activist campaigns directed by hedge funds, said Anne Simpson, the pension’s managing investment director for board governance and sustainability. But Exxon’s unwillingness to engage with shareholders called for bolder action, Ms. Simpson said, noting the critical difference with Engine No. 1 was the fund’s focus on future performance. Calpers not only voted for the Engine slate but also introduced the fund to Climate Action 100+, a group of investors overseeing a combined $54 trillion in assets.

“Engine No. 1 was an outlet for other shareholders to let out their frustrations,” said Ali Saribas, a partner at shareholder advisory firm SquareWell Partners. “The stars were aligned.”

—Christopher M. Matthews contributed to this article.

Write to Justin Baer at justin.baer@wsj.com and Dawn Lim at dawn.lim@wsj.com