The damage, though, was severe. Melvin’s loss that month was 54.5%, or roughly $6.8 billion, one of the swiftest and steepest declines for a hedge fund since the financial crisis of 2008.

Now, the 43-year-old hedge-fund manager is attempting a feat that might prove more difficult than his earlier rapid rise to the top ranks of money managers. He is trying to make back all that money. As of the beginning of last February, Melvin had to post a 120% return to do so.

Melvin charges investors an annual management fee of 2%, and takes a cut of up to 30% of all profits, among the highest incentive fees in the industry. That can be extremely lucrative in profitable years, but the economics tend to be punishing on the way down. Like most hedge funds, Melvin must first make back all of its losses before it can resume taking incentive fees from investors who suffered losses.

Many hedge funds dealing with such sizable losses close down rather than toil for years to reach the point at which they can start charging again. The departures of investors and employees that often follow can make a recovery even less likely.

In an effort to avoid becoming a target of individual investors again, Mr. Plotkin adjusted his tactics. Now, when Melvin bets against companies through strategies such as shorting their stocks, it does so in ways that mostly don’t trigger requirements to disclose its positions. He has reduced the size of the fund’s short wagers to minimize the likelihood of big losses.

He also has pared back the amount of leverage, or borrowed money, Melvin uses. That has lessened the magnitude of potential losses—but also diminished any gains, making it harder to swiftly make up lost ground.

By year-end, the fund was part way back. For the full year, Melvin was down 39.3%, but far below the average 11.9% gain for stock-picking hedge funds, according to industry-research firm HFR. The total return of the S&P 500, in comparison, was 28.7%.

This year has gotten off to a rocky start amid volatile markets. Hedge funds betting on fast-growing companies have suffered losses. Melvin lost about 17% in the first three weeks of the year, according to people familiar with the returns.

Mr. Plotkin has told friends he is trying to ignore the constantly updating tally in his head of how much he must make back, and focus simply on performing.

Before last year, Mr. Plotkin had compiled one of the best investing track records around. Between 2014, when he founded Melvin with $900 million in capital, and 2020, the fund returned an average of 30% annually, after fees.

As of 2020, Mr. Plotkin had more than $1 billion invested in his fund. He has homes in Manhattan and the Hamptons, and a Miami Beach residence purchased for $44 million. He owns a stake in the NBA’s Charlotte Hornets.

Mr. Plotkin grew up in Portland, Maine, the son of a grocery-store executive and a part-time Hebrew school teacher. He worked at a series of jobs, including at a small hedge fund, before landing a position in 2006 as a portfolio manager at

Steven Cohen’s

SAC Capital Advisors. He became a top moneymaker there, and was one of only a few SAC portfolio managers to make money in 2008 when credit markets seized up.

In 2013, SAC pleaded guilty to criminal insider-trading charges and agreed to pay $1.2 billion in fines and stop managing outside money. Mr. Plotkin left SAC several months later, with his whole team. SAC invested when the new fund launched. By 2019, Point72 Asset Management—the new name for SAC—had more than $1 billion invested with Melvin.

Mr. Plotkin bets on companies he believes will grow relative to market expectations, and he shorts those he believes are in decline, a fairly typical approach among hedge funds.

Other hedge-fund managers regard Melvin as especially good at shorting, which involves borrowing shares and selling them in hopes of buying them back later at a lower price and pocketing the difference. Melvin profited by shorting retailer J.C. Penney Co. and renewable-energy company SunEdison Inc., both of which filed for bankruptcy in recent years. In 2015, gains from Melvin’s shorts made up two-thirds of the fund’s 67% returns, before fees.

“A lot of [portfolio managers] say they do something and they explain a process, and then, when you observe them, they don’t do that,” said

Perry Boyle,

Mr. Cohen’s longtime deputy, since retired, who had been head of equities at SAC and then at Point72.

Of the hundreds of portfolio managers who worked for Mr. Cohen over the years, he said, “Gabe was the most disciplined and process-oriented PM we had run across.” Mr. Boyle invested in Melvin last year, after the January losses.

Mr. Plotkin started 2021 on a high note. The previous year, Melvin gained 52.5%, after fees. The firm celebrated by flying employees from New York in a private plane for a beach party in Miami Beach, where Mr. Plotkin and his family had moved during the pandemic. The plane was chartered for Covid safety reasons, said one person familiar with the firm.

Still, executives at Melvin noticed something disquieting in the fourth quarter of 2020. The stock prices of some companies it had bet against, including videogame retailer GameStop, were unexpectedly rising on bullish social-media chatter.

Melvin had been betting against GameStop since 2014. It profited as the shift toward downloaded and streaming videogames caused the bricks-and-mortar retailer’s stock to drop. The gaming industry had its best year in 2020, but GameStop lost $215.3 million, following a larger loss the prior year.

Last year, between Jan. 1 and 22,

Goldman Sachs Group Inc.’s

basket of the most widely shorted stocks gained a total of 25%, compared with a total-return of 2.4% for the S&P 500. The stocks Melvin was shorting were up even more, according to people familiar with the firm. By then, the firm had lost roughly 30%, astonishing for a top fund so early in the year.

On forums such as Twitter and Reddit’s WallStreetBets, individual investors urged one another to inflict pain on Melvin by buying shares and bullish call options in companies it was believed to be shorting. From an average 87 mentions a day on Twitter and Reddit in the first seven days of that January, “Melvin” or “Gabe Plotkin” totaled an average of nearly 48,000 a day during the last seven, according to media-intelligence company

Meltwater.

Melvin hired a security firm to review hate mail sent to its office in Manhattan. Pranksters were sending pizza and GameStop gift certificates.

Ken Griffin’s

hedge fund Citadel, which itself had come close to the brink during the 2008 financial crisis, had profited previously by snapping up all or parts of the portfolios of hedge funds in distress. On Jan. 25, a senior investment executive there called Mr. Plotkin with an offer to invest.

Mr. Plotkin phoned Point72’s Mr. Cohen for advice, according to people familiar with the matter. Mr. Plotkin believed the additional money could help him weather the volatility. Mr. Cohen said taking the money sounded like a good idea. By the end of the call, Mr. Cohen had offered to invest more, too.

Hours later, Citadel, its partners and Point72 invested a total of $2.75 billion in exchange for a share of Melvin’s revenues over the next three years. The investment allowed Melvin to reduce its leverage rather than sell out of its positions.

The next two days were so brutal for Melvin that people inside Citadel and Point72 wondered if the infusion would be enough. Stocks were moving in ways that defied logic. Between Jan. 1 and their highs during the month’s final week, GameStop surged 1,745%;

AMC Entertainment Holdings Inc.,

839%, and headphone maker

Koss Corp.

, 1,761%. On some days, AMC, a struggling movie-theater chain, replaced

Apple Inc.

as the most actively traded stock in the entire market.

More than 10 of the stocks Melvin was shorting rose more than 100% in a 10-day stretch, Melvin later told clients.

Melvin also was losing money on previously profitable stakes in companies as other hedge funds experiencing losses were dumping their shares in those companies to reduce their risk.

Mr. Plotkin and his top deputies, including Chief Operating Officer

David Kurd

and partner

Greyson Clymer,

stayed up late into the nights plotting their strategy for survival.

They sold down stakes in some companies, exited the short bets they could and cut Melvin’s leverage to the lowest level since its launch, while trying to preserve what they could of the portfolio. The firm was out of its GameStop short position by the market’s close on Jan. 26.

Other prominent hedge funds had double-digit percentage losses in January, too, but Melvin’s wounds were the deepest, according to industry executives.

Melvin executives told clients the rules of the game had changed overnight. In a call near the end of January, Mr. Plotkin told investors that Melvin’s process was sound, just not geared for an aberrant, social-media-fueled tidal wave no one could foresee. He outlined changes Melvin would make to safeguard against another such episode.

Almost no clients pulled out their money over the course of the year, people familiar with the firm said. Melvin took in billions of dollars more from new and existing clients, a show of faith in Mr. Plotkin that also provided a new source of management and performance fees.

Melvin gained 14% in February 2021, raising hopes among some investors for a quick recovery.

Little has come easy since. Hedge funds struggled last year as markets repeatedly shifted between favoring fast-growing companies and inexpensive stocks, and concerns about inflation and Covid triggered broad stock selloffs. The companies most widely held by hedge funds trailed the S&P 500 by 16 percentage points since February, that basket’s worst performance on record, Goldman wrote to clients in mid-November.

Still, Melvin made money in six of the past seven months. Stocks Mr. Plotkin had bought or added to in 2020 and 2021—part of a “reopening” trade anticipating that pent-up demand would lift travel and entertainment providers—were paying off. The stock prices of live-entertainment company

Live Nation Entertainment Inc.

and theme-park operator

SeaWorld Entertainment Inc.

rose last year. Hotel operators were able to increase prices even as occupancy remained low.

He also wagered correctly that consumer demand for goods would stay strong in the face of government stimulus initiatives, lifting stocks such as

Victoria’s Secret

& Co. and

Bath & Body Works Inc.

Bets on payments and software companies contributed to gains, too.

In December, Mr. Plotkin and his team concluded that Omicron was likely to be relatively mild and that they should maintain their wager on a broader economic recovery. Melvin held on to its positions in companies such as hotel company

Hilton Worldwide Holdings Inc.

and online travel agent

Expedia

Group, returning 3.7% that month.

From February through December, Melvin notched a 33.2% return, soundly beating most other such funds. But because losses that January shrank so sharply the pool of money it was investing, the gains during the rest of the year didn’t bring the firm close to making up the earlier losses.

The dynamics that took a toll on stock-picking funds late last year have only intensified this year. The selloff in technology and growth companies, and the corresponding pullback from the stock market by other hedge funds suffering losses, have buffeted Melvin’s portfolio.

Back in 2020, Mr. Plotkin spoke about resilience on “All That Glitters,” a podcast co-hosted by his personal trainer. Referencing a big loss he suffered at SAC Capital in 2007, he recalled that his team, instead of panicking, went back to work and recouped the losses by year’s end.

“One of the great things about, whether it’s sports or the stock market, you’re knocked down a lot,” he said. “I mean, it’s tough. You’re going to go through some good times and some bad times. It’s a very humbling game.”

Write to Juliet Chung at juliet.chung@wsj.com