GroupSense soon began fielding more such requests from victims’ law firms and insurance companies, which reached up to 10 a week by the end of last year. The company charges flat rates of $12,000 to $25,000 based on clients’ revenue.

“We did not jump in,” Mr. Minder said of the market for ransom negotiation, adding that it is a loss leader for his firm’s other services. “We got dragged in kicking and screaming, basically.”

The growing prevalence and complexity of ransomware has spurred a cottage industry of first responders to counter it. Startups have launched to communicate with hackers or transmit payments using cryptocurrencies, while large cyber companies have hired personnel or acquired specialty firms to help clients respond to and recover from such incidents.

Ransomware took on new prominence this month after a hacking group known as DarkSide targeted Colonial Pipeline Co. and forced a six-day shutdown of the largest conduit for fuel on the East Coast.

Colonial Pipeline Chief Executive Joseph Blount told The Wall Street Journal Wednesday he decided to pay the hackers about $4.4 million in bitcoin hours after receiving a ransom note.

A representative didn’t comment on whether the company negotiated the price down. “We needed to do everything in our power to restart the system quickly and safely,” he said in a statement.

Victims must quickly weigh such payments against potentially crippling computer outages or hackers’ extortion threats in the form of leaked internal data. In the worst cases, negotiators help victims cut a deal they never wanted to make.

“I do approach it as a business deal,” said Karen Sprenger, chief operating officer and lead ransomware negotiator for cyber firm LMG Security. While victims often are understandably angry during such talks, Ms. Sprenger tries to remain detached.

“You can’t go into it showing emotion,” she said.

Some cyber experts warn the new muscle is still hard-pressed to match increasingly professionalized hacking groups behind a ransomware boom that U.S. officials have labeled a national security threat. Some security firms also are wary of the risks from engaging with criminals or facilitating payments that could run afoul of sanctions.

“The industry is, in certain areas, hitting capacity limits,” said Eric Friedberg, co-president of Stroz Friedberg, an incident-response consulting firm owned by insurance company

Aon

PLC.

While Mr. Friedberg said Stroz Friedberg advises executives on trade-offs, the firm works with specialty negotiators with a granular knowledge of ransomware groups’ tactics and trustworthiness, key factors for victims deciding how to respond.

“Is what’s important to the company the speed of resolution or lowering the ransomware payment?” Mr. Friedberg said. “Those are often mutually exclusive goals, and they’re usually board level decisions.”

As executives debate their strategy, negotiators at firms such as Booz Allen Hamilton Inc. play a game of cat and mouse with hackers through emails or online chats, said Jerry Bessette, senior vice president of the consulting firm’s cyber incident response program. The goal is to glean insights into what data hackers may have stolen, stalling for time as victims probe the impact and try to restore their systems using backups.

“Time is money on both sides,” said Mr. Bessette, who estimated that upward of 80% of his team’s investigations involve ransomware, up from about half a few years ago.

Making ransom payments doesn’t necessarily mean companies are able to unlock their data. The decryption tool hackers provided to Colonial Pipeline didn’t allow the company to fully restore its systems, the Journal reported.

Such snafus are uncommon for credible hacking groups, said

Charles Carmakal,

chief technology officer of cybersecurity firm

FireEye Inc.’s

consulting arm, Mandiant. But he said his team is sometimes able to extract information from unworkable decryptors and design its own.

FireEye advised Colonial Pipeline after the recent hack but Mr. Carmakal declined to comment on it. The publicly traded firm doesn’t negotiate directly with hackers or make payments, he said.

Arlington, Va.-based GroupSense does facilitate such transactions, working with third-party brokers to convert victims’ cash into cryptocurrency and transferring it to hackers.

Banks increasingly scrutinize transactions reaching seven figures, said Mr. Minder, the company’s chief executive. But many of his clients include small businesses, such as a print shop and florist, negotiating ransoms of tens of thousands of dollars.

“It’s very different from when you get into the room with a large company—they’ve got a whole committee working on this thing,” he said. With small companies, he added, “You’ve got the guy who started the business 10 years ago and could lose it all.”

Write to David Uberti at david.uberti@wsj.com