The IRS, calling the Brockman offshore entities involved “shams,” claims in the court filings that most of the transferred funds were dividends on which Mr. Brockman should have paid taxes.

Mr. Smith isn’t a party to the court dispute, but the documents show that he was involved in the planning of the transaction and that a Cayman Islands entity named Vista Equity Fund III LP facilitated the transaction. He personally signed the deal-closing documents on behalf of that entity, the documents show.

An attorney for Mr. Smith,

Emily Hughes

of Kirkland & Ellis LLP, said that the government “never has alleged that Robert Smith engaged in any wrongdoing in connection with the transaction, nor that Mr. Smith knew that Mr. Brockman structured the Hotrod transaction for the purpose of illegally avoiding taxes.” She said Mr. Smith relied on the representations of the other parties involved, as well as the opinion of a nationally recognized law firm, that the transaction wasn’t taxable.

Mr. Brockman, the owner of a large automotive-software firm, was indicted in October 2020 on charges that he used a web of offshore entities to evade taxes on what the Justice Department said was about $2 billion in income, most of it derived from investments in Vista funds. Prosecutors have called it the biggest tax-fraud case brought against an individual in U.S. history.

The indictment said Mr. Brockman “directed and caused” a transaction code-named Hotrod as part of his alleged criminal tax-evasion scheme, but it didn’t go into details about the transaction.

Mr. Brockman, 80 years old, has denied the allegations and has denied owing any taxes, including on the Project Hotrod transaction. He also has claimed to be unable to stand trial due to advancing dementia. Prosecutors claim he is exaggerating a mental decline. Both sides are awaiting a ruling from a federal judge on Mr. Brockman’s competency.

Mr. Smith, who was also a target of the government’s probe, reached a nonprosecution agreement around the time of Mr. Brockman’s indictment. He admitted to engaging in an illegal scheme to evade taxes on more than $200 million in income from Vista’s first fund—in which Mr. Brockman was the sole investor—and agreed to pay $139 million in back taxes and penalties. Mr. Smith also agreed to cooperate against Mr. Brockman.

Mr. Smith has told Vista investors he regretted his conduct, but he said it was purely a personal tax matter and that the Justice Department had “never claimed that Vista or any Vista funds were involved or under investigation.”

Vista, based in Austin, Texas, specializes in buyouts of business-software companies. It was founded in 2000 and has grown to manage more than $86 billion in assets. In January, Vista announced a roughly $13 billion deal to buy

Citrix Systems Inc.,

one of the biggest recent software buyouts. The firm’s success has made Mr. Smith the richest Black person in the U.S., with a wealth Forbes pegs at $6.7 billion.

Vista is in the midst of trying to raise what would be its biggest-ever fund, a vehicle that could total as much as $24 billion. As part of that effort, the firm has been emphasizing to prospective investors that its bench of investment talent extends beyond Mr. Smith, according to people familiar with the conversations.

Mr. Smith has tried to put the tax settlement behind him. He has appeared on television to offer financial advice to young people and has tried to focus attention on his many charitable initiatives, including one aimed at reducing debt burdens for students at historically Black colleges and universities.

The IRS filed the documents showing the intricacies of the Hotrod transaction in late January in federal court, as part of a separate civil case in which Mr. Brockman has sued the IRS over the agency’s early efforts to collect $1.4 billion in tax, penalties and interest it claims he owes.

In the civil case, the IRS pegged at $2.7 billion the amount of income on which Mr. Brockman failed to pay taxes—more than the $2 billion cited by the Justice Department when Mr. Brockman was indicted. The Project Hotrod income accounts for most of the difference, the IRS documents suggest.

In the case, the IRS filed hundreds of pages of supporting documents, including materials seized from an encrypted server kept by Mr. Brockman. The documents include a 2002 letter from Mr. Brockman with an attachment outlining the proposed Project Hotrod transaction.

It called for a non-U.S. corporation managed by Mr. Smith to form an offshore entity called Vista Equity Fund III, according to the documents.

The fund would use money raised from a “private offshore mutual fund” to help buy majority control of Mr. Brockman’s U.S. holding company from his main Bermuda trust, using the change in control to transfer more than $600 million to Mr. Brockman’s offshore structure without incurring U.S. withholding taxes, according to the documents.

But the IRS contends that Mr. Brockman secretly controlled the offshore mutual fund, which was the only outside investor in Vista Equity Fund III, making the whole arrangement a sham.

When the transaction was completed in July 2004, Mr. Smith signed deal documents as Vista Equity Fund III’s “manager,” and those documents show that any correspondence for that entity was to be directed to him at Vista’s U.S. headquarters. The transaction was reversed in 2006, with Vista Equity Fund III selling its shares back to the Brockman offshore entity that originally owned them and receiving $32 million more than it initially invested, according to the documents filed by the IRS.

Several years after the deal, Mr. Brockman went to a Washington tax lawyer’s office to personally shred documents related to Project Hotrod, deeming them “super sensitive,” according to an email he sent associates that the IRS included in the recent filings.

Vista’s role in the deal “is very unusual,” said

Gregg Polsky,

a tax-law professor at the University of Georgia who examined the court filings for The Wall Street Journal. “You wouldn’t normally think of a private-equity fund manager like Robert Smith participating in something like this.”

Although Vista Equity Fund III sounds like a typical private-equity fund, it wasn’t. Mr. Smith’s lawyer and a Vista spokesman said the entity was set up solely for this transaction and wasn’t formed or controlled by Vista. They didn’t say who formed Vista Equity Fund III, but the IRS alleges the entity’s general partner was controlled by Mr. Brockman.

“Neither Vista nor Mr. Smith had any ownership or financial interest” in the fund, and didn’t receive profits on the Hotrod deal, said Ms. Hughes, the lawyer for Mr. Smith.

Cayman Islands records reviewed by the Journal show that Vista Equity Fund III was formed in October 2003 and dissolved in February 2009.

It is a different entity from the similar-sounding $1.3 billion Vista Equity Partners Fund III LP, a buyout fund that the private-equity firm marketed to outside investors and finished raising in 2007. Mr. Brockman committed $100 million to that vehicle, alongside other institutional investors, the documents show.

Generally, fund managers disclose prior deals and funds to investors in subsequent funds, so they can gauge past performance. The Vista spokesman said it didn’t disclose the existence of Vista Equity Fund III to its investors because the entity was involved in a “single recap” transaction directed by Mr. Brockman, and thus wasn’t relevant to its investing track record.

Write to Mark Maremont at mark.maremont@wsj.com and Miriam Gottfried at Miriam.Gottfried@wsj.com