Those threats from the court-appointed monitor, James Stanton, have included saying he would leverage his friendship with the judge overseeing the case, the people said. After previously certifying Shenzhen-based ZTE’s compliance with the settlement agreement on an annual basis, Mr. Stanton in June pushed for the extension, saying he had evidence that the company violated the terms of its probation, without providing further details, they added.

Mr. Stanton has told people involved in the monitorship that he is “close personal friends” with U.S. District Judge

Ed Kinkeade,

the judge who appointed him, and that together they would do “what’s needed” to make sure ZTE complied with the Justice Department’s settlement. Company lawyers and Justice Department officials viewed that language as inappropriate and potentially unethical, people familiar with the matter said.

Mr. Stanton said there were “extensive inaccuracies” in The Wall Street Journal’s reporting and declined to comment further, saying the matter was under seal. Judge Kinkeade said, “No one speaks for me but me,” declining to comment further. A Justice Department spokeswoman declined to comment. A spokeswoman for ZTE didn’t respond to requests for comment.

ZTE has argued that there is no legal basis for extending Mr. Stanton’s monitorship past the current probationary period, and prosecutors have chosen not to dispute the company’s position, the people familiar with the matter said. Such a decision would normally fall to prosecutors, with the company granted the right to respond.

Mr. Stanton’s firm has made tens of millions of dollars a year from the monitorship, the people said, and his bills have risen in recent months as the spat over the term has escalated.

The dispute puts the Biden administration in the delicate position of possibly siding with a Chinese company that the U.S. charged in 2017 with illegally exporting sensitive U.S. technologies to Iran, while it also looks to maintain a tough U.S. government line against Chinese telecom firms over potential espionage risks.

U.S. officials have long alleged that Chinese telecom providers, including ZTE and its larger rival, Huawei Technologies Co., pose risks to U.S. national security through their close relationships with the Chinese government and potential backdoor access they can provide to their telecom networks.

Both ZTE and Huawei have said they don’t use their gear to spy, but the Chinese companies have taken contrasting approaches to U.S. regulatory interventions. While ZTE has reached settlements and submitted to oversight to maintain its ability to export devices that rely on U.S. technology, Huawei has aggressively fought sanctions and charges by the Justice Department, arguing they are politically motivated, and remains subject to far-reaching bans on key components.

ZTE still lags behind Huawei competitively despite its more conciliatory approach. ZTE has long trailed Huawei, its main Chinese rival in the global market for smartphones and telecommunications gear. Once a top-five vendor of smartphones in the U.S., ZTE is now a minor seller of handsets globally.

Mr. Stanton’s monitorship stems from a 2017 $900 million settlement under which it admitted to orchestrating a six-year conspiracy to acquire U.S. technology, send it to Iran and mask its involvement through a network of front companies.

Although most Justice Department settlements with companies don’t include a monitor, prosecutors impose the extra scrutiny in some cases to ensure compliance with the terms of the settlement. Such monitorships can be costly, and, once agreed to, companies have little control over the activities of the individual chosen to oversee them.

In most cases, Justice Department policy requires a monitor to be chosen in consultation with the company involved, with the company selecting three candidates and prosecutors picking the one they deem the most qualified.

In the ZTE case, Judge Kinkeade in 2017 took the unusual step of rewriting the plea agreement to allow him to appoint and oversee Mr. Stanton, who didn’t have prior experience in sanctions law. Mr. Stanton, who described the judge as his mentor in the dedication of a book he wrote, had previously been appointed by Judge Kinkeade as a special master in a medical-device product-liability case.

The former Justice Department official who oversaw ZTE’s plea agreement, David Laufman, sharply criticized that unorthodox arrangement.

“This train wreck is a long time coming.” he said Friday after the initial publication of this article. “While it was entirely appropriate for ZTE to submit to a rigorous monitorship, it’s also essential that the process for appointing monitors be above reproach and that monitorships are conducted in a responsible manner with appropriate oversight and accountability.” The Justice Department didn’t immediately respond to a request for comment.

In 2018, the Trump administration accused ZTE of violating the terms of its 2017 deal, and as punishment banned U.S. companies from selling to ZTE. The Commerce Department subsequently struck a new deal with ZTE that required the firm to pay an additional fine of $1 billion, replace its board of directors and senior leadership, and fund another team of lawyers to separately monitor the company for 10 years under the aegis of the Commerce Department.

Soon after that, Judge Kinkeade extended by two years the term of Mr. Stanton, the criminal monitor, which had been set to expire in 2020.

After ZTE earlier this year refused to voluntarily extend his monitorship, Mr. Stanton said the company wouldn’t be able to “unscramble the egg” and demanded that ZTE turn over more than 150 sets of documents and make dozens of employees available for depositions within 30 days, the people familiar with the matter said. The document and deposition requests sparked a mad scramble at three different law firms representing ZTE to meet Mr. Stanton’s demands, the people said.

Federal prosecutors are continuing to pursue two different investigations involving ZTE, according to people familiar with the matter, a foreign-bribery inquiry and a visa-fraud investigation, both of which the company is cooperating in.

Mr. Stanton has insisted on investigating those issues himself, even though they aren’t directly related to export issues and the Justice Department is already investigating them, the people familiar with the matter said.

The Biden administration has continued to ratchet up pressure on Beijing’s 5G ambitions overseas in recent months, offering financial incentives and other enticements to countries willing to shun equipment from Huawei and ZTE, as they build next-generation cellular networks.

It has also continued to pursue a criminal case against Huawei, accusing it of violating U.S. sanctions on Iran, stealing trade secrets, and other crimes, after it reached a deferred prosecution agreement with Huawei’s chief financial officer last month.

The dispute over ZTE’s monitorship could make it more difficult for the Justice Department to impose a monitor in Huawei’s case, or those involving other Chinese companies, former prosecutors said.

“If Chinese companies are going to do business in the U.S., they need to be brought along from a compliance perspective,” said Ryan Fayhee, a former national-security prosecutor who is at the law firm Hughes Hubbard & Reed. “Because of ZTE’s experience, it’s going to be much, much more difficult to resolve cases that really do merit a monitor.”

—Byron Tau contributed to this article.

Write to Aruna Viswanatha at Aruna.Viswanatha@wsj.com and Dylan Tokar at dylan.tokar@wsj.com