Paul Marciano, who the activist also wants removed from his position as chief creative officer, has been publicly accused of sexual misconduct by multiple women in recent years. Legion argues that Maurice Marciano is also to blame for turning a blind eye to his brother’s behavior.

Guess said Legion Partners has “resurfaced information related to one of the company’s co-founders, the details of which come from public information, including the company’s own disclosure on these matters.” It added, speaking also on behalf of the brothers: “We have strongly refuted these allegations fully in the past and are contesting them vigorously.”

In the letter, Legion applauds recent steps taken by Guess Chief Executive Carlos Alberini including cost cuts and improvements to its supply chain. It goes on to note that it doesn’t believe the company can reach its potential with the “legal, reputational and moral risk” posed by the Marciano brothers’ continued involvement.

Guess said in its statement that it has also made improvements in the styling and quality of its products with a focus on sustainability, and improved the look of its stores, website and marketing. It said the company is successfully executing its turnaround under the direction of the board and management team and will continue to engage with Legion.

Legion in the letter writes that Paul Marciano has been publicly accused of sexual misconduct by at least 11 women, including four in the past year. It details some of the claims, including those of a former model who accused him of rape in a lawsuit filed in January 2021.

In early 2018, model and actress Kate Upton, who had modeled for the Guess brand, publicly accused Paul Marciano of sexually and emotionally harassing women, including herself. At the time, Mr. Marciano denied the allegations.

Later that year, Paul Marciano resigned as executive chairman and agreed to leave the company in 2019 after an internal investigation determined he exercised “poor judgment” in some situations involving models and photographers. At that time, the company said in a filing it and Mr. Marciano, who didn’t admit wrongdoing, had reached settlements with five people totaling $500,000.

But when it tapped Mr. Alberini as CEO in 2019, Guess said Paul Marciano would remain chief creative officer at the request of the board, which was then chaired by Maurice Marciano.

Legion also noted in the letter that the insurance carrier that underwrites Guess and Paul Marciano sued a little over a month ago to absolve itself of any responsibility for covering claims related to Paul Marciano’s alleged “pattern” of “wrongful acts.”

Guess, established by the brothers in 1981, is credited with helping redefine denim, though it has struggled to find its place in recent years as consumer tastes and shopping habits shift. The company’s shares are roughly where they were 16 years ago. Los Angeles-based Guess has a market value of around $1.4 billion.

Legion, also based in Los Angeles, has roughly $500 million under management. It has teamed up with other activists on campaigns at

Bed Bath & Beyond Inc.

and

Kohl’s Corp.

in the past few years.

Write to Cara Lombardo at cara.lombardo@wsj.com