Leading up to Berkshire’s annual meeting on Saturday, proxy advisers Glass Lewis & Co. and Institutional Shareholder Services Inc. have recommended that investors withhold their votes for board members.

While many of the complaints aren’t new and none of the shareholder proposals are likely to pass, Berkshire’s lackluster returns in recent years have made it more vulnerable to criticism amid a growing wave of investor interest in corporate sustainability issues.

The shareholder movement to press companies on climate change, social progress and governance continues to gain steam in the U.S., emerging as a key selling point for money managers in their efforts to keep client money.

Under Mr. Buffett’s leadership, the firm boasts 20% compounded annualized gains from 1965 to 2020, outperforming the S&P 500’s 10.2% gains including dividends during the period. Berkshire’s total returns over the past three- and five-year periods were 12% and 14%, respectively, compared with the index’s 19% and 18%.

“Berkshire has gotten a pass in part because of its historically strong financial performance,” said

Simiso Nzima,

head of corporate governance at Calpers.

Berkshire has continued to stress its continued focus on the long game. Mr. Buffett, who is chief executive and chairman of the company, built up a diverse portfolio of mostly U.S. businesses and investments meant to perform over decades, not to compete with a volatile market buoyed by booming tech stocks.

Calpers, the nation’s largest public-pension fund with $444 billion in assets, co-sponsored a shareholder proposal imploring Berkshire to provide more disclosures on climate-related risks and opportunities.

The pension fund is also withholding its votes to re-elect members of the board’s audit and governance committees on grounds of failing to meet shareholder demands over climate-risk disclosures. It said it was concerned that the board lacks new members, doesn’t engage with shareholders and isn’t letting investors vote on executive pay plans.

“If you don’t refresh the board, you don’t have a next generation of directors able to learn from the long-serving directors before they leave the board,” Mr. Nzima said.

Berkshire declined to comment ahead of the company’s Saturday meeting.

Neuberger, a privately held money manager with more than $429 billion in assets, also said it would vote for several shareholder-led proposals related to environmental, social and corporate-governance issues, often abbreviated as ESG.

“One would think that if companies have a responsibility to look out for the environment or deliver good on social issues and governance, that Berkshire might be a leader in these areas,” said

Michelle Giordano,

a Neuberger analyst who follows the company. “But it doesn’t seem like they are.”

Berkshire said in its annual proxy statement that while it agreed companies had a responsibility to manage climate risks, it preferred to let its various operating units commit to their own environmental policies. Mandates from a small corporate office, the company wrote, would infringe upon the autonomy that has helped those businesses thrive under Berkshire’s ownership. Berkshire Hathaway Energy, for instance, already produces a sustainability report.

Calpers has also pledged to support a proposal requiring the company to report its efforts to diversify its staff.

Berkshire said the diversity-report proposal improperly suggests that “there is a standardized technique for each of Berkshire’s more than 60 operating businesses to address diversity, equity and inclusion.”

“It would be unreasonable to ask for uniform, quantitative reporting for the purposes of comparing such dissimilar operations in different geographic locations,” Berkshire wrote.

Glass Lewis and ISS recommended shareholders vote for the ESG proposals and withhold votes for certain directors.

“This year there’s a lot more attention given from mainstream investors on ESG issues,” said

Courteney Keatinge,

a senior director of ESG research at Glass Lewis.

Another factor is at play: Berkshire shares are slowly changing hands.

Mr. Buffett’s longstanding plan to shrink his stake in the company over time has shifted more Berkshire shares to big institutional investors, said

Lawrence Cunningham,

a law professor at George Washington University who has written extensively about the company.

About 70% of Berkshire’s shares are owned by individuals, many of whom are longtime holders loyal to Mr. Buffett, Mr. Cunningham said. And many don’t care whether Berkshire lacks a corporate sustainability report or an investor-relations team at the ready to answer their questions.

“Berkshire’s unusual and valued family of individual shareholders may add to your understanding of our reluctance to court Wall Street analysts and institutional investors,” Mr. Buffett wrote in his most recent letter to shareholders. “We already have the investors we want and don’t think that they, on balance, would be upgraded by replacements.”

The gradual uptick in institutional ownership, though, might already be empowering professional managers to press Berkshire on governance matters. When Mr. Buffett and his estate sell off his remaining shares, it is likely those money managers will hold an even bigger stake in the company, Mr. Cunningham said.

“There will be a dawning of significant leadership and structural change, and these holders are preparing for that battle,” Mr. Cunningham said.

Write to Justin Baer at justin.baer@wsj.com