To mitigate these risks, there are things baby boomers and others can do now to prepare for any problems. In addition, big do-it-yourself investing and trading venues like Vanguard Group, Fidelity Investments and

Charles Schwab Corp.

SCHW -1.31%

are strengthening some of the ways they detect possible signs of decline. Among other things, all three firms check for clients’ difficulty navigating security protocols or need for frequent password resets. In such cases, a designated family member might be informed.

Vanguard also checks client-call recordings for keywords—such as “confused” and “dementia”—that might signal trouble.

Big money

The dollars at stake are big.

Boomers, who are now between the ages of 57 and 75, own more than half the estimated $50 trillion in total U.S. household financial assets, according to Cerulli Associates. Of that $26 trillion, roughly a quarter, or $6.5 trillion, is invested on a do-it-yourself basis, Cerulli analyst

Scott Smith

says. A big majority of that DIY total resides at Vanguard, Fidelity and Schwab.

In the next 10 years, nearly half of the oldest boomers face the likelihood of some cognitive impairment. The rates for mild cognitive decline and dementia rise from a combined 12% for ages 70 to 74 to 45% for those 80 to 84, according to a 2017 report by the Center for Retirement Research at Boston College. Even a mild decline “can rapidly erode financial capacity,” the center said.

The challenging math of asset allocation and withdrawal rates from 401(k)s and individual retirement accounts creates “an enormous problem that we are only vaguely aware of,” says

David Laibson,

an economics professor at Harvard University who co-wrote a study that found financial skills peak at age 53. Without mechanisms in place to delegate such decisions, investors with cognitive decline may “hold on to the reins ever more tightly and steer the horse over the cliff.”

Going solo

Do-it-yourself boomers may be more vulnerable in some ways because they’re calling the shots solo, without help from wealth advisers. So if they go off the rails, no one else may know. “That’s the danger with do-it-yourself investors—they may be overconfident,” says

Michael Finke,

a professor of wealth management at the American College of Financial Services.

Officials at do-it-yourself venues say they have ways, albeit limited, to spot signs of decline. Vanguard, for one, has tripled the head count of its 14-member investor-protection group over the past five years. Officials at Vanguard say 70% of instances where cognitive decline is flagged occur without any sign of fraud or elder abuse.

For those investors, the risk might be veering away from some “set-it-and-forget-it” stock-and-bond allocation into, say, risky margin debt. Or it may be that they have a stack of unpaid bills that they lose track of and somebody has to intervene to sort things out. Or it may be just that their finances are a chaotic mess.

Meredith Stoddard,

Fidelity’s vice president of life events planning, says she knew of one investor who died holding “56 accounts at different firms.”

The Vanguard do-it-yourselfers tend to follow the low-cost index-fund approach of the group’s late founder, John Bogle. Many of Fidelity’s wound up managing their workplace 401(k) portfolios in retirement. And Schwab’s are often the kind of single-stock traders who emerged in the dot-com market of the late 1990s.

Unfortunately, there’s no one-size-fits-all plan to deal with cognitive decline. People’s family situations and estate size and complexity vary. Some people may distrust their own children, complicating a power-of-attorney designation.

Since February 2018, brokerage firms have been required to ask customers to designate a “trusted contact” who can be notified of possible problems. But a survey released last October indicated that less than 25% of firms’ clients have provided a name. Another rule adopted then gives firms greater power to step in and temporarily halt disbursements when fraud is suspected.

A ‘dark topic’

Baby boomers themselves also can take steps to prepare.

First, though, they have to be willing to acknowledge the potential risk. A 2014 survey co-sponsored by the Merrill Lynch unit of

Bank of America Corp.

BAC -0.35%

found that fears about dementia outweighed all other possible illnesses combined. “This is a dark topic. Most people are scared of it,” says

Surya Kolluri,

head of thought leadership for Bank of America’s retirement group.

One key element of a plan is identifying a person or service provider who can help manage one’s financial affairs, preferably under the kind of legal authority embodied in a power-of-attorney or trust. Another is collecting for that person—either in a binder or an internet vault—a list of goals and all the relevant financial account numbers and passwords, as well as regular monthly bills and important records.

“Having a plan and picking someone who understands your objectives is really important,” says

Anna Rappaport,

a retirement-security advocate and former Mercer consultant. “The do-it-yourself person needs to have help in a transition, and that might be very difficult to do. People are often totally in denial and don’t want to do anything about it.”

Subtraction test

Baby boomers can also get clues to their cognition levels. There are simple tests, such as starting with 100 and subtracting by 7 repeatedly. Other more-formal tests include the Short Test of Mental Status, the Montreal Cognitive Assessment or the Mini-Mental State Examination. (Sample Mini-Mental question: What is today’s date?) While meant to be administered by doctors, they can also be found online.

Some apps can even detect possible cognitive issues and alert designated individuals, such as EverSafe, which tracks payment patterns for signs of trouble, offered by Fidelity and

Raymond James Financial Inc.

RJF -1.33%

An app called SyncUP Drive, available through wireless carrier

T-Mobile,

measures changes in driving patterns, another possible sign of decline.

This year AARP, the over-50 group, is sponsoring a “Thinking Ahead Roadmap” with a how-to summary for shifting oversight of finances to a “financial advocate” and alternate—with power of attorney—who may be family members, outside professionals such as accountants, or both. Just starting the conversation “can feel like the hardest step,” says lead author

Marti DeLiema

of the University of Minnesota.

Encouragingly, adults with Alzheimer’s, the most common form of dementia, generally become more passive and exercise less initiative as their illness progresses, reducing the chance of investing disasters, according to

Daniel Marson,

former director of the Alzheimer’s Disease Center at University of Alabama at Birmingham. However, Dr. Marson added, they remain vulnerable to scams and exploitation.

Allan Roth,

a financial planner who is treasurer of the John C. Bogle Center for Financial Literacy, recommends that boomers try to simplify their finances before possible decline sets in. They can tell their advocates to shift assets into a low-cost robo-advisory service, or a single fund with a preset mix of stocks and bonds, if tax consequences allow. Mr. Bogle himself recommended shifting from 60% stocks to 50% later in retirement. Or they can suggest the advocate choose a financial planner or registered investment adviser for money-management help.

Mr. Smith, a former financial reporter for The Wall Street Journal, is a writer in New York. He can be reached at reports@wsj.com.