Mortgage rates have been very low for a very long time. Even though they have risen significantly over the past year as the pandemic has loosened its grip and inflation has picked up, the average rate on a 30-year mortgage has risen from 3.2% to just 4.4% over the past year, according to Freddie Mac. That compares with what had been a historically low average of 4.1% during the decade before the pandemic struck and 6.7% in the decade before the 2008-09 financial crisis.

The persistence of low mortgage rates has reshaped people’s perceptions of what is a reasonable price to pay for a house. Consider: The monthly payment on a 30-year mortgage for $500,000 carrying a 6.7% rate is $3,226, but it is $2,176 at 3.25%, which CoreLogic estimates that as of January was the median rate on outstanding mortgages. The difference helps explain why home prices have been able to rise so much without killing demand. As of December, U.S. home prices were 31% higher than two years earlier, according to the Federal Housing Finance Agency.

The decline in mortgage rates during the pandemic set off a refinancing surge that dramatically lowered payments for many homeowners. A significant increase in mortgage rates would lead to two big problems for the housing market: First, it would make it harder for people hoping to buy their first homes at current prices; and second, it would make it difficult for existing homeowners now paying low rates to move because moving would mean giving up a lower interest rate for a higher one. Some will keep their homes much longer than planned, denting businesses that rely on housing turnover. The faster rates go up, the more it will hurt.

Economic conditions are already less than ideal for home buying. The Fannie Mae Home Purchase Sentiment Index fell in January to its lowest level since May 2020. Mortgage rates aren’t the only factor at play right now in housing, though. Many members of the large millennial generation were slow to step into homeownership, in part as a result of the setbacks they experienced following the 2008-09 financial crisis.

Despite the high and increasing financial hurdles, they are finally taking the plunge. The pandemic has reshaped attitudes about owning a home, while high savings and a strong job market are giving people the wherewithal to buy. It isn’t as if the housing market hasn’t weathered a significant increase in rates before. In the 1970s mortgage rates doubled. And yet, over the decade ended in 1979, the number of total homes rose by 25%—then the result of a boomer push for homeownership.

But with higher rates, the prices people are willing to pay for homes could come under pressure. While it is rare for prices to fall nationally—the housing bust that helped precipitate the 2008-2009 financial crisis was a rare exception—there have been periods when prices haven’t kept up with inflation, which in effect made homes more affordable over time.

The search for affordability could create shifts in what types of homes people buy and where they buy them. Some of these shifts are already under way. Existing-home sales in February were down 8.3% in the West from a year earlier, according to the National Association of Realtors, but up 3% in the South. where the median home price was nearly $200,000 less. Demand for homes in the more-affordable exurbs was picking up even before the pandemic hit and has continued to swell given more-flexible work arrangements.

The low number of available homes for sale—in February there were only enough existing homes on the market for 1.7 months of sales compared with an already-low 3.1 months two years earlier—should continue to provide a good backdrop for home builders. But they might benefit from focusing more of their efforts on building smaller homes in more-affordable areas—especially if more people who already own their homes are forced to ditch the low mortgage rates they have locked in. After decades of growth, the average square footage of new single-family homes sold in the U.S. started easing in 2015, according to the Commerce Department.

Higher interest rates might not lead people to give up on their dreams of owning a home, but they could lower expectations on what kind of home that is.

Write to Justin Lahart at justin.lahart@wsj.com and Laura Forman at laura.forman@wsj.com