His novel solution: a paper catalog mailed to thousands of homes.

“The print cost plus the postage costs should actually do as well or better than it does to acquire customers online,” Mr. Stecko said. The catalog costs about $86,000 to produce, though he is waiting for more sales to come in before judging its success.

The tech ecosystem that powers ecommerce has simplified the way aspiring merchants set up shop, from web design and inventory management to email marketing and sales-tax collection.

Shopify Inc.,

a provider of such services, said 1.75 million merchants used its platform last year, more than double the number two years earlier. But the lower entry barriers and rising cost of online advertising are making it harder for existing and new sellers to cut through the crowd and find more customers.

Sellers and consultants say that in addition to marketing tactics like search-optimizing web pages and targeting users on social media, brands must also take the more elaborate steps of developing community among customers or an identity that consumers deem authentic. Other online sellers are testing out distinctly analog ways of reaching new customers, like printed catalogs or bricks-and-mortar shops.

“There are certain things that have gotten easier and that makes other things hard,” said

Rick Watson,

chief executive of the ecommerce consulting firm RMW Commerce Consulting Inc. Advertisements aren’t as effective as content—like email newsletters or YouTube videos—created by the brands to entertain or inform, he said.

Chris Young,

who founded the cooking website ChefSteps in 2012 and launched Joule, a sous-vide cooking tool, is building a new kitchen-gear brand. He is adapting lessons learned in his first company, building up an online audience with content before releasing a product for sale.

Mr. Young said the tools available to launch his new company, Combustion Inc., are simpler than the ones he used in 2012 when building ChefSteps, which he later sold. With Combustion, he is developing a wireless thermometer that predicts foods’ cooking and resting time. He has focused on cultivating a mailing list of interested customers by leveraging his reputation in the cooking world and creating content for platforms like YouTube to drum up interest until the product launch. Standing out is still a challenge, he said.

“It’s vastly easier for an individual, without taking a bunch of investor dollars, to build a brand, create a product and sell it to customers,” Mr. Young said. “But the bar has become a lot higher to be unique.”

One of his latest attempts to meet that bar was a video where he chills a steak using liquid nitrogen before deep frying it. He promoted the video to online followers on an email list, and by working with other creators with large audiences to mention his work. With 3,000 views and 150 new sign-ups to his email list in the first week, Mr. Young estimates that the video, which cost around $1,000 to produce, will have been profitable.

New entrants to online selling face steep odds. “From a consumer vantage point, there’s a lot of clutter,” said

Michelle Evans,

digital retail analyst at the research firm Euromonitor. “You have to sort through who best to buy from and what to buy.”

Owners of online shops that have built up their businesses for years say they have an advantage in affinity from existing customers and proven methods for finding more.

Randy Owen,

founder of ThermoWorks Inc., has been selling kitchen thermometers for two decades, and the handheld staple of chefs’ kitchens has garnered approval from publications like America’s Test Kitchen and food personalities like

Alton Brown.

Mr. Owen hasn’t sold on

Amazon.com Inc.

since 2015, citing lack of access to customer data and the fact that Amazon sold similar-looking thermometers at a lower price. Still, sales have since increased to over $75 million a year, including a jump during the pandemic as more people cooked at home, he said.

ThermoWorks has to work constantly to continue ranking high on relevant search terms on

Alphabet Inc.’s

Google and also monitors product listings on other platforms to ensure they aren’t using his brand’s name.

Since people can’t buy ThermoWorks on Amazon, the retail giant sells search ads for the thermometer maker’s product names. Mr. Owen said ThermoWorks’ legal options there are limited. An Amazon spokesman said the site is designed to show, through search results and ads, products that customers can buy.

“If a particular product is not available, we present them with similar products from our vast selection,” the spokesman said.

While ThermoWorks may lose some sales to shoppers buying other thermometers on Amazon, Mr. Owen said, many eventually find his site. “They’ll find a copycat that lasts them a few months before they come to us,” he said.

Other brands are evaluating how to capitalize on new spending patterns resulting from the pandemic.

Parachute Home Inc., a 7-year-old bedding company, has branched out into other home goods, such as towels and loungewear. It has also opened 11 stores, which do bustling sales but also serve as a branding tool, said founder and CEO

Ariel Kaye.

The company maintains a mix of online ad spending and has complemented those ads with a printed catalog for the past four years. Ms. Kaye said that the catalog, sent via a direct marketing agency to several million new households last year, has helped build brand awareness. Recently, Parachute sought to widen its audience with advertisements on streaming video services such as

Walt Disney Co.

’s Hulu.

Ms. Kaye said that her costs to acquire customers have fallen recently since people have spent more sprucing up their homes during the pandemic. “There’s a high-intent customer at home, looking for a refresh,” she said.

Mr. Stecko of 80sTees.com said that his acquisition costs have actually risen, due in part to rising digital advertising costs. “The saying in digital marketing is, whoever can spend the most wins,” he said.

Another annoyance is that, from time to time, his ads on social-media platforms are flagged by automated systems for policy violations. One example: A shirt celebrating wrestler Sgt. Slaughter references a move called the “Cobra Clutch.” Mr. Stecko suspects the algorithms may think he is listing animals for sale, which is prohibited on many platforms.

He hopes his catalog—mailed out last month to 100,000 past customers plus 15,000 potential new ones he purchased from a database—can drum up business. He can’t calculate profitability until he sees how many orders are shipped to homes that received catalogs, though he said he is optimistic, especially after years duking it out in digital advertising.

“It’s one of the few advertising options where a random algorithm issue can’t shut you down completely,” he said.

One side benefit: The catalog—which includes a $7 credit and a poster by a respected comic-book artist—is also for sale on his site, and has become a hot item in its own right.

—For more WSJ Technology analysis, reviews, advice and headlines, sign up for our weekly newsletter.

Write to Paul Ziobro at Paul.Ziobro@wsj.com