"You don't need huge assortments to shop anymore. Never did you need a huge assortment," the retail legend said on "Mad Money." "Focus on the best."
Drexler, who left J. Crew in January, is now chairman of upstart clothing retailer Alex Mill, which was founded by his son, Alex Drexler. The company's creative director, Somsack Sikhounmuong, worked for 16 years at J. Crew and Madewell.
Alex Mill was founded in 2012 but relaunched earlier this year, with an emphasis on classic wardrobe staples. It sells mostly online. It currently has one physical store, in New York City.
"You can't leave the store or the website not looking good in a sense," Mickey Drexler said. "It's always been something I've felt strongly about. Style, taste, value and quality. I've repeated those words for the 40 years I've been running companies."
First, he said malls naturally lead to price markups.
"When there are two profits, two markups instead of one, it already inflates the perceived value of goods because customers know," said Drexler, who helped turn around Gap in the 1990s.
He said malls also face declining traffic due to cell phones because "if you take your device and go online, I don't know if half the goods are available at a better price by just punching a few buttons."
The third issue comes down to creativity, Drexler said.
"I think, really importantly, the lack of passion for great product and creativity is also a very important reason the malls are not doing as well. There is not enough excitement to fill them up," he said. "They're not doing enough to make them creative, compelling places you want to hang out in with the exception of some."