Zimmer was fired as the company's executive chairman last week. On Monday he submitted a letter resigning from the board.
Zimmer says in the letter that it's clear from his firing that the board is determined to avoid addressing his growing concerns with recent board decisions and the company's direction.
Zimmer, 64, built Men's Wearhouse Inc. from one small Texas store using a cigar box as a cash register to one of North America's largest men's clothing sellers with 1,143 locations.
He also became one of TV's most recognizable pitchmen, with the slogan "You're going to like the way you look. I guarantee it."
Men's Wearhouse fell 62 cents to $35.20 in afternoon trading.
MORE HBJ HEADLINES
Brexit vote slams world, U.S. financial markets Yellenís Fed to play long game dealing with Brexit fallout House Speaker Ryan proposes simpler tax code Maine governorís wife gets waitressing job Global markets in tailspin after British vote to exit EU Stocks crash as UK vote to quit EU shocks investors