Zimmer was fired as the company's executive chairman last week. On Monday he submitted a letter resigning from the board.
Zimmer says in the letter that it's clear from his firing that the board is determined to avoid addressing his growing concerns with recent board decisions and the company's direction.
Zimmer, 64, built Men's Wearhouse Inc. from one small Texas store using a cigar box as a cash register to one of North America's largest men's clothing sellers with 1,143 locations.
He also became one of TV's most recognizable pitchmen, with the slogan "You're going to like the way you look. I guarantee it."
Men's Wearhouse fell 62 cents to $35.20 in afternoon trading.
MORE HBJ HEADLINES
Profits plunge at major U.S. oil companies U.S. rig count drops 11 this week to 420, another all-time low There’s rare good news on the retirement front Canadian dollar rises above 80 cents U.S. mark OPEC oil output surges as Iran looks to regain market share U.S. wage increases subdued in first quarter