Zimmer was fired as the company's executive chairman last week. On Monday he submitted a letter resigning from the board.
Zimmer says in the letter that it's clear from his firing that the board is determined to avoid addressing his growing concerns with recent board decisions and the company's direction.
Zimmer, 64, built Men's Wearhouse Inc. from one small Texas store using a cigar box as a cash register to one of North America's largest men's clothing sellers with 1,143 locations.
He also became one of TV's most recognizable pitchmen, with the slogan "You're going to like the way you look. I guarantee it."
Men's Wearhouse fell 62 cents to $35.20 in afternoon trading.
MORE HBJ HEADLINES
Everett's ATS CEO named HBJ Executive of the Year Briefs: BECU, car dealerships raise money for Housing Hope Target faces backlash on stance on transgender bathrooms Driver in fatal duck boat crash in Boston cited 10 times for speeding St. Louis jury awards $55M in Johnson & Johnson cancer suit Corporate earnings keep falling, but there are signs of life