Martin broke the scaphoid bone of his left hand during the first stage on July 1. He stayed in the race, aiming for a possible win in the first long time trial.
But the powerful German had a flat in Monday's 25.8-mile time trial to Besancon and finished 12th.
"It's difficult for me to step out of the Tour and leave my teammates here to keep on fighting," Martin said. "But it's the right thing to do now, and I know that even the guys will understand. I will go home and I will try to recover as much as I can."
Everything went wrong for Martin from the start of the Tour. Before breaking his hand, he had a punctured tire in the opening prologue that dashed his hope of claiming the yellow jersey.
His main objective this season is winning the time trial at the London Games on Aug. 1 to add to the gold medal he claimed at the world championships last year.
Martin, who spent a lot of time at the back of the peloton since his injury, said he did not want to risk aggravating it when the race hit the Alps.
"I don't want things to get even worse going into the mountains," Martin said. "I cannot imagine sitting in the back of the peloton suffering every day a little bit more as I did during this week. I really tried, but now it's time to think about my complete recovery. I have still three weeks before the Olympics."
Omega Pharma Quick Step doctor Helge Riepenhof said Martin had to withdraw from the Tour to maximize his chances of performing well in London.
"From a medical point of view, it is now time to give the scaphoid fracture a chance to heal," he said in a statement. "We also have to avoid further issues to his body because of being limited in his movement while wearing a brace. This is the only possible decision to be able to perform at the London Olympics without damaging structures such as the lower back or knees."
Martin was expected to fly home later Tuesday.
The Tour peloton is enjoying a rest day before Wednesday's 10th stage from Macon to Bellegarde-sur-Valserine, the first Alpine leg featuring the grueling ascent of the Col du Grand Colombier.
More Sports Headlines
Beat goes on: Red Sox rout Mariners 22-10 Day has 2-shot lead over Spieth going into final round of PGA AquaSox fall 4-3 to Emeralds in 12 innings. Mayhew wins again at Evergreen Speedway Ichiro unofficially passes Cobb with 4,193rd career hit Chargers announce 4-year extension for quarterback Rivers Serena falls to 18-year-old in Rogers Cup semis Djokovic reaches Montreal final
Our new comment system is not supported in IE 7. Please upgrade your browser here.