The Authors Guild had sued Google in federal court in Manhattan 2005, claiming the company was not making "fair use" of copyright material by offering searchable snippets of works in its online library.
Among the plaintiffs was former New York Yankees pitcher Jim Bouton, author of the best-seller "Ball Four."
Google already has scanned more than 20 million books, most of them out-of-print, for the project. It includes the collections of the New York Public Library, Library of Congress and several major universities.
The guild was seeking $750 for each copyrighted book that was copied -- damages that Google estimated could climb to more than $3 billion.
In a statement, Google said it was pleased with the decision.
"This has been a long road and we are absolutely delighted with today's judgment," the company said. "As we have long said, Google Books is in compliance with copyright law and acts like a card catalog for the digital age giving users the ability to find books to buy or borrow."
There was no immediate response to phone messages left with the Authors Guild.
In his ruling, Circuit Judge Denny Chin found that Google's project would not "supersede or supplant" books because it's not meant to be used for reading them.
Google's project "does not engage in the direct commercialization of copyrighted works," wrote Chin, who began hearing the case when he was a district judge. "Google does not sell its scans, and the scans do not replace the books."
The judge said that Google Books will benefit authors by increasing book sales and become a valuable research tool for students and researchers who need to find their works.
"In my view, Google Books provides significant public benefits," the judge wrote. "It advances the progress of the arts and sciences while maintaining respectful consideration for the rights of authors and other creative individuals, and without adversely impacting the rights of copyright holders."
In 2011, Chin rejected a $125 million settlement between Google and lawyers for authors and publishers that was opposed by the search engine's rivals, consumer watchdogs, academic experts, literary agents and even foreign governments.
At the time, the judge said creation of a universal library would benefit many but would "simply go too far."
MORE HBJ HEADLINES
Watchdog: Too few air traffic controllers where needed most A $32B tally, but Boeing's 787 costs don't bother Wall Street Czech airline to buy 16 Boeing 737 Max jets Lockheed Martin separating unit, combining it with Leidos Apple forecasts rare sales drop Obama administration loosens Cuba embargo with new measures