Hannah Noll’s paper for her M.S. in Library Science degree, Where Google Scholar Stands on Art: An Evaluation of Content Coverage in Online Databases (PDF link, 300kb)…

“This [ 2008 ] study evaluates the content coverage of Google Scholar and three commercial databases (Arts & Humanities Citation Index, Bibliography of the History of Art, and Art Full Text/Art Index Retrospective) on the subject of art history. Each database is tested using a bibliography method and evaluated based on Peter Jacso’s scope criteria for online databases. Of the 472 articles tested [ * ] , Google Scholar indexed the smallest number of citations (35%), outshone by the Arts & Humanities Citation Index which covered 73% of the test set. This content evaluation also examines specific aspects of coverage to find that in comparison to the other databases, Google Scholar provides consistent coverage over the time range tested (1975-2008) and considerable access to article abstracts (56%). Google Scholar failed, however, to fully index the most frequently cited art periodical in the test set, Artforum International. Finally, Google Scholar’s total citation count is inflated by a significant percentage (23%) of articles which include duplicate, triplicate or multiple versions of the same record.”

* tested with a set of “article citations authored by a pre-selected set of art historians” via 12 names “culled from the Dictionary of Art Historians“, according to the paper. Authors had to be British or American, and born after 1925.

It’s interesting that Noll rejects keyword searches as a test measure…

“Searching by a compiled list of subject terms did not seem appropriate for testing Google Scholar. Google Scholar lacks a system of controlled vocabulary and search results reflect in many cases a full-text search of the document, whereas traditional databases only search the title and abstract keywords of a record.”

… yet Noll might have easily used intitle:”title of the article” with Google Scholar, to find specific articles. The intitle: search modifier is not mentioned in the paper. Instead Noll used a wider author search, then trawled the results for the target titles, but admits of this method of using Google Scholar that…

“some articles may have been impossible to find by using the author search.”