OVERVIEW AND CONCLUSIONS

Over a two-year period, the Getty Information Institute (formerly Getty Art History information Program) sponsored and carried out a major study of end-user online searching by humanities scholars. Complete logs of the searches and output were captured, and the twenty-seven scholars involved were interviewed in depth. An overview of the study and its results is presented, with particular emphasis on matters of interest to academic librarians. Implications are drawn for academic library reference service and collection development, as well as for cataloging in the online and digital environment.

The results of the study have appeared in a series of five articles ^1-5 to date. It is the purpose of this article to 1) provide an overview of the entire project,

2) review major findings of the study, and 3) draw implications of the study research for the practice of academic librarianship in reference and online searching services, collection development, and cataloging.

Marilyn Schmitt, Deborah Wilde, and Susan Siegfried, of the Getty Information Institute, who originated and carried out much of the study, brought me into the project to design the second year’s interview schedule and to analyze this immense body of data. (Wilde and Siegfried are no longer with the Institute. Vanessa Birdsey, Nancy Bryan, Brian Sullivan, Jeanette Clough, and Katherine Smith also assisted in various parts of the study.)

In the space of this necessarily brief article, it is feasible neither to recapitulate all the results nor, especially, to present all the supporting data that appeared in five lengthy articles elsewhere. The purpose here is to extract the information of use to the busy academic librarian; the reader interested in more detail will be directed to the appropriate article(s). As can be seen in the notes, the articles are numbered Reports #1-5 to to make it easier to distinguish and locate them; they will also be referred to by number here as well.

Succeeding sections describe the study methodology, state the content areas of each of the five prior reports, extract the principal results from the five prior articles, and draw implications of these results for academic library service to humanities researchers.

Literature reviews are provided in the various earlier reports, especially #1-3. The reader interested in learning more in general about humanities information seeking is directed to the recent review article by Watson-Boone,⁶ the earlier extensive review by Stone,⁷ and research by Wiberley & Jones.^8-9

Methodology

Visiting Scholars at the Getty Research Institute for the History of Art and the Humanities¹⁰ are invited for a one-year stay, and come from all over the world. To do research at the Research Institute is a prestigious opportunity, and the Institute draws a range of qualified researchers, from people finishing their doctorates to internationally renowned senior scholars. In return for the opportunity to do unlimited free (i.e., subsidized by the Getty) online searching as end users on about 60 humanities and social sciences databases, participating scholars agreed to be interviewed and to have their search logs captured for study. During 1988-89 and 1989-90, all Visiting Scholars at the Research Institute were invited to participate.

Over the two years, about two-thirds of them agreed to participate, as well as five spouses (scholars in their own right), for a total of twenty-seven participants, seventeen male and ten female. (One of the first-year scholars stayed a second year; see Report #2 for information on where her second-year data are or are not included in various tallies.) Scholars came from all over Europe and the United States. Eleven were native English speakers; English language proficiency of the non-native speakers ranged from adequate to excellent. Research interests of the scholars ranged across the arts and humanities, with some working in the social sciences.

All took a one-day training session in DIALOG searching with a DIALOG trainer. The scholars also had available some other opportunities for search assistance, which they utilized, but, overall, the study design encouraged scholars to do their own searching and further self-education once the training was over. See Report #2 for more details on general study design.

Scholars recorded their own natural language descriptions of their queries at the beginning of their searches. A specially written computer program captured these statements, as well as the entire search, including printout results. In other words, full searches and search results were printed out locally, rather than at DIALOG; the program kept a record of 1) the user’s natural language query statement (meaningless to DIALOG), 2) all DIALOG search statements by the end user, as well as 3) all printout retrieval results.

Terminology used in natural language statements and search formulations was analyzed in detail. Methodological issues associated with terminology analysis are discussed in Report #1. An elaborate set of about one hundred codes was developed to analyze the various aspects of DIALOG searching that was done by the scholars, including use of commands and use of Boolean logic. The codes are presented in Report #2. Scholars were also interviewed in depth. See Report #3 for interview schedules used and methodological issues involved in the interviewing.

Terminology used in the natural language statements and in search formulations was analyzed in detail; these data form the bulk of what is reported on in Report #1. The character of the vocabulary, and the implications for cataloging and online searching, were surprising in some ways. Cataloging is now in flux as we enter the era of digital resources. We may be called upon to make substantial changes in descriptive and subject cataloging for this new age. Results in Report #1 may be of interest to catalogers designing for new approaches to subject cataloging, particularly.

Analysis of length and character of the online searches--including use of search commands and Boolean logic--forms the heart of Report #2. Search records were also sampled through time to determine if the scholars demonstrated a learning curve with experience. Librarians engaged in online searching and bibliographic instruction may find the results in Report #2 revealing.

Results of the interviews with scholars were presented in Report #3. Scholars were interviewed on their experiences with online searching, the role the searching took in their broader research activities, their reactions to DIALOG, and their attitudes about the future of end-user online searching. Reference librarians may find Report #3 of particular interest.

As one who has taught science and engineering literature and has some sense of the culture and values of the sciences, I was struck by the distinctive characteristics of the humanities’ culture, values, and expectations, as against those of the sciences, as they appeared in our study.

I comment on these differences in Reports #1, 4, and 5, in particular. Much of the database world has been developed on the science model. In Report #4, I drew implications from the study for the design of databases and other information resources for humanities scholars. Librarians may find the points in Report #4 of value for their future collection development in reference departments or main collections.

Report #5 addressed the possible impact of features of online searching in the humanities on the (characteristically science-based) theory of information science, especially information retrieval theory and Bradford’s Law.

In this section, study results of interest to academic librarians will be extracted from the various reports and discussed. Points made are not necessarily associated with any one prior report.

•The distribution of amount of end-user online searching by the scholars falls out into a familiar pattern of a few using it a lot, and most using it little.

It has commonly been found that where end users have the opportunity to learn online searching and search on their own, some few will take it up with enthusiasm, while, in a sharply descending curve of use, many will use it little or not at all after training.¹¹ This pattern emerged with the Getty scholars. All who participated were required to attend a one-day training session for DIALOG. Yet, of the twenty-seven scholars, only five spent a total of seven or more hours (each) of actual online searching time over their year’s stay after their training session, i.e., only five even searched about as long as or longer than their training session lasted. Five others did no further searching on their own at all after their training session, and most of the rest searched less than two hours.

With regard to their background use of computers, the scholars had had vanishingly little prior experience with online searching, but many had used online catalogs before, and most had begun using personal computers for writing.

•Positive reactions to online searching dealt mostly with the power of DIALOG to cover many topics or bodies of literature; negative comments dealt mostly with difficulty with the command language and lack of desired resources available online.

Scholars liked Dialindex, DIALOG’s database of terms covering all databases at once, and OneSearch, which allowed scholars to search several databases at once. One said "I feel like an explorer....I enjoy it." Another called online searching "Las Vegas for the intellectual," referring to the surprises he found in the output. This same scholar became experienced and comfortable enough with searching that he recognized and commented on the power of Boolean logic to search on combinations of two or more topics.

Negative comments expressed the beginner’s hesitancy in using the system, as well as some of the problems that have repeatedly been touched on by critics of the conventional online Boolean-type systems. The scholars had difficulty converting their queries into Boolean logic; they found the DIALOG commands confusing, not natural. Searchers found it "too complicated," "daunting", and "cumbersome," and one compared it unfavorably to UCLA’s online catalog, ORION, which he found easier to understand. Another scholar said, "I resist what strikes me as a kind of industrialization of scholarship."

A separate interview schedule was used for those scholars who had not searched at all during the months after their training session. They said they had little need, and/or did not want to spend the time experimenting with it. Not finding an hour or two for cost-free experimentation with DIALOG over the course of the better part of a year suggests something more than just lack of time or immediate need--perhaps discomfort, or lack of fundamental interest.

Overall, judging both by the scholars’ comments and the character of their searching, most did not become skilled enough in searching to get maximum benefit from online searching. All search statements were coded for "probable errors" and "certain errors." In the case of the latter, something about the search statement would have guaranteed failure; with regard to the former, there might conceivably have been some reason to do what the searcher did, but it did not appear to be a wise way to search. Over the two years, 17.0 percent of all search statements contained one or more probable or certain errors, roughly evenly divided between the two classes of error. This is not too bad a rate, considering that most scholars did not search enough to build up expertise. Further, errors did not seem to be a major inhibiting factor for the searchers. In fact, one of the most active searchers had the highest personal error rate: 32.8 percent of search statements contained one or more errors.

At the same time, the searching done was not very sophisticated either. Search terms used were mostly (62.5 percent) single-word terms, and only 3.1 percent of the "Select" statements contained parentheses, which are a mark of more sophisticated searching. However, use of the "building block" technique was emphasized in the scholars’ training. This technique of entering the elements, or "building blocks," of the query one at a time is an excellent one for beginners to use, and may account for the apparent simplicity of the scholars’ search statements.

Three of the twenty-two scholars who searched subsequent to their training never used a single Boolean operator; another five used only ANDs when they did do Boolean searching. So these eight, a third of the active searchers, were not able to take advantage of the common searcher technique of inputting "hedges" (strings of terms ORed together) of multiple term variants for a single concept, in order to get full coverage of a topic.

The other main body of scholars’ criticisms of online searching concerned the contents of the databases. Available databases often proved not to be what the scholars wanted. They wanted access to more European literature, to earlier literature, and to primary research materials. Most databases cover only literature going back to the early 1970’s, which is often just a small part of the range that a humanities scholar is interested in. Earlier literature is generally not in machine-readable form, so is much more expensive for the vendor to mount. (See Report #4 for some suggested ways to meet both scholar needs and vendor interest in the bottom line.)

There were a number of spontaneous comments to the effect that the scholar searchers were pleased to find something of value at all in the searching --not because they doubted the effectiveness of the retrieval system, but rather because they considered themselves experts on their subjects of interest, and therefore did not expect to come across anything they did not already know.

The standard assumption behind much of information retrieval theory is that the person making a query will be familiar with little or none of the literature found in a bibliographic search. In fact, this assumption is so fundamental that it is hardly ever articulated. People would not be searching if they already had found what they wanted, goes the assumption. That may be reasonably accurate for many non-humanities researchers, but for a variety of reasons having to do with fundamental differences in the very nature of humanities scholarship and scientific research (discussed in depth in Reports #4 and #5), it may be more common for the scholar to know, and know very thoroughly, the literature respecting his or her subject of inquiry, than is the case for the scientist.

Other research on humanities scholars has found that scholars use the indexes that are the basis for bibliographic databases little in print form either, and gather much of their key references through a variety of other channels, such as references in articles, specialized bibliographies, colleagues, etc. ^12-13 One might nonetheless make the straightforward assumption that once scholars had discovered the power and scope of online database searching, they would happily move to it and abandon older, more scattershot methods. This may yet happen, but did not occur during the year the scholars had their free opportunity, with the possible exception of one or two of the most enthusiastic searchers.

Another reason for low usage could be that the scholars had already done their research before arriving, and came only to write during their year at the Getty. This proved not to be the case. The fifteen scholar-participants during the second year were asked about the status of their research projects. It turned out that the fifteen were engaged in thirty-nine different projects among them, at all stages of development. We found no discernible association between stage of project and amount of online searching.

Upon being asked, several of the scholars said they expected that online databases would raise the expectations of thoroughness in scholarly preparation in the field. Nonetheless, at the time of their interview, only two scholars had actually looked into how they might be able to continue searching once they returned home.

Five scholars mentioned the value of online searching to explore interdisciplinary topics or topics in neighboring disciplines. So it would appear that in a very clear-cut way, the use of DIALOG served indeed to benefit the scholars at the margins of their interests. They often found little at the core of their interest area(s), because they had developed extensive bibliographies of materials in those areas over the years, and over many projects and papers. But the power of online searching enabled them to discovered their topic being discussed in another field, and some of the best finds were materials found outside their own discipline.

•Scholars’ search mode preferences varied over five distinct options.

One of the most striking results of the study was the wide range of preferences the scholars expressed for modes of online searching. The following question was asked of the second year’s fifteen visiting scholars: "Do you prefer to do online database searching yourself or would you rather have librarians or research assistants do it for you?"

The interview style was conversational and scholars responded in a variety of ways to this question. Their preferences can be summarized as follows, with "searcher" referring to those who searched by themselves at some point after their training session, and "non-searcher" referring to those who did not search by themselves after their training session:

•Self searching alone: two searchers, one non-searcher

•Self with someone nearby to answer questions: four searchers, zero non-searchers

•Self with someone sitting next to one: two searchers, one non- searcher

•Intermediary searching with scholar sitting there giving feedback: two searchers, one non-searcher

•Intermediary searching without scholar present: one searcher, one non-searcher

•No response: one searcher

This remarkable range of preferences suggests that librarians doing online searching may need to expect to work out a variety of arrangements with their users, rather than assuming that one arrangement will work best.

•The logical, engineering-oriented design of online systems is generally not well matched with the talents of the humanities scholar.

The humanities scholar is a genius at detecting the trend or nuance that is not explicitly expressed in the text being studied, at reading through and between the lines. The design of current online retrieval systems, by contrast, is intended to be explicit, rigorously consistent, and logico-mathematical in design. In analyzing the data, we were struck, again and again, by the "simple" logical and other errors made by these brilliant scholars as they attempted to master DIALOG searching.

One small example is telling: In the DIALOG manual available at the time (this has since been changed), the "N" operator was shown as "nN". The "N" operator is one with which the searcher can ask the system to search for two words in either direct or reverse order. The "nN" model was meant to show that the searcher could put a number before the N operator to indicate how many words could be between the two search words and still retrieve the record. For example, "Child (1N) psychology" would mean that phrases "Child psychology," and "Psychology of children," among others, would both be retrieved by such a formulation. One of the scholars input this operator directly as "nN" in a search, however, without converting the "n" to a number, and consequently retrieved nothing.

Now this usage is essentially algebraic. The small letter can be substituted for by many numbers. In my experience teaching online searching, I have been struck by the ease with which students with a strong mathematics or engineering background pick up Boolean searching, and the difficulty that many, though not all, people with a humanities background have in learning this skill. The humanities scholar might well point out that the engineer is never again required to repeat the poem he memorized in ninth-grade English. Why, then, should the humanities scholar be required to remember ninth-grade algebra?

The humanities scholar will experience online searching differently. Referring to online searching as "the industrialization of scholarship," as mentioned earlier, is the sort of conceptualization of the searching experience not likely to be foremost in the mind of the engineer. Another scholar liked the idea of having her own password, which, she said, "is a kind of intimate relation to the machine." Others were conscious of a sense of being connected in to a network--one scholar commenting positively on that sensation, and one negatively. This sophisticated awareness of social and aesthetic dimensions of this particular human activity of online searching is bound to have some impact on the reactions of scholars to its use.

•The character of humanities search terms varies considerably from that of the sciences. Humanities thesauri should probably be designed on different principles from conventional thesauri, and humanities search interfaces should be designed differently as well.

Stephen Wiberley argued, beginning in 1983,^14-15 that humanities subject terms are mostly not the characteristically vague, hard-to-define terms that we generally assume they are. He provided convincing evidence that terms of importance to humanities scholars in reference books and indexes are often highly precise proper names.

In this study, both the vocabulary in the natural language descriptions of queries given by the scholars, as well as the actual terms used in searching, were analyzed in detail (Report #1). The data confirmed Wiberley’s findings and demonstrated a number of other differences from conventional thesaural vocabulary and search vocabulary as well.

In the natural language statements, the scholars’ descriptions of their queries were expressed in the most natural, native way, not yet converted into search statements. Ninety-one percent, or 150, of the 165 natural language statements indicated a subject search of some kind. These subject queries were the focus of my analysis.

After reviewing the data, these categories of subject terms were developed: 1) works or publications as subject, 2) individuals as subject, 3) geographical term, 4) chronological (date) term, 5) discipline term, 6) other proper term, and 7) other common term. "Discipline term" refers to terms comparable to academic disciplines, such as "history," "philosophy." "Other proper" refers to proper nouns (normally capitalized words) that do not fall into any of the preceding categories, and "other common" referred to any common (uncapitalized) terms that do not fall into the preceding categories.

Data analyzed from these 150 subject statements were then compared to the text of thirty-eight real natural and social science queries that had been gathered for a National Science Foundation-funded study of online searching carried out by Tefko Saracevic and others.¹⁶ For our study, the Saracevic queries were analyzed in the same way as the Getty natural language statements. The data are summarized in Table 1 below. Figures in each row represent the percentage of queries that contained one or more terms of the designated type. (More detail on the methodology of counting these terms is found in Report #1.) Indefinitely many terms and term types can be in a single query.

The dramatic difference in terminological profile between the two bodies of data is immediately evident in the table. All the Saracevic queries contained one or more common terms; only fifty-seven percent of the Getty queries did. Consider a typical Saracevic study query: "the relationship and communication processes between middle-aged children and their parents."¹⁷ All terms in this query are common; there are none of any other type. This is typical of the science queries. Note what small percentages of the queries in the Saracevic sample contain any other type of term.

Now consider one of the Getty queries: "Image of the tree in literature, art, science, of medieval and renaissance Europe." This query contains geographical, chronological, and discipline terms, in addition to the common terms. As noted above, only fifty-seven percent of the Getty statements contained any common terms at all. On the other hand, high rates of all the non-common term types appear in the Getty column, in sharp contrast to the data in the Saracevic column. This general pattern carried over into the actual search terms used by the scholars as well.

Much of the theory of thesaurus design focuses on the development of what are here called "common" terms. Given the profile of the science data, and the historic importance of science databases in online searching, this is understandable and reasonable. But it is also clear that if the needs of humanities scholarship are to be well served, much more attention needs to be given to the design of terminology of the other types as well. (The study results inadvertently confirmed the value of two Getty databases that were already in development at the time of the study, the Union List of Artist Names, and the Thesaurus of Geographic Names.)

Good design of such terminology in the online searching context is not obvious. Consider dates, for example, which would seem to be the most straightforward possible type of search term. However, the producers of the database Historical Abstracts experimented for several years before finding optimal ways of coding and searching on dates to provide good retrieval under a variety of circumstances. (See more extensive discussion in Bates.¹⁸)

The broad categories of subject term found in this study lend themselves well to a faceted approach to index terminology. (The choices that had been made earlier to design the Getty’s Art & Architecture Thesaurus on a faceted basis are strongly confirmed by this study’s data.)

Likewise, the scholars frequently attempted to create search terms that were, in effect, strings of facet elements. Unfortunately, because they did not fully understand the use of Boolean logic and proximity operators, and because current indexing is often not well suited to such strongly faceted subject matter, they frequently produced strings that would not work online, for instance:

"humanities(w)(method? OR methodology)(w)(comparison usa europe)"

Suppose that the searcher could instead have entered elements of this query into a prompt screen with labeled slots for geographical term, date, topic, named individuals, etc. In that case, the system could carry out the Boolean work; the user need only know what he or she knows best--the elements of the query.

The indexing in many current databases does not recognize and build on these facet elements well, thus making it difficult to form a good search statement, even when one is an expert searcher. Many of the humanities queries work like the example above, i.e., they consist of several elements, each one of which is conceptually quite broad ("humanities," "Europe"), but when the elements are all brought together (in effect, ANDed), they form a highly specific query. Yet precisely because of the breadth of many of these terms, indexers are discouraged from using them. Database producers need to recognize that with queries of the distinctive sort found in the humanities, indexing terms and policies may need some adaptation to produce indexing that is optimally effective for online users.

Implications for Library Service

Keeping in mind the relatively small sample upon which these conclusions are drawn, consider the following points regarding academic library services for humanities scholars:

•As online searching has been powerful for discovering work of interest for a scholar in other disciplines, give particular attention to providing interdisciplinary information in online searching for humanities scholars. Use multi-disciplinary databases like DIALOG’s Dialindex and ISI’s citation databases.

•Recognize that humanities searches are often composed of several facet elements. As the indexing for many databases is not optimally designed for queries of this sort, good online database searching in the humanities may actually be harder than that for the sciences, even for the skilled online intermediary, and will almost certainly be difficult for the typical humanities end user.

•Expect a wide array of preferences in humanities users for database searching arrangements--from the scholar searching alone, through various mixes of scholar and intermediary working together, to the intermediary doing the search alone for the scholar.

•Look to acquire reference materials in CD-ROM or other automated forms that show a sensitivity to the unique characteristics of humanities scholars and scholarship, specifically, that provide indexing sensitive to the typical kinds of facets of interest to scholars, that have very user-friendly interfaces, and that contain resources with historical and geographical depth.

1. Marcia J. Bates, Deborah N. Wilde, and Susan Siegfried, "An Analysis of Search Terminology Used by Humanities Scholars: The Getty Online Searching Project Report Number 1," Library Quarterly 63 (Jan. 1993): 1-39.

2. Susan Siegfried, Marcia J. Bates, and Deborah N. Wilde, "A Profile of End-User Searching Behavior by Humanities Scholars: The Getty Online Searching Project Report No. 2," Journal of the American Society for Information Science 44 (June 1993): 273-91.

3. Marcia J. Bates, Deborah N. Wilde, and Susan Siegfried, "Research Practices of Humanities Scholars in an Online Environment: the Getty Online Searching Project Report No. 3," Library and Information Science Research 17 (Winter 1995): 5-40.

4. Marcia J. Bates, "The Design of Databases and Other Information Resources for Humanities Scholars: The Getty Online Searching Project Report No. 4," Online and CDROM Review 18 (Dec. 1994): 331-40.

5. Marcia J. Bates, "Document Familiarity, Relevance, and Bradford’s Law: The Getty Online Searching Project Report No. 5," Information Processing & Management, in press.

6. Rebecca Watson-Boone, "The Information Needs and Habits of Humanities Scholars," RQ 34 (Winter 1994): 203-16.

7. Sue Stone, "Humanities Scholars: Information Needs and Uses," Journal of Documentation 38 (Dec. 1982): 292-312.

8. Stephen E. Wiberley, Jr., and William G. Jones. "Patterns of Information Seeking in the Humanities," College & Research Libraries 50 (Nov. 1989): 638-45.

9. Stephen E. Wiberley, Jr., and William G. Jones. "Humanists Revisited: A Longitudinal Look at the Adoption of Information Technology," College & Research Libraries 55 (Nov. 1994): 499-509.

10. The J. Paul Getty Trust is a private operating foundation. Two of its programs are the Getty Information Institute (formerly the Getty Art History Information Program) and the Getty Research Institute for the History of Art and the Humanities (formerly the Getty Center for the History of Art and the Humanities). These two entities collaborated on this research.

11. Bryan Pfaffenberger, Democratizing Information: Online Databases and the Rise of End-User Searching (Boston: G.K. Hall, 1990): 111-12.

12. Wiberley and Jones, "Patterns of Information Seeking."

13. Susan S. Guest, "The Use of Bibliographical Tools by Humanities Faculty at the State University of New York at Albany," Reference Librarian 18 (Summer 1987): 157-72.

14. Stephen E. Wiberley, Jr., "Subject Access in the Humanities and the Precision of the Humanist’s Vocabulary," Library Quarterly 53 (Oct. 1983): 420-33.

15. Stephen E. Wiberley, Jr., "Names in Space and Time: The Indexing Vocabulary of the Humanities," Library Quarterly 58 (Jan. 1988): 1-28.

16. Tefko Saracevic and Paul Kantor, "A Study of information Seeking and Retrieving. II. Users, Questions, and Effectiveness." Journal of the American Society for Information Science 39 (May 1988): 177-96.

17. Ibid., 195.

18. Marcia J. Bates, "Implications of the Subject Subdivisions Conference: The Shift in Online Catalog Design. In The Future of Subdivisions in the Library of Congress Subject Headings System: Report from the Subject Subdivisions Conference Sponsored by the Library of Congress, May 9-12, 1991, edited by Martha O. Conway. (Washington, D.C.: Library of Congress Cataloging Distribution Service, 1992): 92-98.