| Index
Site Visit and Quarterly Reports
send comments or questions to: l-neuma1@uiuc.edu |
Understanding Use in the Real WorldAnn Peterson Bishop
Abstract--This paper considers a number of issues related to measuring and understanding use of innovative electronic document systems. It presents experiences related to the implementation of a digital library comprised of scientific and technical journal articles, and concentrates on one particular barrier to use: the difficulties potential users have in logging in to an electronic document system. I. INTRODUCTIONDuring the course of the Digital Libraries Initiative (DLI) project at the University of Illinois, testbed designers and the social science team have worked to create a digital collection of science and engineering journals that faculty and students would really use. With the deployment of our web-based client, DeLIver, we are now entering the rather frightening territory of measuring the success of all our efforts. Will people use our system? How can we pinpoint barriers to use? This paper considers a number of issues related to measuring and understanding use of innovative electronic document systems:
I will address these issues by describing our recent experiences with implementing DeLIver. These are the kinds of issues that must be dealt with as we try to identify and resolve barriers to the use of electronic documents, especially in new systems whose capabilities are unfamiliar to most users. This paper presents our experiences in an anecdotal fashion and concentrates on one particular barrier to use: the difficulties potential users have in logging in to a digital library.
II. BACKGROUNDThis paper draws on my experiences as head of the Social Science Team for the NSF/DARPA/NASA Digital Library Initiatives (DLI) project at the University of Illinois [see the project homepage at: http://dli.grainger.uiuc.edu/]. The DLI testbed contains the fulltext of recent articles from over 50 scientific and technical journals, primarily in the disciplines of engineering, computer science, and physics. One innovative aspect of our DLI testbed is its capacity, through Standard Generalized Markup Language (SGML) and enhanced search features, to support retrieval of newly foregrounded document components: information in individual parts can be disaggregated from the surrounding textual package and retrieved for use in a way not possible with traditional bibliographic retrieval systems. Through DeLIver, the web interface to the DLI testbed [http://dli.grainger.uiuc.edu/ deliver.htm], researchers at the University of Illinois can search for terms in particular document components (e.g., MIT in the author affiliation, spectrum in a figure caption) to make searching more precise. They can also view certain components that have been extracted from the complete document. Currently, users have the option of viewing the abstracts, figures and tables, bibliographies, and author affiliations before viewing the fulltext of the document. DeLIver also offers several ways for users to link to additional material. They can link to abstracting and indexing service records from INSPEC and Compendex for articles they find in DeLIver, click on a reference in an article they have found and jump directly to the fulltext of that item, and link to articles that cite a document they have retrieved. The University of Illinois DLI Social Science Team has performed formative and summative user studies to improve our testbed design, document use, and develop an understanding of faculty and student work and communication practices in the changing information infrastructure. We have gathered data through:
Our work represents an integrated research program for studying testbed users that combines broad study of use with deep study of social phenomena. Throughout the project, we have tried to integrate our research with the work of the Testbed Development Team, which is the group charged with designing and implementing DeLIver.
III. USABILITY TO USEThe initial trial version of DeLIver was tested in the Summer of 1997, with October 1997 as the official Òroll-outÓ date for users across our campus. To access DeLIver, prospective users must first enter their University of Illinois network identification number in an online "NetID form." This allows us to offer the publishers of material in the testbed a reasonable assurance that access is restricted to campus affiliates, in accordance with our agreement. In addition, prospective users must complete a registration form that provides us with the basic demographic data that will help us learn more about who is using DeLIver. Moving from needs assessment and testing to deployment of DeLIver, the Social Science and Testbed Development Teams were faced with new issues: How should we gauge success? How many users are enough? How do we interpret reasons for lack of use? We faced a period of initial panic when we consulted our web log data on attempted DeLIver accesses for the first several weeks following our full public roll-out. For the period from November 1-14, 1997, we learned that:
Our first task, then, was to address the barriers to use presented by these two forms and figure out how to gain maximum use in the future. The Social Science and Testbed Teams met to discuss options and their implications. Four basic alternatives were proposed. The first was to simplify functionality of the search interface, so that those people who did enter their network IDs and proceed through the registration process would be more likely to actually use the system. We considered removing all advanced search features, such as Boolean searching. The second alternative proposed was to cut our authentication and registration processes completely, offering, in effect, a "free sample" of DeLIver during the last few weeks of the semester, when several project members anticipated heavy demand for the system. A third option was more moderate: to streamline the authentication and registration processes. We began thinking about how to reduce the userÕs burden by shortening and clarifying these two forms. The fourth alternative we considered was to step up publicity around anticipated hubs of use. We reasoned that if we could market the system more aggressively to those people most likely to have a natural demand for it, we could attract more serious users who would be willing to wade through our authentication and registration processes and would naturally become heavy system users. Visits to classes in appropriate disciplines and affixing stickers announcing the availability of web versions of articles to the print copies of the corresponding journals were the mechanisms judged to offer the greatest potential for success. We concluded our meeting by agreeing that we would pursue the third and fourth alternatives before pursuing more drastic measures. What surprised me at this meeting was that, in fact, different project participants had different answers to the seemingly simple question of why we would want to reduce barriers to use. Their answers reflected the odd placement of our DL testbed as a hybrid research/demonstration/production system. I believe the hybrid nature of our system probably reflects the character of many other digital libraries and electronic publishing ventures. Those who viewed DeLIver as a production system wanted to "make the system better" by making it easier to access and use. Those who viewed it as a demonstration system were more concerned that we generate high use for its own sake. And those who viewed our testbed as a research system emphasized the goal of gaining more users so that we could learn more about digital libraries and their use. Of course, it was really more a matter of recognizing all of these goals and negotiating among them. In order to help choose from among our alternatives, Testbed and Social Science Team members pooled their expertise and experiences to try to identify the most likely explanations for the limited number of attempted accesses and the high percentage of people who were bailing out when confronted with our NetId and registration forms. A number of possibilities were proposed, based on results of usability tests and user interviews, our own test uses of DeLIver, and our general knowledge of library system use:
The last consideration brings up the important issue of how to interpret attempted (and to some extent, actual) uses in digital libraries. Use of libraries has traditionally been measured in various ways, such as tabulating the number of registered patrons, the number of visits to the library, the frequency of use of the library's online catalog, and the number of items checked out of the library. Web sites, on the other hand, typically tally the number of "hits" they receive. How should use of a digital library be measured? One particular issue we considered was the difficulty of gauging purposeful use in our testbed. Reference librarians in academic libraries often tally patrons' questions and can easily sort out questions that do not represent patron use of library materials, such as "Where is the drinking fountain?" But when it comes time to interpret attempted access data for DeLIver, we have no way of ascertaining whether an attempted access is really associated with an intended use. In some cases, as noted above, the person who comes across the DeLIver website may just be surfing the web, stopping at any site that looks mildly interesting. If that person abandons their access attempt, to what extent should that be considered as a thwarted use? Our view of use also changes in the kind of hybrid system represented by DeLIver. Different types of use are associated with DeLIver as either a research, demonstration, or production system. People logging in to test or demonstrate the system are easily discounted, conceptually, as "real users" although care must be taken to allow such uses to be accurately tagged in transaction logs. But the case of a new user "playing" with the system to get a sense of its contents and how to use it is harder to categorize and account for when measuring system use. The issue of how to measure use relates directly to arriving at a characterization of success for a digital library or electronic publishing system. Even if we record the number of users and uses of DeLIver, how do we arrive at a reasonable conclusion regarding our achievement? How much use of their material should occur before our publishers can deem their electronic publishing ventures a success? In approaching this issue, we considered what we already knew about DeLIver users from our earlier studies. We also drew on our accumulated knowledge about library use among scientists and engineers, generally. First, we knew from focus groups, interviews, and transaction logs that graduate students were the heaviest users of DeLIver. Thus, we reasoned that we should measure our success in garnering use against the total number of graduate students in disciplines relevant to the material in our collection. In our case, this meant that we were looking at a prospective heavy user population of only about 1000 people. We then thought more carefully about journal use behavior among those in our potential pool of users. First, how many of those graduate students regularly read the journals in DeLIver? Of those who do, how many of them already have personal or office subscriptions to the paper issues, making access through DeLIver from their desktops less of an imperative? For those who would benefit most from online access at their desktops, with what frequency and at what times of year do they normally search for or browse through journals? Many graduate students who participated in our interviews, for example, noted that they only consulted journals at the beginning or end of a project, or at a few key junctures in the semester. Our calculation of the number of attempted accesses for DeLIver, then, should be viewed within this context. In interpreting our success, we should compare the number of attempted uses to the estimated number of attempted uses of the equivalent paper journals within the same period of time. Several of our system designers thought that usage would be extremely high in the final weeks of the Fall semester and were especially disappointed by the number of attempted accesses during this time. The librarians on our project and Social Science Team members who had interviewed faculty and students, however, countered that their experiences suggested that this was not a peak time for journal usage: people were either studying for exams or completing projects for which they had already gathered literature. Another important consideration in interpreting levels of attempted use is the sort of trajectory that is typical for the adoption of any new information system. Based on our earlier interactions with potential users, we reasoned that it takes more than just a few weeks or even a few months to become aware of a new library system and to learn enough about it to be comfortable with its use. In addition, librarians whom we interviewed commented that use would probably be limited because of potential users' lack of commitment to any system portrayed as an "experimental" service. They argued that neither they nor potential end users would be willing to invest the amount of time needed to become familiar with a new system if they felt that the system would be removed when the project supporting it ended. In attempting to put our statistics on attempted use into an appropriate context, we also realized that additional comparative data would be helpful. Not only should we compare the number of graduate students who registered to use DeLIver to the number of graduate students in relevant disciplines on campus and to an estimate of their use of corresponding paper journals, we should also compare the number of students who registered to use DeLIver to the number of students who have used our libraryÕs online catalog. Finally, we noted that only 280 users had registered for the custom client version of DeLIver that ran only on certain terminals in the library from August 1996 to October 1997. So we have seen substantial increase in the use of our system since its migration to the web. IV. CUT AND TRY ENGINEERING: STREAMLINING OUR FORMSOnce we felt that we had identified barriers to DeLIver access and use, and reasonable approaches to removing them, we set about stepping up our publicity and improving our access processes. Presented below in Table 1 is a summary of the changes we made and the resulting statistics for subsequent user access attempts. Several features of the table deserve particular note. One is the variation in the number of attempted accesses at different times of the year. Another is the apparent success of our attempts to streamline the authentication process and corresponding NetID form. Finally, it appears that shortening the registration form had little effect on increasing the number of people willing to complete it. V. CONCLUSIONSThis paper has presented, in an informal manner, a description of how we identified, interpreted, and attempted to remedy access and use barriers associated with our digital library testbed. The issues we confronted are likely to arise in the implementation stage of any electronic library or publishing system that incorporates user authentication and registration processes. We feel that we gained important insights about the ÒhybridÓ nature of information services that incorporate, virtually simultaneously, goals and characteristics of research, demonstration, and production systems. The hybrid nature of such systems has ramifications for both measuring and interpreting use. We found that integrating evidence from various user studies--such as needs assessment interviews, TABLE I SUMMARY OF FORM CHANGES AND ACCESS STATISTICS _____________________________________________ Time period: Dec. 9-19, 1997 Time period: Jan. 1-23, 1998 Time period: Jan. 23-Feb. 18, 1998 Time period: Feb. 18-Apr. 9, 1998 _____________________________________________ usability tests, and transaction logs--was crucial in allowing us to both identify barriers to use and develop reasonable expectations related to use. ACKNOWLEDGMENTThe research represented in this paper was sponsored by the NSF/DARPA/NASA Digital Libraries Initiative under contract number NSF 93-141 DLI. The paper draws on the work of project members affiliated with the University of Illinois Social Science Team (S. Leigh Star, Laura Neumann, Emily Ignacio, Robert Sandusky, Cece Merkel, Eric Larson), as well as work of project members engaged in the construction of our testbed (Bruce Schatz, William Mischo, Tim Cole, Thomas Habing, Donal O'Connor). |