| Index
Site Visit and Quarterly Reports send comments or questions to: l-neuma1@uiuc.edu |
User Research and the NSF/ARPA/NASA Digital Library Initiative (DLI) Projects: Capsule Descriptions of Work in Progress
October 13, 1995
The aim of the six DLI projects is to conduct R&D needed to develop and deploy digital libraries as part of the global information information infrastructure. The projects began in Fall 1994 and will be carried out over a four-year period. While each project team will deploy a digital library testbed, their R&D agendas, technologies, collections, and intended users vary substantially (Digital libraries, 1995). Thus, while all devote some attention to studying system use and users, the nature and level of their user research efforts naturally vary. A capsule description of each project, based primarily on the initial DLI press release from the National Science Foundation is presented below. An overview of each project's user research component, based on material from project homepages as well as on discussions with, and other material provided by, project members, is also presented. Some common issues that seem to cross the DLI user research efforts are the need to develop new methods for tracking distributed "virtual" users, difficulties in integrating data from various quantitative and qualitative sources, determining the appropriate mix of study foci (from system usability to institutional impact to policy implications), dealing with a large and heterogeneous user population, and discovering the most effective mode for collaborating with both users and system designers. With encouragement from project sponsors, we've established an informal DLI-wide User Research Working Group. The motivation for the working group stems from our sense that the six DLI projects are similar enough that we can learn from each others' experiences. In addition, we've found that each user research group has different strengths. While our group, for example, is especially strong in ethnographic approaches to studying system use, other groups have had more experience with conducting usability tests and designing system instrumentation. For this first year, our interactions have been limited in scope and informal in nature. We get together twice a year at the DLI synchronization meetings to report on our progress and discuss common issues, have set up a mailing list for working group members, and are gradually getting to know each other. We also try to make some information about our findings available on our respective project homepages.
Key Individuals in User-Based Research: Scott M. Stevens, PI, User Interface Development-Testbed Evaluation Overview of Project Scope and Goals:
User-Based Research (see Stevens, 1995). The creation of a multimedia library that is unlike anything in existence today demands special attention to the development of an interface that fosters both comfortable and effective use. User-based research in this project will focus on three issues: how to allow users to select the most relevant video segment when their search returns a number of "hits"; how to allow users to adjust the duration of video segments to meet their needs; and how to facilitate skimming of video segments. User studies will play an important role in the development and testing of mechanisms such as the presentation of multiple imicons (intelligent moving icons) in separate windows, a context-sizing feature enabling the user to adjust the duration of segments for playback, and a skimming feature that lets the user select and abstract relevant video and audio information. The Informedia prototype will be placed in cooperating laboratories and schools. Use will be monitored and the system will be instrumented to automatically log and produce statistical reports of usage activities. To further refine the system, the researchers will also conduct protocol analyses of individual users' sessions and focused interviews with those who have used the prototype.
Key Individuals in User-Based Research: Nancy Van House, head of group conducting user needs assessment and evaluation Overview of Project Scope and Goals:
User-Based Research (see Van House, 1995) A significant focus in this project is learning about the interaction of the digital library with work practices of environmental planners. At issue is how people's work is adapted to the system, as well as how the system can adapt to work practices. The user research group has identified several levels of work that must be explored in order to understand how digital library use is situated in the work context: the task environment (including its organizational, political, disciplinary, social, and economic aspects), the task of environmental planning, the user's specific goals and tasks, the user's information acts (how one goes about seeking and utilizing information), and digital library use (how one interacts with the system itself). The development of methods--drawn from usability analysis, work-centered system design, and library evaluation--that are appropriate to understanding these various levels of work and iteratively informing the design of the Electronic Environmental Library's functionality, interface, and collection is an important methodological contribution of this project. Testbed users will be drawn primarily from professionals involved in environmental planning, with the general public a second priority. Currently, the researchers are conducting in-depth interviews, focus groups, controlled observations (with videotaping), and the collection of user comments and feedback (Van House, 1995, draft). One important issue already identified by the user research group is that of customization: how customized can and should a digital library be in order to best serve different professional communities, with varying levels of expertise in both system use and domain knowledge?
Key Individuals in User-Based Research: Barbara P. Buttenfield, head of User Evaluation Team Overview of Project Scope and Goals:
User-Based Research: This user research team is focusing on developing a better understanding of user requirements and providing an empirical basis for interface design (Buttenfield, 1995, draft). At present, the target user study populations are those who patronize the various libraries where the prototype system has been installed. One interesting challenge in this project has been developing an evaluation program and instruments that can accommodate versions of the testbed that are running on different technology platforms. Currently, the team is combining a number of data collection techniques, including questionnaires, videotaping of testbed use sessions followed by think-aloud user reports as they view the videotapes, and session logging. Interactive online logging that involves querying system users in response to specific moves they make in using the prototype is an interesting methodological contribution of this research. In addition, the researchers have used their findings not only to provide feedback to designers, but to develop tutorials for Project Alexandria. Longitudinal experiments with the same user groups have been conducted to try to get a handle on how user requirements for spatial information change over time and across revised versions of the system.
Key Individuals in User-Based Research: Vicky Reich, project member exploring user research approaches and implications Rebecca Lasher, head of the Library Issues Working Group (whose members come from all six NSF/ARPA/NASA DLI projects) Overview of Project Scope and Goals:
User Research: In the Stanford project, research is aimed not so much at the construction of a particular digital library, but at developing tools that others can use to integrate electronic library services, programs, and resources, creating their own virtual libraries (Reich, 1995, draft). Computer scientists and librarians are working together to create the kinds of interoperability technologies that will support the information work of scientists in the NII environment, providing integrated access to personal and public repositories, commercial and noncommercial resources, a range of search tools, and new services such as annotation providers or format conversion facilities. While much of this research is devoted to the development of underlying technologies, there are also plans for testing user interfaces later in the project. Librarians associated with the project are contributing their expertise related to both user and provider work practices in library settings. They are interested in exploring such things as how the new technologies will be integrated into traditional library services (Lasher, 1994) and how community and situational aspects of library spaces might be recreated in the virtual environment (Reich and Weiser, 1994).
Key Individuals in User-Based Research: Thomas A. Finholt (leader) and Joan Durrance (member) of the Deployment, Testing, and Evaluation team's Evaluation Group Overview of Project Scope and Goals:
User Research (see the UMDL proposal and the Deployment, Testing, and Evaluation team's Jan.-June 1995 Progress report). The user communities for the UMDL will include researchers, university and high school students, and the general public. Critical to the evaluation effort are ongoing programs of outreach, training, and assistance to promote digital library use. This project's user research aims at both providing direct user feedback to system designers and developing models of how primary source scientific information and communication tools are used to "scaffold" learning and research processes. Currently, deployment and evaluation is focusing on schools in the Ann Arbor area. Evaluation is integrated with curriculum development and deployment of digital library resources in selected settings. Project members are working with teachers and librarians to develop and implement projects involving the use of networked resources at the sites. Use data is being collected through system monitoring, field notes and videos from classes, interviews with teachers and media specialists, written evaluations from students and teachers, and focus group interviews with teachers and school media specialists. Deployment in the Ann Arbor Public Library, the New York Public Library, and a New York high school are scheduled for 1995/96. Methodological issues encountered in this project include dealing with the heterogeneous nature of UMDL users: Which users should researchers select for sampling and instrumenting system use? Another issue is that design decisions made to satisfy one type of user, like space science faculty members, might not suit other groups, such as high school students or teachers. The evaluation group is also considering how conflicting demands may be arbitrated when users don't bring shared assumptions to the system (Finholt, 1995, draft). Ann Bishop, coordinator of the testbed evaluation team Susan Leigh Star, lead researcher for sociological studies
A fundamental component of our project is research and evaluation related to digital library use. Our team's goals are to:
To pursue these goals, we have developed an integrated research program that combines broad study of use with deep study of social phenomena. Our sociological studies aim at developing methods and models for understanding the nature of particular information interactions, individual work and learning processes, institutional and professional dynamics, and community phenomena in the distributed, digital realm. Over the course of the project, we will conduct observations of engineering work and learning activities and how they intersect with the use of distributed, digital information. Individual and group interviews will be conducted with a range of potential and actual testbed users from the engineering community. We will conduct usability tests of various components and versions of the DL prototype and experiment with economic models and charging mechanisms. Extensive data on use will be gathered through large-scale user surveys and system instrumentation. To date, we have conducted focus group and individual interviews with faculty and students, observations of the use of existing online engineering library systems, usability tests of the thesaurus component of the testbed. By situating the study of testbed use within the broader context of professional work and social practices, we believe we will gain more robust insights into the functions and features that will make our testbed more effective. In addition, we will get a sense of large-scale changes in work and cognition afforded as the nation's entire information infrastructure begins to change. Since our digital library corpus is based on documents structured with SGML, one unique contribution we hope to make is to learn something about the way that unbundled and malleable document components affect knowledge production in engineering communities, as well as information service provision by libraries and publishers. Two conceptual and methodological challenges that we are confronting are triangulation and scalability. By triangulation, we mean not just the use of multiple methods, but the need to take multiple views of the complex phenomenon of building infrastructure. Scalability refers to the problem of understanding use of distributed repositories by a large number of remote users whose work is conducted in both online and offline arenas. Just as we need new prototypes for constructing digital libraries for the NII, we need new prototypes for the conduct of social science research in the NII environment. We hope to begin exploring the boundaries between the online and offline interactions that occur between system researchers and users.
REFERENCESButtenfield, B. P. (1995, draft). User evaluation for the Alexandria Digital Library Project. [Discussion document submitted for the 1995 Allerton Institute: How We Do User-Based Design and Evaluation for Digital Libraries: A Methodological Forum]. Digital libraries [Special issue]. (1995). Communications of the ACM, 38(4). Finholt, Thomas A. (1995, draft). Understanding digital libraries: Possible lessons from the analysis of collaboratories. [Discussion document submitted for the 1995 Allerton Institute: How We Do User-Based Design and Evaluation for Digital Libraries: A Methodological Forum]. Gaston, B. (1994, Sept. 27). NSF announces awards for digital libraries research. Washington, DC: National Science Foundation. Lasher, R. (1994, Oct. 11). Library issues for the Joint Initiative Digital Library Projects. Unpublished manuscript. Reich, V. (1995, draft). Allerton discussion document. [Discussion document submitted for the 1995 Allerton Institute: How We Do User-Based Design and Evaluation for Digital Libraries: A Methodological Forum]. Reich, V., & Weiser, M. (1994). Libraries are more than information: Situational aspects of electronic libraries. Serials Review, 20(3), 31-37. Van House, N. A. (1995). User needs assessment and evaluation for the UC Berkeley Electronic Environmental Library project. In F. M. Shipman, III, Richard Furuta, & David Levy (Eds.). Proceedings of Digital Libraries '95: The second annual conference on the theory and practice of digital libraries (pp. 71-76). College Station, TX: Texas A&M University. Van House, N. A. (1995, draft). Current project: UC Berkeley's NSF/ARPA/NASA Digital Libraries Project. [Discussion document submitted for the 1995 Allerton Institute: How We Do User-Based Design and Evaluation for Digital Libraries: A Methodological Forum]. |