| Index
Site Visit and Quarterly Reports send comments or questions to: l-neuma1@uiuc.edu |
Social Science Research- Social Science TeamQuarterly Report (work from May through July, 1995) Faculty and students in the Graduate School of Library and Information Science, Sociology, and Economics are continuing with their user-based research related to the DLI. The main activities in this quarter have been some initial development of a conceptual framework to serve as a theoretical base for their work, the completion of an online user registration form, the development of specific procedures for capturing and analyzing Mosaic transaction log data, observations of the use of existing digital resources at the Grainger library, and progress in the integrated analysis of findings from all of their activities. One of the major conceptual challenges that the evaluation team has been working on over the last quarter is triangulation. Triangulation means taking multiple views of the complex phenomenon of building infrastructure, and using multiple methods to do so. But the project itself involves a kind of triangulation of different concerns in order to make a workable system, including those of designers, librarians, users, and publishers. Triangulation is not simply a matter of adding views or data points together. Data deriving from different methods, or design decisions taken from different viewpoints, or usage by those with different needs, always means a negotiated process. Rather than try to resolve these difficult questions by fiat or formula, the evaluation team has begun to conceptualize its own work in terms of different viewpoints, each with its own epistemological focus. These are the viewpoints of users' work, the analysis framework, and study methods, with a fourth "meta view" as the process of integrating and triangulating these views, which are described briefly below. Each of these views also scales, from focusing on digital library use at the individual level (encompassing more cognitive tasks), to wider net- or Web-wide phenomena. At the individual work level, tasks such as actual browsing of information and information retrieval are important. Analytically here, the important questions concern those cognitive/conceptual changes for users in moving to digital form, including such anticipated changes as different metaphors and images (e.g., "navigating" in cyberspace vs. "wandering" the stacks). Moving from the individual in front of the screen, the next level we encounter is that of the flow of work, where the digital library is embedded within a work space. Analytically, the questions move here to understanding the links between an individual's workspace and work flow and the features of the system. One example of this sort of question concerns how an individual's "ethno-classification" and personal library and filing system is affected by the digital library. In addition to work spaces and work groups, our project has interesting institutional implications, including occupations and the institutions of the extant physical libraries affected. Analytically, the challenge for this level of focus is to understand the changing distribution of skills posed by the virtual library environment and to understand the nature of organizational transformations. What will happen to the ways librarians, engineers, and publishers currently organize their work and operations at the institutional and professional levels? At the widest level of scale with which we are working in this project, the digital library interfaces with the World Wide Web, and involves a large number of people (whose ties to each other and to the information in the system may be looser than those influenced by shared proximity, tasks, goals, or institutions) in complex cognitive tasks such as information retrieval. Analytically, here, there are many questions in the realm of what we call "sociology of infrastructure," that is, what is the nature of large-scale changes in work and cognition afforded as the entire information infrastructure begins to change? The evaluation team is currently developing two primary mechanisms for automatically collecting DLI usage/user data. First, many of the DLI software components are being instrumented to collect detailed transaction logs of each user session. Second, an automated DLI user registration process will collect demographic information about each DLI user and provide a confidential mechanism to link the data in the transaction logs to individual DLI users. These data are being collected to serve two primary purposes. The first purpose is to provide various sorts of management data. This includes summary data to the project management and the outside world concerning the number of users and their aggregate behavior as well as providing system performance data to ensure that the DLI is operating within acceptable tolerances. The second objective met is the collection of detailed data on the individual DLI users and their individual behavior. During this quarter, the researchers developed and began pilot testing a World Wide Web user registration form. The form collects contact information as well as information about the user's professional background and the extent of the user's familiarity with common computing and communications systems. Work in system instrumentation this quarter has resulted in the completion of specifications for collecting and analyzing transaction log data from Mosaic. The evaluation team is currently working with NCSA staff to de-bug the instrumented version of Mosaic prepared according to their mutually agreed upon specifications. Next quarter, they aim to begin user testing with an instrumented Mosaic. The goal will be to gain experience with collecting these data before incorporating an instrumented Web browser into the DLI prototype. In addition, they will use the instrumented Mosaic to begin collecting data on general Web use by members of the engineering community. The evaluation researchers have also been working on plans for instrumenting each of the other software components (e.g., database management systems, database search engines, thesauri, and SGML and other data format viewers) that together will comprise the DLI system. During any single DLI session, several loosely coupled systems are brought into play in order to search for, retrieve, and display documents for the user. By logging the user's interactions with each of these systems, the researchers can focus on one of the systems in isolation or all of the systems as components of a digital library. A higher level methodological question they hope to address is the extent to which these types of transaction logs can be used effectively to understand the behavior, needs, etc., of individual users instead of using methods such as interviews and field observations. The major data collection effort of the evaluation team this past quarter has been to observe patrons using existing computer systems related to the retrieval of fulltext material: Mosaic, Engineering Index on CD-ROM, the expanded online catalog, and IEEE's fulltext journal system on CD-ROM. The purpose of these observations is provide information requested by the DLI testbed team, who noted that some of their design decisions would be influenced by knowing such things as:
Answers to these questions will also contribute to longer term sociological research on the changing nature of information infrastructure in engineering. Members of the evaluation team have observed a number of Grainger patrons and transcribed the observation logs of their search sessions. Each observation session also includes asking each system user the following questions:
These observations were begun recently and the researchers have not yet collected a great deal of data. They have developed a content analysis scheme and begun coding their initial results. Preliminary analysis reinforces the findings from earlier focus group interviews, individual interviews, and reference desk observations: patrons do not usually have a good grasp of either the content or nature of the systems that they are using; they employ a more or less random searching strategy with general information (i.e., subject searches with uncontrolled vocabulary); and they have little patience for reading instructions or asking for help when they get stuck. When patrons did not know what a database contained, they would simply enter their subject search terms and see what was returned rather than trying to find a source of information to tell them if the contents of the database were relevant to their search subject. Another major area of social science research activity this quarter has been in the area of data analysis. Members of the evaluation team have begun the integrated analysis of data from all of their research efforts, using the grounded theory approach. Grounded theory involves reviewing all sources of data, coding the most frequent occurrences of actions and perceptions within the data and asking under what conditions these actions emerge. These coding schemes are then abstracted into memos that identify and discuss emerging themes and concepts, which are then applied to data sources (previous and new) and re-examined. This is an iterative process directed at inductively generating concepts and theories directly from the data collected. The short term goal of the evaluation team's work is to not only provide feedback to the testbed team on issues that they specifically ask about, but also to watch for "the unexpected." For those observations and interviews most directly related to user interactions with library systems, summaries of coded data are given to the testbed engineers immediately so that they benefit from direct user input as they design the digital library testbed. The coding schemes used to produce these summary reports, such as the content analysis scheme developed for the Grainger observations, are tuned to the specific goals and results of that particular data collection activity. As more data are collected-- from interviews, observations, transaction logs, user surveys, etc.--individual coding schemes are expanded, refined, and integrated and can be applied to other data sources as well, and more integrated and analytic feedback can be presented to system designers. Further, the integration of all of the evaluation data in this manner helps the researchers generate theories about information infrastructure, knowledge gathering and sensemaking, and communication in the engineering community. A second major data analysis activity pursued during this quarter is the exploration of Hyper-G as a tool to support the researchers' grounded theory work. Other standard software packages for qualitative data analysis do not seem to provide the flexibility and power desired. Further, they are inadequate for dealing with the multimedia data the team is generating. The evaluation team has continued planning for the first stage of deployment of the prototype DLI testbed, which will provide the opportunity to collect data on system use through both controlled usability tests and more naturalistic observations of use. As noted in last quarter's report, several public sites for deployment at the University of Illinois have been identified. The sites have been selected to allow the study of system use within both library and workteam environments, and across several engineering disciplines, by various segments of the academic engineering community, such as faculty members, graduate and undergraduate students, and librarians. We hope that the next two quarters will see both usability testing of the prototype along with the collection of data about system usage in these actual work and learning situations through observations, interviews, and the solicitation of user feedback. Co-PI Bishop continues as the evaluation team leader and Star continues as the technical lead for ethnography and research. In addition to their responsibility for evaluation research related to the University of Illinois project, they also play an important role in promoting evaluation efforts across the other DLI projects. Bishop serves as chairperson of the DLI-project- wide working group on user-centered evaluation and of the upcoming Allerton Institute on "How We Do User-Centered Design and Analysis of Digital Libraries: A Methodological Forum." As hoped, this small invited meeting has attracted participants from the six DLI projects and from other major digital library projects, as well as other renowned researchers from sociology, anthropology, psychology, computer science, and library and information science. Bishop also serves on the program committee of DL96, an interdisciplinary conference sponsored by a number of organizations, including the ACM. During this quarter, Bishop participated in a panel on the DLI initiative at the ACM's Computer Human Interaction (CHI-95) conference, conducted a session on digital libraries as part of a faculty institute at Skidmore College in Saratoga Springs, New York, and hosted an expert forum on user research at the DL95 conference in Austin. Emily Ignacio and Laura Neumann, research assistants on the evaluation team, participated in DL95 and are collaborating with Bishop and Star on Allerton activities. Neumann also took the lead in developing a homepage for the DLI evaluation team that facilitates the dissemination of information related to their work to the public. Star serves on the steering committee for the Allerton Institute. She has participated in several national meetings related to furthering humanities and social science efforts for digital libraries. Publications Bishop, A., et al. (in press). Building a Digital Library for the Academic Engineering Community: Implications of User Research for Higher Education. In Higher Education and the NII: Proceedings. Washington, DC: Coalition for Networked Information. L. Star, editor, "Cultures of Computing" (Sociological Review Monograph). Oxford: Basil Blackwell, 1995. L. Star and K. Ruhleder, "Steps toward an Ecology of Infrastructure", Proceedings of CSCW 94. New York: ACM Press. Pp. 253-264. Revised version accepted for Information Systems Research, special issue on Organizational Transformation, edited by JoAnne Yates and John Van Maanen. L. Star and G. Bowker, "Work and Infrastructure," (short contribution), Communications of the ACM (Sept. 1995) 38: 41. Presentations Bishop, A. Presentation as part of a panel on the DLI. ACM's Computer- Human Interaction Conference 1995 (SIGCHI), Denver, Colorado, May 10. Bishop, A. Seminar on the DLI and other current digital library activities. Presented at the Skidmore Faculty Institute on information technology, Saratoga Springs, New York, May 26. Bishop, A. Hosted expert forum on user-centered digital library research. Digital Libraries 95, Austin, Texas, June 13. Bishop, A. and Joseph Squier. Artists on the Internet. Annual Meeting of the Internet Society, Honolulu, Hawaii,June 27. Estabrook, L. Presentation on DLI social science research, as part of a panel at the 1995 Mid-Year Meeting of the American Society for Information Science, Chicago, Illinois, May 25. Star, S. L., Plenary speaker, "Infrastructure and Work, " to Conference on Mediated Activity in Organizational Contexts, School of Education, University of Helsinki, Finland, January 1995. Star, S. L., invited attendee to NSF special meeting at Stanford, April, 1995, on digital libraries and science and technology studies. Star, S. L., "The Illinois Digital Library Project" and "Standardization" to the Institute for Informatics and Norwegian Computing Centre, University of Oslo, Norway, June, 1995. also participated in a seminar there on standards and the internet, and made a presentation on standards and communities. |