| Index
Site Visit and Quarterly Reports send comments or questions to: l-neuma1@uiuc.edu |
John and Jane Q. Engineer: What About Our Users?Emily IgnacioLaura Neumann Bob Sandusky 1 November 1995 INTRODUCTIONA fundamental component of our DLI project is sociological research and evaluation related to information seeking and use. This piece, synthesized by Laura Neumann, is a joint effort by members of the social science team (Emily Ignacio, Bob Sandusky, and Laura Neumann, specifically) to take a first attempt at integrating, making some sense of, and sharing the large corpus of data that our team has collected over the past year. The nature of the data and the questions that we were asked to address led us to use a qualitative approach to analyze the data. Each of us, coming at the data from different angles and backgrounds has something different to contribute to the final outcome of this first round of analysis. Additionally, each section of this paper addresses a different level of addressing the issues, each contributing to a multi-dimensional picture. This large set of data is the result of several separate data collection efforts (see Appendix A). First, there are ten interviews that Laura Neumann conducted with engineering and physics professors at this university in the fall of 1994. They were asked questions about their use of libraries, computers and their general information finding habits. Second, Emily Ignacio and other members of the sociology team cooperated in three focus groups asking professors, graduate students and undergraduates about their journals and computer use and how they found information that they needed. Third, Laura Neumann did a series of observations during the winter of 1994-1995 of the Grainger reference desk and noted the questions that patrons brought to the reference librarian as well as how these questions were answered. Fourth, Laura Neumann and Emily Ignacio did a series of observations and interviews with Uni High students using the computer room and library to find out how they found information. And finally, observations were conducted by Bob Sandusky, Emily Ignacio and Laura Neumann in the Grainger reference area of how patrons were using the different search systems (including Mosaic) to find information. These data were brought together and were analyzed with several different questions in mind: This paper is an attempt to share what we have been learning and hearing from our respondents. We are not suggesting that the points brought up are unknown; indeed, we are very aware that our work only supports much of what has already been written. We do think, however, that members of the larger Digital Library project, and specifically the testbed team, would be interested to know what we are finding. In addtion, the statements and suggestions that we mention are those of our respondents; we realize that some of them are not feasible, and others are being worked on already, but again, we thought that the people working on this project would be interested in hearing what our respondents are telling us. We hope that this document will be the starting point in a longer discussion between the Social Science team and other parts of the Digital Library project, especially the testbed team. We want our research to contribute to more fully understand the issues.
SECTION 1: POINTS OF INTERESTIn the first section, organized by Emily Ignacio, each question is addressed briefly, pulling out specific quotes and points made in the data on each topic. These are snippets, condensations, and pieces from interviews, focus groups and observations. They were chosen because they are the comments that were repeated time and time again and were found ina variety of sources of data. In addition, people's comments on what they wish a system would do for them and other interesting insights are mixed in. Problems with current systems (library/Illinet/IBIS/CD ROMs): They want to be able to customize their searches - they want to have a lot of control over the searching process: People have also said that they want the computer to be able to keep track of what they have searched, in a sort of running search history AND be able to present them with references that they will be interested in based on this search history. People want to be able to search "by sight." They worry about the loss of the ability to browse. They have asked if they would be able to visualize the appearance of a book and if there were going to be "virtual shelves." The spatial arrangement of the library is very important. Many professors have stated that they were used to the arrangement of the books in the old engineering library and still have to get a feel for where things are at Grainger. Faculty and students mentioned that they would like to be able to search by: System should be able to catch spelling errors and offer synonyms or controlled vocabulary terms for key words. System should be able to allow them to "snowball" search forward and backward - students and faculty want to see how many times a specific article had been cited since its publication, and be able to link to the papers cited in the references. How they like to do searches (not in any particular order): Most people who are not novices, and even some novices simply need to locate something that they already have a reference for. They are not often doing general searches, and this is particularly true of graduate students and professors. Steps that people follow when doing reading journal articles: 1. Read the abstract or first paragraph Several faculty members spoke of a list serve which sends out abstracts. They skim the abstracts and when they find something that they like, they send out a request, and the full article is sent in 10 seconds. They then print the articles out so they can read it at their leisure. In fields where very recent data are important, services such as these are very popular and greatly relied upon. 2. Read the introduction and the conclusions to see if the author is writing about anything interesting. 3. Read the bibliography - to see if the author is citing any important people or articles. Also, they want to make sure that they know what's out there and that they are not repeating an experiment that has already been done. 4. If any of the above things are helpful, then read the entire article. Why access to a printer and copier is necessary: They like to print out the full-text of articles because they can highlight them and read them at their own leisure - they're not bound by the placement of the computer. Even students printed out lecture notes that were available on-line because they liked to bring the notes to class so they could follow the lectures better.
People like to flip back and forth between sections of the paper. They also like to highlight and mark important quotes. Some students mentioned that they keep two copies of the same article: then they can have one "clean" version of the article and mark the other article up. The quality of the printout matters as well: one student gave his reason for downloading his files while using the INSPEC system instead of printing them: "because of the quality and speed of the printer (it is a crappy little dot-matrix printer)", but notes the time is probably not an issue, seeing as it is taking quite a while for the disk to do the downloading. He mentions that he will take this disk to his office where he has a laser printer. Additionally, he explains that "most people use the IEEE system for the free printouts...The worst thing about the Grainger is the photocopy machines. Often you end up walking up and down the whole building looking for one with paper and no paper jam. But when you go and tell the people at the reference or the circulation desk that there is a problem with all the printers in the building, half the time they won't even go and fix them. That is why the IEEE system is really appealing." Why they thought we should we have access to full-text retrieval: Why they use journals: Relevant parts of journals (in no particular order): Other important things besides journals (in no particular order): Note: everything about doing research is based on the socialization process of becoming a professional:What topics to use: Use of the system: How to find topics: (see above for narrowing topic) Other things people mentioned they would like:
SECTION 2: FRAMING THE ISSUESThis section, by Laura Neumann, attempts to offer a more "meta-level" or a slightly more broad view of what seems to be falling out of the data. This section is not only about finding useful quotes or points; instead, quotes are contextualized, which offers some broader ideas on how to think about search strategies. Like Section 1, this analysis is based on all data collected. Search strategies: How do people find things? Novices Undergraduates and the Uni students are lumped together in this category. Most of their searching is done by keyword and topic. They also want to search by location, they want to be able to search on the collection by locality, i.e. search only through the holdings of the undergraduate library. The Uni students actually can do this, search only the Uni library. They focus less on the author and the journal, and seem less discriminating about the quality of what they retrieve and rather focus on the amount of effort it takes to obtain the document. They do some limiting by year. They do some browsing, and particularly for the Uni students, the search on the Internet in non-directed and creative ways, remembering paths, contexts and the look of information rather than the traditional specifics that items are usually searched for by. It seems that these people are looking for information on a general area rather than specific pieces of information. They do not have a deep awareness of the formal search strategies and systems that are available. In their inexperience, they do not have a deep awareness of the field of possible information that they are searching and seem to move rather randomly about in it. Intermediates Graduate students and newer professors fall into this category. These people are getting a feel for their field and thus have different search tasks and strategies than either the novices or the experts in addition to sharing the search strategies of both. These people have the peculiar task of placing themselves and their work in their field, finding their niche and the niche of others. They do keyword or topic searches on occasion, but, since they have a better feel for the information they need and want, are more discriminating in the items they will accept and retrieve. They are more concerned with finding the correct keyword and express a strong desire for a synonym finder, a thesaurus and an acronym finder to be readily available. In their searching of a general area, they do more browsing of a directed sort, for example reading through all the recent issues of one particular journal. In their searching for a particular piece of information and in their awareness of quality issues, they do more snowball searching, following the references back from one article. They express a desire first, to have live reference links- to be able to only click on a reference and be brought to that paper, and secondly, to be able to follow references forward in time, to see who has cited this paper since it was written. As more experienced people, intermediates seem to be more familiar with the formal search systems available to them and use them fairly heavily. [why are these most useful to this group?... the particular pitch of the contents] They also mention using the "find" functions of software to aid them in getting to where they want to be. They are able to pick out what information is most useful to search on and where to go to find what they are looking for (a greater familiarity with the contents of databases.) Intermediates are in the early formation stages of their own personal information finding infrastructure. At this point they are beginning to develop workarounds and rely more heavily on the formal structures available to them. Intermediates seem to also be more apt to ask others for information, for pointers or suggestions as to how to go about finding things or search strategies. As graduate students, they are in a unique position to do so, as professors point out, a large part of their job is guidance of the graduate students in getting to know the field. Experts Experts are those people with a great deal of experience in their field. I define them as people who have built up their own information infrastructures in which they have refined their workarounds to the point that information comes to them. These people rarely do keyword or subject searches, they know their field well enough that such a general search is probably not useful. They instead focus on finding specific pieces of information. They have a "feel" for their field, but they do browse on occasion just to see what is out there. They do less going out and finding information, as I said, they have established their personal infrastructure so that the information they need comes to them in one way or another. Colleague and friend networks are very important in getting the information that they need. Their personal information infrastructure can be understood as located along several different continua: For experts, the office as an information universe becomes an "archaeological dig." It can range from fully ordered (everything carefully files and recorded) to what looks like utter chaos (piling, shuffling, lack of formal easily recognizable system). The process of finding things in the office is about discovery... there is serendipity, the "methods" of piling as cataloging and indexing, friends with whom information is exchanged very informally, colleagues with whom more guarded exchanges occur. The office has various inputs of information: the phone, the door, the mail slot, the Internet connection. There are information flows within the space of the office, information traverses a path from entry to placement in different areas, to replacement, and over again. People navigate their offices and find things in a process that includes hunches vague recall about the "look", the "feel or flavor," the color, the density, the length, the accessibility, the source, and the type of the information, among other things. What is done with retrieved information? What work is the system supporting? Why do people go and get things? This question also seems as if it would vary on Novices As people finding their place into and in a field, the novices' tasks differ greatly from others'. The undergraduate and high school students are doing work for their courses, first find and narrowing down a topic for research, then finding references and writing the paper. They also use articles as models for their own work, they are trying to find out what a paper should look like, or what their research should look like. Occasionally they will use a retrieved piece of information to snowball search for other information, but it does not seem that this is common. Most of the information that they need for their courses is given to them in their textbooks or course notes, but sometimes they will go to journals to get a more detailed piece of information. These people, again, are working in a more general searching mode. Intermediates Intermediates, again, graduate students and new professors, due to their lack of experience in the field are using information in yet a different way. They are using it to place themselves and others within and between fields. They are gaining and understanding of what is out there and what is relevant. They are developing standards of quality with which to evaluate information. They use some documents to point them to other documents through the reference sections. They also use journal articles in particular to model their own work: to model their methods and techniques in their work and to model their writing. They look to articles as examples of publishable material which they strive to follow. They do literature searches for papers they write, for thesis research, for background information on experiments that they are doing. Additionally, intermediates will look to journals to retrieve single bits of information, a formula that they do not want to personally derive, a specification, a graph. These people are in the general search arena as well as looking for specific pieces of information. Experts Experts, again, due to their knowledge of the field and experience do less general searching to find all about something and do more retrieving of specific bits of information or specific documents. They are also using these documents to keep up with their field, or to retrieve a specific number or fact that they need for comparison with their own work.
SECTION 3: MORE QUESTIONS TO BE ASKEDThis section, by Bob Sandusky, is an attempt to bring in other related topics and other questions that might be explored and new angles on how to investigate the issues in question. Rather than making individual points or trying to take a meta-level view, this section refers to a more personal experience of exploration and ideas that were drawn in as the exploration went on. The Search Context This is about the contexts within which people do on-line searches at Grainger. I want to focus on the person's motives, the references they have in hand when they sit down at the terminal, what their quest is, what motivates them, what sort of help they have had in the past (training, experience) or are using this time, which systems they are using and why. What results do they obtain, in which format, and what do they do with them. Maybe a term to use is the externalities of the search process. I'm not looking at the details of how the system works, the specific search strategies used by the user, etc. Another way I make sense of this is by using the black box metaphor. I'm trying to focus on the inputs and outputs of the black box. The black box is the particular system(s) and commands and responses. I'd like to look at what's offered to the black box as input / user knowledge / starting points and what comes out when they logout or stand up and walk away. I can't make any claims for causality here: there are too few subjects per system to make it a valid sample, and we aren't putting in controls to make this an experiment. Primary data for this is the observations done during this summer in the Grainger reference area. The interview and focus group transcriptions are also informing this little bit of analysis. The first parameter of the search context is the motive for the quest. When a user comes to the library or logs onto an on-line information system, there is motivation. The quest that the user is embarking on is not likely to be trivial. Even the most straightforward of these systems (probably IO+/LCS) is not easy to use. I think that the typical user isn't going to begin using these systems casually or for entertainment (vs. e-mail or WWW). So, the question is what motivates people to expend the energy to log on or come to the library to use these systems? So far, the Grainger observations show a variety of motives for searching, from job searching, to experimenting with unfamiliar systems (just sort of messing around with them to see what they can do), to directed browsing, to looking for very specific subject matter, to looking for specific information objects (like an article) [see details below]. In contrast, some of the interviews and focus group results I've looked at seem to talk more about normative motives like writing articles and proposals (faculty and graduate students) or completing assignments (undergraduates). The real life observations are useful in identifying a wider range of motivations for usage. This is interesting to me also because many of the Grainger observations were done without asking the users directly what their motives were. Their actions suggest some of the motives listed above while others told us specifically in response to our questions. The next issue is the references people have in hand when they begin a search. It seems to me that most searches that are directed at all would begin with some specific starting point: a URL, a search term, a vaguely or specifically conceptualized topic or subject, and author, a citation, an article with citations, a title, a journal, etc. Several of the Grainger observations were of people who did not seem to have any concrete references (on paper). Some of those without paper seemed to be fairly well focused (perhaps because the user had the subject /author/etc. in his or her memory) but a few seemed very scattered, almost random. The latter seem to be more likely to be experimenting with the systems to get a sense of what they are like or what they can do. How strong of a relationship is there between motive and having a reference at hand; a starting point? The interviews seem to get at this at least a little, I will look there, also. Training or experience in the past and getting help is next. So far, we haven't asked anyone about training or experience they have had with specific systems. In a few observations, we have seen the users talking to librarians or other users about the systems. In a couple of cases they have interacted with the observers! Also of interest is the use of L.C. subject heading lists (on IO+) and use of thesauri: generally, I call this getting help with words. I also think in terms of help from on-line sources and help from off-line sources. Librarians, other users, books, pamphlets are examples of off-line help. Help screens (and maybe BI) are examples of on-line help. Perhaps the interviews and focus groups will provide more data here. It seems to me that the on-line help and information at the terminals is extremely thin, and not of much use. If someone really wants to know how a system works and / or what information it covers, how do they do that? It's interesting to look at which systems people choose to use and why they use them. Is their choice based upon familiarity and comfort? Capabilities (a lot of people like printing the full text for free from the IEEE system)? Knowledge and awareness of the coverage of various systems (I doubt it because I have a hard time figuring this out)? Don't have a lot of concrete information about the why here, we can observe pretty easily what they choose to use. What they take away from the search session is important, too. This is usually easy to observe, but we have only asked some of the users we've observed exactly what they plan to do with the results. The interviews and focus groups might provide some normative types of responses. I'll look there, too. Sometimes people seem to take the citations from something like I.E. or IEEE and go to IO+ to check holdings. A lot of people just walk away. Do they follow up? How?
CONCLUSIONThis work is only a first attempt at organizing, making sense of, and sharing the information that the Social Science team has been collecting. It has been generated through the use of qualitative methods, specifically grounded theory. The goal of this theory is not to generate statistics but rather to reveal trends, commonalties and important points in the data and then to situate these in a larger analytical framework. Each member of the sociology team comes from a slightly different background, as should be obvious from reading this paper, but we see these different views as helpful in revealing multiple facets of the issues. We hope this document will lead to discussion between the Social Science team and interested parties on the rest of the Digital Library project, especially the testbed team. We not only want to discuss the findings in this document, but also the method that we used to analyze the data, how it is or is not useful, and what is to be done in the future. This paper is an attempt to share what we have been learning and hearing from our respondents. We are not suggesting that the points revealed are novel; indeed, we are very aware that our work only supports much of what has already been written. We do think, however, that members of the larger Digital Library project, and specifically the testbed team, would be interested to know what we are finding. Secondly, the statements and suggestions that we mention are those of our respondents. We realize that some of them are not currently technologically feasible, and that others are being worked on already, but again, we thought that the people working on this project would be interested in hearing what our respondents are telling us. We find their comments and actions fascinating and wish to use what we are finding to help build a useful digital library. |