| Index
Site Visit and Quarterly Reports send comments or questions to: l-neuma1@uiuc.edu |
DLI USER STUDIES AND EVALUATIONAnn Bishop, abishop@uiuc.edu
GOALS:
METHODS:
SYSTEM INSTRUMENTATION Transaction Log Format:
Some specific transactions (examples): 2001 19960301 151011 00003122 255.255.12.1
2109 19960301 151043 00003122 255.255.12.1
2110 19960301 151152 00003122 255.255.12.1
2119 19960301 151221 00003122 255.255.12.1
2120 19960301 151228 00003122 255.255.12.1
2121 19960301 151256 00003122 255.255.12.1
2161 19960301 152207 00003122 255.255.12.1
Transaction Log -Session date Snapshot of Illinois Digital Library User Activity The following tables present some of the information available about early usage of the Illinois Digital Library (IDL) Testbed. These tables are based upon data collected through IDL registration process and by instrumentation included in the ‘custom client’. The registration process was developed by the IDL Social Science Team, and the custom client was developed by the IDL Testbed Team. The client instrumentation was developed through close collaboration between the Social Science and Testbed teams. Several terms appearing in the tables need to be identified in order to clarify the meaning of data. A "Patron" is a Illinois Digital Library user who self described themselves during the registration process as neither a member of the Illinois DLI nor an employee of the Grainger Engineering Library. This distinction allows us to focus on the usage of the testbed and client by ‘real’ users and separate out the activities of librarians conducting demonstrations or DLI project members performing troubleshooting or testing. Most of the tables here reflect activities performed by patrons only. A "Practitioner" is defined as a full member of a scientific community of practice. This category includes registrants who identified themselves as professors of any rank, scientists, engineers, and post-docs. The category labeled "Other" includes those whose role is more likely to be in support of the working faculty, scientists, and engineers, such as academic professionals and librarians. The field names (Computer Science, Physics, etc.) should be self-explanatory. In the future we will subdivide the "General Engineering" category into subcategories that reflect the specific contents of the IDL (e.g. Civil Engineering; Electronic / Computer Engineering, etc.). The numbers of registrants and sessions are relatively low, probably due to (1) the client deployment, which is limited to a small number of workstations located at Grainger Engineering Library, the Beckman Institute Library, and a microelectronics research laboratory, and (2) the short time which has elapsed since the client and the associated instrumentation became operational. The data presented below were gathered between August 1996 and February 1997. The most fundamental numbers are that 144 patrons have registered between August 1996 and February 1997, the vast majority from Grainger Library. There have been 191 patron sessions recorded in this time period. Patrons change the article component settings in about 5% of the sessions recorded so far. This may seem low, but is more than the 0% use of similar facilities reported by the CORE project. –Robert Sandusky
Table 1 Summary of the distribution by field and career level of IDL registrants (August 1996 through March 1997). This table includes all registrants: patrons, project members, and library employees.
Table 2 Summary of the number of sessions conducted by IDL patrons (August 1996 through March 1997)
Table 3 Summary of the number of searches conducted by IDL patrons (August 1996 through March 1997)
Table 4 Summary of the number of times IDL patrons examined the article component search settings (August 1996 through February 1997) Data are given in the form N/M, where N is the number of times the settings were examined and M is the number of sessions in which examinations occurred
Table 5 Summary of the number of times IDL patrons changed the article component search settings (August 1996 through February 1997) Data are given in the form N/M, where N is the number of times the settings were changed and M is the number of sessions in which changes occurred
Table 6 Summary of the number of patrons registered for the IDL, by field and by gender (August 1996 through February 1997)
DLI Spring ’97 Partners Workshop At the publisher’s meeting last year, the social science team asked participants to write down any questions they had that related to the user studies that we on the social science team have been working on. What we were trying to find out was what the publishers were interested in knowing about use and user issues surrounding the DLI testbed. The following are some of these questions. Some questions are not included because either they were duplicates of other questions or we do not yet have the data needed to answer them. Answers provided to the questions are drawn from data collected through semi- structured interviews, focus groups, and observations with potential users and others related to the digital library project. More data specific to the testbed system will be generated from transaction log analysis as more people use the system. Do searchers become more adventurous as they use the system more...that is, do they try more features or stick with those they previously used? We don’t yet have enough use of the system to determine this, however, other data collection efforts find this is an issue with other systems. In my interviews and observations, I have noticed that when a system is new, people are sort of playing with it-- using it for as many things as possible, stretching it in new ways. However, this passes and users settle into patterns of use that are often based on features that they know, with some exceptions. For the most part, users we have studied are quite computer- savvy and they have found what they consider the optimal use for particular programs and use them in that specialized manner. If a new program does a new function particularly well, I have the impression that the new function stands a good chance of being used so long as it is a function that they need in their work. What I would consider a more pertinent question related it this is which functions would be most useful to our audience? Assuming the system is functional (doesn’t crash, has proper documentation, people can fairly easily figure out how to use it, has that critical mass of documents...), he functions that are most useful will probably be used, regardless of how new they are in relation to the more "traditional" functionalities. How important is archival vs. current awareness use? Do people expect and want current stuff only? Absolutely not. People do want current stuff, and it is very useful, however, this is what most systems have. What many people have mentioned in interviews and focus groups is that they wish that the databases covered a longer time span. Most fields have databases available (formal or informal) that have more current materials. The space in the market in current stuff is that the system has to offer better features than what is out there already, however as there are few systems with anything except current stuff, there is a much larger market space for this. There is a definite call for this. A related question, however, is what "archival" materials are for each field. In sub- atomic physics, this may mean work published between 1945 and1995. In physicists dealing with general relativity, they would like to see Einstein’s work available. Then for computer scientists, anything from earlier than December of 1996 is "old." The meanings of "archival" and "current" are very fluid and need to be carefully examined for each field. Do users download and print? Once the article is retrieved, what do users do with it? Users will download, it seems, only if it is necessary for printing. There are a variety of different habits regarding when and why people print out materials. In interviews and observations there has been a complete range of people- from those who print out and photocopy and keep in a accessible location just about everything that could possibly be useful to them as well as their colleagues, to people who print nothing. Some of the dimensions to consider are: What is done when accessing article:
What is done with a print out:
Why did the person make a copy:
What will it take to make it desirable for users to read the article on- line?
1. better monitors-- higher resolution
3. value added features of electronic medium: |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||