DLI USER STUDIES AND EVALUATION

Ann Bishop, abishop@uiuc.edu
Emily Ignacio, eignacio@uiuc.edu
Cecelia Merkel, c-merkel@uiuc.edu
Laura Neumann, l-neuma1@uiuc.edu
Robert Sandusky, sandusky@alexia.lis.uiuc.edu

GOALS:

• Provide ongoing user feedback for developing DLI system features and functions
• Document and analyze extent and nature of testbed use, satisfaction, and impacts
• Identify reasons for use and non-use of the testbed
• Contribute to theoretical understanding of the changing information infrastructure
• Develop and assess new methods for capturing user data in the distributed repository environment

METHODS:

• Observation and interviews
• Focus groups
• Usability testing
• Online user surveys
• System instrumentation

SYSTEM INSTRUMENTATION

Transaction Log Format:

Some specific transactions (examples):

2001 19960301 151011 00003122 255.255.12.1

Log in to DL system

2109 19960301 151043 00003122 255.255.12.1

Erase all search terms

2110 19960301 151152 00003122 255.255.12.1

AND must also contain

2119 19960301 151221 00003122 255.255.12.1

Execute the search

2120 19960301 151228 00003122 255.255.12.1

Search results: provide details on retrieved set of documents

2121 19960301 151256 00003122 255.255.12.1

Display results

2161 19960301 152207 00003122 255.255.12.1

Quit the DL system

Transaction Log
Session Summary Contents

-Session date
-Session time
-UserCode
-Session duration
-Client used
-IP Address
-User type (Patron, DLI Project Member, Grainger Employee)
-Number of full-text searches
-Number of author searches
-Total number of searches (aggregate)
-Number of short entry records viewed
-Number of SGML documents viewed
-Number of PDF documents viewed
-Total number of documents viewed, any format (aggregate)
-Flag: were help, hints, or demos used?
-Flag: was document disaggregation used at any time during the session
-List of document components used (defaults only; other particular settings)
-Number of times bibliography list viewed from short entry display
-Number of times author affiliation list viewed from short entry display
-Number of times figure / table caption list viewed from short entry display
-Number of figures / tables viewed from short entry display
-Number of times linked to journal name / issue from short entry display
-Number of times printed short entry data
-Number of times linked to WWW
-Number of times linked to Faculty database
-Number of times linked to Engineering Index database
-Number of times 'display results' chosen
-Number of times linked to journal title via top screen's pull-down menu

Snapshot of Illinois Digital Library User Activity

The following tables present some of the information available about early usage of the Illinois Digital Library (IDL) Testbed. These tables are based upon data collected through IDL registration process and by instrumentation included in the ‘custom client’. The registration process was developed by the IDL Social Science Team, and the custom client was developed by the IDL Testbed Team. The client instrumentation was developed through close collaboration between the Social Science and Testbed teams.

Several terms appearing in the tables need to be identified in order to clarify the meaning of data. A "Patron" is a Illinois Digital Library user who self described themselves during the registration process as neither a member of the Illinois DLI nor an employee of the Grainger Engineering Library. This distinction allows us to focus on the usage of the testbed and client by ‘real’ users and separate out the activities of librarians conducting demonstrations or DLI project members performing troubleshooting or testing. Most of the tables here reflect activities performed by patrons only. A "Practitioner" is defined as a full member of a scientific community of practice. This category includes registrants who identified themselves as professors of any rank, scientists, engineers, and post-docs. The category labeled "Other" includes those whose role is more likely to be in support of the working faculty, scientists, and engineers, such as academic professionals and librarians. The field names (Computer Science, Physics, etc.) should be self-explanatory. In the future we will subdivide the "General Engineering" category into subcategories that reflect the specific contents of the IDL (e.g. Civil Engineering; Electronic / Computer Engineering, etc.).

The numbers of registrants and sessions are relatively low, probably due to (1) the client deployment, which is limited to a small number of workstations located at Grainger Engineering Library, the Beckman Institute Library, and a microelectronics research laboratory, and (2) the short time which has elapsed since the client and the associated instrumentation became operational. The data presented below were gathered between August 1996 and February 1997. The most fundamental numbers are that 144 patrons have registered between August 1996 and February 1997, the vast majority from Grainger Library. There have been 191 patron sessions recorded in this time period.

Patrons change the article component settings in about 5% of the sessions recorded so far. This may seem low, but is more than the 0% use of similar facilities reported by the CORE project. –Robert Sandusky

Table 1

Summary of the distribution by field and career level of IDL registrants (August 1996 through March 1997). This table includes all registrants: patrons, project members, and library employees.

Table 2

Summary of the number of sessions conducted by IDL patrons (August 1996 through March 1997)

Table 3

Summary of the number of searches conducted by IDL patrons (August 1996 through March 1997)

Table 4

Summary of the number of times IDL patrons examined the article component search settings (August 1996 through February 1997)

Data are given in the form N/M, where N is the number of times the settings were examined and M is the number of sessions in which examinations occurred

Table 5

Summary of the number of times IDL patrons changed the article component search settings (August 1996 through February 1997)

Data are given in the form N/M, where N is the number of times the settings were changed and M is the number of sessions in which changes occurred

Table 6

Summary of the number of patrons registered for the IDL, by field and by gender (August 1996 through February 1997)

DLI Spring ’97 Partners Workshop
3-4 April, 1997
Social Science Team
prepared by Laura Neumann

At the publisher’s meeting last year, the social science team asked participants to write down any questions they had that related to the user studies that we on the social science team have been working on. What we were trying to find out was what the publishers were interested in knowing about use and user issues surrounding the DLI testbed. The following are some of these questions. Some questions are not included because either they were duplicates of other questions or we do not yet have the data needed to answer them.

Answers provided to the questions are drawn from data collected through semi- structured interviews, focus groups, and observations with potential users and others related to the digital library project. More data specific to the testbed system will be generated from transaction log analysis as more people use the system.

Do searchers become more adventurous as they use the system more...that is, do they try more features or stick with those they previously used?

We don’t yet have enough use of the system to determine this, however, other data collection efforts find this is an issue with other systems. In my interviews and observations, I have noticed that when a system is new, people are sort of playing with it-- using it for as many things as possible, stretching it in new ways. However, this passes and users settle into patterns of use that are often based on features that they know, with some exceptions. For the most part, users we have studied are quite computer- savvy and they have found what they consider the optimal use for particular programs and use them in that specialized manner. If a new program does a new function particularly well, I have the impression that the new function stands a good chance of being used so long as it is a function that they need in their work.

What I would consider a more pertinent question related it this is which functions would be most useful to our audience? Assuming the system is functional (doesn’t crash, has proper documentation, people can fairly easily figure out how to use it, has that critical mass of documents...), he functions that are most useful will probably be used, regardless of how new they are in relation to the more "traditional" functionalities.

How important is archival vs. current awareness use? Do people expect and want current stuff only?

Absolutely not. People do want current stuff, and it is very useful, however, this is what most systems have. What many people have mentioned in interviews and focus groups is that they wish that the databases covered a longer time span. Most fields have databases available (formal or informal) that have more current materials. The space in the market in current stuff is that the system has to offer better features than what is out there already, however as there are few systems with anything except current stuff, there is a much larger market space for this. There is a definite call for this.

A related question, however, is what "archival" materials are for each field. In sub- atomic physics, this may mean work published between 1945 and1995. In physicists dealing with general relativity, they would like to see Einstein’s work available. Then for computer scientists, anything from earlier than December of 1996 is "old." The meanings of "archival" and "current" are very fluid and need to be carefully examined for each field.

Do users download and print? Once the article is retrieved, what do users do with it?

Users will download, it seems, only if it is necessary for printing. There are a variety of different habits regarding when and why people print out materials. In interviews and observations there has been a complete range of people- from those who print out and photocopy and keep in a accessible location just about everything that could possibly be useful to them as well as their colleagues, to people who print nothing. Some of the dimensions to consider are:

What is done when accessing article:

save location (bookmark)---- not
print-------- don’t print
add to personal homepage of "hotlinks"-------- not

What is done with a print out:

keep printout---------- trash/ recycle/ loose
use printout again---------- never touch printout again
pass to others-------------- keep to self

Why did the person make a copy:

portability
can write on
permanence
file, organize and use as mental cue
other qualities that paper has

What will it take to make it desirable for users to read the article on- line?

-value- added features of browser?
-rendering quality?
-electronic note- taking?
-hyper- link capabilities?
-availability of multi- media objects?

Yes! All of the above, but in order of importance (as mentioned by the various respondents):

1. better monitors-- higher resolution
2. more manipulatable documents... replicate paper in

note taking
marking
"stacking" (organizing physically)

3. value added features of electronic medium:

searching
copying
hyperlinking
multi- media stuff