send comments or questions to: l-neuma1@uiuc.edu

Laura Neumann and Ann Bishop

Digital Libraries Initiative, University of Illinois

Data Processing Clinic

March 23, 1998

Social Science Team:

Leigh Star, Emily Ignacio, Robert Sandusky Cece Merkel, Eric Larson, Rebecca Engsberg, Madonnalisa Gonzales

DLI User Studies And Evaluation

• Conduct needs assessment
• Provide ongoing user feedback
• Document and analyze extent and nature of testbed use, satisfaction, and impacts
• Contribute to theoretical understanding of the changing information infrastructure
• Develop and assess new methods for capturing user behavior

An Integrated Research Program: Studying Testbed Users

Combine broad study of use with deep study of social phenomena

• Observation of engineering work and learning activities
• Interviews and focus groups with a range of potential and actual testbed users
• Usability testing
• User registration
• System instrumentation (the creation of testbed transaction logs)
• Large-scale user surveys

Usability To Use

Moving from testing to deployment of DeLIver, we are faced with new issues:

• How should we gauge success? How many users are enough?
• How do we interpret reasons for lack of use?

Roll-Out Panic: Will Anyone Use Our DL?

Web stats (Nov. 1-14, 1997)

Of 1540 attempted accesses

1276 (83%) stopped at NetID form

Of 186 people who entered NetID

91 (49%) stopped at registration form

How Can We Reduce Barriers To Use?

Which option will succeed? What are implications?

• Simplify functionality
  • Simple search only
• Cut authentication and registration
  • "Free sample" during peak usage period
• Streamline authentication and registration
  • Shorten and clarify
• Step up publicity (pinpoint hubs of use)
  • Classroom visits and journal stickers

Why Reduce Barriers?

Make a better system!

Demonstrate high use!

Learn more about DLs and use!

What Is Causing Bail Outs?

• People aren't UIUC affiliates
• Lack of awareness
• Tech problem: browser doesn't support cookies
• Registration form on web = fee
• Privacy concerns
• Lack of real need -- just surfing
• Confusion -- "NetID? What do I do now?"
• Registration form is too long

How Should We Measure Use?

Type of registered user (are some users better than others?)

Frequency of use

Number of documents accessed

Number of search sessions conducted

How Should We Gauge Success?

• 1000 grad students in Engr, Physics, CS

• Read those journals
• Don't have own paper subscription
• Searching/browsing habits
• New system timeframe (awareness, learning, and commitment)

• How many students ever use online catalog?
• How often are the paper journals used?
• No. of registered users for custom client (280 from Aug. 96 to Oct. 97)
• Virtually no use of components in CORE Cut And Try Engineering

Streamlined NetID Form

Of 1540 attempted accesses

• 1276 (83%) stopped at NetID form

Of 186 people who entered NetID

• 91 (49%) stopped at registration form

Of 462 attempted accesses

• 259 (56%) stopped at NetID form

• 35 (31%) stopped at registration form

Of 560 attempted accesses

• 240 (43%) stopped at NetID form

Of 320 people who entered NetID

• 49 (27%) stopped at registration form

Usability Testing

• Standard approach
  • Sit user in front of system
  • Assign realistic task
  • Ask user to "think aloud"
  • Debriefing interview
• Real use and group discussion
  • Person uses system on own for real work
  • Keeps record of first impressions
  • Group convenes to discuss while viewing system
• Guided use
  • User invited for individual help with own search
  • Researchers assist with search and record the interaction
• Situated usability study
  • Users identified through TLs
  • Interviewed in own office about work
  • Screen shots as discussion prompts

DeLIver Exit Poll

__ Find articles on a particular topic

__ Find articles by a specific author

etc.

__ Play with the system

__ Other:

Did you accomplish what you wanted to from this session?

No, not at all ....................................... Yes, accomplished everything I wanted

Please rate DeLIver on this session:

Ease of use Poor .............. Excellent

Power of search capabilities Poor .............. Excellent

Reliability of system Poor .............. Excellent

Value of contents Poor .............. Excellent

Suggestions, comments or questions:

Lessons Learned

• Triangulation is crucial
• Building on past data and methods
• Full understanding (e.g., login bail-outs)
• Define place of user support and marketing

More Lessons Learned

Registration

• Tradeoff: analytical power vs. limited use
• Few fakers
• Decline to answer --
  • 25% Ethnicity
  • 10 Age
  • 1 Sex

• Parsing individual actions
• Constructing a session
• Don't use defaults

• Low response rates on web
• Which style?
• Which delivery option?