FROM USABILITY TO USE:

FROM USABILITY TO USE:

MEASURING THE SUCCESS OF TESTBEDS

IN THE REAL WORLD

Emily Ignacio and Ann P. Bishop

University of Illinois

Moving from needs assessment to deployment of DeLIver, we are faced with new issues:

How should we gauge success? How many users are enough?
How do we interpret reasons for lack of use?

Also evolved new way of working to address these issues:

Testbed and Social Science Team in closer synch
Bringing different methods and sources of evidence to bear

Presentation will be a candid reflection:

On problems related to measuring use of a deployed testbed
And how we are addressing them

ROLL-OUT PANIC: WILL ANYONE USE OUR DL?

Web stats (Nov. 1-14, 1997)

Of 1540 attempted accesses

1276 (83%) stopped at NetID form

Of 186 people who entered NetID

91 (49%) stopped at registration form

HOW CAN WE REDUCE BARRIERS TO USE?

Which option will succeed? What are implications?

Simplify functionality

Simple search only

Cut authentication and registration

Free sample during peak usage period

Streamline authentication and registration

Shorten and clarify

Step up publicity (pinpoint hubs of use)

Classroom visits and journal stickers

WHY REDUCE BARRIERS?

Testbed as hybrid

Make a better system!

Demonstrate high use!

Learn more about DLs and use!

WHAT IS CAUSING BAIL OUTS?

Pooling expertise and experiences

People aren't UIUC affiliates

Lack of awareness

Tech problem: browser doesn't support cookies

Registration form on web = fee

Privacy concerns

Lack of real need -- just surfing

Confusion -- NetID? What do I do now?

Registration form is too long

HOW SHOULD WE MEASURE USE?

Number of registered users

Type of registered user (are some users better than others?)

Frequency of use

Number of documents accessed

Number of search sessions conducted

HOW SHOULD WE GAUGE SUCCESS?

% of potential users

1000 grad students in Engr, Physics, CS

Likelihood of use

Read those journals

Don't have own paper subscription

Searching/browsing habits

New system timeframe (awareness, learning, and commitment)

Use compared to what?

How many students ever use online catalog?

How often are the paper journals used?

No. of registered users for custom client (280 from Aug. 96 to Oct. 97)

Virtually no use of components in CORE

CUT AND TRY ENGINEERING:
STREAMLINED NETID FORM

Web stats (Dec. 9-19, 1997)

Of 462 attempted accesses

259 (56%) stopped at NetID form

Of 113 people who entered NetID

35 (31%) stopped at registration form

Fewer people are bailing out at the NetID form, but ....

Why are there so many fewer attempted accesses?

Why do more complete the registration form?

Introduction: The First Two Years or "I am user, Hear me Roar"

primary goal: to help system designers create usable and functional Digital Library.

wanted users’ needs to be designed into the system

methods reflected our goals

methods used during needs assessment phase:

focus groups

interviews

observations of physicists in work setting

observation of people using other systems

The Shift to Usability Tests (the third year)

several teams; different goals, deadlines, and methods of evaluation

the wake up call

same deadline; needed to fuse different goals and methods of evaluation

usability studies (three different kinds):

realistic tasks
give people URL, let them evaluate it at home, hold a focus group
help students with their assignments and analyze their problems with the system (Mike Twidale)

teams began to merge

worked together to build interface and plan marketing strategies

The "Threat" of Use: after the roll-out

transaction logs and large scale surveys given more emphasis than usability tests, other methods

new obstacle: how do we measure "use"?